How to evaluate an AI pentesting tool

Evaluating an AI pentesting tool can feel like a lot to check. In Intruder’s State of AI Pentesting 2026 report, the biggest concerns among security and engineering leaders were data privacy, safety on production systems, whether auditors would accept the report, and how far they could trust the results.

Most of these can be settled with a few straightforward questions to vendors. Here’s what to ask, and what a good answer sounds like.

TL;DR

9 questions to ask an AI pentesting vendor

What happens to my source code during an AI pentest?

Giving an AI pentest access to your source code, known as white-box testing, lets it understand how your application actually works instead of probing it blind. That produces more accurate results with fewer false positives, and it gives you an advantage attackers don’t have, since they can’t see your code. In return, your code should be:

• Processed in an isolated environment, only for the duration of the test
• Deleted when the test ends, apart from any excerpts genuinely needed
• Never used to train AI models

Is AI pentesting safe to run?

Confirming a vulnerability can mean exploiting it, which can change or delete data. No pentest, human or AI, is risk-free, so you want a vendor that recommends testing a staging environment and has clear safeguards that:

• Keep the agent within the agreed scope
• Avoid destructive actions
• Enable the test to be halted immediately

Can I see what the AI pentest actually tested?

Not knowing what was tested was the top concern in our survey, with a quarter of security leaders saying it would damage their trust in AI pentesting more than anything else. An AI pentest report should make the scope of the test clear, including:

• The environment and source code that were tested
• The user roles tested
• What was out of scope, and any limitations
• The vulnerability categories covered

Each finding should also point to exactly where the issue sits in your code, so you can trace it back yourself.

Will my auditors accept the AI pentest report?

If the report is going to an auditor or a customer security review, ask for a sample first (here’s ours). It should include:

• An executive summary and full technical detail
• Supporting evidence for each finding
• Impact, likelihood, and remediation guidance

Frameworks like SOC 2 and ISO 27001 focus on evidence of testing rather than whether a person or an AI carried it out, so a report that covers all of this can support them.

How are false positives and hallucinations handled?

Most teams already have enough false positives to deal with, and with any AI tool, hallucinations are an understandable concern too. Every finding should be validated against the application, by running the exploit or attack path, before it's reported. Validation should also produce proof-of-concept code your team can run, so you can confirm an issue yourself and hand developers exactly what's wrong.

What kinds of vulnerabilities can AI pentesting find?

The real value of AI pentesting is in the issues that don’t have a signature, so check that testing covers issues like:

• Authorization flaws, such as IDORs and privilege escalation
• Business logic issues, like manipulating prices or skipping payment steps
• Multi-step vulnerabilities, where several smaller issues chain together
• The standard vulnerability classes, from injection to authentication weaknesses

For real examples of each, see our guide to AI web app pentesting.

Who’s behind the AI pentesting tool?

AI pentesting has attracted a lot of new vendors, and an AI pentest is only as good as the expertise built into it. Look for:

• Experienced, qualified pentesters who built and trained the agents
• A track record in security, such as published research or credited vulnerability discoveries
• An established company you’d be comfortable giving access to your code

How much setup and expertise does AI pentesting need?

Nearly a quarter of security leaders say a lack of internal expertise is holding back adoption. In practice, running an AI pentest can be as simple as connecting your code repository and adding entry points and credentials, with the agent working the rest out from your code.

Can I retest a fix after an AI pentest?

In our research, fewer than two in five organizations retest a fix as soon as it ships, and nearly half take up to two weeks. With AI pentesting, a retest can run as soon as the fix has been made. Look for retesting included in the price, so cost doesn’t decide whether a fix gets checked.

Get started with AI pentesting

Intruder's AI pentesting runs white-box web app pentests on demand, validates every finding before it's reported, and delivers an audit-ready report the same day. Our AI Pentest Terms set out exactly how we handle your code, data, and testing safety. Get started or book some time to chat.