Quick answer: A trustworthy AI pentest report rests on a single distinction: detection is not verification. When a scanner says "there may be SQL injection here" it is stating a hypothesis, a claim. To call that same finding "confirmed" you need independent evidence: a measured delay in time based SQL injection, the raw reflection of an injected script in XSS, a recorded request and response. An evidence based report ties every finding to a verdict (confirmed, likely, hypothesis), a piece of evidence, a severity, a remediation step, and a false positive note. Certainty without evidence is forbidden, and saying "could not be verified" is far more valuable than an inflated "critical vulnerability" claim.

What makes a pentest report valuable is not the number of vulnerabilities found. It is being able to stand behind every sentence in it. When a client opens the report they see a line marked "critical" and steer their team, their hours, and their budget by that line. If the line rests on a scanner's guess rather than real evidence, the client burns effort for nothing while the genuine flaw slips by near the bottom of the list. AI assisted scanning both reduces and amplifies this risk: built well, it verifies faster and more consistently than a human, built badly, it turns hundreds of "maybe" sentences into "critical" in seconds. This article explains why an AI pentest report must be evidence based and when a finding can truly be called "confirmed".

The difference between detection and verification

Detection is a signal that a system may carry a particular flaw. The scanner sends a single quote to a parameter, sees a database error message in the response, and says "there may be SQL injection here". That is a correct start, but not a finished job. The same error message can come from an error handler that never passes the input to the database at all, or the page might print that message under any condition. The signal is real, the conclusion is uncertain.

Verification turns that signal into independent evidence. In time based blind SQL injection the method is clear: you inject a measurable delay into the query, you measure the response time, and you compare it against a control request. If a normal request returns in 120 milliseconds while the injected request consistently returns in 5 seconds, you now hold a reproducible measurement, not a guess. In reflected XSS the evidence is your injected script coming back into the page unencoded, raw, and executable. In SSRF the evidence is the record of a request arriving at a server you control, sent by the target.

The difference in one sentence: detection says "I suspect", verification says "I measured, here is the proof". This is exactly the point where a traditional scanner and an AI assisted approach part ways, a distinction we cover in detail in our difference between AI assisted vulnerability scanning and a traditional scanner article.

When a finding is called "confirmed"

A finding counts as "confirmed" only when there is independent, reproducible evidence that the flaw can actually be triggered. A single signal, a single error message, a single version match is not enough. In practice three verdict levels keep the work honest and tell the reader, in one word, how certain each finding is.

Verdict Meaning Evidence required Example
Confirmed The flaw was actually triggered Independent and reproducible evidence (measured delay, raw reflection, outbound request) An injected 5 second delay measured consistently
Likely Strong indicator but not triggered Indirect evidence, partial behavior, safe confirmation not possible Error message leaks but the WAF blocks injection
Hypothesis Only a signal or a match Just a version, header, or signature match "Server: nginx 1.18" version match

To call something confirmed, the evidence must be reproducible by a third person following the same steps. A version match on its own is a hypothesis, because the distribution may carry a backported patch. Ways to make these verdict levels measurable and auditable are explored in more depth in our how to measure a security AI: benchmark, hallucination, evidence replay article.

The elements of an evidence based report

In an evidence based report every finding arrives with the same five elements. A missing element means an incomplete finding.

The first is the verdict: confirmed, likely, or hypothesis. In one word the reader should grasp the weight of the finding. The second is the evidence or proof of concept (PoC): a measurement table in time based injection, the reflected payload and a screenshot in XSS, and in every case the raw request and response. Evidence must be concrete enough that whoever opens the report can reproduce it themselves. For the ethics of PoC development and the responsible disclosure framework, see our exploit development, PoC ethics, and responsible disclosure article.

The third is severity. It is given through a framework such as CVSS but adjusted for context: an internet facing endpoint that requires no authentication is heavier than a copy of the same flaw sitting on an internal network. The fourth is remediation: not a generic sentence like "validate input" but a step specific to that technology and actually applicable. The fifth is the false positive note: under what conditions the finding could be wrong, and what assumptions it rests on. That note is the measure of the author's honesty. For an example flow that unites evidence, PoC, and remediation into one discipline, our verified vulnerability, false positive free security testing, PoC and remediation article is a good reference.

Good report versus bad report: a comparison

Seeing how the same scan can become two different reports is the clearest way to show the difference.

Criterion Bad report Good report
Language "SQLi found, critical" "Time based SQLi confirmed, 5 second delay measured"
Evidence Scanner screen output Reproducible request, response, and measurement
Severity The scanner's ready made label Context adjusted, justified score
Uncertainty Hidden, everything is "critical" Stated openly, verdict level given
False positive Never discussed A separate note on every finding
Prioritization Long, undifferentiated list Confirmed and impactful ones first

A bad report drowns the client: hundreds of lines, all "critical", none proven. A good report hands the client a roadmap: start with these three confirmed flaws, because here is their measurable evidence.

Why over-claim is dangerous

Over-claim means presenting a finding with no evidence as "confirmed" or "critical". It does two kinds of serious damage.

The first is loss of trust. The client receives a "critical SQL injection" report, mobilizes the team, then discovers the flaw is actually not exploitable. Once this happens, the other genuine findings in the report lose their credibility too. The only real capital a security report has is its trustworthiness, and once spent it does not come back.

The second is wrong prioritization. Limited hours and budget flow into an inflated finding, while a genuinely exploitable but plain looking flaw is left waiting. Noise buries the real signal. When a scanner produces hundreds of "maybes", filtering and prioritizing them becomes a job in itself, which we address in our overproduction, noise, and finding prioritization in AI scanning article. An AI being able to show why it classified a finding the way it did is part of this trust. The chain of evidence behind the verdict must be visible, a topic we discuss in our why explainable AI is critical in security article.

Honesty: saying "could not be verified" is legitimate

This is the most misunderstood part of evidence based reporting: saying "could not be verified" is not a failure, it is a statement of honesty. At the end of a pentest a single finding can have three honest outcomes: confirmed (evidence exists), could not be verified (authorization, time, or environment did not allow evidence), and safe (tested, no flaw found). All three are legitimate outcomes, and each can appear plainly in the report.

The one thing that is forbidden is certainty without evidence. The sentence "we could not verify it but it is probably vulnerable, let us mark it critical" destroys the report. The correct language is: "this indicator was observed, for this reason it could not be triggered in a safe environment, an authorized retest is recommended." That sentence neither misleads the client nor hides the real risk. The boundary is always the same: stay within authorization and scope, and never write "I measured" about something you did not measure.

The KAOS and DSET approach

KAOS, DSET's local artificial intelligence engine, builds its reporting on this distinction: detection first, then independent verification. When a flaw signal is caught, KAOS marks it as a hypothesis, then tries to trigger it within safe and authorized boundaries. In time based injection it measures the delay and compares it against a control request, in reflected XSS it confirms the payload comes back raw. Only when this independent evidence is obtained does a finding rise to the "confirmed" level.

KAOS has an over-claim brake: without evidence a finding stays at "hypothesis" or "likely" and is never reported as "confirmed". Version matches are listed as "may be affected", not "definitely vulnerable". Every finding is presented together with its evidence, its severity justification, its remediation step, and its false positive note. The goal is not to inflate the finding count but to give the client a verdict list they can act on with confidence. You can review the whole of KAOS's scanning approach on our KAOS artificial intelligence cyber security scanning tool page.

Frequently asked questions

If a scanner says it "found a critical SQL injection", is that not confirmed? No. The scanner's output is a detection, meaning a hypothesis. To count as confirmed it needs independent, reproducible evidence that the flaw can actually be triggered, for example a consistently measured injected delay in time based injection. Without that measurement the finding is at the "likely" or "hypothesis" level and should be shown as such in the report. The scanner's label does not substitute for verification evidence.

What should every finding contain in an evidence based report? Five elements: the verdict (confirmed, likely, or hypothesis), the evidence or proof of concept (PoC), a context adjusted severity, an applicable remediation step, and a false positive note. The evidence must be concrete enough that whoever reads the report can reproduce it themselves. When these five elements are present together, the reader grasps both what to prioritize and how certain the finding is at a single glance.

Does writing "could not be verified" make the report look weak? On the contrary, it strengthens it. "Could not be verified" is an honest outcome and usually has a concrete reason: an authorization limit, a time constraint, or the impossibility of triggering the flaw in a safe environment. Stating this openly is far more valuable than presenting an unproven finding as "critical", because once certainty without evidence collapses, the whole report loses its credibility. An honest "could not be verified" is offered alongside a recommendation for an authorized retest.

Sources

If you want measurable evidence behind every line of your report, contact DSET. Our team provides evidence based penetration testing and reporting from the Ankara Hacettepe Teknokent center.