How to Measure a Security AI: Benchmarks, Hallucination Rate and Proof Replay
How to tell whether a security AI is genuinely good. Honest evaluation with solvable benchmarks, hallucination rate, proof replay and false positive measurement.
Engineering / KAOS AI · DSET Engineering Team · 13 October 2026 · 12 min read
Quick answer
The only way to separate a security AI's marketing promise from its real capability is to measure it. Four metrics are decisive: success rate on solvable benchmarks (can it find known vulnerabilities), hallucination rate (does it invent vulnerabilities that do not exist), proof replay (can it later prove what it found) and false positive rate (does it produce noise). A good security AI reports all four transparently. Saying "very smart" is not a metric.
Why an impressive demo misleads
A demo can look impressive on a cherry picked target. Real value is repeatable success on unknown and diverse targets. That is why serious evaluation rests on standard benchmarks. Difficulty sets whose solutions are known but not shown to the model reveal whether the model truly reasons.
Metric 1: benchmark success rate
The security world has vulnerability solving benchmarks. These are controlled targets containing known vulnerability classes. The question of what percentage of these targets a model solves correctly is a basic capability measure. What matters is balanced success across diverse categories (injection, identity, logic, memory), not a single category.
Metric 2: hallucination rate
This is the most dangerous failure of a security AI. If the model reports a vulnerability that does not exist, the team chases something unreal and trust collapses. The hallucination rate measures how much of what the model produces is not real. In general purpose models this rate can be high on security topics. A domain trained engine aims to keep it low and, most importantly, must be able to say "I do not know" when unsure.
Metric 3: proof replay
If a finding is real today, it should be provable tomorrow too. Proof replay is recording a finding with an anchor and being able to re run it later. This provides three things. It proves the finding was actually made. It measures that the issue closed after patching. And it tracks the issue's state over time, is it still live, did it change, did it close.
Metric 4: false positive rate
A false positive is a finding reported as real that is not. A high false positive rate makes a tool practically useless because the team drowns in noise. A good security AI cuts false positives at the source, for example distinguishing by context whether a security header is actually weak, rather than declaring every absence a vulnerability.
The four metric table
| Metric | What it measures | Good direction |
|---|---|---|
| Benchmark success | Finding real vulnerabilities | High and balanced |
| Hallucination rate | Invented findings | Low, able to say I do not know |
| Proof replay | Provability | Every finding replayable |
| False positive | Noise | Cut at the source |
Principles of honest evaluation
Three principles matter when evaluating a security AI. First, independence, the model should not test itself, there should be a separate verification layer. Second, repeatability, the same test should give the same result. Third, transparency, metrics should not be hidden, weak points should be reported too. KAOS applies these principles in its own evaluation and admits its own weak points, because inflated self assessment collapses in the field.
Defensive value
The right metrics let an organization know which security AI to trust. Choosing by measurable proof rather than a marketing promise protects both budget and trust. If a tool openly shares its false positive and hallucination rates, it has already passed a test on honesty.
Frequently asked questions
Is a high benchmark score enough on its own? No. High success but high hallucination is useless in the field. The four metrics must be evaluated together.
Can the hallucination rate be zero? In practice it approaches zero, but claiming exact zero is an overstatement. What matters is that the model can honestly stop when unsure.
Why is proof replay critical? Because it tracks a finding's state over time. The question of whether a patch actually worked is answered by replayed proof, not a signed report.
Meet KAOS
Every capability in this article runs live inside the KAOS security intelligence. KAOS is an autonomous security engine built in Türkiye by the DSET Engineering team, with inference kept local. Request a demo with our Hacettepe Teknokent based team to measure your attack surface.
Kimliğinizi doğrulayın
Yetkilendirilmiş erişim alanı. Tüm giriş denemeleri kayıt altına alınır.