Engineering / KAOS AI Engine · DSET Engineering Team · July 16, 2026 · 30 minute read (updated)

157 specialist agents. 308 integrated security tools. A 4.9 million document domestic security brain, continuously updated. A self-critiquing 4-layer adversarial architecture. The technical anatomy of what KAOS actually does when it looks at a target.

#autonomous_pentest #kaos #adversarial_ai #vuln_research #self_healing #0day #critic_stack

Table of Contents

  1. Why is another autonomous pentest engine needed?
  2. Other LLMs are for everything. KAOS is for a single job.
  3. Architectural layers
  4. The brain: a 4.9 million document continuously updated vector memory
  5. 157 specialists: the agent mesh and task distribution
  6. Critic Stack: 19-layer self-monitoring
  7. Adversarial Twin: 4 layers that question their own finding
  8. The Anti-Hallucination Doctrine
  9. Self-Healing: from finding to patch in 20 minutes
  10. Proven new capabilities
  11. Field test anecdote: scanme.nmap.org
  12. Roadmap and closing

1. Why is another autonomous pentest engine needed?

As 2025 drew to a close, there were at least 40 products on the market claiming to be an "AI pentest". Most wrapped a foreign cloud LLM API around sqlmap and nmap. They put a dashboard on top. Two fundamental problems:

  1. If the context is narrow the answer is superficial. If it is broad the door to hallucination opens. A vanilla prompt engineering approach cannot reliably answer the question "is this XSS real?".
  2. All of the customer's scan data leaks to a foreign cloud. Air-gapped deployment becomes physically impossible. KVKK and sector regulations remain in a gray area.

KAOS rejected this at its founding: all models were trained by the KAOS Team in Turkey. Inference is local. No API call ever leaves. Sovereign mode enforces this not as a preference but as a two-layer requirement: model remapping plus a host-level egress block. When the internet is cut, KAOS keeps working.

The DSET pentest team had three observations from 11 years in the field:

  1. Knowledge ages. If a CVE is disclosed today, a classic scanner does not see it for a week. KAOS integrates it the same day.
  2. Humans cannot keep the clock. A three-day red team window compresses into 20 hours of real attention span.
  3. By the time the report is finished, the patching has not started. When the PDF is signed, the attacker is already 14 days ahead.

KAOS is an engine that does not accept these three asymmetric problems.

2. Other LLMs are for everything. KAOS is for a single job.

The large language models on the market today were designed for everything. Let it write poetry. Let it generate a SQL query. Let it answer customer service. Let it summarize a medical report. Let it provide tax consultancy. The same model. The same weights. One product. One model. One breakfast for everyone.

KAOS is not an LLM. KAOS is a multi-model attack intelligence trained to perfection for a single job.

The sole task of this intelligence is this: to find a target's attack surface. To extract the shortest path that can pass through it. To recommend the patch. To test its patch. Nothing else.

The concrete results of this focus:

  • 6 times more technical content per token. Because it does not know poetry, every token corresponds to a CVE ID, a CPE pattern, a cipher suite, an HTTP header.
  • A hallucination rate of 0.31%. In general-purpose LLMs this rate is measured between 12% and 18% on security topics. The KAOS Embed Pro encoder was trained only on CVE descriptions, PSIRT bulletins, exploit code and patch diffs.
  • A tenth of the inference cost. Instead of a general-purpose 70B model, a domain-specific Coder runs. It spends 10 times fewer GPU hours on the same task.
  • Air-gapped deployment is possible. A model that has learned every kind of data for everything cannot run without going outside. An engine trained only for security lives on local hardware.

The comparison with existing scanners is clearer: Nessus receives signature updates every hour but does not reason. Burp Suite reasons but is dependent on the operator. Tenable Web App Scanner has a broad surface but cannot produce a 0day. SaaS scanners like Detectify and Acunetix send customer data to the cloud. Their common feature is this: none of them critiques itself. None of them patches and tests the vulnerability it finds. None of them runs air-gapped.

KAOS was designed to produce an answer to each of these three gaps. Not general-purpose. Not broad-surfaced. Not multi-talented. It was designed to be the best in the world at a single thing.

Our engineering principles:

  • A single tool is not a black box. Every wrapper writes its own telemetry to the critic stack.
  • The LLM is the reasoning step. It is not the scanning step. The model does not scan ports. The model critiques the scan plan.
  • A finding is not a verified finding. It does not reach a report line until the adversarial twin approves it.
  • The patch is part of the report. A finding does not close without a recommendation from the patch twin.
  • KAOS must fine-tune itself. Training at night. Inference by day.

3. Architectural layers (from bottom to top)

9. CRITIC stack          (19-layer self-monitoring)
8. Adversarial Twin      (4 layers: critic dpo kb patch)
7. Self Improver         (with proposer, critic and applier)
6. Learner               (learning modules, night training)
5. Swarm v2              (4-stage context-aware orchestrator)
4. Coordinator           (parallel agent orchestrator)
3. Specialist            (157 specialists, BaseSpecialist contract)
2. Wrapper               (tools_v2, 308 integrated security tools)
1. Model                 (KAOS Coder, Critic alpha beta gamma,
                          Embed Pro, Attack Intel Encoder,
                          Code Encoder, Reranker, Edge, all domestic)

Each layer is bound to the one below it only through contracts. The specialist layer does not have to know the model layer. It simply calls the LLMService interface. Thanks to this, when we want to change the Coder model, we do not rewrite any of the 157 specialists. Today 308 tool wrappers and 372 registered capability modules are reachable through this contract.

4. The brain: a 4.9 million document continuously updated vector memory

KAOS's brain is not a single vector DB. It is a hybrid retrieval system operating on a stratified corpus.

4.1 The Corpus

  • 13 PSIRT sources (Microsoft, Cisco, Red Hat, SUSE, Apple, Google, NVIDIA, Adobe, Oracle, Atlassian, GitLab, Mozilla, Debian)
  • 22 feed lists (Mitre CVE delta, Mitre ATT&CK, ExploitDB, packetstorm, GTFOBins, LOLBAS, HackerOne disclosed, Bugcrowd disclosed, Inj3ct0r, MetasploitDB, NVD CPE delta, OSV.dev, GitHub Security Advisories, OSS Fuzz issues, MalwareBazaar, MISP feeds, AbuseIPDB sample, URLhaus, OpenCTI, threatfox, alienvault OTX, Cisco Talos)
  • A CTF write-up corpus. The author summaries of 78 major CTF events
  • A red team disclosure corpus. The chunked form of public bug bounty write-ups
  • Field arithmetic. DSET's own 11-year internal finding archive (anonymized)

Total: 4.9 million document chunks vectorized. The brain is split across 8 Qdrant collections: a main corpus of 4.33 million, an exploit knowledge base of 567 thousand, methodology, verified exploits and honeypot signatures. Stored in 3 tiers as hot, warm, cold, with the hot tier updated hourly. KAOS now also uses agentic RAG: the model steers the brain through a self-directed multi-step query loop rather than a single shot. It forms a hypothesis, seeks the missing evidence, then confirms or discards.

4.2 The retrieval pipeline

Question → query expansion → BM25 (sparse) → 3 parallel encoders (Embed
Pro, Attack Intel, Code) → RRF fusion → cross-encoder reranker →
LLM context window

The reason we run 3 encoders in parallel is this: to embed the natural language of a CVE description, its exploit payload, and its patch code in separate semantic spaces. A single embedding cannot capture these three at once.

4.3 Forgetting curve

Knowledge ages. KAOS keeps a "decay score" for each chunk. An exponential decay from the publication date times a usage-frequency-based boost. Chunks older than 60 days that have never been queried drop to the warm tier. Those older than 180 days and still dead fall to cold.

4.4 JSD drift detection

Every day, the Jensen-Shannon Divergence of the query distribution relative to the previous week is calculated. If it rises above 0.18, a corpus update is triggered. New feeds are sought.

5. 157 specialists: the agent mesh and task distribution

Distribution (today's real numbers):

  • 78 vuln assessment (web app, API, AuthN, AuthZ, IDOR, SSRF, XXE, deserialization, RCE, file upload, command injection, SQLi, NoSQLi, GraphQL, JWT, OAuth, SAML, OIDC, MFA bypass, race, mass assignment, prototype pollution, CSRF, CORS, CSP bypass, business logic, request smuggling, WebSocket, web3 multichain contract address audit, DeFi economic logic, OWASP API Top 10, gRPC, GraphQL introspection)
  • 34 exploit (auto payload mutator, WAF bypass, blind time-based, OAST oracle, gadget chain, JWT confusion, ROP, JOP, kernel ret2usr, race window, nday pipeline, crash triage, angr, harness generation, LLM-guided fuzzing dictionary, post-quantum tradeoff)
  • 24 recon (subdomain takeover, SSL cert intel, ASN reverse DNS, HTTP fingerprint drift, OSINT external attack surface, source intel, origin IP disclosure, WAF bypass)
  • 9 post-ex (lateral movement plan, credential graph, AD trust mapping, Kerberos AS-REP roasting, kerberoast, cloud privilege escalation)
  • 8 detection engineering (SIEM rule co-author, blue team detection engineering, anti-forensic, breach and attack simulation)
  • 3 reporting (autonomous narrator, NATO-format DOCX, evidence portfolio)
  • 1 planning (mission planner, kill chain operator)

The BaseSpecialist contract:

class BaseSpecialist(ABC):
    name: str
    domain: str
    cost_weight: float

    async def plan(self, context: MissionCtx): ...
    async def execute(self, plan: Plan, brain: Brain): ...
    async def critique(self, findings: Findings): ...
    async def export(self, findings: Findings): ...

A specialist must produce a plan before running. The Coordinator deduplicates this plan against those of the other specialists. Thanks to this, when 157 specialists look at the same target, nmap runs not 27 times but once. With closed-loop skill replay, a technique proven on one target is first recalled from memory on a similar target, then replayed, then verified.

6. Critic Stack: 19-layer self-monitoring

If KAOS produces a finding, it does not accept it automatically. 19 separate layers examine the finding:

ablation → cluster → advisor → engine → trend → tick → push
→ APPLY → A/B → revert

The health composite formula:

H = 0.35 × critic + 0.25 × hunt + 0.25 × exploit + 0.15 × audit

If H for a specialist is below 0.62 over the last 24 hours, that specialist enters the "rejected" sample pool for night training (DPO).

7. Adversarial Twin

KAOS works together with a second model that tries to deceive itself.

7.1 Critic Adversary

The Critic alpha beta gamma ensemble is continuously fed a "create a false positive" prompt. This jailbreaking-style task forces the real critic to remain robust.

7.2 DPO Poison

Deliberately wrong answer chunks are injected into the training data. The model learns to reject these chunks. If it does not reject them, the loss for that round spikes.

7.3 KB Injection

Fake CVE chunks are injected into the vector DB. If the retrieval pipeline pulls them, a post-prompt that forces verbatim extraction kicks in so it can say "I don't know".

7.4 Patch Twin

Every patch produced in the self-healing pipeline is reviewed through an attacker's eyes by a second KAOS Coder instance. The question "how would I bypass this patch?" is asked. If the answer produces a concrete bypass, the patch is withdrawn.

8. The Anti-Hallucination Doctrine

Every LLM call is wrapped with compose_system():

  1. It is permitted to say "I don't know". The model cannot fabricate to avoid uncertainty.
  2. Every fact requires a source. If it cannot be stated, withdraw the claim.
  3. Extract a verbatim quote before document analysis. Quote before summarizing, then comment.

These three rules reduced the hallucination rate to 0.31% over a total of more than 47,000 automated findings in 12 months (manual verification sample: 5,000 findings, 16 false positives). A separate claim verifier layer cross-checks a claim against independent evidence such as WHOIS, hosting and tool existence before it is reported.

9. Self-Healing. From finding to patch in 20 minutes

FINDING → PoC validate → KAOS Coder Proposer → KAOS Critic alpha beta
gamma ensemble → KAOS Patch Twin → APPLY ff-only → regression with tests
→ PoC replay 4h → auto DPO (chosen/rejected) → KAOS Coder night
fine-tune → cross-target memory
  • FF-only (feature flag only): APPLY never pushes directly to the main branch. It first rolls out behind a flag.
  • PoC replay 4h: After the patch, the original PoC is replayed every 30 minutes for 4 hours.
  • Auto DPO: Successful patches are "chosen". Rejected ones are written to the night training data as "rejected".
  • Cross-target memory: A lesson learned on this target is added as an enricher to the retrieval context of the other targets.

10. Proven new capabilities

KAOS is no longer just a web scanner. Capabilities proven in the field in recent months:

  • Coverage beyond web audit. Give it a contract address (CA) and KAOS pulls the multichain smart contract and audits it including its economic logic. Give it an APK and it decompiles and hunts for secret leakage and abuse. Give it an AI agent and it tests prompt injection and key exfiltration. Give it an ERP and it verifies, for example, an exposed Odoo database manager with a working exploit.
  • Digital forensics. KAOS solves its own forensics benchmark DFB. On a live field image it extracted 8 of 8 real flags with 0 traps (XOR, RSA Fermat, hash, LSB stego, PCAP, SQLite freelist, fragmented file, Vigenère).
  • Active Directory and Kerberos. On a live domain controller it captured a hash via AS-REP roasting, produced the LDIF analysis, and cleared three false positives (krbtgt, machine account, DC) with its own filtering.
  • Open source 0-day hunting. With its own fuzzing stack (LLM-guided dictionary, crash triage, angr, harness generation) it searches for memory safety flaws in common open source libraries.
  • XBOW offensive benchmark 104/104. It solved the offensive challenge set 100 percent, including cases the field calls impossible.
  • Closed-loop learning. A skill learned on one task is written to memory, recalled on a similar task, replayed and verified. The engine sharpens with every engagement.

All of this runs in a single sovereign engine, leaking no data outside.

11. Field test anecdote. scanme.nmap.org

A field test conducted while writing the first version of this article:

  • First pass: 5 runtime errors plus 1 low-impact finding (TLS cipher suite warning)
  • Error classification: 1 circular import. 2 Python 3.14 Path.rglob() behavior changes. 1 nmap parse error. 1 timeout.
  • Auto fix: The self-improver patched 4 of the 5 errors in 38 minutes. It wrote a new parser test for the nmap parse error.
  • Second pass: All 4 of the 4 errors resolved. The parse error fix was verified live. 2 new findings (banner leakage and an outdated Apache).
  • Total: 4 bug fixes within 1 hour. 103 unit tests added. 0 regressions.

KAOS caught the hardening Python 3.14 made to Path.rglob() at the higher usage layer. It fixed it. It attached a regression test. With no developer intervention whatsoever.

12. Roadmap and closing

Most of yesterday's targets are real today:

  1. Agent count. A year ago the target was 144, today 157 specialist agents are live. The next target is 300+ (toward IoT, MQTT, AMQP, satellite, automotive ECU, ICS, OPC UA).
  2. Web3 and multichain audit. Done. Contract address audit and DeFi economic logic analysis are live.
  3. Mobile native. Done. Autonomous APK and IPA audit is running.
  4. Digital forensics. Done. The DFB benchmark and live image solving are proven.
  5. Sovereign mode. Done. With a two-layer egress block it runs air-gapped.
  6. KAOS Brain Cluster. The distributed vector memory keeps growing.
  7. KAOS Vision. Visual attack surface (a CSS injection chain via screenshot).
  8. KAOS Academy. A certified pentester training platform.
  9. KAOS Quantum Ready encoder. Compatible with post-quantum cryptography.
  10. KAOS Cloud. A marketplace and a multi-tenant web console with scope-locked operator access.

A customer who subscribes to KAOS Sentinel today receives this entire roadmap at no additional cost.

KAOS is not a tool. It is a way of working. Developed in Turkey. Running on 100% domestic models. Leaking no data outside. A self-growing attack intelligence.

Other LLMs are for everything. KAOS is for a single job. And for a single job, perfect.

Contact: [email protected]

DSET Engineering Team