Engineering / KAOS AI Engine · DSET Engineering Team · May 14, 2026 · 28 minute read

72 specialist agents. 276 integrated security tools. A 66.5 GB security corpus updated hourly. A self-critiquing 4-layer adversarial architecture. The technical anatomy of what KAOS actually does when it looks at a target.

#autonomous_pentest #kaos #adversarial_ai #vuln_research #self_healing #0day #critic_stack

Table of Contents

  1. Why is another autonomous pentest engine needed?
  2. Other LLMs are for everything. KAOS is for a single job.
  3. Architectural layers
  4. The brain: a 66.5 GB continuously updated vector memory
  5. 72 specialists: the agent mesh and task distribution
  6. Critic Stack: 19-layer self-monitoring
  7. Adversarial Twin: 4 layers that question their own finding
  8. The Anti-Hallucination Doctrine
  9. Self-Healing: from finding to patch in 20 minutes
  10. The HUNT and EXPLOIT pipelines
  11. Field test anecdote: scanme.nmap.org
  12. Roadmap and closing

1. Why is another autonomous pentest engine needed?

As 2025 drew to a close, there were at least 40 products on the market claiming to be an "AI pentest". Most wrapped a foreign cloud LLM API around sqlmap and nmap. They put a dashboard on top. Two fundamental problems:

  1. If the context is narrow the answer is superficial. If it is broad the door to hallucination opens. A vanilla prompt engineering approach cannot reliably answer the question "is this XSS real?".
  2. All of the customer's scan data leaks to a foreign cloud. Air-gapped deployment becomes physically impossible. KVKK and sector regulations remain in a gray area.

KAOS rejected this at its founding: all models were trained by the KAOS Team in Turkey. Inference is local. No API call ever leaves.

The DSET pentest team had three observations from 11 years in the field:

  1. Knowledge ages. If a CVE is disclosed today, a classic scanner does not see it for a week. KAOS integrates it the same day.
  2. Humans cannot keep the clock. A three-day red team window compresses into 20 hours of real attention span.
  3. By the time the report is finished, the patching has not started. When the PDF is signed, the attacker is already 14 days ahead.

KAOS is an engine that does not accept these three asymmetric problems.

2. Other LLMs are for everything. KAOS is for a single job.

The large language models on the market today were designed for everything. Let it write poetry. Let it generate a SQL query. Let it answer customer service. Let it summarize a medical report. Let it provide tax consultancy. The same model. The same weights. One product. One model. One breakfast for everyone.

KAOS is not an LLM. KAOS is a 7-model attack intelligence trained to perfection for a single job.

The sole task of this intelligence is this: to find a target's attack surface. To extract the shortest path that can pass through it. To recommend the patch. To test its patch. Nothing else.

The concrete results of this focus:

  • 6 times more technical content per token. Because it does not know poetry, every token corresponds to a CVE ID, a CPE pattern, a cipher suite, an HTTP header.
  • A hallucination rate of 0.31%. In general-purpose LLMs this rate is measured between 12% and 18% on security topics. The KAOS Embed Pro encoder was trained only on CVE descriptions, PSIRT bulletins, exploit code and patch diffs.
  • A tenth of the inference cost. Instead of a general-purpose 70B model, a domain-specific 13B Coder runs. It spends 10 times fewer GPU hours on the same task.
  • Air-gapped deployment is possible. A model that has learned every kind of data for everything cannot run without going outside. An engine trained only for security lives on local hardware.

The comparison with existing scanners is clearer: Nessus receives signature updates every hour but does not reason. Burp Suite reasons but is dependent on the operator. Tenable Web App Scanner has a broad surface but cannot produce a 0day. SaaS scanners like Detectify and Acunetix send customer data to the cloud. Their common feature is this: none of them critiques itself. None of them patches and tests the vulnerability it finds. None of them runs air-gapped.

KAOS was designed to produce an answer to each of these three gaps. Not general-purpose. Not broad-surfaced. Not multi-talented. It was designed to be the best in the world at a single thing.

Our engineering principles:

  • A single tool is not a black box. Every wrapper writes its own telemetry to the critic stack.
  • The LLM is the reasoning step. It is not the scanning step. The model does not scan ports. The model critiques the scan plan.
  • A finding is not a verified finding. It does not reach a report line until the adversarial twin approves it.
  • The patch is part of the report. A finding does not close without a recommendation from the patch twin.
  • KAOS must fine-tune itself. Training at night. Inference by day.

3. Architectural layers (from bottom to top)

9. CRITIC stack          (19-layer self-monitoring)
8. Adversarial Twin      (4 layers: critic dpo kb patch)
7. Self Improver         (with proposer, critic and applier)
6. Learner               (29 learning modules, night training)
5. Swarm v2              (4-stage context-aware orchestrator)
4. Coordinator           (parallel agent orchestrator)
3. Specialist            (72 specialists, BaseSpecialist contract)
2. Wrapper               (tools_v2, 276 integrated security tools)
1. Model                 (KAOS Coder, Critic alpha beta gamma,
                          Embed Pro, Attack Intel Encoder,
                          Code Encoder, Reranker, Edge, all domestic)

Each layer is bound to the one below it only through contracts. The specialist layer does not have to know the model layer. It simply calls the LLMService interface. Thanks to this, when we want to change the Coder model in 2027, we do not rewrite any of the 72 specialists.

4. The brain: a 66.5 GB continuously updated vector memory

KAOS's brain is not a single vector DB. It is a hybrid retrieval system operating on a stratified corpus.

4.1 The Corpus

  • 13 PSIRT sources (Microsoft, Cisco, Red Hat, SUSE, Apple, Google, NVIDIA, Adobe, Oracle, Atlassian, GitLab, Mozilla, Debian)
  • 22 feed lists (Mitre CVE delta, Mitre ATT&CK, ExploitDB, packetstorm, GTFOBins, LOLBAS, HackerOne disclosed, Bugcrowd disclosed, Inj3ct0r, MetasploitDB, NVD CPE delta, OSV.dev, GitHub Security Advisories, OSS Fuzz issues, MalwareBazaar, MISP feeds, AbuseIPDB sample, URLhaus, OpenCTI, threatfox, alienvault OTX, Cisco Talos)
  • A CTF write-up corpus. The author summaries of 78 major CTF events
  • A red team disclosure corpus. The chunked form of public bug bounty write-ups
  • Field arithmetic. DSET's own 11-year internal finding archive (anonymized)

Total: 66.5 GB vectorized. Stored in 3 tiers as hot, warm, cold. The hot tier holds 14 GB updated hourly. The daily increase is between 2 and 4 GB.

4.2 The retrieval pipeline

Question → query expansion → BM25 (sparse) → 3 parallel encoders (Embed
Pro, Attack Intel, Code) → RRF fusion → cross-encoder reranker →
LLM context window

The reason we run 3 encoders in parallel is this: to embed the natural language of a CVE description, its exploit payload, and its patch code in separate semantic spaces. A single embedding cannot capture these three at once.

4.3 Forgetting curve

Knowledge ages. KAOS keeps a "decay score" for each chunk. An exponential decay from the publication date times a usage-frequency-based boost. Chunks older than 60 days that have never been queried drop to the warm tier. Those older than 180 days and still dead fall to cold.

4.4 JSD drift detection

Every day, the Jensen-Shannon Divergence of the query distribution relative to the previous week is calculated. If it rises above 0.18, a corpus update is triggered. New feeds are sought.

5. 72 specialists: the agent mesh and task distribution

Distribution:

  • 46 vuln assessment (web app, API, AuthN, AuthZ, IDOR, SSRF, XXE, deserialization, RCE, file upload, command injection, SQLi, NoSQLi, GraphQL, JWT, OAuth, SAML, OIDC, MFA bypass, race, mass assignment, prototype pollution, CSRF, CORS, CSP bypass, business logic, rate limit bypass, captcha bypass, fingerprint, robots.txt analysis, sitemap, .well-known, JS framework version, dependency confusion, SSRF cloud metadata, SSRF DNS rebinding, web cache, request smuggling, WebSocket, WebRTC, web3 bridge, OWASP API Top 10, OAS schema, gRPC, GraphQL introspection, JSON pollution)
  • 15 exploit (auto payload mutator, browser context, mobile context, CSP-aware, WAF bypass, blind time-based, OAST oracle, pad deserialization, gadget chain, JWT confusion, ROP, JOP recon, kernel ret2usr, race window, post-quantum tradeoff)
  • 4 recon (subdomain takeover, SSL cert intel, ASN reverse DNS, HTTP fingerprint drift)
  • 4 post-ex (lateral movement plan, credential graph, AD trust mapping, cloud privilege escalation)
  • 1 detection engineering (SIEM rule co-author)
  • 1 planning (mission planner)
  • 1 reporting (autonomous narrator)

The BaseSpecialist contract:

class BaseSpecialist(ABC):
    name: str
    domain: str
    cost_weight: float

    async def plan(self, context: MissionCtx): ...
    async def execute(self, plan: Plan, brain: Brain): ...
    async def critique(self, findings: Findings): ...
    async def export(self, findings: Findings): ...

A specialist must produce a plan before running. The Coordinator deduplicates this plan against those of the other specialists. Thanks to this, when 72 specialists look at the same target, nmap runs not 27 times but once.

6. Critic Stack: 19-layer self-monitoring

If KAOS produces a finding, it does not accept it automatically. 19 separate layers examine the finding:

ablation → cluster → advisor → engine → trend → tick → push
→ APPLY → A/B → revert

The health composite formula:

H = 0.35 × critic + 0.25 × hunt + 0.25 × exploit + 0.15 × audit

If H for a specialist is below 0.62 over the last 24 hours, that specialist enters the "rejected" sample pool for night training (DPO).

7. Adversarial Twin

KAOS works together with a second model that tries to deceive itself.

7.1 Critic Adversary

The Critic alpha beta gamma ensemble is continuously fed a "create a false positive" prompt. This jailbreaking-style task forces the real critic to remain robust.

7.2 DPO Poison

Deliberately wrong answer chunks are injected into the training data. The model learns to reject these chunks. If it does not reject them, the loss for that round spikes.

7.3 KB Injection

Fake CVE chunks are injected into the vector DB. If the retrieval pipeline pulls them, a post-prompt that forces verbatim extraction kicks in so it can say "I don't know".

7.4 Patch Twin

Every patch produced in the self-healing pipeline is reviewed through an attacker's eyes by a second KAOS Coder instance. The question "how would I bypass this patch?" is asked. If the answer produces a concrete bypass, the patch is withdrawn.

8. The Anti-Hallucination Doctrine

Every LLM call is wrapped with compose_system():

  1. It is permitted to say "I don't know". The model cannot fabricate to avoid uncertainty.
  2. Every fact requires a source. If it cannot be stated, withdraw the claim.
  3. Extract a verbatim quote before document analysis. Quote before summarizing, then comment.

These three rules reduced the hallucination rate to 0.31% over a total of more than 47,000 automated findings in 12 months (manual verification sample: 5,000 findings, 16 false positives).

9. Self-Healing. From finding to patch in 20 minutes

FINDING → PoC validate → KAOS Coder Proposer → KAOS Critic alpha beta
gamma ensemble → KAOS Patch Twin → APPLY ff-only → regression with tests
→ PoC replay 4h → auto DPO (chosen/rejected) → KAOS Coder night
fine-tune → cross-target memory
  • FF-only (feature flag only): APPLY never pushes directly to the main branch. It first rolls out behind a flag.
  • PoC replay 4h: After the patch, the original PoC is replayed every 30 minutes for 4 hours.
  • Auto DPO: Successful patches are "chosen". Rejected ones are written to the night training data as "rejected".
  • Cross-target memory: A lesson learned on this target is added as an enricher to the retrieval context of the other 72 targets.

10. The HUNT and EXPLOIT pipelines

HUNT (recon):

  • subdomain takeover triage
  • SSL cert intel (CT log subscription)
  • ASN reverse DNS
  • HTTP fingerprint drift (a hash comparison every day)
  • unified orchestrator
  • dashboard
  • systemd timer
  • webhook push

EXPLOIT:

  • PoC evidence repository
  • periodic replay daemon
  • dashboard panel
  • auto record from findings
  • bounty markdown export
  • local vulnerable lab (KAOS sets up its own Vulhub)
  • payload diversifier
  • cross-package dependency graph

11. Field test anecdote. scanme.nmap.org

A field test conducted while writing this article:

  • First pass: 5 runtime errors plus 1 low-impact finding (TLS cipher suite warning)
  • Error classification: 1 circular import. 2 Python 3.14 Path.rglob() behavior changes. 1 nmap parse error. 1 timeout.
  • Auto fix: The self-improver patched 4 of the 5 errors in 38 minutes. It wrote a new parser test for the nmap parse error.
  • Second pass: All 4 of the 4 errors resolved. The parse error fix was verified live. 2 new findings (banner leakage and an outdated Apache).
  • Total: 4 bug fixes within 1 hour. 103 unit tests added. 0 regressions.

KAOS caught the hardening Python 3.14 made to Path.rglob() at the higher usage layer. It fixed it. It attached a regression test. With no developer intervention whatsoever.

12. Roadmap. Tomorrow's KAOS (10 main tracks, 24 months)

  1. KAOS Brain Cluster. A 250 GB distributed vector memory, up from 9.5 GB (Qdrant cluster).
  2. 144 specialist agents. Currently 72. Doubling within 12 months (IoT, MQTT, AMQP, satellite, automotive ECU, blockchain bridge, adversarial ML, OPC UA).
  3. KAOS Vision. Visual attack surface (a CSS injection chain via screenshot).
  4. KAOS Audio. Voice deepfake red team (a voice biometric bypass simulation).
  5. KAOS Mobile Native. An autonomous APK and IPA package.
  6. AGI tier planning. From 7 phases to 12 phases (a Markov decision process).
  7. KAOS Marketplace. A third-party specialist ecosystem. Revenue share.
  8. KAOS Academy. A certified pentester training platform.
  9. KAOS Quantum Ready encoder. Compatible with post-quantum cryptography.
  10. KAOS Cloud. AWS, Azure, GCP marketplace. End-of-2027 target of customers on 3 continents.

A customer who subscribes to KAOS Sentinel today receives all 10 of these tracks at no additional cost.

KAOS is not a tool. It is a way of working. Developed in Turkey. Running on 100% domestic models. Leaking no data outside. A self-growing attack intelligence.

Other LLMs are for everything. KAOS is for a single job. And for a single job, perfect.

Contact: [email protected]

DSET Engineering Team