DFB Methodology
A soundness-aware, anti-forensics-stratified benchmark for digital forensic tools and AI agents. This page is the living methodology and the working draft of the accompanying research paper.
1. The gap
Existing forensic tool testing (NIST CFTT, CFReDS, DFRWS challenges, vendor CTFs) is largely static, recall-only and predates AI agents. None of them stratify cases by anti-forensics difficulty, and crucially none measure soundness: a tool's resistance to planted false evidence. As anti-forensics proliferates and autonomous AI agents enter casework, the field lacks a reproducible, adversarial, soundness-aware benchmark. DFB fills that gap.
2. The model
A challenge is a forensic artifact, a hidden ground-truth question set with objective answers, an automatic grader, an anti-forensics level and a version hash. Ground truth is generated in an instrumented VM: we play out the scenario under full recording, so every action is logged and the answer key is authoritative. All data is synthetic.
3. Anti-forensics stratification (AF-0..AF-4)
| Level | Adversary |
|---|---|
| AF-0 | Clean artifact, no obfuscation. |
| AF-1 | Basic deletion and hiding (slack, ADS, simple wipe). |
| AF-2 | Timestomping, log clearing, steganography, encryption. |
| AF-3 | Hidden volumes, journal/USN wipe, fileless memory, planted false trail. |
| AF-4 | Multiple techniques chained, anti-tool structures, deception designed to defeat advanced suites. |
4. Scoring: the soundness signature
- Recall. Difficulty- and AF-weighted sum of correct findings.
- Soundness (precision). TP / (TP + FP). Reporting a planted false trail as real, or claiming to recover the genuinely unrecoverable, is heavily penalized.
- Confidence calibration. Tools report confidence; overconfident wrong answers are penalized further.
- AF-Resilience. The anti-forensics level at which a tool's accuracy collapses.
- Court-grade output. Not just answers: a structured forensic report (findings, evidence, methodology) is graded for defensibility.
Certification tiers: Bronze, Silver, Gold and DSET Forensics Certified, issued as a cryptographically verifiable badge.