Quick answer: AI security incident response is the process that defines what to do when an AI system is attacked, deceived or behaves unexpectedly. Classic incident response handles a server or network breach; an AI incident is different because the symptom is often not a crash but the model silently making wrong or harmful decisions. The most common incidents are: an agent directed to a bad action with prompt injection, a model corrupted with data poisoning, data leakage through model inference and the model producing unusual output. The root approach follows the same order as classic response but is adapted to AI: contain the incident, preserve the model and inputs, determine the scope, find the root cause and return the model to a safe version; all done while preserving evidence and under human oversight.

Organizations now use AI in critical processes: customer service, decision support, security. But an AI system can also be attacked or deceived. What is done in such a case? This article explains AI specific incident response.

Why an AI incident is different

In a classic security incident the symptom is usually clear: a server crashes, a network traffic spikes, a file is encrypted. In an AI incident the symptom is often silent: the model keeps running but makes wrong or harmful decisions. An agent may have done an unwanted action because it was deceived; a model may have learned wrong from poisoned data. This silence leads to the incident being noticed late. So AI outputs must be continuously audited with explainability and monitoring.

AI specific incident types

Incident Symptom Source
Prompt injection The agent does an unwanted action Prompt injection
Data poisoning The model systematically decides wrong Data poisoning
Model inference Data leakage Model attacks
Excessive agency abuse A destructive action triggered Excessive agency
Unusual output Unexpected or harmful response Manipulation or error

The common challenge of these incidents is that most of the evidence is volatile and distributed: inputs, model state and decision logs can vanish after the incident passes.

AI incident response steps

1. Contain and preserve

If the affected AI system keeps doing destructive actions, it must be safely stopped or its privileges restricted. Inputs, outputs and model state must be preserved according to digital evidence and chain of custody principles.

2. Determine the scope

The impact of the incident must be determined: which actions the agent did, which decisions the model made, which data may have leaked. The agent's tool call logs are critical at this stage.

3. Find the root cause

The source of the incident must be detected: a prompt injection, poisoned data or an excessive privilege. Without knowing the root cause the fix remains incomplete.

4. Return to a safe version

If the model was poisoned it must be returned to a clean version, the agent's privileges reviewed and the flaw closed. The model's past versions and training data must be preserved for this rollback.

5. Lesson and remediation

After the incident, controls that prevent the same manipulation from recurring (input separation, privilege restriction, monitoring) must be added.

The KAOS and DSET approach

DSET runs AI security incidents with a process that combines AI speed with human expert oversight. The local AI engine KAOS scans agent logs, model outputs and inputs for fast triage, catches unusual behavior and helps extract the root cause. But every finding with evidentiary value is verified under expert oversight and a chain of custody. Because KAOS runs offline, sensitive model and incident data is not sent outside. The result is a defensible response that makes a silent AI incident visible.

Frequently asked questions

How do I know my AI has been attacked? Usually not by a crash but by unusual or wrong decisions. If the model keeps running while producing systematically wrong output, an agent did an unexpected action or there are signs of data leakage, an incident may be occurring. So AI outputs must be continuously monitored.

Can a poisoned model be fixed? Often the safest way is to return the model to a clean version from before poisoning. So the model's past versions and training data must be preserved. Retraining without finding the root cause can bring the same poisoning back.

What is evidence in an AI incident? The agent's tool call logs, the inputs to the model, the outputs produced and the model state are evidence sources. Because this data is volatile it must be collected with a chain of custody right after the incident; if delayed, the trace can vanish.

Sources

For incident response and post attack examination for your AI systems, contact DSET. We provide security with KAOS and expert oversight from our Ankara Hacettepe Teknokent laboratory.