Quick answer: Adversarial ML (adversarial machine learning) is the practice of misleading AI models with deliberately crafted inputs. Its best known type is the evasion attack: at inference time, small changes imperceptible to a human are added to the input to force the model into a wrong decision. For example, invisible distortion added to an image can make a classifier misidentify an object entirely. This differs from data poisoning: poisoning happens in training, evasion happens at inference.

How do adversarial attacks work?

Machine learning models evaluate patterns in input mathematically, not like a human. This gives the attacker an opportunity: by studying the model's decision boundary and making very small but targeted changes to the input, they can fool the model. These changes, imperceptible to the human eye, are effective enough to flip the model's decision.

The danger of these attacks is that the input looks normal. A security camera's object recognition model, a fraud detection model or a content filter can be silently bypassed with an adversarial input. The attack produces no obvious error, it just causes a wrong decision.

Training time and inference time attacks

Type When How
Data poisoning During training Injecting malicious samples into training data
Evasion During inference Targeted alteration of input to mislead the model
Model extraction During inference Recovering the model or data through queries

This distinction determines the defense. Poisoning is prevented by the integrity of training data. Evasion is about the robustness of the running model.

Defense against evasion attacks

  1. Increase robustness by training the model with adversarial examples (adversarial training).
  2. Catch anomalous inputs with input validation and anomaly detection.
  3. Monitor sudden confidence drops and inconsistencies in model output.
  4. Do not leave critical decisions to a single model, add human review or a second control.
  5. Rate limit the model API to make it harder for an attacker to probe the decision boundary.

The KAOS and DSET approach

DSET tests the robustness of machine learning models against adversarial attacks in an authorized way and ties evasion paths to evidence. These tests are especially critical for models that make security decisions (fraud detection, content filter, object recognition). Our KAOS security engine evaluates whether a model can be misled with input. Our goal is not a theoretical risk but showing whether the model can actually be bypassed.

Related reading: AI data poisoning, OWASP machine learning security Top 10, AI model security and model theft, MITRE ATLAS AI threat matrix, MLSecOps machine learning pipeline security, explainable AI.

Frequently asked questions

Are adversarial attacks and data poisoning the same? No. Data poisoning happens during training: the attacker corrupts the model from the start by injecting malicious samples into training data. Adversarial attacks like evasion happen at inference: the model is already trained, and the attacker misleads it in the moment by targeted alteration of the input. Their defenses also differ.

Can a small input change really fool a model? Yes, and that is what is dangerous. Because machine learning models evaluate input mathematically, a distortion small enough to be imperceptible to the human eye but targeted can completely flip the model's decision. Because the input looks normal, the attack stays silent.

Which models are open to adversarial attacks? In principle all machine learning models are open to some degree, but the most critical are those that make security decisions: fraud detection, content moderation, biometric recognition and object recognition. When these models are bypassed there is a direct security consequence, so their robustness should be tested.

Sources

To test the robustness of your security decision models against adversarial attacks, you can talk to our team at Ankara Hacettepe Technopark. Contact DSET.