Summary: Corrupting the data a model learns from — at training time, or at retrieval time in production. Named explicitly in AI Act Article 15(5) and listed in the OWASP Top 10 for LLMs.
Sources: raw/Data-poisoning/ (8 files); see individual citations
Last updated: 2026-07-28
Definition
“AI data poisoning is when an attacker manipulates the outputs of an AI or machine learning model by changing its training data. The attacker’s goal… is to get the model to produce biased or dangerous results during inference” (source: What is AI data poisoning_.md, Cloudflare).
CMU’s SEI adds the insider dimension: “Data poisoning occurs when an insider or adversary modifies training data to influence the performance or operation of a model” (source: CMU chain-of-custody article).
Two axes, often conflated
Axis 1 — targeted vs untargeted (source: Cloudflare):
- Direct / targeted — “skew or alter a model’s output only in response to particular queries,” leaving it otherwise normal. Example given: tricking an email security filter into allowing specific malicious URLs.
- Indirect / untargeted — degrade general performance or bias the model broadly. Example: “a foreign adversary… might want to bias general-use LLMs towards giving out misinformation within a particular country.”
Axis 2 — training-time vs runtime (source: Trilateral Research):
- Training-time poisoning “affects the data used to build the model itself.”
- RAG and prompt injection poisoning “attacks production systems by manipulating the documents, emails, or knowledge bases that AI assistants retrieve” — described as “the more immediate threat for most GenAI adopters.”
The second axis is the one that matters operationally. Most organisations do not train models; all of them feed documents into AI workspaces. See rag-and-runtime-poisoning.
Attack types
From Cloudflare’s taxonomy:
- Backdoor poisoning — “introduces a hidden vulnerability into the model so that, in response to certain specific triggers known to the attacker, it behaves in an unsafe way.” Dangerous precisely because the model “will otherwise behave normally.”
- Mislabeling — altering labels, leaving the underlying data untouched.
- Data injection and manipulation — adding, altering or removing records to bias the model.
- Availability attack — degrading overall performance, “to slow down or crash the model.”
CMU makes the mislabeling point vivid: “after the data is labeled, the attacker could modify some labels, leaving the images themselves unaltered.” Integrity checks on the data files would pass.
How poison gets in
Cloudflare names three routes:
- Insider attack — “more difficult to detect and stop than attacks by an external third party.”
- Supply chain attack — poisoned upstream datasets propagating to “any model using that data for training and fine-tuning.”
- Unauthorized access — via lateral movement, phished developer credentials, and so on.
CMU frames the pipeline as the attack surface: data generation → curation → training, where “each instance presents an opportunity to alter the data while each person involved presents a potential insider threat.”
Measured impact
An experimental study on CIFAR-10 and an insurance-claims fraud dataset found poisoning “reducing classification accuracy by up to 27% in image recognition tasks and 22% in fraud detection models,” with defences (statistical anomaly detection, adversarial training) “restoring accuracy levels by an average of 15-20%” (source: detectingDatapoisingAttack.md, Kure et al.).
Note the defences restore most but not all of the loss — and this is a controlled study with known poisoning.
Recovery is the hard part
CMU is blunt: while “there is burgeoning research in machine unlearning, which could be used to recover from a data poisoning attack if you know what was poisoned, it is still more effective to retrain the model, a task itself that is extremely expensive. Since recovery is meager at best, prevention is the optimal approach.”
This is the strategic case for chain of custody over detection.
Where it sits in the frameworks
| Framework | Placement |
|---|---|
| AI Act Art 15(5) | Named — “attacks trying to manipulate the training data set (data poisoning)“ |
| NIST AI RMF | Under Secure and Resilient, alongside adversarial examples and exfiltration |
| MITRE ATLAS | Resource Development — poisoning is preparation, effects surface later |
| OWASP Top 10 for LLMs | LLM04 Data and Model Poisoning — adds malicious pickling and the sleeper-agent framing |
| CoSAI | Dedicated CACAO playbook: “Detecting training data poisoning” |
An unusual framing: poisoning as protest
One source covers Nightshade and similar tools as defensive use by creators — “is ‘data poisoning’ a new form of civil disobedience?” (source: In the face of rampant AI, is 'data poisoning' a new form of civil disobedience_.md). Trilateral describes Nightshade as enabling creators “to subtly alter images so models trained on them learn incorrect associations,” e.g. teaching a model that “‘handbag’ should be classified as a ‘toaster’.”
Worth keeping in view: some poisoning in the wild is rights-holders defending their work, not adversaries attacking yours. It degrades your model identically.
Related pages
- poisoning-thresholds — how much poison is enough, and why the sources disagree
- poisoning-defenses
- rag-and-runtime-poisoning
- Retrieval and memory poisoning