AI Security Second Brain

Cascading Hallucination Attacks (T5)

Properties1
descriptionFalse information generated by an agent, then reinforced through its own memory, reflection or inter-agent communication until it becomes operative fact across a system.

4 min read

⭐ Star on GitHub

Summary: False information generated by an agent, then reinforced through its own memory, reflection or inter-agent communication until it becomes operative fact across a system.

Sources: raw/Agentic/Agentic-AI-Threats-and-Mitigations-1.1.pdf

Last updated: 2026-07-29

The term, and why OWASP coined it

“Cascading hallucinations occur when an AI agent generates inaccurate information, which is then reinforced through its memory, tool use, or multi-agent interactions, amplifying misinformation across multiple decision-making steps” (source: Agentic-AI-Threats-and-Mitigations-1.1.pdf, p. 14).

Ordinary hallucination is a single wrong output. LLM09 Misinformation covers it. The agentic variant differs in that the error does not stay put. Two amplification paths (p. 24):

  • Single-agent — “hallucinations can compound through self-reinforcement mechanisms such as reflection, self-critique, or memory recall, causing the agent to reinforce and rely on false information across multiple interactions.” The reflection capability that improves reasoning also launders the error into a conclusion the agent has now “verified.”
  • Multi-agent — “misinformation can propagate and amplify across agents through inter-agent communication loops, leading to cascading errors and systemic failures.”

The worked illustration: “if one agent misinterprets a financial transaction anomaly as legitimate, subsequent agents may validate and act on this misinformation, propagating an incorrect decision across an automated workflow” (p. 15). Each downstream agent’s validation increases apparent confidence while adding no independent evidence.

Attacker-driven, not just accidental

The taxonomy entry frames T5 as an exploit, not merely a failure mode: attacks “exploit AI agents’ inability to distinguish fact from fiction, allowing false information to propagate, embed, and amplify across interconnected systems.” Attackers “can manipulate AI-generated outputs to trigger deceptive reasoning patterns, embedding fabricated narratives into decision-making processes, which can persist and escalate over time, especially in systems with persistent memory and cross-session learning” (p. 24).

So T5 has two entry points — seed a hallucination deliberately, or wait for one and let the architecture do the work.

Four scenarios

Pp. 24–25:

  1. Sales Orchestration Misinformation Cascade — false product details injected into a sales agent’s responses accumulate in long-term memory and logs, so misinformation worsens across future interactions. Note the logs: the audit trail becomes a poisoning vector.
  2. API Call Manipulation and Information Leakage — hallucinated API endpoints introduced into context cause the agent to generate fictitious API calls, leading to data leaks. This is the mechanism behind slopsquatting — see secure coding patterns.
  3. Healthcare Decision Amplification — a false treatment guideline that “progressively builds upon previous hallucinations,” producing dangerously flawed recommendations.
  4. Foreign Exchange Market Manipulation — false exchange-rate information leading agents to negotiate at unrealistic values, causing capital loss and market instability.

The mitigation problem

The T5 table entry (p. 17) prescribes output validation, behavioural constraints, multi-source validation, feedback-loop corrections, and “secondary validation of AI-generated knowledge before it is used in critical decision-making processes.”

Then it concedes the catch:

This will face the same constraints of scaling AI as discussed in Overwhelming Human In the Loop and would require similar approaches.

That is the honest core of T5. Verification does not scale at the rate agents generate claims. If a human validates every AI-generated fact you have removed the reason for the agent; if you use another agent to validate, you have added another hallucination source to the cascade. The document does not resolve this — it points at T10 and adopts the same risk-tiered posture: automate low-risk, escalate high-risk.

Playbook 2 (pp. 35–36) shares its controls with T1, since both are memory-mediated. The T5-relevant ones:

  • Probabilistic truth-checking against trusted sources before committing to long-term storage, and re-checking stored knowledge to detect drift
  • Knowledge lineage tracking — maintain historical references of how a belief evolved, enabling forensic investigation of how misinformation spread
  • Limit propagation from unverified sources — an agent should not use low-trust input for decisions
  • Version control and rollback for knowledge updates
  • Multi-agent or external validation before cross-session commits

Lineage tracking is the most valuable and the least commonly built. It does not prevent the cascade, but it is the only control that lets you find the origin afterwards and roll back everything downstream of it.

Graph View

Backlinks