Summary: Two distinct disciplines share the name — using AI to accelerate IR, and responding to incidents that target AI systems. This vault cares mainly about the second, but the sources insist on treating them together.
Sources: raw/AI-incident-response/What is AI Incident Response_ A Practical Overview.md; raw/AI-incident-response/Incident response for AI systems.md; raw/AI-incident-response/Defending AI Systems...md
Last updated: 2026-07-28
The two meanings
“AI incident response is a security discipline that covers two converging areas: applying artificial intelligence to speed up how teams detect, investigate, and contain threats, and the specialized process of responding to incidents that target AI systems such as models, training pipelines, autonomous agents, and inference endpoints” (source: What is AI Incident Response…md).
The argument for not separating them: “A SOC team investigating a compromised cloud workload may discover the attacker accessed an AI training dataset, requiring both AI-enhanced investigation speed and AI-specific containment procedures… separating them creates the exact blind spots attackers exploit.”
Also worth keeping straight: “‘automated’ and ‘AI-driven’ are not the same thing. Automated IR means scripted SOAR playbooks that execute predefined steps when a condition fires, like blocking an IP address. AI-driven IR means systems that reason through evidence, adapt their investigation path based on what they find, and make contextual decisions.”
There is a third meaning entirely, from the AI Incident Database: a public official response by the entity responsible for an incident. See aiid-incident-responses.
The baseline everyone extends
NIST SP 800-61r2 — preparation, detection and analysis, containment and recovery, post-incident activity. Both MITRE ATLAS and CoSAI are “now adapting these frameworks for AI-specific threats” (source: What is AI Incident Response…md).
What actually breaks
Microsoft identifies four properties of AI systems that defeat traditional IR (source: Incident response for AI systems.md):
- Taxonomy gaps — harm categories like “generating dangerous instructions, enabling misuse through natural-language interfaces, and targeting specific groups that confidentiality/integrity/availability frameworks don’t cover. Triage loses signal when incidents default to generic buckets.”
- Context-dependent severity — “A model producing inaccurate medical guidance presents a fundamentally different risk than the same model producing inaccurate trivia.”
- Ambiguous root cause — “Problematic behavior can emerge from the interaction of training data, fine-tuning choices, retrieval inputs, and user context. Investigation may narrow contributing factors without isolating a single defect.”
- Telemetry gaps — “Standard security monitoring doesn’t capture anomalous output patterns, classifier confidence shifts, or unexpected post-update behavior.”
The framing sentence: “Traditional incident response assumes deterministic systems. A given input produces a given output, a defect maps to a line of code, a patch can be verified by replaying the failure. AI systems don’t work this way. Outputs are probabilistic. A gap in a safety classifier can produce thousands of harmful outputs before any reviewer sees the first one.”
Detail in microsoft-ir-readiness.
The threat classes
CoSAI’s list of AI threats that “don’t fit neatly into conventional security categories” (source: Defending AI Systems…md):
- Prompt injection
- Memory poisoning
- Context poisoning
- Model extraction
- Jailbreaking
All five are catalogued from the attacker’s side in AI Attack Techniques — see retrieval and memory poisoning and weight-level attacks.
CoSAI’s framing of why this is hard: “Think about how you’d respond to a typical data breach: you’d isolate compromised systems, analyze logs, patch vulnerabilities, and restore from backups. But what happens when an attacker doesn’t break into your system — they simply ask your AI chatbot the right questions?”
Detail in cosai-framework.
Containment is architecture-specific
The containment question has no generic answer. CoSAI poses it as: “Should you roll back to a previous model version? Purge poisoned memory? Rebuild your vector database?” (source: Defending AI Systems…md).
Microsoft’s staged model answers with sequencing rather than a single action — see microsoft-ir-readiness:
- Immediate containment, first hour: “Block known-bad inputs, activate content filters, restrict access.”
- Expand and strengthen, first 24 hours: broaden mitigations “to cover related variants using automated pattern analysis.”
- Fix at source, days to weeks: classifier updates, model adjustments, systemic changes. “Use tactical allow-block lists as triage tools, not durable solutions.”
The regulatory constraint on containment
AI Act Art 73(6) limits what you may do before notifying authorities: the provider “shall not perform any investigation which involves altering the AI system concerned in a way which may affect any subsequent evaluation of the causes of the incident, prior to informing the competent authorities of such action” (source: raw/AI-regulations/Regulation - EU - 2024_1689.md).
Rolling back a model, purging memory or rebuilding a vector store are all alterations of that kind. They remain permitted — but the notification comes first, and it has to be in the playbook, not improvised at 3am. See serious incident reporting.
Reporting clocks to design against
| Regime | Trigger | Deadline |
|---|---|---|
| AI Act Art 73 | Widespread infringement / Art 3(49)(b) | 2 days |
| AI Act Art 73 | Death | 10 days |
| AI Act Art 73 | General | 15 days |
| Code of Practice C9 | Critical infrastructure disruption | 2 days |
| Code of Practice C9 | Serious cybersecurity breach (exfiltration, cyberattack) | 5 days |
The 5-day cybersecurity tier exists only in the Code of Practice. For a GPAI systemic-risk provider, weight exfiltration is a 5-day reportable event with no Art 73 equivalent.
The part most IR programmes miss
Responder wellbeing. “Personnel handling AI safety incidents are exposed to harmful content in ways that differ materially from analyzing malware. Address wellbeing before a crisis: scheduled rotation, structured cognitive breaks, peer mentoring, and collaboration with content moderation teams whose exposure management practices apply directly” (source: Incident response for AI systems.md).
And the honest trade-off: “Rotation and cognitive break protocols reduce effective team capacity — a deliberate investment in sustained performance, not a free addition.”