Summary: Attacking the human oversight layer — by exhausting reviewers until approval becomes reflex (T10), or by exploiting the user’s trust in the agent to steer the person rather than the system (T15).
Sources: raw/Agentic/Agentic-AI-Threats-and-Mitigations-1.1.pdf
Last updated: 2026-07-29
Human oversight is the fallback control behind most of this document — high-risk actions require confirmation, AI-generated code needs review, financial operations need approval. T10 and T15 are the threats to the fallback itself.
T10 — Overwhelming Human in the Loop
“Overwhelming Human-in-the-Loop (HITL) occurs when attackers exploit human oversight dependencies in multi-agent AI systems, overwhelming users with excessive intervention requests, decision fatigue, or cognitive overload” (source: Agentic-AI-Threats-and-Mitigations-1.1.pdf, p. 30).
The structural cause is a mismatch in scaling: “this vulnerability arises in scalable AI architectures, where human capacity cannot keep up with multi-agent operations, leading to rushed approvals, reduced scrutiny, and systemic decision failures.”
This is the document’s sharpest point, and it generalises past T10. Any mitigation of the form “require human approval” degrades as the agent count rises, and the attacker controls the rate. The document raises it as “the critical question of scaling AI safely” (p. 15) and returns to it when conceding that T5 mitigations “face the same constraints of scaling AI.”
Three scenarios (p. 30):
- Human Intervention Interface (HII) Manipulation — compromising the human-AI interaction layer itself: “introducing artificial decision contexts, obscuring critical information, and manipulating perception.” The reviewer is diligent and still wrong, because the display lies.
- Cognitive Overload and Decision Bypass — excessive tasks, artificial time pressure, complex decision scenarios, inducing fatigue and rushed approvals.
- Trust Mechanism Subversion — gradually introducing inconsistencies to degrade human trust in the AI, “creating uncertainty in decision validation and reducing system oversight effectiveness.”
Scenario 3 runs opposite to T15 and is worth noting: too little trust breaks oversight as surely as too much. A reviewer who has learned the agent is unreliable stops reading its explanations.
The RPA example threat model gives the concrete version: “an attacker uses a prompt injection to escalate thousands of low-priority approval requests, causing reviewers to rubber-stamp high impact fraudulent transactions” (p. 49).
T15 — Human Manipulation
“In scenarios where AI agents engage in direct interaction with human users, the trust relationship reduces user skepticism, increasing reliance on the agent’s responses and autonomy” (p. 19). Attackers “coerce agents to manipulate users, spread misinformation, and take covert actions.”
The target is the person. “The implicit trust in AI responses reduces scepticism, making this an effective method for social engineering through AI” (p. 30). The agent is the delivery channel and its perceived authority is the exploit — the same message from an unknown sender would be treated with suspicion.
Two scenarios (pp. 30–31):
- AI-Powered Invoice Fraud — indirect prompt injection into a business copilot replaces legitimate vendor bank details with the attacker’s. “The user, trusting the AI’s response, unknowingly processes a fraudulent wire transfer.”
- AI-Driven Phishing Attack — a compromised assistant generates a message instructing the user to click a malicious link disguised as a security update. Account takeover follows.
Both appear again in the Enterprise Co-Pilot threat model (p. 44). Scenario 1 is the most operationally realistic attack in the entire document: the payload is a changed account number, the delivery is a trusted summary, and nothing about the interaction looks anomalous.
The document connects T15 to T7: “related to deceptive behaviors is the human manipulation we see by agents exploiting the trust humans develop, especially with conversational agents in co-pilot settings” (p. 15). T7 is the agent deceiving on its own initiative; T15 is an attacker directing it.
Mitigations — Playbook 5
Pp. 40–41.
Proactive — reduce fatigue
- AI trust scoring to prioritise review queues by risk level
- Automate low-risk approvals; reserve human oversight for high-impact tasks
- Limit AI-generated notifications; frequency thresholds on requests and approvals
- Dual-agent verification before an agent modifies its own operational goals
- AI-assisted explanation summaries so reviewers decide faster on clearer information, with mechanistic explainability frameworks suggested to scale this (citing arxiv.org/html/2404.14082v1)
- Adaptive workload distribution across reviewers
Reactive
- Goal consistency validation to block unintended behavioural shifts
- Track goal modification request frequency per agent
Detective
- Cryptographic logging and immutable audit trails
- Real-time anomaly detection on decision workflows
- Log human overrides and analyse reviewer patterns for bias or misalignment
- Flag decision reversals in high-risk workflows
The T10 table entry (p. 18) adds the central idea: adaptive trust mechanisms — “dynamic AI governance models that employ dynamic intervention thresholds to adjust the level of human oversight and automation based on risk, confidence, and context,” with hierarchical collaboration automating low-risk decisions and prioritising human attention for high-risk anomalies.
For T15 specifically (p. 19): monitor that agent behaviour aligns with its defined role, restrict tool access, limit the agent’s ability to print links, and filter manipulated responses using guardrails, moderation APIs “or another model.”
“Limit the agent’s ability to print links” is the most concrete, immediately implementable control in the document — narrow, cheap, and directly defeats T15 Scenario 2 and the link-rendering exfiltration in the Enterprise Co-Pilot model (p. 43).
The unresolved tension
Playbook 5’s proactive controls all reduce the number of decisions reaching a human. That is the correct response to T10 and it necessarily widens the automated path — which is where T15, T2 and T6 operate. Risk-based triage moves the exposure rather than removing it, and the residual question is whether the risk classifier can be manipulated by the same attacker. The document does not address that.
Related pages
- agentic-threat-taxonomy — T10 and T15 in context
- intent-breaking-and-deception — T7, the agent-initiated form of deception
- cascading-hallucination — T5, which hits the same scaling ceiling
- agentic-mitigation-playbooks — Playbook 5 in full
- PITAX persuasion and framing