How to Threat Model an AI Agent

An AI agent threat model extends chat and RAG risk with action: tools that send email, modify databases, browse the web, call MCP servers, and chain side effects across multiple turns. The model proposes plans; your runtime must enforce what actually executes.

This guide threat-models agent architectures — tools, permissions, external content, credentials, high-impact actions, human approval, and runtime monitoring. Start with what is AI agent security for vocabulary. For chat-only features, use threat model an LLM application first.

Agent vs chat-only threat surface

Agents compound OWASP LLM06: Excessive Agency with LLM01 injection — manipulation plus capability.

Agent architecture diagram

User task
   ↓
Agent orchestrator (your code)
   ↓
LLM proposes: text and/or tool calls
   ↓
┌──────────────────────────────────────┐
│ Pre-execution gates                  │
│  • input / injection scan on context   │
│  • Agent Action Guard on tool proposal │
│  • human approval (optional)           │
└──────────────────────────────────────┘
   ↓ allow
Tool runtime (APIs, DB, MCP, browser)
   ↓
Tool result (untrusted) → scan → back to LLM

The orchestrator is the authorization authority. The model is an untrusted planner.

Assets specific to agents

Trust boundaries for agents

Tool output injection is LLM01 via the return path — not only user chat.

Threat model worksheet (agent template)

Extend the table for your tool inventory. One row per tool × abuse pattern is often clearer than generic rows.

Tool inventory exercise

List every tool the agent can call:

For each write action, define:

• Allowed without approval?
• Required arguments schema?
• Tenant scope source (session, not model)?
• Action Guard policy_id?

See AI agent permissions and least privilege for agents.

Pre-execution control flow

Every proposed tool call should pass:

async function executeToolProposal(proposal: ToolProposal, ctx: SessionContext) {
  const guard = await callUnifiedGuard({
    checks: ["agent_action"],
    agent_action: {
      tool_name: proposal.tool,
      action: proposal.action,
      arguments: proposal.args,
      context: summarizeContext(ctx), // no secrets
      policy_id: ctx.policyId
    }
  });

  if (guard.decision === "block") {
    logAgentEvent({ decision: "block", request_id: guard.request_id });
    return { error: "action_blocked" };
  }
  if (guard.decision === "review") {
    await approvalQueue.enqueue(proposal, guard.request_id);
    return { status: "pending_approval" };
  }
  return runTool(proposal, ctx);
}

Pair with Agent Action Guard. Text guards do not replace action policy.

Human-in-the-loop placement

Route to humans when:

• Financial or legal commitments
• Destructive operations (delete, purge, mass update)
• Ambiguous review verdicts from action or output guards
• First-time tool use for a tenant tier

Human-in-the-loop agent actions — block unsafe content from reaching reviewers.

External content and browsing agents

Agents that browse inherit web page prompt injection:

• Fetch through sanitizing proxies
• Cap response size; strip active HTML
• Scan extracted text before re-prompting
• Do not let the model choose arbitrary URLs without allowlists (secure agent web browsing)

Credentials and secrets

• Never pass raw API keys in tool arguments assembled from chat
• Use short-lived tokens scoped per tool
• Action guard should block when secrets appear in context or serialized args
• Rotate credentials if a session logged unsafe on secrets scan

Runtime monitoring for agents

Log metadata per tool attempt (agent runtime monitoring):

{
  "event": "agent_tool_guard",
  "session_id": "sess_abc",
  "tool_name": "database",
  "action": "update",
  "decision": "review",
  "matched_rule": "Writes require approval",
  "guard_request_id": "req_guard_xyz",
  "execution": "queued"
}

Correlate with injection scan request_id on tool outputs.

MCP-specific threats

If tools arrive via Model Context Protocol:

• Harden MCP servers at deployment boundary
• Enforce tool permissions in host, not in model
• Screen MCP prompt injection on results

MCP standardizes wiring — not policy.

Testing agent threat mitigations

Automate defensive fixtures — AI security test suite. Periodic red team exercises for novel chains.

Prioritization matrix

Before enabling write tools in production:

1. Action Guard on every proposal (A1, A2)
2. Tenant scope from session (A8)
3. Tool output injection scan (A4)
4. Human approval for destructive/high-value (A7)

Before enabling browse/fetch:

5. URL allowlists + response scan (A4, A9)

Related checklists

• AI agent security checklist
• Source and sink security
• Safe destructive agent actions

Summary

Threat model AI agents by inventorying tools and credentials, marking untrusted content at every re-prompt, enforcing pre-execution action policy with Agent Action Guard, scanning tool outputs for injection, routing high-impact operations to human approval, and monitoring with metadata-only logs. Injection that merely annoys in chat can cause incidents when agents have write access — design the orchestrator as the enforcement point.

Mitigate agent threats with Action Guard · AI agent security checklist