Threat-Model an AI Agent: Sources, Sinks, Authority, and Blast Radius

An AI agent is not secured because its system prompt says to ignore malicious instructions.

It is secured when a malicious instruction can reach the model, influence a plan, and still fail to obtain the identity, authority, data, network path, or approval required to cause harm.

That distinction matters in 2026. The OWASP Top 10 for Agentic Applications now gives teams a shared taxonomy for goal hijacking, tool misuse, identity abuse, supply-chain failures, unexpected code execution, memory poisoning, cascading failures, and other agent-specific risks. NIST’s large-scale agent red-teaming study analyzed more than 250,000 attack attempts across 13 frontier models and found at least one successful attack against every model tested.

The useful conclusion is not that agents are impossible to secure. It is that the threat model cannot stop at the model.

This is the first guide in the Agent security engineering series. It gives you a threat-modeling method that produces enforceable controls rather than a list of scary prompts.

Start with paths, not prompts

OpenAI’s source-to-sink framing is a useful security primitive. An attacker needs a source they can influence and a sink that becomes dangerous when the agent uses it in the wrong context.

Sources include:

• webpages, search results, emails, documents, tickets, chat messages, and code comments;
• MCP resources and tool results;
• recalled memory and retrieved knowledge;
• another agent’s task, result, or Agent Card;
• project-local configuration and skills loaded before a user understands their effect.

Sinks include:

• sending data to a third party;
• changing external state;
• executing code or shell commands;
• writing durable memory;
• delegating to another agent;
• spending money, issuing a refund, deleting data, or changing access.

A prompt injection is therefore not the whole incident. It is one edge in an attack path:

attacker-controlled source
  -> content enters context
  -> model treats data as instruction
  -> plan requests a capability
  -> runtime supplies identity and authority
  -> tool reaches a sink
  -> harmful effect succeeds

Threat modeling should ask which edges are necessary and which deterministic control can break each one.

Inventory six things

Before drawing attack paths, inventory the system in six columns.

Do not label an entire connector “trusted.” Trust is more granular:

• The connector code may be reviewed while the data it retrieves is attacker-controlled.
• The MCP server may be authenticated while one resource contains an injection.
• The user may be authorized while the agent is not authorized to act with the user’s full privileges.
• The tool may be safe for reads while one argument turns it into a write or an exfiltration sink.

The unit of analysis is a path through capabilities and data, not a product name.

Model the action as a risk vector

A single low/medium/high label hides too much. ContextOS represents action risk across multiple dimensions:

type ActionRisk = {
  effect: "none" | "local_state" | "external_state" | "physical_world"
  authority: "agent" | "service" | "user_delegated" | "human_approved"
  reversibility: "read_only" | "reversible" | "compensatable" | "irreversible"
  interaction: "api" | "browser" | "computer" | "agent_to_agent" | "human_handoff"
  data_scope: "PUBLIC" | "INTERNAL" | "CONFIDENTIAL" | "RESTRICTED"
  decision_ttl_seconds?: number
}

Two actions can both be classified as network and still have radically different risk. Fetching a public status page and uploading a restricted incident report are both network interactions. The second combines sensitive data, an external sink, and delegated authority.

Use the vector to decide:

• whether the capability is surfaced at all;
• which arguments must be constrained;
• whether user delegation is required;
• which approval mode is the ceiling;
• whether the effect needs idempotency or compensation;
• how long the authorization decision remains valid;
• which eval scenarios must exist before release.

Draw the dangerous paths

For each high-value sink, work backward to every source that can influence it.

Now add preconditions. A realistic exfiltration path usually needs all of these:

1. attacker-controlled content reaches the context;
2. the model follows it;
3. sensitive data is available in the same run;
4. a communication sink is surfaced;
5. the destination is attacker-controlled or redirectable;
6. policy allows the call;
7. the call does not require meaningful approval;
8. the runtime permits the network effect.

Model robustness can reduce step 2. Runtime security should make several of steps 3 through 8 independently false.

A worked refund-agent threat model

Assume a support agent can read tickets, look up transactions, draft replies, and propose refunds. Refunds above a threshold require finance approval.

The naive architecture gives the model all four tools and relies on a prompt: “Only refund eligible customers.”

The governed architecture compiles a narrower run:

intent: support.refund.evaluate
safety_mode: delegated
delegated scopes: tickets.read, payments.read
surfaced tools: ticket.read, payment.lookup, reply.draft
withheld tool: payment.issue_refund

If eligibility is established, the next decision can request the refund capability. The Tool Gateway then checks:

• the agent subject and workload identity;
• the delegated user’s scopes;
• tenant equality;
• refund amount and transaction ownership;
• required evidence references;
• approval-mode ceiling;
• finance approval and its expiry;
• idempotency key and previous mutation receipt.

The ticket can still contain an injection. The model might even propose the wrong action. But the attack path does not reach the payment sink because the first run has no refund capability and the second boundary re-evaluates identity, evidence, policy, approval, and arguments.

Put deterministic controls at every boundary

The model can participate in detection and review. It should not be the only enforcement point.

This is defense in depth with different failure assumptions. A content classifier may miss an injection. A model may follow it. An approval reviewer may be tired. The system should still constrain credentials, destinations, arguments, effects, and blast radius.

Turn the threat model into tests

A threat model that ends as a diagram will drift. Every dangerous path should produce at least one test.

• Source tests: injections in HTML, PDFs, email threads, tool results, code comments, and recalled memory.
• Authority tests: expired claim, wrong audience, revoked agent, cross-tenant principal, over-broad child scope.
• Argument tests: amount over limit, unapproved recipient, redirecting URL, path traversal, wildcard resource.
• Effect tests: duplicate write, partial failure, timeout after mutation, compensation failure.
• Evidence tests: missing source, stale source, fabricated reference, untrusted memory promoted as policy.
• Repetition tests: the same scenario attempted many times and with adaptive variations.

The evaluator verdict should be deterministic over the recorded outcomes. If a high-severity external effect succeeded once, the gate fails once. Do not average it away with hundreds of benign runs.

The design-review questions

Ask these before giving an agent a new capability:

1. Which attacker-influenced sources can reach this run?
2. Which sensitive data can coexist with those sources?
3. What is the most dangerous sink the run can reach?
4. Whose authority is used at that sink: agent, service, user delegation, or fresh human approval?
5. Can source content change the destination, recipient, amount, path, or scope?
6. Which control still holds if the model follows the malicious instruction?
7. Can the same effect happen twice after a retry or timeout?
8. Can a child agent or remote tool widen authority?
9. Can an operator revoke the agent, credential, tool, or destination immediately?
10. Which trace proves the denial or the authorized effect later?

If the answer to question 6 is “the prompt,” the design is not ready.

Research base

• OWASP Top 10 for Agentic Applications 2026 for the agent-specific risk taxonomy.
• NIST: Insights into AI Agent Security from a Large-Scale Red-Teaming Competition for current evidence that all tested frontier models yielded at least one successful hijacking attack.
• OpenAI: Designing AI Agents to Resist Prompt Injection for the source-to-sink and constrained-impact framing.
• Anthropic: How We Contain Claude Across Products for the separation between model, environment, and external-content defenses.
• ContextOS: Security and Compliance, Governance, Adapter Mesh, and Evaluation and Observability.

What to read next

• Prompt Injection Is a Boundary Problem
• Secure the MCP and Tool Supply Chain
• Red-Team Agent Hijacking
• Agent security engineering series