Agent Context as a Security Boundary

We Need to Start Treating Agent Context as a Security Boundary

Darktrace recently demonstrated an attack technique that deserves more attention than it is getting: conversation history poisoning.

Researchers modified locally stored conversation history so an AI agent believed it had previously been authorized to perform offensive security work.

That raises an important enterprise AI security question.

How much of the context an agent uses to make a decision can actually be trusted?

Agents don’t operate from a system prompt alone. They use conversation history, memory, retrieved data, tool results, instructions and increasingly information supplied by other agents and external systems.

All of that context can influence what an agent decides to do next.

We already authenticate users, validate API requests, protect credentials and control access to applications. But an agent can have a legitimate identity, valid credentials and appropriate permissions while making a decision based on context that has been manipulated.

That makes context integrity part of the security architecture.

Identity tells us who or what is acting.

Authorization determines what it can access.

Governance establishes what should be allowed.

Context influences what the agent believes about the situation.

Runtime controls determine whether the resulting action should proceed.

As agents become more autonomous, I think enterprises will need to establish provenance for that context.

Where did this information come from?

Has it changed?

Who or what supplied it?

Should the agent trust it?

Does it still represent the authority originally granted?

This becomes even more important when context starts moving between memory systems, RAG pipelines, MCP servers, tools and other agents.

Securing the agent isn’t enough if we can’t trust the information the agent is using to make its decisions.

Why this matters: Manipulated context can influence an agent’s behavior without changing its formal identity or permissions.

Decision it changes: Agent security architecture should include integrity and provenance controls for memory and context, alongside identity, authorization and runtime enforcement.

Who should care: CISOs, enterprise architects, AI platform teams, application security leaders and AI governance teams.

What I would ask next: Before an AI agent acts on something in its memory, can your architecture prove where that information came from and that it hasn’t been changed?

Back to Insights