Agents Have Boundary Issues

A look at the five isolation boundaries where agent security fails, and how they map to real-world agent deployments and controls

By now, it is clear that agents are moving from demos and research prototypes into real systems, reading files, calling tools, browsing the web, editing state, and coordinating with other agents over long-running workflows.

I’ve been writing about this shift for a while now, from breaking down the OWASP Top 10 for Agentic Applications to examining Agentic AI Threats and Mitigations, and one theme keeps surfacing across all of it, which is that agent security failures that look wildly different on the surface tend to share the same underlying causes.

That said, the research literature on agent security has been fragmented, organized around attack types, application domains, and benchmarks, which makes it hard to explain why prompt injection, tool misuse, and memory poisoning so often rhyme with one another. A new research publication titled “Isolation as a First-Class Principle for LLM-Agent System Safety” takes a crack at fixing that, organizing the entire space around five isolation boundaries and asking one question of every attack and defense, where does the loss of isolation first occur?

I really like their 5 boundary framing and found it useful for thinking about potential agent risks and failures.

In this article, I will walk through the five boundaries the researchers lay out, the risks and defenses at each one, and then map the taxonomy onto the agent deployment patterns and common security controls (sandboxes, hooks, posture management, runtime monitoring, and hard boundaries among others) that practitioners are actually wrestling with.

---

---

Isolation as a First-Class Principle

The paper’s core argument helps simplify what can feel like a complex domain of AI and Agents.

The authors treat isolation, meaning the separation of user inputs, tool access, execution channels, inter-agent communication, and environment-originated context, as the organizing principle for agent safety. In their words,

“agent safety improves when system boundaries are explicit and enforced structurally, rather than left to prompt instructions alone.”

That single sentence is key, because it captures the difference between what many of us have taken to calling soft guardrails and hard boundaries. Soft guardrails are instructions, system prompts, and model training, all of which are probabilistic and all of which can be talked out of their behavior by a sufficiently motivated adversary.

I actually went into detail in a recent video where I discuss AI jailbreaks, system prompts and more as they relate to soft guardrails as security controls:

Hard boundaries are structural, things like permission systems, sandboxes, network egress controls, and deterministic policy checks that sit outside the model and don’t care how persuasive the injected text is. This mirrors the conversation I had with Luke Hinds on the podcast in Your AI Agent Is Running As Root, where he made the point that application-layer controls can be circumvented while kernel-level enforcement cannot be jailbroken.

The taxonomy places the agent core (policy, planning, and memory) at the center, surrounded by five interfaces where things can go wrong. Each boundary is summarized as:

• The user-agent boundary concerns whether user content remains data or becomes control.

• The agent-tool boundary concerns how external capabilities are accessed.

• The agent-execution boundary concerns the transition from reasoning to action.

• The agent-agent boundary concerns communication and coordination across multiple agents.

• The system-environment boundary concerns how the agent system interacts with external content and state as a whole.

  The authors classify prior work by its “primary safety boundary,” the point where the loss of isolation first occurs, which keeps things clean even though most real attacks end up crossing several boundaries before they’re done.

Let’s walkthrough each of the primary boundary types and how the paper discusses them.

Boundary 1 - User-Agent

The first boundary asks whether an agent can keep user content from becoming privileged control.

In a well-isolated system, user input should remain a request, a query, or task data, while system and developer instructions keep higher authority. Failure begins when user-controlled content starts to steer internal policy, which is the classic direct or indirect prompt injection and jailbreak story, where as the authors put it, “a low-privilege source can behave like a high-privilege one.”

What I appreciated is that the survey pushes well past single-turn jailbreaks. Multi-turn attacks exploit gradual steering across a conversation, implicit clues, and long-context overload, leading to the observation that “many systems look safer in one-shot evaluation than they are in realistic sessions.”

The attack surface also expands in multimodal settings, where user-provided images can carry visual or typographic instructions that bypass text-oriented safeguards. And perhaps most relevant for anyone running agents with persistent memory, the strongest recent trend is persistence, with in-context poisoning and memory injection showing that user influence can remain after the original interaction ends. This maps directly to Memory & Context Poisoning from the OWASP Agentic Top 10, which I covered in depth in my breakdown after serving on the Agentic Security Initiative’s review board for the project.

Defenses at this boundary fall into three groups, making authority separation explicit through structured queries, signed prompts, and DSL-style interfaces, hardening the model itself against jailbreaks through safety classifiers and semantic smoothing, and repairing after degradation through unlearning, editing, and refusal-boundary control.

The takeaway from the authors is that user-agent safety “must move beyond short prompts and static refusal scores toward long-session robustness, multimodal authority separation, and recovery from persistent compromise.”

Boundary 2 - Agent-Tool

The agent-tool boundary is where I suspect most enterprise pain will actually materialize, given the explosion of MCP servers and tool ecosystems we’ve watched over the past two years. Clutch Security actually had one of the better reports on this, titled “MCP Servers: What We Found When We Actually Looked”.

They found a 2,000 growth of MCP servers in a single year (not counting 2026), which were widely connected to enterprise services and overwhelmingly running on developers endpoints rather than enterprise servers, making them a major supply chain risk.

The framing here is that “tools should extend what the agent can do without taking over how it decides.” The basic failure mode is that tool-returned content gets treated as trusted instruction, which is the indirect prompt injection story every security team has now heard, but the paper enumerates a fuller failure set, noting the system can fail:

“by choosing the wrong tool, passing unsafe arguments, trusting malicious output, or composing plausible calls into an unsafe workflow.”

The MCP-specific findings deserve attention from anyone deploying agents against tool catalogs. In MCP-style ecosystems, the model sees tool descriptions, capability advertisements, and metadata before it ever invokes a tool, and the authors point out that “metadata is not neutral from the model’s perspective. It can shape preferences, change routing, and bias decisions before real tool output even appears.”

Benchmark work such as MCP Security Bench shows that attacks against the protocol layer are not an edge case but a natural extension of tool-mediated prompt injection. The below image from the paper shows various examples of tool attacks vectors.

As tool use standardizes through protocols and orchestrators, the boundary no longer sits only at the moment of API invocation, but also at discovery, ranking, metadata interpretation, and workflow construction. I flagged tool poisoning against MCP as an emerging concern back in my Agentic AI Threats and Mitigations piece, and the research has since caught up in a big way.

There’s also a trajectory-level insight here that practitioners should be thinking about, which is that a single tool call may look harmless while the full trajectory becomes unsafe. This is why organizations need security tools that watch the entire trajectory, not providing security rigor for individual tool calls in isolation.

The security target is not one prompt or one API call but the full trace of calls, arguments, observations, and intermediate state. The broader lesson, in the authors’ words, is that “safer tool use depends less on the model inferring the right behavior from natural language, and more on interfaces that make capability scope, trust level, and action semantics explicit.”

That is about as clean an articulation of least privilege for agents as you’ll find.

Boundary 3 - Agent-Execution

The agent-execution boundary concerns the point at which internal decisions become real actions, and failure begins when the system treats model output as ready-to-execute behavior without enough mediation.

As the paper puts it, this boundary “turns control errors into operational impact.” A model producing unsafe text is one kind of problem, while an agent that clicks the wrong button, runs unsafe code, or submits the wrong form is another entirely.

I’ve been making this argument repeatedly, in my blogs, videos and my lectures at CMU’s AI programs, pointing out that the potential blast radius and risk to organizations is far great from agents with autonomy and the ability to take actions on production systems than it was for chatbots related to hallucinations or unintended responses.

The browser and GUI agent findings are sobering. Benchmarks like SafeArena and ST-WebAgentBench show that web agents can produce unsafe outcomes through clicks, submissions, and navigation, and the research shows that refusal-trained models remain vulnerable once they act through an interface rather than a chat window. In other words, the safety training that holds up fine in a chat box degrades when the same model is grounded in an interface and taking actions.

This is also a major factor that has led to organizations such as Gartner outright recommended organizations do NOT adopt agentic browsers, and the technology itself having a slower adoption curve than in say, coding agents.

This is where the control conversation gets practical, because defenses at this boundary increasingly focus on containment rather than only prevention, including constrained execution, zero-trust architectures, policy-executable safeguards, and sandbox ecosystems.

This is exactly the design philosophy behind the sandboxing and permission models in modern coding agents, and behind hooks, the deterministic pre- and post-action checkpoints that products such as Claude Code expose so teams can enforce policy in code rather than in prompts (Anthropic’s agentic coding best practices are a useful reference here).

The paper also nods to the broader industry direction of decoupling the brain from the hands, separating the model that decides from the runtime that acts, so unsafe decisions can still be checked, delayed, or blocked before they create side effects. The authors’ bottom line is that safe agent design must treat execution “as a governed interface, not as the automatic continuation of model output.”

This is also the exact premise behind AARM (Autonomous Action Runtime Management), the CSA-powered specification for agent runtime security where I serve as a Co-Lead, and which I recently covered in AARM and the Case for Standardizing the Agent Runtime Security Category.

The paper's conclusion that execution must be a governed interface is essentially AARM's founding argument, that the stable security boundary for agentic AI isn't the model, the prompt, or the orchestration layer, but the action execution boundary, and that deterministic controls need to be enforced there, outside the agent's reasoning loop.

The reason for this of course is that soft guardrails such as system prompts are imperfect (as any security control is), and models are non-deterministic by nature, so we need deterministic controls outside the model to enforce rigorous security.

Boundary 4 - Agent-Agent

The agent-agent boundary concerns what happens when multiple agents communicate, delegate, debate, and share intermediate state.

In a well-isolated system, one agent’s message should remain a bounded contribution rather than an unverified control signal for others. The memorable line from the paper is that:

“a message is not just information. It can also be a carrier of control.”

The research here validates concerns I’ve raised repeatedly about multi-agent architectures as well as the reality that were in the infancy of inter-agent protocols, such as A2A.

The Prompt Infection research cited in the survey found that one compromised agent can pass malicious instructions to others, turning ordinary coordination into an attack channel, and the progression the authors describe is worth quoting because it reads like an incident report waiting to happen, “first one compromised message, then repeated propagation, then communication-level cascade.”

Topology matters too, since network structure, routing rules, and memory sharing determine how fast compromise travels and how hard it is to contain. Work like Trojan Hippo shows that shared memory is itself a high-risk surface, because poisoned memory can be weaponized for later exfiltration or control.

The threat moves from bad messages, to bad agents, to bad shared state, at which point rollback becomes much harder than in a single-agent system. Readers who have gone through my OWASP Agentic Top 10 breakdown will recognize Insecure Inter-Agent Communication and Cascading Failures living at exactly this boundary.

On the defense side, efforts such a Project Agent Memory Guard are starting to materialize. It’s aim is to stop AI agents from being weaponized through their own memory by sitting between the agent and the memory store, screening every operation through a pipeline of detectors and declarative policies.

The section closes with a line I’d encourage every architect building multi-agent systems to keep in mind:

“collaboration is not automatically a safeguard. Without isolation, it can become a mechanism for amplification.”

Boundary 5 - System-Environment

The final boundary concerns how an agent reads and reacts to the outside world.

In a well-isolated system, webpages, retrieved passages, documents, emails, interface elements, and memory artifacts should remain observations rather than hidden commands. Failure begins when environment-originated content is absorbed into agent context and then treated as if it had authority, which is the indirect prompt injection problem that has haunted LLM-integrated applications since the earliest research on the topic.

I continue to cite Google DeepMind’s paper on this titled “AI Agent Traps”, which discusses how the information environment itself from the web now:

becomes a vulnerability where adversarial content can manipulate, deceive or exploit visiting agents.

The survey shows this problem is broader and more persistent than first expected. For web and computer-use agents, the environment is not just read but clicked, navigated, and executed against, leading to the conclusion that “web environments are active adversarial surfaces, not passive information sources.”

Visual prompt injection work extends this to interface appearance itself, showing computer-use agents can be misled through how a page looks rather than what its text says. RAG systems expose another variant, where the issue is often corrupted evidence rather than explicit instruction, with work like PoisonedRAG and BADRAG showing attackers can manipulate knowledge bases so the agent reasons from compromised support, and the disclosure findings cut both ways, since “the environment can both push corrupted knowledge in and pull private knowledge out” through data extraction and membership inference attacks.

It’s easy to see how the adversarial content can quickly become a major problem for agents and this realization sits at the heart of what I call the Security vs. Utility tradeoff for AI Agents. The utility an agent has in terms of autonomy, tools and the ability to take actions, the more problematic it becomes security wise as well.

Defenses here are evolving from detection toward provenance, with approaches like Spotlighting and Task Shield aiming to mark, isolate, or filter environment-originated instructions before they silently become control input, and newer work adding causal attribution and trustworthy evidence selection.

The lesson from the authors is that system-environment safety “will likely depend less on stronger generic refusal and more on better source authentication, provenance tracking, memory hygiene, and context attribution.”

When Boundaries Fail Together

The taxonomy’s real payoff is in cross-boundary analysis.

Serious failures often cross boundaries, and the common pattern the authors describe is sequential escalation, where user input first overrides control at the user-agent boundary, then steers tool use, and finally triggers unsafe execution.

Another pattern starts from the environment, where a malicious webpage or poisoned memory item enters through the system-environment boundary and later propagates into tool calls, agent communication, or action traces. The unit of analysis is the full control path, not a single prompt or tool call, which is why “local robustness at one interface does not guarantee system-level safety.”

The forward-looking agenda the paper lands on is what the authors call isolation-by-construction, building interfaces where the boundary remains visible to the system itself.

Inputs from users, tools, peer agents, and external content should remain distinguishable, capability access should be scoped, propagation paths should be observable through trace-level monitoring, and recovery should be a core requirement once compromise reaches memory or shared state.

They’re also candid about open problems, noting that many benchmarks still test single boundaries while real failures are cross-boundary, and that the field still lacks stable abstractions for authority, trust, and privilege in full workflows.

From Taxonomy to Deployment Patterns

Academic taxonomies are only useful if they map to how agents actually show up in the enterprise, so let’s do that mapping.

As I discussed in my recent video breaking down OWASP’s State of Agentic AI Security and Governance report, agents are landing in organizations through a few distinct patterns, endpoint agents such as coding CLIs and agentic browsers running on user machines, custom or homegrown agents built on cloud platforms and orchestration frameworks, and SaaS-embedded agents that arrive inside the productivity and business platforms you already own.

Each pattern weights the five boundaries differently.

Endpoint agents live and die at the agent-execution boundary, since they run with the user’s permissions, credentials, and file access, which is exactly the “running as root” problem Luke Hinds and I dug into, and why sandboxes, hooks, and OS-level enforcement matter most there.

Custom-built agents concentrate risk at the agent-tool and agent-agent boundaries, because you are the one wiring up MCP servers, tool catalogs, memory stores, and multi-agent topologies, and every one of those design decisions either preserves or erodes isolation.

SaaS-embedded agents shift the weight toward the system-environment and user-agent boundaries, since you control neither the model nor the runtime, and your exposure comes through the content, connectors, and data the agent can reach inside the platform.

This is also where the emerging crop of agent security tooling slots in, and I find it more useful to think about it in terms of two complementary functions rather than vendor categories or acronyms.

The first function is preventative and posture-focused, inventorying agents across all three deployment patterns, understanding their tools, identities, entitlements, and data access, and driving least privilege before anything goes wrong, which in taxonomy terms means knowing where your boundaries are and how much isolation each one actually enforces. Think of it akin to “Shift Left” for Agentic AI, aimed at posture, configurations, policies and more, before agents are running in production, or identifying deviations from desired posture for agents already deployed.

The second function is runtime monitoring and response, watching agent behavior for goal hijacks, anomalous tool use, and cross-boundary propagation, which maps directly to the paper’s calls for trace-level monitoring, attribution, and recovery.

Neither function replaces the hard boundaries themselves, but posture tells you where isolation is weak and runtime monitoring tells you when it has failed, and the paper's cross-boundary escalation patterns are a solid blueprint for what your runtime detections should actually be looking for.

For those wanting a fuller landscape view, I’ve started doing a video series breaking down the OWASP Agentic AI Top 10 one-by-one such as my first video below where I cover Agent Goal Hijack.

You can check out the videos for my breakdown of the OWASP Agentic AI Top 10 pairs naturally with this paper, since nearly every risk in that list is a boundary failure wearing a different name.

Closing Thoughts

It of course is debatable whether the industry needed another taxonomy, and the authors themselves acknowledge theirs is not the only valid way to organize the literature.

That said, I find the boundary-centric framing helpful, because it explains why so many differently-named attacks share the same structure and it gives practitioners a common language that spans research papers, OWASP guidance, and vendor categories alike. This builds on the excellent work of OWASP and others, such as my friend Rock Lambros, who’s blog you should follow if you haven’t already, as well as leaders like Ken Huang who created MAESTRO, for threat modeling agentic systems.

The message security teams should take away is one I’ve been hammering for a while now, which is that we cannot prompt or guardrail our way to secure agents. Isolation has to be designed in, enforced structurally, and monitored continuously, and organizations adopting agents across endpoints, custom builds, and SaaS platforms would do well to ask one question of every deployment:

Where are the boundaries, and what actually enforces them?