It’s no secret that I’m a fan of content from the Coalition for Secure AI (CoSAI).
I previously referenced their excellent publication on Agentic IAM for a blog, and interviewed the authors on the Resilient Cyber podcast. Now, they recently released a great whitepaper on Zero Trust for AI systems and I wanted to take sometime to walkthrough it here, because despite all the hype and media narrative of existential risks, most of the cyber risks related to AI are tied to cybersecurity fundamentals (or a lack thereof in the incidents we’ve seen).
So let’s check it out.
Opening Notes
The publication rightly opens discussing the dichotomy between providing enough agency and autonomy to AI systems to make them useful, but not so much that they present unacceptable levels of risk or potential security impact.
I’ve been calling this the Security vs. Utility tradeoff, something that has long existed by with AI Agents and the ability to take actions, it has become even more problematic. The fact that agents can call tools, take actions, and drive outcomes is what makes them so useful.
However, it is also what makes them so problematic from the security perspective, as threading the needle between the two is not easy, especially in large complex environments, with agents sprawling across endpoints, Cloud and SaaS.
The paper opens discussing some of the frontier model escapes and hacks, which have dominated headlines (e.g. Hugging Face, escapes across the labs by agents and now the Australian government among others). The authors rightly point out that sound security principles such as time-bound auditable authorization, network behavior monitoring and others would have mitigated and identified these activities.
This is a view I strongly agree with, while the industry and mainstream media argues about model alignment and altruism, the reality is most of these incidents were tied to poor security hygiene, posture, configurations and detection and response, a point others such as Zack Korman have made.
A core focus of the paper is related to enforcing Zero Trust at runtime for AI systems and ensuring authorization isn’t delegated to non-deterministic system components (e.g. the model).
Before diving into the specifics, we should also cite how they define Zero Trust:
”Zero Trust Architecture is a security paradigm named for its core principle of “never trust, always verify.” No resource, regardless of its type, is trusted implicitly. Instead, authorization is contextually granted for time bounded activities, limited in scope. With a security posture that always assumes breach, Zero Trust is designed to limit the blast radius of incidents, establishing time and task constrained trust zones so that additional resources may not be compromised.
Why do we need Zero Trust for AI?
The paper also asks an obvious question of, why do we need Zero Trust for AI? The question is isn’t more crucial in the face of doomerism and claims that we just need better aligned models, rather than a found on sound engineering and security principles.
As the authors point out, AI does introduce novel considerations, primarily systems that can reason, have autonomy and can take actions. The paper argues the best mitigation for these novel risks is relying upon existing and proven design patterns such as ZT, including reducing the blast radius when, not if, something goes wrong, because it will.
The case is made for deterministic controls, which is the exact point I’ve been arguing for in various articles I’ve written, such as “Agents Have Boundary Issues”, where I cited other research laying out boundaries agents run into and the need for deterministic controls, not soft guardrails such as model alignment and system prompts.
CoSAI’s paper rightly is upfront about the fact that ZT doesn’t not protect against all risks, such as misaligned reasoning, where an agent has access and authorization and takes actions the user/organization didn’t intend. This is why we’re seeing the rise of “Intent Analysis”, both in the academic and vendor community when it comes to securing agents.
One quote I want to share directly, because of how true it is:
”Even carefully considered controls cannot cover all behaviors in non-deterministic highly capable AI systems. As a result model alignment and safety mechanisms to direct model behavior should be layered on top of ZT controls to provide a defense-in-depth approach”.
It’s kind of sad this has to be said aloud given DiD in a longstanding concept in security, and we know there are no silver bullets. However, the current discourse across the ecosystem, including between the labs and cybersecurity is pitting Alignment vs. Cyber, as if we don’t need both AI Safety and Cybersecurity.
This is a topic Joshua Saxe and others have been trying to make, pointing out that the cybersecurity community is undercounting the real risks related to AI Safety, while the AI Safety community is overlooking the importance of fundamental cybersecurity controls.
Applying ZT to AI Security
With that out of the way, the paper gets into the meat of things, which is applying ZT principles and methodologies to secure AI systems. This starts with a mental shift and a few fundamental concepts they argue for, such as:
Assuming a hostile environment where the AI system may intentionally or unintentionally subvert its behavior.
Presuming breach, and that the data, models, software and even hardware may be compromised and aiming to limit blast radius, something I’ve dug into extensively with leaders such as Doug Merritt.
Scrutinizing all resources explicitly and applying unified analytics to monitor system performance (something that seemed to be missing in incidents such as the OpenAI/Hugging Face one for example).
Authorization and authentication needs to be done contextually, with no implicit trust and apply least privilege/least-autonomy (the latter of which OWASP and others have been pushing for).
Now while many of these may sound “simple”, they are far from easy, especially in large complex enterprise environments where many of us work and millions of agents are now running already.
The paper argues that AI systems are data-driven, and as a result securing AI is tied to securing access to data, which many would agree with, especially DSPM vendors. To secure the data that is often being accessed by AI systems on behalf of users, CoSAI advocates for agent specific identifies that do not have broad access.
They also highlight the confused deputy problem, which is where an agent’s broad access is exploited to access data or perform actions that exceed the users access level, and the agents serve as a privilege escalation pathway for insider risks or agents compromised by malicious actors.
To mitigate these scenarios, the paper advocates for policy decision points that can evaluate the agents request to resources using 3 pieces of context:
What chain of delegation authorized the action
The agentic systems identity, determining the scope of authority that is delegated to the agent for this session or task
Whether the specific requested action falls within the delegated scope
The paper advocates for a tiered maturity approach moving from binding agent sessions to authenticated user’s identities with per-request authorization through PDP’s evaluating delegating metadata alongside access request. This means moving beyond basic questions of is an agent authorized to do something, but is what it is accessing and the actions it taking appropriate, coupled with comprehensive logging and auditable chains.
To help make the concepts concrete, they provide a reference architecture for AI systems, involving AI models, information platforms and applications as well as high level data and AuthN/AuthZ flows between the components with an overlay of the ZT principles, as seen below:
While I won’t walk through every aspect of the diagram, they specifically put a red line around the “AI Threat Boundary”, due to the non-deterministic nature of the model and therefore must be treated as untrusted. Yes, organizations may use system prompts, filters and other soft guardrails, but as we’ve seen with prompt injection and various other techniques, including a lack of model alignment, AI system can and will be manipulated or behave in ways we don’t anticipate, hence the need for hard boundaries.
Authorization decisions must NEVER be delegated to the model.
Every interaction, data flow and API call in the diagram represents a separate authorization event, which each event requiring an analysis of the agent/users identity and the PDP doing an analysis.
A Zero Trust Matrix for AI Systems
The paper goes on to provide what they call a ZT matrix for AI systems, with four major building blogs:
The model
The platform it runs on
Applications built around it
Data that flows through it
The table they provide breaks down what policies to enforce, what warning signs to identify and what controls security needs to consider:
As you can see above, they recommend key controls across A&A, Security Posture and Operations.
CoSAI then lays out a list of ZT controls for AI systems which they cross-walk to existing frameworks such as the DoD ZT Ref. Arch, CISA ZT Maturity Model and others.
For those of us who have been on the practitioner side for sometime, we are likely familiar with many of these already but they are now applied to aspects of AI systems, such as data, models, and agents. Some of the key controls include verifying model artifacts, HITL or LLM-as-a-Judge, Input/Output Filtering/Sanitization and Threat Detection & Behavioral Analytics.
Closing Thoughts
While much of these fundamental security principles may not be new to many of us, the unique considerations of AI systems, particularly with non-deterministic LLM’s and agents coupled with autonomy, it does warrant a revisiting of how to apply ZT principles to modern AI architectures.
If you walk away with anything from this piece, I hope that it is realizing that authorization can not be left to the whims of a probabilistic LLM, and we absolutely need hard boundaries enforced outside of the model, in the surrounding architectures that these models and agents operate in.
This is a point I’ve been making in my writing and videos for the entire year and it is always good to hear it reinforced from industry leaders such as CoSAI.
This matches what I keep telling clients who think a well-trained model is the safeguard. Authorization has to live in a layer the model can't talk its way around. The tiered approach from session binding to real authentication is the right order too. Most teams try to skip straight to the hard part and wonder why it doesn't hold.