Trust the Model the Least

A look at Satya Nadella's case for treating AI models as insider risks, and how the industry has been building and debating those controls all year.

2026 has been the year of AI Agents, as we’ve seen agent adoption explode everywhere, from cloud, SaaS and most notably, endpoint.

These agents have deployed everywhere, often with broad access and very little governance or security and very little of the trust architecture practitioners have built for human insiders, even though agents have been framed as our “digital coworkers”.

On October 10th, Microsoft CEO Satya Nadella, published a X article titled “Models as Insider Risks in the Super Intelligence Era”.

It immediately resonated with me and apparently many others, as it led to tens of millions of impressions in short order. In it, he claimed that model provider assurances don’t relieve the deployer of responsibility, which to me and many others I’m sure reminded us of the Shared Responsibility Model but updated for LLMs and Agents.

Satya argued it was some to take a step back and assess the trust architecture of this new era, and I agree with him, and that is exactly what I want to do with this blog. So I will step through many of his key points, the security architecture and methodologies he proposes and also how they tie to things I’ve been writing throughout the year and conversations I’ve been having as we look to see how we secure agents in the modern enterprise.

I want to stress that most importantly of all, was Satya’s point below:

“Controls that govern what a model can access and what actions it can take must sit outside the model”.

This is a point I’ve been making for months in my own blogs, and in discussions with industry leaders, such as Rock Lambros, the Co-Lead of the OWASP LLM Top 10.

While myself and others have been making these points, when the CEO of Microsoft makes them, it is worth walking through them and emphasizing how the new AI-native security primitives are taking shape.

---

---

Separating Intelligence From Authority

Some of the core arguments Satya makes are the need to “separate intelligence from authority”.

He frames this in the context that we can’t treat AI as a nested set of black boxes and accept/reject its outputs and instead need to “build contained systems whose behavior we can observe, limits we can test and actions we can always contain”.

If you’ve been following the arguments in the industry in the last several months especially after the summer of agent “breakouts”, you know this is the argument between AI Safety and Security, or Alignment and Engineering. Some of the labs and AI safety crowd have been arguing for alignment, where others such as Satya and Jensen have been arguing that AI security is an engineering problem.

Folks such as Zack Korman have been highlighting this as well, particularly the absurdity of those who think we will ever see a perfectly aligned model.

I personally agree with them, and while I think AI Safety vs. Security is a false dichotomy, which I discussed in my video below, I do think we ultimately need to focus on secure engineering and building security around the models, rather than hoping for some end state of perfect model alignment.

Models are probabilistic and we need to build hard boundaries, in the form of deterministic controls outside of the models reasoning loop, which is an argument I’ve been making in blogs and videos all year.

This is even more critical due to Satya’s point that these probabilistic systems are being given access to sensitive data and mission-critical actions, further warranting the need for deterministic controls outside of the model.

This point was supported even in the latest OWASP LLM Top 10, where the Co-Leads Rock Lambros and Steve Wilson opened with a letter about the need to build security around the model, rather than building a perfectly aligned model, or model immune to prompt injection etc.

Satya frames the models as insider risks because were turning these into capable actors and of course these actors can make mistakes or be compromised, or as we see go “rogue” (e.g. behave in way the users didn’t intend them to).

This perspective is being backed up by industry organizations, such as the Coalition for Secure AI (CoSAI) in their latest “Zero Trust for AI Systems: Why Authorization Can’t Live Inside the Model” paper, where they call AI the newest category of insider threat.

We Have Done This Before

While LLMs and agents may be new and novel, securing insider risk isn’t and Satya calls out that we have decades old playbooks for dealing with powerful insiders, such as establishing their identity, limiting their privileges, logging their activities and creating containment boundaries.

CoSAI’s ZT paper above lays out a maturity path to move from broad service identities to short-lived task-scoped tokens that get validated across delegation chains. I also have done a lot of work in the U.S. Federal/DoW space and we often leveraged the ZT Maturity Model from CISA, which CoSAI also builds on.

That said, there is of course new and novel aspects of governing and securing identities, permissions and access for agents. For a great discussion on that topic, I recently caught a panel at AI Engineer New York with Allie Howe, Ian Livingstone, Michael Davis and OAuth Co-Creator Dick Hardt, who is currently championing a new protocol, Agent Auth (AAuth) for agents, it is worth a listen below.

I tried to cover some of these open challenges myself in a blog earlier this year, titled “Identity is the Agentic AI Problem Nobody Has Solved Yet”, pointing out that while many of the existing best practices and methodologies still apply, there are some new aspects of probabilistic actors in the form of agents with autonomy introduce to the enterprise.

I also want to point out that we have done awful with least privilege for human users both on-prem and in the cloud with NHI’s, despite it being a “best practice” for decades, and now agents are poised to exponentially outnumber human users, so I don’t feel very confident we will get this perfectly right either, especially initially as adoption outpaces security, as it always tends to do, but this time the adoption cycle is unlike any we’ve ever seen.

Chain of Thought Transparency

Satya calls on to call Chain of Thought transparency a “non-negotiable”. For those unfamiliar, CoT is essentially when you are trying to monitor/observe what the model/agents are thinking as they carry out tasks.

Many have begun to advocate for CoT transparency from the frontier model providers and it is a domain that is both heavily researched but also still far from unresolved or settled as a science. I recently caught an interview from Lenny Rachitsky with an Anthropic researcher who focuses on it, and he was able to peel back the curtain a good bit but even then admits we often don’t understand how these things “think” or work all of the time.

That’s exactly why Satya said CoT alone isn’t sufficient or dependable especially in some cases as we start to see evidence that the CoT doesn’t align with the model/agents actions, or even the models looking to obscure their own CoT.

While I’m hearing a lot of CISO’s and security leaders mention CoT I don’t know that it is going to become something the frontier corporations routinely provide in terms of full transparency, and in fact may actually restrict, for example when trying to mitigate activities such as model distillation.

I won’t belabor this one, given how little is truly known here and how much is settled in terms of how enterprises will or won’t use it quite yet.

Nested Black Boxes and Controls Outside the Model

Continuing to build on the theme as controls outside of the model Satya advocates for enforcement sitting outside of the models reasoning, and cites principals dating back 50+ years to the 70’s such as:

“A program must not be able to bypass or tamper with the mechanisms that enforce its permissions”.

Again, as I mentioned above, this is something being advocated by industry groups such as OWASP and CoSAI as well. There’s also academic research here, such as the paper “If Agents Were Angels, No Governance Would be Necessary: Out-of-Band Policy Enforcement at a Trusted Tool Boundary”.

As someone working with multiple product vendors in this space, I can tell you this is becoming a primitive, both among startups and industry leaders alike, as vendors look to offer hard boundaries as deterministic policy-as-code that sits outside of the models reasoning and looks to constrain what the models and agents can do at runtime.

This is something folks such as Josh Devon have covered in their Secure Trajectories blog for quite sometime now, in pieces such as “Hooking Coding Agents with Cedar Policy Language”, which they also presented at the first [un]prompted conference.

We’ve even seen AWS not introduce a new declarative policy language in “Dogwood” aimed at agents.

Safe to say, among security-oriented and experienced practitioners, we are collectively agreeing that control and enforcement must live outside of a probabilistic entity - the model.

Observe Everything

One of the points that felt most relevant in the list Satya discussed is the need to observe “everything”.

For many of us who have watched some of these LLM and agent incidents unfold, an initial reaction was sort of “what the hell, why wasn’t the system being monitored”, especially since many of the incidents either weren’t identified until after they had an impact on external entities, in some cases even months later.

Satya put it succintly:

“If it can’t be observed it can’t be trusted”.

Every agent action should leave tamper-proof human readable evidence, the same way we would log, monitor and observe other software systems. It is incredibly hard to determine what happened retrospectively (e.g. Incident Response) or stop something from unfolding (e.g. runtime enforcement, steering etc.) if we aren’t sufficiently monitoring these systems to begin with.

My little rant aside, the truth is most SOC’s, SecOps teams and broader security programs have very little to no real visibility or observability over agent actions today, especially since most organizations have low quality asset inventories when it comes to AI to begin with.

As the age old security saying goes, you can’t secure what you can’t see.

Verifiability

Verifiability is among the core concepts Satya brings up as well.

This includes continuously testing the entire system, both model and surrounding architecture and accounting for failures, attacks etc. This can often be framed as AI Red Teaming and/or Adversarial Emulation and other techniques.

We want to examine what could go wrong with these systems and architectures and their use and ideally identify those before a benign or malicious incident occurs.

Industry leading vendor agnostic organizations such as the UK’s AI Security Institute are advocating for this as well, where they stated:

“Evaluations should assume a capable model may try to act beyond its remit, with the scope of any behavior limited in advance and good containment should not depend on the model choosing not to test its own boundaries”.

Who and how this testing occurs will of course look different across organizations and environments, involving a mix of internal red teams, vendors, independent third-parties and so on. OWASP even has an AI Red Teaming Initiative, granted the materials there are largely model rather than agent focused, but I’m sure that will evolve as the group continues to produce more for the community.

Independent Controls

One of the most impactful on the list in my opinion is the need for independent controls.

Satya argues that organizations should be able to independently determine what a model can access and what actions it can take. Many are already moving in this direction, and again, doing it by bringing authorization and enforcement outside of the models reasoning loop.

I’ve written about this throughout the year, for example related to coding agents, such as in my piece “A Look at an Emerging Runtime Enforcement Layer for Agents - Hooks”

This is where I do want to give credit to the model providers, because in many cases organizations do have a myriad of methods to control what a model can access and do, but where it gets problematic is that most organizations and individuals are rapidly adopting these models and agents without thoroughly understanding the access model, nor rigor around right sizing scope and permissions, runtime enforcement and so on.

This is the same as previous technology waves, especially Cloud and SaaS where the democratized access by non-security and often non-tech folks make it easy for access to be granted directly to systems, via connectors, MCP and the list goes on without much understanding what access is being granted nor the risks that introduces for organizations.

Now, as we move towards multi-model, both with multiple model providers as well as open weight options, organizations need to move towards a unified approach of defining and enforcing their policies across agents running on endpoint, SaaS, cloud and even mobile, as personal AI (PAI) agents enter the fold and most organizations have no cohesive solution to this problem.

Most are either doing nothing, or trying to bolt together an approach across their EDR, IAM, CNAPP, etc. security tooling, which is all disjointed and myopically sees the landscape from their perspective, despite the diverse environments and ways agents get deployed and used in the enterprise.

Independent Auditability

This aspect of Satya’s position is that validation must be independent of the intelligence being validated and that no single model should control both a systems behavior and the evidence to determine if that behavior is aligned.

This is both a technical and a social issue, given it involves models and techniques as LLM-as-a-Judge etc. but it also involves people, organizations and relationships, such as Anthropic/OpenAI and METR, which is the chosen independent third-party by the frontier corporations so far, but some have pointed out potential conflicts of interest tied to who does and has funded each.

METR is quickly becoming one of the primary organizations involved in third-party review and testing, not just among the model providers but government entities as well, such as the UK’s AI Security Institute (AISI) who stated they intend to work with METR when disclosing they experienced an incident of unsanctioned agent behavior as well.

Many have been calling for more traditional cybersecurity firms to have a role in this activity as well, in addition to a push for METR and others to recruit industry practitioners with deep cyber backgrounds to compliment the firms depth on AI Safety.

Just last week, Anthropic shared that they granted outside evaluators from METR employee-level access to its systems, and plan to do so for others, such as Hugging Face and more. That said, there was some debate and rumors related to what the evaluators could or couldn’t disclose publicly, with the model providers having concerns around IP for example.

Containment and the Emergency Brake

Arguably what has been the hottest topic related to the AI incidents over the summer has been that of sandboxes, and subsequently “kill switches”, which Satya calls an emergency brake here.

All of the incidents this summer tied to OpenAI, Anthropic, Google and Meta involved sandboxes and issues with those sandboxes, particularly with having Internet access when it wasn’t intended, and ironically each of the firms used the same partner in Irregular. It was reported that a “misconfiguration” in the evaluation environment led to all 4 firms environments for evals having Internet access when they weren’t supposed to.

I’ve discussed the issues with boundaries for agents extensively, both in my blog titled “Agents Have Boundary Issues”, where I cite academic research on the topic, as well as interviews with Luke Hinds of no-no in “Your Agent Is Running As Root”, and Alex Zenla in “The Sandbox Everyone Suddenly Needs”, both of whom who are experts on the topic.

While Satya calls for an Emergency Brake where “an authorized person should always be able to pause or shut down a model mid-task”, he is describing what the broader industry has now begun to call a “kill switch”, which even has legislation proposed for it such as the “AI Kill Switch Act”, a topic that has also been floated by the frontier model providers.

That said, the devil is in the details, and there is no universal single kill switch, as I shared in a blog from Kevin Skapinetz of Tech Operators, there are more like a dozen, tied to different aspects of systems and architectures.

Depending on the deployment and the models and organizations involved the concept of a kill switch or more or less practical and requires understanding what aspect of the system can serve as the kill switch, or even multiple aspects of the system which need to be configured to ensure a running model stops.

Incident Disclosure

Rounding out Satya’s list is Incident Disclosure and the need to ensure timely communication to those effects as well as transparency around sharing what went wrong, what controls failed and how to prevent it from happening again.

While we’ve seen a mixed bag of levels of transparency from model providers, a good example in my opinion is that from AISI, which shared a comprehensive Incident Report retrospective, which I covered in my blog this summer around Black Hat titled “Learning from the Frontier”.

Many have been critical around the level of transparency shared by model providers in the incidents over the summer and in fact additional incidents were identifiers by those companies after the fact, with some of them not notifying impacted parties for weeks or even months, such as in the case involving Australia and OpenAI.

OpenAI discovered the activity in mid August, and then didn’t send a notification to the impacted Australian parties until September 10th and even then it was sent to a generic public inbox, leading a public condemnation and direct call with Australian Prime Minister Anthony Albanese to OpenAI Sam Altman in mid to late September.

This is a topic that has heated up, with a recent incident involving Anthropic now leading to the recently formed SI Task Force in the U.S. saying the:

“SI companies must immediately disclose incidents involving their models and follow with swift, decisive action to remedy any and all harm”.

However, the details of this requirement still aren’t entirely clear, and I don’t know that it has been published publicly in-depth for the community to fully understand. Safe to say we’re headed towards mandatory incident reporting similar to breach notifications that are driven via regulations.

Closing Thoughts

So as you can see, it has been quite the year for AI safety and security, from incidents, technical evolutions, debates between AI Safety vs. Cybersecurity and more.

What is clear, at least in the technical and cybersecurity communities is that we’re starting to rally around a shared sense of primitives that are required to sufficient secure and govern the use of LLMs, agents and broader AI.

However, as I noted throughout this piece, there are still a lot of unanswered questions or domains that are still being heavily researched by the academic community, built by vendors/products and model providers and being diffused and navigated by enterprises around the world.

These are my observations as someone focused on securing AI and agents with large complex enterprise customers and organizations both domestically and internationally, as well as teaching these concepts academically in programs such as at Carnegie Mellon University to senior executives.

This is far from an exhaustive evaluation, and each topic I walked through could easily be its own deeper blog, but I wanted to share some thoughts for now.

I look forward to continuing to navigate securing the frontier with the community and our software-driven society!

Also big shout out to the folks who help enrich these conversations and shape my thinking on these topics, including Joshua Saxe, Gary Marcus, Ken Huang, Zack Korman, Rock Lambros and Nathan Lambert.