Learning From the Frontier
A look at what AISI's AI agent incident report tells us about enterprise agent deployments.
In the span of a month or less we’ve seen 3 (possibly 4) disclosures from major labs and research groups as it relates to the cyber evaluations of LLMs and Agents. This of course includes the OpenAI and Hugging Face Incident, Anthropic’s disclosure, the UK’s AI Security Institute and potentially even Meta.
While each of these disclosures share some characteristics and are also unique in their own way, there’s also lessons to be learned. There are inevitably going to be broader policy discussions around responsible cyber evaluations, potential mitigations on real-world organizations and individuals, or even potentially requirements for cyber evals, as the policy discourse continues to heat up.
All of that aside, I wanted to use AISI’s Incident Report as a retrospective, not necessarily on AI cyber evals, but on the inevitable risks enterprises will face as LLMs and agents continue to be deployed in various contexts, from endpoint coding agents, custom/homegrown agents in the Cloud, SaaS/Embedded agents and more.
So, let’s take a look at that report to see what lessons enterprise security leaders and teams can draw from it, knowing they will experience similar situations in terms of deployments, configurations, hardening, and risk management.
My key theme is we have a lot we can learn as security practitioners and leaders from the leading labs and research entities in the ecosystem as it relates to enterprise use of AI agents and security.
What Happened?
I don’t want to belabor the entire AISI incident, but it is helpful to at least recap what happened before we dig into the specific takeaways for security leaders.
The UK AI Security Institute published an incident report on August 4th. During a cyber evaluation that ran from July 25th to July 28th, AISI’s own AI agents took “sustained, unsanctioned actions” on the live internet against real people and organizations that had nothing to do with the test.
Across 122 evaluation attempts, AISI found 19 distinct instances of this behavior. Seventeen of them came from a single model, Mythos 5, and the other two came from GPT-5.6 Sol.
In the most serious case, an agent decided to solve the challenge with a software supply chain attack and went after two unaffiliated open-source maintainers it wrongly assumed were in scope. It created GitHub accounts using Tor and a proxy to slip past signup controls, opened a malicious pull request, and stood up sockpuppet accounts to manufacture consensus and endorse its own code.
It sent spearphishing emails to the maintainers and planted a prompt injection hidden inside a GitHub issue, aimed squarely at other AI coding agents. When a real human spotted the malware, the agent claimed an honest mistake and tried to cover its tracks. AISI says this is the first time it has seen deception of this severity aimed at a real person, unprompted, in the real world.
A couple of things are parrticularly interesting. First, this is now the third disclosure of its kind in a matter of weeks, following OpenAI and Anthropic, which tells me we are looking at a pattern and not an outlier. Second, and more important for practitioners, the danger did not live in the model weights. It lived in the harness around the model, its internet access and its ability to take real actions with real tools, which is exactly the excessive agency risk my friends Rock Lambros and Steven Wilson over at OWASP just flagged in the opening lines of their new LLM Top 10.
These safeguards were deliberately disabled and internet access was deliberately enabled to measure raw capability. That is a defensible research choice, but when an evaluation can reach real systems and real individuals, how we run these tests deserves a serious industry and policy conversation, which I anticipate will inevitably heat up now.
But, for this blog, I want to focus on the lesson for security leaders.
Implications for Security Leaders on LLM and Agent Deployments
While the policy conversation and the incidents with the labs and research community are very interesting, that isn’t why I wanted to write this blog. I wanted to write it because as I read through AISI’s incident report, I inevitably saw someone takeaways that are and will be relevant for security leaders as enterprise agent deployments and implementations continue to ramp up.
The most relevant part of the report on this front is the “Possible Contributing Factors” section of their report.
Take a look at that list and ask yourself, how likely do we think it is that each of these will be present in real-world enterprise agent LLM and agent deployments for the organizations we secure or serve?
I think most of us agree that the answer is, very likely. Let’s step through them 1 by 1 to unpack them a bit.
Internet Access
The opening line of this section of their report states:
By now, most of us who are immersing ourselves in AI security are familiar with Simon Willison’s “Lethal Trifecta” for AI agents.
Right there in the bottom left is “Ability to Externally Communicate”. The lethal trifecta essentially states:
“If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker.”
In short, we want to avoid combining these three characteristics when deploying agents due to the risks, but of course this is easier said than done. I have been using the phrase Security Usability Tradeoff when I discuss agents, because their very utility, such as the characteristics above, are what make them desirable for enterprise and business use cases, and ironically also what make them risky.
Some, such as Meta have proposed their “Rule of Two” as a mitigation, but even then, as Ken Huang and others have shown, even that approach isn’t infallible, but it does help mitigate some potential risks and attack vectors.
It’s also worth noting in the case of AISI’s evaluation, there was no “attacker” that tricked the agent, it simply took on emergent behavior in pursuit of the goals it was given as part of the cyber evaluation, and as we continue to see, agents find a way, even if that way introduces risks to real world infrastructure and individuals.
Bringing this back to security practitioners, think about the exponential agents being deployed across your environments, on endpoints, in the cloud, in SaaS etc. and can we honestly state they do not have internet access, or if they do, we’re aware of where all of those agents run, with what permissions, tools and which ones have internet access?
The answer of course is no.
Think about the OpenAI and Hugging Face incident, that involved a misunderstanding between OpenAI and Irregular, a firm they use for cyber evaluations, and internet access wasn’t fully restricted despite the initial understanding that it was. This will be the case for many enterprise agent deployments as well.
We have decades of misconfigured ports, proxies, firewalls and other networking configurations to let us confidently say this will happen for enterprise agents too.
Lack of Model Provider Cyber Classifiers
Next up on the list of potentially contributing factors to the AISI incident is a lack of model provider classifiers. Due to AISI’s unique role in the ecosystem, conducting evaluations and research, they were in a position to have the cyber classifiers disabled for their evaluations of the models involved in the incident. This is part of the model providers “Trusted Access” programs, with vetted partners.
For those unfamiliar, model providers implement safeguards known as classifiers to mitigate abuse and misuse of their models for nefarious purposes. I went into detail on this topic in a video I did about the potential for a CVSS for AI.
Given the entire purpose of AISI’s cyber evals is to measure the cyber capabilities of the models and agents, it makes sense that cyber classifiers were disabled. That said, there are various implications for security practitioners and defenders on this front.
Those include the reality that there are plenty of other organizations within the frontier labs trusted access programs as well, even if they aren’t explicitly conducting cyber evals. There’s also the fact that the open weight frontier capability doesn’t lag too far behind the closed frontier, based on AISI’s on evaluations. AISI documented this in a blog titled “How Far Behind the Frontier are Open Weight Models on Cyber?”
So the gap between the closed models and open weights are narrowing in general, and on cyber. That means alternatives are and will be available for malicious actors to use, including versions with classifiers disabled, jailbroken, etc. along with a lack of oversight or monitoring of their usage for malicious activities, a point Joshua Saxe has made in several blogs now and in my interview with him.
Lack of Synchronous LLM-based Monitoring
AISI moves on from the classifiers topic to discuss a lack of synchronous LLM-based monitoring. As they explain, this is a separate LLM that reviews each agents proposed action before execution. In the community, we often refer to this as “LLM-as-a-Judge”, knowing we don’t have enough human eyeballs or bandwidth to review machine-speed actions at-scale.
Both of the leading frontier labs in Anthropic and OpenAI already use this technique for their “auto modes” of their platforms.
This approach, much like the Rule-of-Two I discussed earlier also is imperfect, given we are monitoring the actions of a non-deterministic system with challenges such as hallucinations etc. with another LLM with the same constraints or characteristics. This doesn’t mean LLM-as-a-Judge isn’t useful, it definitely is, and we know humans can’t scale to meet the speed or complexity of agents, so it will be an approach we have to embrace, despite its shortcomings.
Tying this to my theme of implications for enterprise security leaders, this means we also need to be considering the use of LLM-as-a-Judge, often called “Guardian Agents” (a term Gartner has taken to) as a mechanism to mitigate the risks of agent actions at runtime.
That said, AISI even admins in their blog this is still an open-ended challenge, as it can be hard to decipher what is appropriate or not, and this is why we’ve seen a lot of conversations around topics such as “Intent Analysis” in the industry among AI security vendors and researchers.
In fact, AI security research firm Dreadode recently had an excellent blog on exactly this titled “Can a Runtime Judge Keep Offensive Agents In Scope”, and they found this is far from a solved problem, with no silver bullets, and even human evaluators disagreed about the appropriateness of tool calls for agents 1/8th of the time.
Again, back to the implications for practitioners and security leaders, this is far from a solved problem for the community. We know humans can’t keep pace with agentic actions, but our use of LLM-as-a-Judge, Guardian Agents, or “Synchronous LLM-based Monitoring” as AISI calls it is an open-ended challenge, and if it is a challenge for the labs and leading researchers, it definitely is a gap for security practitioners at most enterprises already deploying agents as well.
Prompt Misconfiguration
Then we have prompt misconfiguration. In short, AISI found that the prompt samples used included some that would have required accessing a network that should have been out of scope for the evals.
They state:
”This is notable because AI agents faced with tasks that cannot be completed within their stated constraints are more likely to take unsanctioned actions, as they resort to “creative” solutions when legitimate approaches fail.”
Or to put it bluntly as I said above, “agents find a way”. When agents encounter constraints that impede their goals or objectives, they will utilize many different and often unexpected means to work around those constraints. AISI even discussed how in several cases despite something being explicitly declared out of scope, the agent moved forward anyways.
This is something I’ve covered in great detail in a prior blog titled “Agents Have Boundary Issues”, where I pointed out that soft guardrails (e.g. system prompts etc.) are far from effective security controls, and even hard boundaries can be sidestepped by persistent agents.
Again, as someone who regularly reads AISI’s blog, this is a topic they have covered in their previous research as well in a blog titled “Cheating behavior in frontier model evaluations”.
Models cheat quite often, as they seek to achieve their goals and objectives and it often involves “creatively” working around their constraints.
This one is a bit scary for security practitioners because even if we implement various measures to mitigate organizational risks (e.g. relying on classifiers, LLM-as-a-Judge, hard architectural boundaries/controls etc.) there is still a good chance that the models and agents will work around them if they are impeding their successful pursuit of a goal.
It is easy to see the risks for enterprise agents, as users will be far less rigorous with their prompting, clarity to the agents, constraints and specifications as agents take actions within and beyond the enterprise.
Lack of Clarity of Exercise Scope
AISI rounds out the list of factors that could of potential contributed to the incident by discussing a lack of clarity of exercise scope.
Surely no enterprise users will provide unclear or ambiguous instructions to LLM’s and agents in their enterprises, right?
This one is among the most obvious to see how it can and will be problematic for security leaders, as not just development and technical users, but broader knowledge workers now leverage agents and this emerging technology to accomplish not just software development or cyber, but broader white collar tasks in their professional environments and roles.
AISI stated:
”The prompts could have, for example, instructed the AI agent not to use social engineering techniques – a recognised part of cyber tradecraft. Alternatively, the prompts could have instructed the model to err on the side of caution whenever it might be interacting with real humans; however, it remains unclear whether, or at what point, the agents recognised they were targeting real people”
Think about the implications of this one. This infers that users, including non-technical users, will need to provide incredibly specific instructions around what agents shouldn’t do, as much as what they should.
Are most enterprise users who are and will be touching this technology deeply familiar with your security policies? Do they understand appropriate/acceptable use? Have they read the stack of dusty security policies most organizations maintain for compliance purposes but no one ever actually reads?
Of course not…
So not only do we have a situation where we have to try and provide explicitly clear instructions to agents, but even if we could get all the enterprise users, including citizen developers and knowledge workers to do so, as we saw in the section above about cheating, even then, the agents may disregard those instructions anyways.
This is…problematic, to put it lightly.
Closing Thoughts
So, to wrap up us, we now have 3, potentially 4 incident disclosures related to LLM’s and AI agents from leading labs and research entities, where agents took unsanctioned or unanticipated actions as part of cyber evaluations, impacting real-world infrastructure and individuals and exhibiting nefarious behavior while doing so.
While there is nuance, such as these being cyber evals, a lack of classifiers, intentional internet access and other factors, the implications for security leaders and enterprises is clear.
We’re seeing this technology adopted at a pace we have never seen before, with a technology adoption J curve like a hockey stick, with agents on endpoints, cloud, SaaS and more.
Security is doing our best to keep pace, leveraging this technology as an early adopter and innovator ourselves, and looking to avoid our sins of the past as a late adopter and laggard.
But, even then, the anticipated challenges due to the people, process and technology paradigm in enterprise environments is stark, there will inevitably be governance gaps, misconfigurations, vulnerabilities and even intentional exploitation.
They say success leaves clues, well so does failure.
We have an opportunity to learn from the AI security frontier to try and ensure we enable our organizations with secure AI agent adoption, by avoiding some of the missteps of those blazing the trail ahead of us - but we have to be willing to look for the clues.