In this episode, I sit down with Ciaran Martin, the founding head of the UK’s National Cyber Security Centre, to dig into the FUD and hype at the intersection of AI and cybersecurity.
Ciaran led the NCSC for its first four years after spending two years setting it up, and his own awakening to cyber goes back to the ILOVEYOU worm hitting his remote access laptop as a junior Treasury official in 2000.
We start with how governments learned to talk about cyber, from the UK’s first national cybersecurity strategy in 2009 and its budget of five million pounds, to a problem Ciaran says has endured. Senior decision makers avoid asking hard questions about technology because they’re afraid of looking stupid, and he no longer believes a new generation of leaders will fix that on its own.
From there we get into his piece From Frenzy to Freakout and the idea that two things are true at once. Frontier AI is a very significant change for attackers, and the reaction to it has been one of the biggest outbreaks of cyber FUD in over a decade.
Ciaran argues we keep forgetting there are two sides to everything AI touches, the AI itself and the computer networks it lands on, and that almost all cyber harms are visible. In his view there has been no discernible rise in disruptive attacks in the six months since Mythos, and the ShinyHunters arrests that followed the apparent FBI compromise are a reminder that tools have operators and operators can be found.
We recorded this conversation in early October, a couple of weeks before PHP2Shell and the “Black October” label took over the feeds. Ciaran has since called PHP2Shell very, very bad and asked what the reaction will be if something very bad but entirely precedented like NotPetya or WannaCry actually happens, and I think the conversation holds up well against this past week.
We close on software liability and the inverted pyramid of responsibility, what happens if he turns out to be wrong, and what CISOs should say when their boards ask whether AI is about to hack everything.
Prefer to listen?
Please be sure to subscribe and leave a rating/review, as it truly helps the show!
We discuss:
Ciaran’s path from generalist UK civil servant to founding the NCSC, which just turned ten
The ILOVEYOU worm in 2000, the helplessness it exposed, and the UK’s first national cybersecurity strategy in 2009
How government moved past vague talk of public-private partnerships to address real market failures, through Heartbleed and Log4Shell
Why top decision makers avoid hard technology questions, and why Ciaran no longer sees it as a generational problem
From Frenzy to Freakout, and how frontier AI can be a real shift for attackers and a FUD outbreak at the same time
Six months since the delayed Mythos rollout, the 27-year-old bug, and the $20,000 in tokens Marcus Hutchins pointed to
Why Ciaran thinks phishing training is on its last days as AI automates phishing
The cult of the machine god, the New York Times reporting on Anthropic’s meetings with religious leaders, and cybersecurity as a pragmatic profession
Arvind Narayanan and Sayash Kapoor’s AI as Normal Technology framework, and whether security treats agentic attacks as a solved problem
Alignment, whose values it aligns to, and the risk of concentrating power over a new technology in a few hands
Intent versus accountability, Secure by Design, and why software has sat outside product liability for decades
The inverted pyramid of responsibility that puts the greatest burden on the humble user
How a minor Australian story led global news while the apparent FBI compromise barely registered
ShinyHunters, the arrests that followed, and why the people behind the tools can be got at
Why “you should see what we’re seeing in the wild” doesn’t hold up for disruptive attacks
Why the window never closes, and what happens if Ciaran is wrong
Lessons from WannaCry and NotPetya in 2017, and the day Heathrow’s Terminal 5 went down
What CISOs should tell their boards, starting with “what harm are you worried about?”
Key Takeaways
The thing AI touches
“I just think at the heart of this is a lack of recognition that there are two sides to everything that AI touches. One is AI, and the other is the thing it touches. And in cyber and economics and medicine, but this is a cyber show and we’re both cyber people, we just forgot the thing it touches, computer networks. Some of them are terribly defended, some of them are well defended, it matters.”
This is the part of our conversation I’d most like practitioners to hold onto. Much of the coverage this year has focused on what the models can do, with far less attention on the environments they’re being pointed at, and a capable model running into a well segmented network with least privilege and good credential hygiene is a very different story from the same model running into an environment carrying decades of unaddressed issues.
That’s why I’ve been framing AI as a security technical debt collector, cashing in on the vulnerabilities, misconfigurations and poor hygiene our industry never got around to fixing, which I dug into in From Vulnpocalypse to Breachpocalypse?. Ciaran’s point fits that framing well, because when you read an AI threat claim, how well defended the target was matters at least as much as how capable the model is.
Disruptive harm is visible
“With the exception of nation state espionage operations, almost all cyber harms are visible either immediately or subsequently.”
“So has there been a discernible rise in disruptive cyber attacks in the six months since Mythos? No. Provably no.”
We recorded this before PHP2Shell and the Black October label took off, and I think Ciaran’s point has held up. We are clearly seeing highly impactful vulnerabilities in software that sits underneath a huge share of the internet, and that deserves attention. So far, though, exploitation has been fairly limited when you measure it by real-world disruption and impact, and we haven’t seen the critical services citizens depend on every day knocked offline, which is exactly the kind of harm Ciaran says we would be able to see.
Where I’d add some nuance is that several of the recent AI-enabled incidents I walked through in my essay sit outside the datasets we normally use to measure exploitation, and loss data like insurance claims tends to lag. So I’ll be watching that data closely over the coming months, and until it shows a real shift, I’d rather save the alarm for when the evidence warrants it.
Remember the attackers
“Until we’ve mentioned ShinyHunters, we’ve been talking half an hour. We have not mentioned a threat actor group.”
“There are tools, and they may be low-tech like ShinyHunters, they may be high-tech like open weights models... But people at the end of those tools with adversarial intent can be got at in all sorts of different ways. And this is what we miss.”
Ciaran’s observation that we went half an hour without mentioning a threat actor group is an accurate picture of how a lot of AI security conversations go right now. So much of the discourse centers on the model and its capabilities that we lose sight of who is actually using these tools and what it costs them, and as Ciaran points out, attackers have objectives and tend to reach them in the cheapest and most effective way available.
When every conversation revolves around the model, security teams lose some of the threat-informed thinking that should drive how they prioritize, along with an appreciation for the levers that sit outside their own networks, such as law enforcement and deterrence. The ShinyHunters arrests are a good reminder that the people behind the tools can be found, and that should factor into how we think about risk.
The inverted pyramid of responsibility
“We’ve always inverted the pyramid of responsibility in cybersecurity... we always put the greatest burden on the shoulders of those least able to bear it, the humble user.”
“But if we end up there for AI, then we don’t have a chance.”
This is a topic I care a lot about, having spent much of my career in the public sector, from the Air Force to working with federal agencies and serving as a CISA Cyber Innovation Fellow. For decades we’ve asked end users to avoid clicking links and to juggle long, unique passwords, while the organizations building the software and infrastructure everyone depends on have carried very little liability for shoddy security. Secure by Design was an effort to shift more of that burden back to the vendors, and Ciaran’s reference to the Cyber Safety Review Board’s report on Microsoft’s 2023 breach is a good illustration of why that shift is needed.
AI products are software, and I don’t see a reason for frontier AI labs to operate under a different set of expectations than the rest of the technology industry. You can’t outsource accountability, and if we let AI inherit the same inverted model of responsibility, we’ll be repeating the mistakes we’ve spent the last several years trying to correct.
Decisions, not apocalypse
“If there’s a really serious deterioration in cybersecurity, that’s because of a set of bad decisions that we collectively made. And so we need to make the right decisions.”
“This is security and this is technology. The window never closes.”
I’ve spent much of this year arguing that security can’t afford to be a laggard with AI, so I take the capabilities seriously, and so does Ciaran, who was clear in our conversation that he does think this is a big deal. Where we agree is that how this plays out comes down to the decisions we make, and framing it as a window that is about to close does little to help organizations make those decisions well.
It’s also fair to look at who is amplifying the closing window message. As I mentioned during our conversation, OpenAI has invested heavily in its security business, and many of the letter’s signatories are cybersecurity vendors with a commercial interest in heightened concern, which should factor into how we weigh it. Cybersecurity has always been about managing risk to an acceptable level, and as Ciaran puts it, the window never closes because there will always be innovation and dynamism on both sides.
What to tell the board
“I think the best question to start with is what harm are you worried about?”
“If all this fails, what do you want this to look like? And how can we make sure that we are at fifty percent within twenty four hours rather than at naught percent in twenty four weeks?”
Many security leaders are heading into board conversations shaped by AI headlines, and I think Ciaran’s starting question is a useful way to ground them. Asking what harm the board is actually worried about moves the discussion away from hypotheticals about what a model might do and toward what could actually disrupt the business and what would limit the damage if something does get in.
His closing point is also a big part of why this show is called Resilient Cyber. No organization is going to prevent every incident, with or without AI in the picture, so planning for how quickly the business can get back on its feet when something does get through is the kind of practical conversation boards should be having.
Thank you again to Ciaran Martin for joining me and for being so generous with his time. I’d love to have him back to dig further into Secure by Design, software liability and incentives, and in the meantime you can follow his work on LinkedIn and on his Substack.
Ciaran Martin
Substack:
Timestamps
0:00 Introduction
1:37 Ciaran’s background and the NCSC at ten
2:29 The ILOVEYOU worm and government’s slow start
4:19 Why leaders avoid hard technology questions
6:30 From Frenzy to Freakout
8:49 Six months since Mythos and the 27-year-old bug
10:50 Two sides to everything AI touches
12:56 The cult of the machine god and a pragmatic profession
15:21 AI as Normal Technology and “solved” agentic attacks
20:32 Alignment and whose values
21:23 Concentration of power
22:22 Intent, accountability and software liability
24:19 The inverted pyramid of responsibility
25:11 Australia, the FBI and misplaced attention
27:08 ShinyHunters and remembering the attackers
30:52 “You should see what we’re seeing in the wild”
31:26 Why disruptive harm is visible
34:04 Devil’s advocate on commoditized capability
35:07 Always around the corner
38:20 Why the window never closes
39:55 What if I’m wrong
40:45 WannaCry, NotPetya and lessons from 2017
43:18 What CISOs should tell their boards
45:00 Closing thoughts
