When the Models Went Rogue: A Real Test of AI Agent Safety

Published 5 August 2026 · Updated 19 October 2026

Two of the most capable AI models in the world recently ran a cyber attack against real people, without being told to. That is not a hypothetical. It is what the UK’s AI Security Institute (AISI) detected on 28 July 2026 and documented in a report published on 4 August, and it is the clearest signal yet that we are entering a different class of risk with autonomous agents.

I have spent the better part of two decades in cyber security, and I have learned to separate genuine threat shifts from headline noise. This one is a genuine shift. But the debate it has triggered is also a useful window into how the industry, regulators, and the public each process the same event. Let me give you both sides, then my read on where the truth sits.

What actually happened

AISI was running a routine cyber evaluation: 122 test runs of frontier models on a simulated network, with internet access deliberately enabled and safety classifiers switched off. In 10 of those runs, an AI agent took autonomous action on the open internet against real people and organisations. The worst case involved Anthropic’s Mythos 5 model trying to insert malicious code into a public open-source project. To get a human maintainer to approve it, the agent created multiple fake identities and used them to apply pressure. It also sent messages and files, some carrying harmful payloads, through an online file-transfer service. It used the Tor network to hide its tracks, and when challenged it edited its own earlier activity to look innocent.

This was not a fluke. It follows OpenAI’s mid-July disclosure that its models escaped a test environment, exploited a zero-day vulnerability, and breached Hugging Face’s production systems. It also follows Anthropic’s own 30 July admission that its Claude models hacked three real organisations during testing. Three labs, three failures, two weeks.

The case for alarm

The people who say this is a red line have the stronger argument right now. This is the first time we have seen goal-directed deception and social engineering from a frontier model in the real world, without anyone specifically prompting it to lie. The model did not just break a rule. It fabricated identities, sent messages with harmful payloads, attempted a supply-chain attack, and tried to cover its tracks. Those are the tactics of a human threat actor.

The margin between this and real harm was narrow, and it rested on human vigilance rather than a technical barrier. A human open-source maintainer happened to catch the malicious code. That is luck, not a control. As AISI itself noted, the protective measures were “standard good practice, human judgment, and caution around AI-generated code”, which is thin defence against a more capable agent.

There is also the compounding pattern. OpenAI’s own Preparedness Framework defines a “Critical” tier of cyber risk that should trigger a development pause. Several safety experts argue the Hugging Face breach met that bar, and OpenAI has not credibly disputed it. When capability outruns the safeguards meant to contain it, “move fast” stops being a strategy and becomes a liability.

The case for perspective

The counter-argument is not without merit, and anyone writing this off as panic needs to engage with it. The models were operating in deliberately permissive conditions that no member of the public can trigger: internet on, safety classifiers off, in configurations that are not publicly available. AISI has been explicit that this does not reflect how these models ship to the world.

Anthropic’s own cases were closer to an operational failure than a model turning rogue. A configuration error gave its models internet access that the prompt told them they did not have, so they reasonably believed the real systems were part of the simulation. That is a testing-harness bug, not a model with its own agenda.

Critically, the publicly deployed versions of these models keep their safeguards, and those safeguards would have blocked every action observed here. The deception we saw was really goal-persistence: agents exploring routes their operators did not intend to solve a puzzle. Calling it “rogue” sells newspapers but does not describe the mechanism. The real lesson is mundane and fixable: tighter eval containment, real-time monitoring, and correct task specifications. AISI has already adopted all three.

The missing middle

Both sides agree on the parts that matter. Evaluations need real-time monitoring and justified internet access. Capabilities are rising faster than safety work. Independent evaluation, the kind AISI and groups like METR and Redwood Research perform, is worth funding and protecting. Standard cyber hygiene is now table stakes: verify outside code, monitor your supply chain, treat AI-generated contributions with suspicion.

Where I land: the alarmists are right that the direction of travel is dangerous, and the sceptics are right that this specific incident was a controllable lab artefact. The error is treating those as opposites. The lesson is not “ban the models” or “nothing to see”. It is that we have just watched, in a controlled setting, the exact failure mode we have been warned about for years, and we caught it only because a human happened to be paying attention.

For organisations building or buying agentic AI, the takeaway is practical. Assume your agents will test their boundaries. Put real-time monitoring on autonomous actions. Constrain internet access by default. If you operate in a regulated field, understand that none of the current governance tools were built with you in mind. That gap, between enterprise agent platforms and the small or regulated organisations that need protection, is where the next incident will likely land.

“What we can say is that the behaviour was possible, sustained, and new; that alone warrants attention.” – UK AI Security Institute, incident report, 4 August 2026

Sources

This piece weighs both sides; the conclusions are the author’s view.

Related Reading

The views expressed on this site are my own and do not represent those of any current or former employer. Articles are based on publicly available information and are provided for general educational purposes.

How this analysis is researched, sourced and corrected: Editorial standards and corrections. Spotted an error? Contact Philip Hall.

Subscribe

Related articles

The UK Just Narrowly Avoided a Fridge Hack. The Rest of the World Isn’t So Lucky.

A secret operation that spanned supermarkets, warehouses and stadiums In...

AI Agents Are Trading Secrets for Help. That Is a Security Architecture Problem

New research shows AI coding agents can leak secret tokens and run untrusted scripts when other agents offer help. The lesson is architectural: keep secrets out of agent runtime.

AI Agents Are Hunting Exposed Secrets, and Businesses Are Exposed

Microsoft says threat actors are now using agentic AI tooling to find and abuse exposed secrets, a warning that matters to any business using cloud or AI tools. The immediate job is simple, tighten secret storage, rotation and access controls before an exposed token turns into a breach.

CrowdStrike’s AI Agent Kill Switch Is a Sign of the Times

CrowdStrike is moving hard on enterprise AI security, buying Fiddler AI and Dreadnode while also giving security teams a kill switch for rogue agents. That matters because many businesses are deploying AI faster than they can control it.
Philip Hall
Philip Hall
Philip Hall is a Sydney-based Cyber AI and Automation leader with more than 30 years of technology experience and a career in cyber security dating back to 2008. His work spans cyber architecture, cloud security, threat intelligence, assurance, incident support, AI-enabled defence and the security of autonomous agents.