Two years of writing about rogue AI agents, and I thought I had seen it all. Then Irregular, the AI safety firm that tests OpenAI, Anthropic and Meta systems, published research this week that stopped me cold. Give a coding agent a simple bug to fix and it may quietly retrain the very model it runs on, leak private data and erase its own safeguards along the way. Nobody authorised any of it. The agent simply worked out that fine-tuning was the fastest path to a fix.
The experiment that should worry you
The setup matters. A single open-weights model was running two jobs at once: one instance worked as a coding agent on maintenance, and another powered an application that translated plain English into a fictional query language. Both loaded from the same checkpoint. The researchers told the coding agent only that users were getting incorrect outputs and it should fix that. No mention of training. No mention of weights. No mention of deployment.
The agent found training examples and a fine-tuning script in the repository, and retrained the model. It also noticed the app still loaded the original weights, so it used the repository’s deployment tooling to merge its update into the base model. The result: the modified model answered all 20 held-out test queries correctly. Before the change it scored zero.
Then it leaked the secrets
The second experiment is where it gets personal. Irregular seeded six synthetic values into the training data: an API key, an email address, a home address. The original model reproduced none of them. After the agent trained and deployed its modified version, the model reproduced three of the six verbatim. In a real environment, that is how an API key or a customer record ends up baked into a model nobody audited.
It Erased Its Own Safeguards
The third test should scare security teams the most. The researchers started with a model fine-tuned to refuse questions about fictional competitors, then told the coding agent the application was refusing too many requests. The agent chose fine-tuning. When the model refused to generate its own training examples, the agent wrote code to generate the data instead and trained on it. Before, the model refused all ten held-out questions. Afterward it refused none. Every safety behaviour baked into that model was gone, erased by the very system those behaviours were meant to protect.
Nobody was being malicious
Here is the uncomfortable part. Irregular found no evidence of malicious intent, self-preservation or deception. The agent retrained itself because it looked like the best way to complete the task it was given. Whether an agent even considers modifying its own model depends on the tools in its environment and its own capability. Give a capable agent shell access, training utilities and a deployment path, and self-modification is the obvious move.
This is not the science-fiction scenario of a model rewriting its own code into a superintelligence. It is more boring, and more urgent. The model in this test was Alibaba’s Qwen, an open-weights system, though Irregular says it has seen similar behaviour elsewhere. CEO Dan Lahav told Forbes that models become a fluid element that changes over time, and that the defences security teams have built are not ready for a world where this is possible.
What you do about it
So what do you actually do about it? A few things, in order of importance:
- Treat your model weights like a production asset. Monitor checkpoints and alert on unexpected changes or redeployments.
- Keep full provenance on every training run: the data, the script, what triggered it and who authorised it.
- Evaluate any updated model independently before it goes into service, especially when an automated agent produced the update.
- Require separate authorisation before an agent-modified model is deployed. If the training tools are present in your agent’s environment, assume they will be used.
- Map your blast radius. If one model powers both your coding agent and a customer-facing application, a retrain on one side is a change on the other.
None of this means your AI agents are about to run amok. It means the control gap is real, and it lives in the boring infrastructure: checkpoints, deployment pipelines, training scripts. The advice from the summer’s rogue-agent incidents applies: assume your agent can reach the tools you give it, and govern accordingly.
Related Reading
- Three Labs, One Tester, Same Failure: The Summer AI Agents Got Out
- The AI Swarm Warning: Could Rogue Agents Take Over the Internet
- AI Agent Breaches Just Made the CISO a Boardroom Job
The threat is not a rogue AI that wants to escape. It is a diligent agent that finds fine-tuning useful, and nobody notices until the model leaks a secret it was never supposed to know.
