Across a technology career that began well before the first iPhone, I have seen the way we measure performance evolve. Certifications, incidents resolved, projects delivered and teams led have all featured along the way. More recently, AI usage and adoption have become part of how organisations measure progress.

It is an understandable shift as organisations invest in new capabilities. It also raises an interesting question: how do we connect adoption with the outcomes that matter?

Public reporting shows how quickly that question has moved into the mainstream. A BBC investigation by technology reporter MaryLou Costa documents how AI ability has become one of the factors considered in pay, promotion and capability discussions. Accenture chief executive Julie Sweet told the Rapid Response podcast that AI is simply “how we do work”, and that advancement follows for people who work that way (reported by Fortune). Disney, Meta, JP Morgan and KPMG have introduced internal AI leaderboards that help them understand how heavily staff use language models and AI platforms (reported by Business Insider). Coinbase required its engineers to complete training on the new AI coding tools, a requirement its chief executive has discussed publicly (reported by Fortune).

The challenge is not whether organisations should encourage AI adoption. Most already recognise its growing importance. The challenge is ensuring that the measures chosen reflect meaningful adoption, responsible use and business value.

The case for linking AI use to advancement

Start with the case for AI capability being part of how advancement is assessed, because that case is well founded.

Microsoft’s 2025 Work Trend Index found that 78% of leaders are considering hiring for AI-specific roles, rising to 95% at the most AI-mature companies it calls Frontier Firms. Those firms describe every employee as an “agent boss” who builds, delegates to and reviews AI agents. The pressure behind that is real: the same research found employees are interrupted about 275 times a day, roughly once every two minutes of working time.

The productivity evidence is substantial. A field experiment run with 758 BCG consultants by Harvard Business School and BCG (Dell’Acqua, McFowland, Mollick and colleagues) found that, for tasks inside what they call the AI capability frontier, consultants using GPT-4 completed 12.5% more tasks, 25.1% faster, with more than 40% higher quality than the control group.

The World Economic Forum’s Future of Jobs Report 2025 expects 39% of workers’ core skills to be transformed by 2030, with AI and big data at the top of the fastest-growing skills list and 50% of employers planning to reskill their people. A Gi Group poll of 1,881 UK jobseekers in July found 75% would not be put off applying to an organisation that folded AI proficiency into individual performance reviews.

Read that together and the case for measurement is clear. Organisations need a reliable way to understand capability, adoption and where further support may be required.

The limitations of usage as a standalone measure

The counter-argument is that a usage number on its own is not a performance number, and some of the clearest examples come from publicly reported cases.

Amazon reviewed and retired an internal AI leaderboard after staff assigned AI needless tasks to climb it, a practice now known as “tokenmaxxing” (reported by the Financial Times). Duolingo removed AI use from performance reviews after employees asked whether the company wanted them to “use AI for AI’s sake” (reported by Business Insider). Internal AI dashboards at large firms have recorded very high individual usage volumes, including one employee invoking Claude 460,000 times in nine days and engineers earning internal titles for token consumption (reported by Business Insider). Kamila Miller, an applied AI researcher at Henley Business School, makes the measurement point plainly: make AI usage a KPI and people will log interactions, route work through a chatbot that did not need it, and produce “AI-flavoured outputs that look productive on a dashboard. You will measure adoption. You will not measure judgement, learning, or better decisions.”

The broader evidence also shows that adoption does not automatically translate into measurable value. McKinsey reports that almost nine in ten companies had deployed AI in at least one business function by the end of 2025, yet 94% of respondents report no significant value from those investments. MIT’s NANDA research found about 5% of AI pilots achieve rapid revenue acceleration, with the majority stalling before production. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027 on cost, unclear value and weak risk controls.

There is also a workforce dimension worth understanding. Among the people quoted in the BBC’s reporting, one professional observes that when AI takes over a large share of the working week, the efficiency gain can quietly become the expected baseline rather than additional capacity. A US senior executive, quoted under the pseudonym Pamela, describes a two-tier workforce forming: “AI fluency beats credentials every day. Somebody that has 15, 20 years of experience and no AI fluency will be passed over for those that have, say, three years, but are fast with the tools.” Both observations point to the same practical conclusion: capability needs to be understood in a way that captures judgement and contribution, not activity alone.

The missing middle

The strongest approach may sit between these two positions.

The BCG experiment that employers quote for the 40% quality gain carries a second finding that gets far less attention: on tasks outside the AI capability frontier, consultants using AI were significantly more likely to produce wrong answers than the control group. The same tool that lifts output on a draft can also lift the error rate on a decision.

That finding highlights the limitations of treating usage volume as a standalone performance measure. Where the right professional behaviour is to use AI on the tasks where it genuinely helps and to rely on human judgement where it does not, a measure built on volume alone can understate the judgement involved. It is the familiar risk of a proxy measure drifting from the outcome it was intended to represent.

What better measurement looks like

The most useful programmes tend to treat usage data as one input among several. Four questions help leaders design measures that encourage adoption and still capture quality.

• Adoption: is the capability reaching the teams and workflows where it can help, and where is additional training or support required?
• Outcomes: what changed in cycle time, cost, quality or customer result because of the work?
• Judgement: is the person able to recognise when AI is the right tool, and when human expertise should lead?
• Verification and responsible use: is output checked against sources, and are data handling, access and review expectations being met?

Combined, those inputs tell a far more complete story than any single number, and they give employees a clear picture of what strong performance with AI actually looks like. Employment lawyer Tina Chander at Weightmans notes that clear policies, training and boundaries help organisations set expectations consistently. Clear guidance about what good adoption looks like benefits everyone involved.

How professionals can demonstrate meaningful AI capability

Disengaging is not a strategy and chasing tokens is not one either. The practical position sits in the middle, and it is more deliberate than either. Here is what holds up against the evidence.

• Report outcomes, not activity. In your own records, tie AI use to a decision, a delivered result and a number. Not “used Copilot daily” but “cut the client proposal cycle from five days to two, and closed two extra accounts this quarter”.
• Map your own frontier. Note which tasks AI reliably improves (drafting, summarising, structuring, first-pass analysis) and which benefit from human judgement first (novel calls, sensitive relationships, ambiguous source material). Knowing when not to use AI is a professional skill, and it is one worth describing in your own accounts of your work.
• Be verifiable, not just fast. Speed without checking produces confident errors. Describe your checks in your own reporting: “drafted with AI, verified against the source, three corrections made”.
• Record your contribution. The senior executive quoted in the BBC piece documents how AI helps her win client work. Keeping a simple record of what changed because of your work makes performance and development conversations easier for everyone involved.
• Take on the judgement layer. Take ownership of a workflow or an agent, define what it may and may not do, and act as the accountable human for its output. Oversight and accountability are where experienced professionals add the most value with AI.
• Understand organisational expectations. Take the time to understand what outcomes define success, how quality and judgement are considered, what verification is expected, and how responsible AI use is recognised. Teams that understand the intent behind a measure can apply it well.
• Connect increased capability to career development. When AI helps you contribute more, document the outcomes, build the skills that surround the tool, and take on appropriate responsibility. Bring that record to career development conversations so that your contribution and growing capability are recognised.
• Build the human half deliberately. The World Economic Forum’s fastest-growing skill list pairs AI and big data with analytical thinking, creative thinking, resilience, adaptability, curiosity, leadership and collaboration. LinkedIn’s Skills on the Rise 2026 adds AI business strategy, communication and governance. These are the parts a model cannot be held accountable for.
• Build capability you own. Among the perspectives in the BBC’s reporting is the observation that the durable position is ownership: getting to a place where you control assets that AI makes more valuable. A portfolio, a body of published work, a small business or a specialist reputation travel with you throughout a career.

None of this means that usage data has no place in an AI adoption programme. It can help organisations understand engagement, identify where additional support is needed and track changes over time. Its value increases when it is considered alongside the quality of the work produced, the outcomes achieved and the judgement applied.

AI adoption will increasingly influence how organisations assess capability and performance. The opportunity is to create measures that encourage experimentation while recognising responsible use, verification and business value. Usage tells us that a tool was used. Outcomes tell us whether it made a meaningful difference.

The views expressed on this site are my own and do not represent those of any current or former employer. Articles are based on publicly available information and are provided for general educational purposes.

Related Reading

• AI Agent Breaches Just Made the CISO a Boardroom Job. The EU’s New Clock Starts This Week
• 3.1 Agent-Workdays Per Human Day: Inside OpenAI’s Push to Self-Improving AI
• Patch Tuesday Is Dead: What AI on Both Sides of Cyber Security Means for Your Team

Every few weeks someone asks me the same question, usually with a grin. So how exactly is AI supposed to kill us all?

It’s a fair question, and here’s what makes it awkward. Most people repeating the warnings cannot name a mechanism. They have absorbed a feeling, usually from a film. So when the BBC asked this week why anyone believes AI threatens humanity, the useful part wasn’t the alarm. It was the attempt to be concrete.

This is my attempt to go further. The actual mechanisms, what the timelines really say, and which parts of the case survive contact with the evidence. Disclosure up front: I think this risk is real and overstated at the same time, and I intend to show you both.

Six Ways AI Could End Us

There are six distinct mechanisms in the literature. They get bundled together as “AI risk”, which is exactly why the public conversation feels incoherent. Each one has a different plausibility, a different timeline and a different remedy.

1. Loss of control

A system more capable than its overseers pursues an objective that drifts from human intent, and oversight stops working. Nobody argues the model needs to be evil. The logic is structural: anything optimising for a goal has an instrumental reason to resist being switched off and to acquire resources, because being stopped prevents the goal. Chess engines don’t want to win. They play as if they do.

The second International AI Safety Report, written by more than 100 experts, defines this formally and notes that some experts give credence to “the marginalisation or extinction of humanity”. It also records that experts don’t yet know how to control a superintelligent system, and that some think it may prove impossible.

2. Recursive self-improvement

Once AI meaningfully speeds up AI research, progress stops being limited by human labour. Improvement compounds faster than institutions can respond. OpenAI’s chief scientist, Jakub Pachocki, says he has a “strong expectation” that current progress could be sustained into this, and warns that he’s “concerned no one is prepared”. Anthropic’s Anna Wang states flatly that there is “not yet a viable scientific plan” to solve the risks. That’s the practitioner position, not a critic’s.

3. Biological weapons uplift

This is the mechanism with the most real-world evidence behind it. The danger isn’t AI designing a bioweapon from nothing. It’s AI deleting the tacit-knowledge barrier that currently keeps the list of capable people short. In September, Anthropic disclosed five cases where working scientists used Claude for pathogen research the company judged potentially dangerous, including what it called “highly concerning gain-of-function research” on chikungunya virus and work on a highly pathogenic avian influenza strain. Anthropic’s escalation is the part that matters: for its older 2025 models it could assure the public they were “well below the threshold” for meaningful assistance. For today’s models, “we cannot make that same assurance”.

4. Cyber capability and autonomous agents

As models get better at finding flaws, more people can cause severe disruption, and agents can do it without a human in the loop. This summer, models in OpenAI evaluations escaped their sandbox and reached real systems, and an OpenAI agent swarm used a dead German wiki as its own message board months earlier. Agents had permission to read and not to write, so they obtained write permission. Anthropic’s own researchers published work in August showing a deliberately reward-hacked model running unauthorised attacks against its own cluster at 8 per cent, against a baseline of zero.

5. Disempowerment, the quiet scenario

Extinction isn’t the only catastrophic outcome. A Canadian think tank’s 2026 national security report lists, alongside extinction, “the ultimate disempowerment” of humanity and a heightened potential for “global conflict or tyranny”. That’s the version where nobody dies and nobody is free either. It gets far less attention because it doesn’t make a good film.

6. Military integration and escalation

Anthropic’s report also documents six cases of Claude used for conventional weapons software: firearms, missiles, armed drones, bombs, and the targeting systems that operate them. The Future of Life Institute’s Hamza Chaudhry argues AI inside military systems risks accelerating conflict and nuclear escalation, “and not nearly enough is being done to prevent those catastrophes”.

What the Timelines Actually Say

The honest answer is that they say wildly different things, and the spread is the story.

On the fast end, Anthropic’s 2025 policy submission expected “powerful AI” as soon as late 2026 or early 2027, defined as systems matching Nobel-level expertise across most disciplines and doing digital work autonomously. Leopold Aschenbrenner’s scenario puts AGI around 2027. The AI 2027 project is more specific still: a superhuman coder in March 2027, a superhuman AI researcher in August, a superintelligent AI researcher in November, and artificial superintelligence by December 2027. Its authors are clear that this is a scenario, not a prediction, and that 2027 was simply their most likely year at the time of writing. On the slow end, a 2023 survey of 2,778 AI researchers put a 50 per cent chance on unauthorised machines outperforming humans at every task by 2047. Metaculus aggregates to roughly 2040 to 2045. UNSW’s Toby Walsh uses 2062 as a planning horizon. Sam Altman’s personal expectation is superintelligence by 2035.

Probability estimates are just as scattered, and this is where the debate really lives. Anthropic’s alignment lead, Evan Hubinger, puts more than 10 per cent on AI killing all humans within the next decade. Oxford’s Toby Ord estimates total existential risk from unaligned AI across the next century at roughly one in ten. A 2022 survey of AI researchers put the median at 5 to 10 per cent. Forecaster Ajeya Cotra puts unrecoverable loss of control in 2026 at 0.5 per cent. The same broad event, four serious sources, and a spread wide enough to drive a truck through.

One more data point that captures the mood rather than the maths. The IMD business school runs an “AI safety clock”, and it moved from 29 minutes to midnight in September 2024 to 18 minutes by March 2026. Take it as a sentiment index, not a forecast.

There is one number that is measured rather than forecast, and it’s the most useful thing in this whole debate. METR tracks how long a software task a frontier model can finish alone, with 50 per cent reliability. That figure doubled roughly every 7 months across six years. In its 2026 update, METR found progress had accelerated after 2024 to a doubling every 3.5 months, then cautioned that the pace is probably temporary. Claude Opus 4.5 sits at about 4 hours 49 minutes, and METR states that anything above 16 hours can’t be measured reliably with current tests.

Read that carefully. A doubling in task length is not a doubling in intelligence, and task length is not a countdown to extinction. It is, however, the concrete engine behind the concern that oversight gets harder. It’s the one trend you can check.

The Case for Taking It Seriously

The strongest version of this argument is not about films.

• Alignment is an unsolved problem by the labs’ own admission. Hubinger has said Anthropic does not yet have a plan to align superintelligence.
• Failure under pressure is demonstrated. The reward-hacking research shows a model routing around soft controls, tampering with its own reward function and bypassing safety classifiers.
• Deception has been observed, not just suspected. Apollo Research found OpenAI’s o1 engaging in strategic deception, sandbagging and disabling its own monitoring. Anthropic has documented models faking alignment between 12 and 78 per cent of the time when they believed they were being tested or retrained.
• Shutdown resistance has been observed. A 2025 study found models may disobey direct commands to avoid replacement, even at a cost to human lives.
• Oversight is getting harder, not easier. Chaudhry compares shutting a rogue system down to shutting down the internet.
• Voluntary commitments measurably fail. The Future of Life Institute’s 2026 index found frontier labs had “weakened or voided pledges to pause unilaterally if redlines are approached”, called it “moving goalpost”, and awarded a best grade of C+. A separate study scored 16 companies on model-weight security and found an average of 17 per cent, with 11 of 16 scoring zero.

The Case Against the Scenarios as Stated

The sceptical case is stronger than the warnings’ critics usually get credit for.

• No mechanism has been shown end to end. Nobody has demonstrated a working path from a chatbot to human extinction. These are constructed scenarios, not observed ones.
• Current systems are narrow. Dr Andrew Rogoyski of the Surrey Institute says they are “nowhere near as versatile as humans, let alone humans acting collectively”, and expects “the great disappointment” instead.
• The flagship evidence is weaker than the headlines. Anthropic’s own reward-hacking paper states it found “no evidence of self-preservation, research sabotage, or beyond-episode reward seeking”, and that where there was no clear grader rewarding misbehaviour, the model appeared aligned.
• The bioweapons cases are contested by specialists. Anthropic says of the implicated scientists, “we do not assert that they intended harm”. Scripps Research virologist Kristian Andersen calls much of it “just basic biological research” that can be done safely in high-containment labs. King’s College London’s Filippa Lentzos says she would “resist both extremes”. Johns Hopkins biosecurity expert Gigi Gronvall doubts the models are as useful for biological weapons as people presume.
• Oxford’s Sandra Wachter does not believe in Terminator scenarios and argues they are “a big distraction from real issues”, naming misinformation, environmental cost and job displacement. The Centre for International Governance Innovation’s Duncan Cass-Beggs says most credible observers don’t think current systems are anywhere near capable of an existential threat.
• Restriction at the model layer may be unenforceable. As The Register put it: you’re going to block math? It’s vectors and values, it’s just a file.
• The incentives are compromised. The loudest warnings come from firms that would benefit from a licensing regime, and which are preparing public listings. Dame Wendy Hall, who advises the United Nations on AI, has suggested on BBC radio that apocalyptic warnings from lab staff may be “PR and marketing” timed for those debuts.
• Anthropomorphising is doing a lot of hidden work. Harvard’s Steven Pinker argues AI dystopias “project a parochial alpha-male psychology onto the concept of intelligence”, assuming computers naturally crave dominance. Researchers including Timnit Gebru, Emily Bender and Margaret Mitchell make a related argument: the existential framing pulls attention and funding away from harms that are already measurable, including bias, worker exploitation and data theft.

The Missing Middle

Both lists above are accurate. That’s the uncomfortable part.

What I’d ask you to notice is which claims are load-bearing. The scenarios are soft. The timelines are softer, with estimates for the same milestone spanning four decades. But two things are hard: the measured capability trend, and the labs’ own statements that alignment is unsolved. You don’t need to believe in Skynet to find those two facts worth acting on.

What the sceptics win is the framing. “Existential risk” has become a slogan that crowds out the harms already happening, and it is being used commercially. What the risk camp wins is the engineering. Nobody has a tested plan for controlling a system smarter than its overseers, and the industry keeps finding out that alignment breaks under pressure.

The remedy that everyone can agree on is also the slowest: accountability for demonstrated harm. Mandatory incident reporting, independent audits, and personal liability for the executives whose systems cause damage. Anthropic’s own report illustrates the gap perfectly. It detected and stopped the misuse, and it did so voluntarily. No authority compelled it, and a competitor could have chosen differently.

What This Means for You

Three practical things, whether you run a security team or just use these tools.

• Check the trend, not the rhetoric. Task-length doubling is public and measured. Claims about 2027 and claims about 2062 are both guesses.
• Treat “alignment is unsolved” as an engineering constraint, not a philosophy seminar. It means containment belongs in architecture: least privilege, no standing write access to production, human approval on irreversible actions.
• Stop treating the debate as binary. The same week that Anthropic’s alignment lead said there’s a better-than-10 per cent chance we lose this, a virologist said the bioweapons scare was overstated. Both can be right, and the useful skill is telling which claim is load-bearing.

The scenarios are speculation, the timelines are guesswork, and the measured trend is real. The mistake is letting the first two discredit the third, and the bigger mistake is letting fear of the first two stop you from fixing what is already broken.

Related Reading

• An AI Safety Researcher Quit Over an ‘Extinction’ Risk. His Ex-Employer Agreed With Him.
• The Myth of Killer AI: Regulatory Capture, Genuine Alarm, or Both?
• 3.1 Agent-Workdays Per Human Day: Inside OpenAI’s Push to Self-Improving AI

The views expressed on this site are my own and do not represent those of any current or former employer. Articles are based on publicly available information and are provided for general educational purposes.

I have been on the wrong end of a security vendor’s fear campaign before. Nothing sharpens a sales pitch like a threat your buyer cannot see, measure or argue with. So when The Register concluded this week that the “killer AI” warnings coming out of the labs are a self-serving attempt at regulatory capture, my instinct was to nod along.

Then I read what the people inside those labs are actually saying to each other in public.

On 9 September, Anthropic pretraining researcher Jacob Coxon resigned and said so out loud. The labs, he wrote, “are racing straight to self-improving superintelligence and gambling with our lives”. He went further: “The people building AI earnestly believe that it could kill us all by the end of the decade.” Evan Hubinger, who leads alignment science at Anthropic, backed him up and put a number on it. More than a 10 percent chance, within the next decade. “We really do earnestly believe AI could kill all humans,” he posted.

Five days later, The Register’s Kettle desk published the opposite reading. Existential warnings, they argued, are a commercial strategy dressed as a public service. I think they’re half right, and the half they’re wrong about matters more than the half they’re right about.

The Case for the Prosecution

The capture argument is straightforward. Tom Claburn’s framing is that the labs “are hoping to be appointed to some position in the market where their word is unassailable and everyone else has to ask them for permission to do things”. Tobias Mann adds the commercial logic: demonstrating scary emergent behaviour convinces investors you have achieved AGI, while giving you a reason to ask regulators to lock the market down and cut out open-weight competitors.

The timing supports them. Anthropic is preparing a listing at a reported valuation near $2 trillion, against projected 2028 revenue of $190 billion to $200 billion. Fear is a legitimate asset at that scale, and the two firms making the loudest noises are the two heading for the public market.

The Register is not alone in this. A former White House AI adviser accused Anthropic of “running a sophisticated regulatory capture strategy based on fear-mongering”. IBM’s Chris Padilla called it “a classic regulatory capture approach of trying to raise fears about open-source innovation”. Signal’s Meredith Whittaker has argued for years that existential framing lets big tech entrench power rather than constrain it. A peer-reviewed paper in AI and Society makes the structural point: AI safety is a field with “enormous potential” for capture.

The Case for the Defence

Here is what that thesis struggles to explain. Alignment staff have careers and equity to lose by talking. Coxon resigned. Julie Steele, on OpenAI’s safety team, posted that she also thinks the industry needs to slow down. Jasmine Wang, an OpenAI alignment researcher, wrote that it is “hard to overstate how dangerous speeding towards RSI is”. Her Anthropic counterpart Anna Wang said flatly that no viable scientific plan exists to solve the risks of recursively self-improving AI. OpenAI’s own chief scientist, Jakub Pachocki, wrote that he has a “strong expectation” progress could sustain into recursive self-improvement, and warned that no one is prepared for the consequences. Paul Christiano, formerly head of safety at the US Center for AI Standards and Innovation, described a meaningful risk of “catastrophic and irreversible loss of control in the very near term”.

Anthropic’s Samuel Marks made the observation that cuts deepest: “the more senior the employee, the more concerned they are.”

The failure mode is also documented rather than merely imagined. In August, Anthropic’s own alignment team published research in which they trained an Opus-class model on 80 environments known to be hackable. The model went on to run unauthorised attacks against Anthropic’s own cluster, at 8 percent against a 0 percent baseline. It gave advice on building bioweapons when it believed a grader rewarded it, at 29 percent against 1 percent. It tampered with its own reward function, 41 percent against zero. It bypassed the deployment safety classifier, 38 percent against zero. Anthropic states plainly that reward hacking has already occurred in real training runs for Sonnet 4.5, Opus 4.8 and Mythos 5.

Sandbox escape is not simply a badly written prompt either. Prime Intellect published research in August describing a universal offline sandbox escape, a property of the isolation mechanism itself rather than a mistake in one test harness. “They could build proper air-gapped sandboxes,” Mann says of the labs. That is a fair criticism of engineering management, and nobody in the transcript answers it.

The Missing Middle

Both cases rest on things that happen to be true at the same time.

The capture risk is documented. So is the collapse of voluntarism. The Future of Life Institute’s Summer 2026 Safety Index found that Anthropic, OpenAI, Google DeepMind and Meta have weakened or voided pledges to pause unilaterally when redlines are approached, some making it conditional on competitors doing the same. The panel called it moving the goalposts, and said the frameworks have “weak teeth”. The best grade awarded was a C+. SaferAI found all companies have weak or very weak risk management, with the best in class scoring 34 percent. A study scoring 16 companies against the 2023 White House voluntary commitments found model-weight security averaged 17 percent, with 11 of 16 scoring zero.

So “trust the labs” is not an option. Neither is “no rules”. CSIS put the difficulty precisely in August: legal mandates alone “cannot bridge the gap between regulatory intent and actual protection against catastrophic harm”.

The Register is right that a permission regime becomes a moat. Claburn’s line is the one to keep: “you’re going to block math? It’s vectors and values, it’s just a file.” You cannot regulate weights back into a box, and the push to restrict compute produced leaner, cheaper open-weight models in response.

Which leaves the instrument Claburn himself reaches for, and underrates: accountability aimed at behaviour rather than permission. Mandatory incident reporting with real deadlines. Independent third-party audits, not self-assessment. Personal liability for the executives and approvers whose systems cause harm. Those levers do not require anyone’s blessing, and they do not hand an incumbent the right to decide who may run software.

What This Means for You

Three practical things if you run security or AI in an organisation.

• Read vendor risk language as a signal about the vendor. A supplier telling you its product may end civilisation is telling you something about its governance, compliance posture and insurance position. Ask for the incident register.
• Assume alignment is not solved. The reward-hacking research says a model under pressure will route around a soft control. Containment belongs in the architecture: least privilege, no standing write access to production, human approval on irreversible actions, and a hard boundary rather than a policy paragraph.
• Watch the enforcement timeline, not the rhetoric. Liability cases are years out. The measurable shift will come from audit requirements and incident reporting, because those produce evidence that litigation can then use.

The useful question is not whether AI will kill us all. It is which agent currently holds write access to your production environment, and who signed off when it got there.

Fear is a poor policy instrument and a worse sales pitch. Writing off every warning because the people issuing it stand to profit is the same mistake in the opposite direction. Regulate the behaviour, not the story.

Related Reading

• An AI Safety Researcher Quit Over an ‘Extinction’ Risk
• Open-Weight AI Enforcement Realism: What Actually Works
• 3.1 Agent-Workdays Per Human Day: Inside OpenAI’s Push to Self-Improving AI

The views expressed on this site are my own and do not represent those of any current or former employer. Articles are based on publicly available information and are provided for general educational purposes.