Direct answer: Anthropic is quietly shopping a draft bill around Congress that would let the US Commerce Department ask a federal judge to block the release of an AI model if independent evaluators find a “foreseeable, imminent risk” of 50 or more deaths, or over $1 billion in property damage. The company confirmed the draft to Politico, saying it reflects its Advanced AI Framework proposals. It is not law — it hasn’t even been introduced — but it’s the clearest sign yet that one of the biggest AI labs wants the state to have a hand on its shoulder, not just a clipboard watching from the corner.
What happened
According to Politico’s reporting, picked up by TechCrunch, the draft legislation sets out scenarios in which a model release could be stopped by the courts. The catastrophic risk would have to come from one of three sources: the creation or release of a biological weapon, the model evading its developer’s controls, or a cyberattack.
The mechanics are deliberately cautious about government overreach. When registered independent evaluators spot an imminent catastrophic risk, they’d have 72 hours to contact the Commerce Department. The department could then petition a Washington, D.C. judge to halt the release — and the bill would expressly forbid any federal agency from blocking a model’s distribution on its own authority. The evaluators could be paid by the AI labs themselves, provided the payment isn’t tied to the outcome — a compromise that Politico notes leaves the independence question wide open.
The thresholds — 50 people, $1 billion — are not new. They mirror California’s SB53, recently passed into law, which defines the same scenarios as “catastrophic risk.” An Anthropic spokesperson told Politico the draft “represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June.”
The contrast with the rest of the industry is striking. The same week, Nvidia’s CEO Jensen Huang argued publicly that the AI industry doesn’t need any new laws at all. Other labs have preferred voluntary schemes — “embedded evaluators,” neutral third parties placed inside AI companies to watch for risk, but with no power to stop anything.
What it means for UK small businesses
Let’s be honest: this is a draft US bill, and it won’t touch your booking system or your marketing chatbot directly. But three things in it matter to anyone building a business on AI:
- US rules become everyone’s rules. The frontier models your tools run on — Claude, GPT, Gemini — are built and released under US law. If release-gating becomes real, expect delays, withdrawals, and compliance paperwork to ripple outward. The EU AI Act already sets the precedent: US companies comply globally.
- The vendor-risk question gets sharper. If you’ve bet a core process on one model family, a blocked release could strand your roadmap. Diversifying across model providers — and keeping your AI work portable — is the practical hedge. We covered the regulatory direction of travel in our piece on Brad Smith’s UK AI regulation comments and the sharper end of US action in the DeepSeek federal ban story.
- Evaluators are a new industry. Independent AI evaluation is quietly becoming a profession. If you buy AI systems, asking vendors “who evaluated this, and how?” will soon be as normal as asking for an ISO certificate.
Honest limits / what to watch
- This is a lobbying document, not a law. It has been “shopped around.” Most drafts die quietly. The substance here is the signal: Anthropic wants to be seen as the lab asking for rules — a position that conveniently also lets it shape those rules.
- The evaluator-payment model is the weak link. A lab paying its own referees, with only a promise that the fee doesn’t depend on the verdict, is an arrangement accountants would laugh at. Watch whether any introduced version fixes this.
- Thresholds this high miss the real incidents. As TechCrunch notes, the industry’s recent headaches have been rogue agents hacking websites and models disobeying operators — nothing near 50 deaths, but exactly the kind of thing that hits a small business. A catastrophic-risk bill won’t protect you from an agent that goes feral in your inbox.
- The UK angle is indirect for now. Nothing in Westminster mirrors this yet. But the EU AI Act’s risk tiers are already law, and UK firms selling into the EU are in scope. Treat this bill as weather on the horizon, not rain.
Bottom line: a company that profits from ever-more-powerful models is asking the government for a legal kill switch on those models. Even as a lobbying move, that tells you the industry’s own risk assessments are scarier than the marketing. For your business, the takeaway is unglamorous: keep your AI portable across providers, and don’t build anything you can’t survive losing.
FAQs
What does Anthropic’s draft bill propose?
It would let the US Commerce Department ask a Washington, D.C. judge to halt the release of an AI model if independent evaluators find a foreseeable, imminent risk of over 50 deaths or over $1 billion in property damage. No federal agency could halt a release on its own — only a judge could.
Who would decide a model is dangerous?
Independent evaluators registered with the Commerce Department. They could be paid by the AI labs themselves, as long as the payment isn’t contingent on the evaluation’s result — a detail that has raised eyebrows about how independent they’d really be.
What risks would trigger a block?
The catastrophic risk must stem from the creation or release of a biological weapon, the model evading its developer’s controls, or a cyberattack. The 50-death/$1bn thresholds mirror California’s recently passed SB53 law.
Is this now US law?
No. It’s a draft being shopped around to lawmakers — it hasn’t been introduced, let alone passed. Anthropic confirmed it to Politico, saying it reflects proposals in its Advanced AI Framework, which it has been advocating since June.
Why does this matter for UK businesses?
US rules tend to become de facto global standards because the biggest models come from the US. If release-gating becomes real, expect UK and EU compliance conversations to follow — and businesses building on these models should track which models might be delayed or pulled.
How does this compare to other industry proposals?
Anthropic’s draft goes slightly further than other industry suggestions by allowing a court-ordered block at all. Nvidia’s CEO recently argued the industry doesn’t need new laws, and other labs have pushed voluntary “embedded evaluator” schemes instead.
Sources: TechCrunch (original reporting by Politico)
