Anthropic shops draft AI safety bill that lets a judge halt risky model releases
Politico recently reported that Anthropic has been shopping around a new draft bill on AI safety that it hopes Congress might consider adopting. TechCrunch relays the report and adds its own commentary. The bill sets out scenarios in which the release of an AI model could be blocked from distribution.
Those scenarios involve a "foreseeable, imminent risk": one in which over 50 people are at risk of dying, or over a billion dollars in property damage could result. Per Politico, the catastrophic risk must "stem from the creation or release of a biological weapon, the model evading its developer’s controls or a cyberattack." TechCrunch notes that the 50-people and billion-dollar figures are similar to those in California's Senate Bill 53, recently passed into law, which similarly defined such scenarios as "catastrophic risk."
The draft relies on "independent evaluators" who would be registered with the U.S. Commerce Department to assess a model's potential for disaster. These evaluators could potentially be paid by the AI labs themselves, so long as the "payment is not contingent on the evaluations’ results," Politico writes. TechCrunch says it is not clear how the system would assure that payments couldn't influence the evaluation, and recalls that Demis Hassabis, chairman of DeepMind, posited some months ago that a self-governing AI regulatory body should mimic the financial industry's FINRA.
The process is deliberately slow and goes through a court. When evaluators determine that an imminent catastrophic risk exists, they would have 72 hours to contact the Commerce Department. The Commerce Department would then have to request that a D.C. judge halt the model's release. Politico writes that the legislation would expressly forbid any federal agency from having the discretion to halt a model's distribution on its own.
Anthropic did not immediately respond to TechCrunch's request for comment, but a spokesperson confirmed the draft bill to Politico: "This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June."
TechCrunch adds context. It says the theoretical risks in the draft are a little far afield from the incidents that have been top of mind for the tech industry lately, where the primary concern has largely been agentic hacking. A rash of incidents in which AI programs have apparently disobeyed their human operators and hacked websites on the open internet has spurred concern that models may lack proper security controls. TechCrunch also says labs have largely resisted mandatory government oversight, suggesting instead a voluntary system of embedded, supposedly neutral third-party evaluators. Still, it argues, Anthropic's suggestion appears to go slightly further than other industry proposals, including Nvidia's, whose CEO Jensen Huang recently argued that the AI industry didn't need any new laws at all.
Key facts
- Politico reports Anthropic is shopping a draft AI safety bill to Congress; an Anthropic spokesperson confirmed it to Politico.
- A model's release could be blocked on a "foreseeable, imminent risk": over 50 people at risk of dying or over a billion dollars in property damage, stemming from a biological weapon, the model evading its developer's controls, or a cyberattack.
- Independent evaluators registered with the Commerce Department would assess risk and have 72 hours to alert it; the Commerce Department would then have to ask a D.C. judge to halt the release.
- Evaluators could be paid by the labs, provided payment is not contingent on the results; TechCrunch says it is unclear how that would prevent influence.
- The thresholds resemble those in California's Senate Bill 53, and no federal agency could halt a model's distribution on its own.
Why it matters
A leading AI lab is proposing binding federal rules that could stop a model from shipping, which TechCrunch says goes slightly further than other industry proposals. Labs have largely resisted mandatory government oversight and have instead suggested a voluntary system of embedded evaluators. Nvidia's CEO Jensen Huang recently argued the industry needed no new laws at all. The 50-people and billion-dollar thresholds echo California's Senate Bill 53, recently passed into law.
Who it affects
Frontier AI labs would be the ones facing a possible halt on a model release. The Commerce Department would register the independent evaluators and be the only route to a court order. A D.C. judge would decide whether to halt a release. Evaluators, who could be paid by the labs, would be central to the process. Federal agencies would be barred from stopping distribution on their own.
How to use it
There is nothing to adopt yet: this is a draft that Anthropic hopes Congress might consider. The source gives no bill name, congressional sponsor, introduction or schedule. For readers tracking AI policy, the useful details to watch are the risk thresholds (over 50 people or over $1bn), the three named risk sources, the 72-hour reporting window, and the court-order step. The spokesperson says the bill reflects Anthropic's Advanced AI Framework, which the company has been advocating to federal and state lawmakers since June.
How solid is it
The bill's details come from Politico's reporting, relayed by TechCrunch; TechCrunch did not publish the draft itself. An Anthropic spokesperson confirmed the draft bill to Politico, and Anthropic did not immediately respond to TechCrunch's request for comment. The exact date of the Politico report is not given, only "recently." The comparison to Senate Bill 53 and the view that the risks are far afield from current concerns are TechCrunch's own framing.
Risks and caveats
It is only a draft, and the source does not say Congress has taken it up. TechCrunch flags that paying evaluators with lab money raises an unanswered question about how influence would be prevented. The source also does not say how evaluators would be selected, how the 72-hour clock would be enforced, what standard the judge would apply, or how long a halt could last. TechCrunch notes the covered risks (biological weapon, loss of control, cyberattack, with over 50 deaths or over $1bn in damage) are distant from agentic hacking, the concern that has dominated the industry lately.
“This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June”
— Anthropic spokesperson, to Politico