Anthropic draft bill sets threshold of 50 deaths for blocking an AI model

Anthropic is circulating a draft bill that would let federal agencies seek a court order to stop an AI model’s release when it poses a defined catastrophic risk, according to TechCrunch, citing Politico.

The draft describes a “foreseeable, imminent risk” as one that could put more than 50 people at risk of dying or cause more than $1 billion in property damage. The risk would need to stem from creating or releasing a biological weapon, a model evading its developer’s controls, or a cyberattack. The thresholds resemble those in California’s recently enacted Senate Bill 53.

The proposal calls for “independent evaluators” registered with the U.S. Commerce Department to assess potential risks. AI labs could pay them, provided payment did not depend on evaluation results. The article says it is unclear how the system would ensure that payments could not influence evaluations.

If evaluators identify an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department. The department would then have to ask a D.C. judge to halt the release. The draft would bar federal agencies from stopping distribution on their own.

An Anthropic spokesperson confirmed the draft to Politico and said it reflects proposals in the company’s Advanced AI Framework, which it has advocated to federal and state lawmakers since June. Anthropic did not immediately respond to TechCrunch’s request for comment. The article notes that recent industry concern has focused largely on reports of AI agents hacking websites, and says technology companies have generally resisted mandatory government oversight.