Anthropic, an AI research lab, has proposed a new AI safety bill aimed at regulating the release of advanced AI models by the U.S. government. The draft legislation sets specific thresholds for intervention, blocking any model if there is a foreseeable and imminent risk of death to 50 or more people or property damage exceeding a billion dollars. These catastrophic risks must arise from scenarios such as the creation or release of biological weapons, AI models evading developer controls, or cyberattacks with severe consequences.

To assess the safety of AI systems, the bill suggests appointing independent evaluators registered with the Commerce Department who would review AI models before deployment. While these evaluators might be funded by AI companies, their compensation would not depend on the outcomes of their assessments. If a model is deemed too dangerous, these evaluators have 72 hours to notify the Commerce Department, which would then seek a judge’s approval in Washington D.C. to halt the model’s release.

The bill notably prohibits any federal agency from unilaterally deciding to block an AI model without judicial authorization, marking a unique aspect of the proposed regulatory framework. This approach contrasts with some industry voices like Nvidia's CEO Jensen Huang, who has recently opposed additional AI laws, arguing that the sector can manage its own safety measures without new government mandates.

Anthropic's proposal aligns with its ongoing advocacy for AI regulation since June and goes beyond some other companies' voluntary safety suggestions, such as embedding evaluators within organizations. Although the bill focuses on extreme risks rather than more common AI security concerns like hacking and rogue behavior, it represents a significant push from an AI lab towards formal legislation aimed at preventing catastrophic outcomes related to artificial intelligence.