Jacob Coxon, an AI researcher who previously worked at both Anthropic and OpenAI, recently resigned from Anthropic citing serious safety concerns. In a widely viewed post on the social media platform X, Coxon accused both companies of recklessly advancing AI technology without adequate safeguards, warning that the teams building these systems genuinely believe AI could lead to human extinction by the end of the decade. He emphasized the transformative and potentially dangerous capabilities of AI, which could soon surpass human intelligence and gain unprecedented power.

Coxon's resignation has reignited ongoing debates within Silicon Valley about the proper pace and control of AI development. As Anthropic and OpenAI prepare for possible initial public offerings, industry insiders are calling for a more cautious approach. OpenAI’s chief scientist Jakub Pachocki acknowledged these challenges in a blog post, advocating for voluntary slowdowns until basic safety standards are established and underscoring the need for global cooperation to manage AI risks more effectively.

Another key AI alignment expert at Anthropic, Evan Hubinger, concurred with Coxon’s assessment, stating that there is over a 10% probability that AI could cause human extinction within the next decade. Both researchers highlighted that despite their employers' best efforts, current alignment strategies are insufficient to guarantee safe development of superintelligent AI, especially with the threat of recursive self-improvement where AI systems might autonomously enhance themselves without human oversight.

The heightened concern has prompted government attention, with lawmakers proposing legislation aimed at regulating AI’s rapid evolution. The bipartisan FRONTIER Act seeks to create regulatory frameworks for advanced AI models, while the Ban Artificial Superintelligence Act calls for a temporary pause in development until safety measures are firmly in place. However, these efforts face mixed reactions, even as public skepticism and pushback against AI infrastructure grow, putting pressure on AI firms to better communicate benefits and risks to the broader public.