This spring, U.S. military officials nearly launched an armed operation against a Chinese vessel based on faulty intelligence generated by an AI chatbot. Military aircraft had already taken off when it was discovered that the AI had hallucinated critical details about the ship's cargo, falsely indicating it was transporting components for a nuclear weapons program. The operation was subsequently aborted just in time, preventing a potential international conflict.

The erroneous intelligence originated from a Special Operations Command analyst who used an AI tool to merge open-source data with classified signals intelligence. The chatbot inaccurately identified the manifest, and the analyst then used the AI a second time to produce a report formatted like an official summary, which was distributed through command channels. This incident highlights the risks of increasingly relying on AI-generated information in high-stakes military decision-making.

As the Pentagon continues to integrate AI to enhance decision speed and maintain strategic advantages over rivals like China, officials express growing concerns about the risks of AI hallucinations. The military sees AI as a key asset in accelerating response times in operations, but the speed and automation can allow critical errors to propagate without sufficient human review. Experts emphasize the necessity of recognizing the uncertainties inherent in large language models, especially for decisions involving potential use of force.

Jake Steckler, a GovAI research scholar and Army veteran, stressed that this event should prompt the implementation of stronger safeguards rather than deter AI adoption. He warned that prioritizing rapid deployment without adequate controls may erode trust among service members and ultimately hinder effective integration. Properly managed, AI tools can still provide significant benefits when used carefully within well-defined operational frameworks.