AI Hallucination Nearly Triggers U.S. Military Operation

Military aircraft were already in the air this spring when U.S. officials discovered that intelligence supporting an armed operation against a Chinese vessel had been generated by an AI chatbot, CNN reported. The operation was aborted at the last minute, narrowly avoiding a potential conflict with China.

The intelligence report circulated during the war with Iran and claimed that the vessel carried components for a nuclear weapons program. According to the report, a Special Operations Command analyst used an AI chatbot to combine open-source information with classified signals intelligence. The chatbot misidentified the ship’s cargo manifest. The analyst then used the tool again to format the mistaken findings as an official-looking summary that moved through command channels.

The episode highlights concerns that AI errors can travel up the chain of command before they are challenged, as the U.S. military seeks to use AI to accelerate decisions and maintain its advantage over China. The Pentagon has said AI can speed up its kill chain, but rapid adoption may also spread hallucinations when human oversight is insufficient.

“It’s important for service members to understand the uncertainty inherent to LLMs,” Jake Steckler, a research scholar at GovAI and a veteran U.S. Army officer, told TechCrunch in a written response. He said safeguards are especially important for targeting, intelligence analysis, and operational planning, while arguing that the incident should encourage safer use rather than abandonment of the technology.