Pentagon Explores AI for Wargaming, Faces “AI Illusion” Risks
The Pentagon is harnessing artificial intelligence in an effort to transform its wargaming capabilities, but this move has raised concerns about the risks associated with what is termed “AI illusion.”
Code Metal, an AI firm based in Boston, has secured an $80 million contract to integrate AI into WarMatrix, the Army’s wargaming program. Reports emerged on Friday about this development. Experts caution that the introduction of AI into military wargaming could expose the U.S. military to misleading information or “AI illusions.”
“Hallucinations are tied to large-scale language models,” stated Elke Schwartz, a political theory professor at Queen Mary University of London, in an interview. “These hallucinations refer to outputs that seem plausible or coherent, yet are factually incorrect or completely nonsensical. While we can perhaps reduce these hallucinations, they can never be completely eliminated.”
The inevitability of AI hallucinations poses significant risks, especially in military contexts. Some analysts have even speculated that AI might have played a role in a tragic bombing incident at an Iranian girls’ school, which officials claim resulted in the deaths of 168 children, the majority of whom were girls.
The investigation into that attack was ongoing as of March 11, but Defense Secretary Pete Hegseth asserted during a broadcast that the U.S. did not deliberately target civilians.
Currently, there are no new updates concerning the attack investigation, according to defense officials.
The Air Force indicated that WarMatrix has been in development since at least 2025, incorporating established military modeling and simulation systems. Now, AI is being utilized to enhance and expedite these existing capabilities.
The Air Force Public Affairs Office did not provide comments upon request, and the Department of Defense was equally reticent.
Michael Horowitz, director at the University of Pennsylvania’s Perry World House, emphasized the importance of rigorous testing and evaluation of AI systems for effective military use. “There’s considerable potential in using these tools to refine wargaming and modernize military planning,” he noted. “Like any tool, continual testing and error checking are vital to ensure AI output fosters better military decision-making rather than hindering it.”
The contract acquisition process reportedly took under a year, with Code Metal’s CEO Peter Morales commenting on the unusually swift pace for the Department of Defense.
Morales noted the excitement of this rapid movement, remarking, “I’ve never seen them move this fast.”
Code Metal did not reply to requests for further commentary.
Schwartz remarked on the serious implications for security and warfare, highlighting that in fast-paced situations, double-checking outputs might not always be feasible. “Imagine an AI analyzing defense documents, possibly suggesting nonexistent targets or misinterpreting enemy intentions,” she said.
This misidentification could lead to tragic outcomes, such as the erroneous classification of a girls’ school as a military target, contributing to civilian casualties.
Adm. Brad Cooper, U.S. Central Command commander, stated in a video update that advanced AI tools were being employed in Operation Epic Fury.
Horowitz expressed that if users of AI in wargames remain aware of their systems’ limitations, including the potential for hallucinations, they could effectively leverage the technology. However, he warned against falling into automation bias, which can lead to blind trust in AI outputs that might be flawed.
In military scenarios, such problematic hallucinations can result in the loss of innocent lives.
Schwartz added that the challenges could deepen when relevant conflict data is scarce, fostering an environment reliant on potentially unreliable AI outputs. “All AI systems can distort reality, and if military planners fail to understand how to gauge AI’s limits, they risk making misguided assumptions and decisions,” she cautioned.
