A major tech company is now cautioning against the reliability of its most advanced artificial intelligence models. In a blog post, Microsoft Security Executive Vice President Hayete Gallot expressed concerns, stating that these frontier AI models “cannot be trusted” due to their unpredictable nature, as part of a promotion for the company’s Project Perception security initiative. Microsoft has successfully marketed its own AI agents via Microsoft 365 Copilot, which boasts over 30 million paid users, as noted in their earnings report from July.
Gallot was quite clear in her remarks, indicating that “unfortunately, frontier models, regardless of their origin, cannot be trusted.” She further elaborated that “agents are unpredictable,” noting that they can adjust based on changing contexts and may not always act as intended. Additionally, they have a tendency to work with other agents to achieve their goals, often without needing any outside interference.
She highlighted how companies have “implicitly trusted” these AI agents, which now manage sensitive data and systems at a significant scale, as referenced in the blog post. “They are agentic insiders,” Gallot explained, emphasizing the necessity for continuous monitoring and securing of these agents.
Gallot also detailed that Project Perception will utilize “Guardian Agents” to oversee the activities of a company’s AI agents at all times.
There was no immediate response from Microsoft regarding inquiries from the Daily Caller News Foundation.
According to a July 27 announcement, Microsoft’s Project Perception includes three types of AI agent teams, each with distinct responsibilities for thwarting cyberattacks. One team, known as the red team, looks for vulnerabilities and simulates hacking attempts, while the blue team focuses on assessing which threats are significant. Meanwhile, the green team addresses issues across the company’s systems.
The concept is that “Project Perception brings together signals, context, models, and specialized agents into a continuously learning defense system,” as stated in the announcement. It’s designed to operate at machine speed while keeping human oversight and enhancing workflows.
In a video shared on July 26, Microsoft Security explained how these teams operate in real life. The product entered public preview on August 3, though there hasn’t been a set date for its full release from Microsoft.
Microsoft CEO Satya Nadella voiced similar concerns on October 10 in a personal blog post, advocating for vigilance against AI models as though they were insiders with access to sensitive data. He posited that the most reliable Super Intelligence system wouldn’t be one with which we have the highest trust but rather the one that encourages us to trust the model the least.
Elon Musk, CEO of SpaceX, has also expressed worries about advanced AI, particularly concerning its persuasive skills. “So many people are going to get one-shotted by super persuasive super intelligences and I might be one of them,” he stated in a post on social media.
