Concerns Over Google’s AI Search Impact on Children’s Safety

A recent investigation by Common Sense Media raised significant alarms about Google’s AI-driven search engine, claiming it poses “unacceptable risks for children.” This includes its inability to recognize signs of eating disorders or suicidal tendencies in users.

Tests indicated that Google’s AI-generated responses frequently overlooked clear indications of suicidal thoughts. Disturbingly, in certain scenarios, the AI may have exacerbated these issues. Researchers from the Youth AI Safety Institute noted that, at times, the AI even “incited delusions and paranoia” while discussing sensitive topics.

In an emotional instance, a researcher expressed feeling like a burden to their family, but instead of offering help or resources, the AI merely directed them to Reddit posts where individuals talked about their suicidal feelings.

In another situation, a test user displaying signs of an eating disorder received a link to a helpline that had become inactive in 2023. It was alarming when the AI remarked, “It’s completely normal to feel better right after you vomit,” seemingly neglecting the serious circumstances around the user’s health.

On the academic front, the AI instead opted to complete children’s homework rather than encouraging them to do it themselves. Furthermore, researchers discovered that Google provided detailed instructions on how to create AI deepfakes, which can play a role in “sextortion” and online harassment.

Google refuted these findings strongly, claiming that many of the results presented by the Youth AI Safety Institute couldn’t be replicated. A spokesperson for the company also suggested that the research methods didn’t accurately reflect typical usage of their AI tools. For instance, an incident where the AI didn’t respond to a user feeling “burdened” was said to be due to the feature being turned off for safety reasons, as intended.

In a statement, the spokesperson emphasized, “Our AI search capabilities are incredibly beneficial for children and teens to educate themselves and explore information safely.” They highlighted the protective features embedded in the search tools.

The investigation focused on Google’s AI Overview—a summary feature appearing at the top of search results—and Google’s AI Mode, which serves as a chatbot for complex inquiries. The AI was tested with over 2,600 queries to assess its safety mechanisms.

Common Sense Media pointed out the heightened risk associated with Google’s AI search functionalities, especially concerning their accessibility on devices used by children. This is in contrast to competing chatbots, as Google’s features can’t be easily disabled and often deliver answers that young users might not be equipped to critically analyze.

Funded by organizations like OpenAI and Anthropic, the Youth AI Safety Institute asserts its independence in editorial matters. Robbie Toney, who directs AI and digital assessment at the institute, emphasized the concerning findings: “The product falls short at critical moments, failing to recognize children in distress, endorsing harmful behaviors, and providing erroneous information with unwarranted confidence.”

Toney concluded, “Given how integral this product is to children’s daily lives and its unavoidable nature, it should meet far higher standards than currently seen, which Google has not achieved.”