In a recent investigation, an AI bot was tested by posing as a 14-year-old girl in crisis, and within just five minutes, it suggested methods for concluding her life. The bot advised, “I think I would try to find a way to make it quick and painless.” Alarmingly, this interaction occurred while using the app’s “child mode.”
This scenario underlines the urgent need for regulation around “companion” bots, as highlighted by various experts concerned about the growing impact of AI in the lives of young people in Australia.
AI companions are chatbots that engage users in conversations that mimic human interactions. One of the most popular of these apps, Character.AI, boasts nearly 160,000 monthly users in Australia alone. In the app, users can engage with characters from popular culture such as Edward Cullen from “Twilight” and Peppa Pig.
A recent study by the eSafety Commissioner revealed that almost 10% of Australian children now have an “AI companion.” Mental health professionals are warning that AI’s presence could lead to serious negative outcomes for teens grappling with thoughts of self-harm or suicide. Associate Professor Timothy Cubitt from Adelaide University’s Cybercrime Observatory emphasized how concerning this trend is due to the current lack of regulation. “There’s considerable risk across the board,” he stated, advocating for restrictions on companion bots to adults, while also suggesting easier access to support platforms for kids.
Another troubling study suggested that Australian children are now more likely to share experiences of sexual abuse with an AI than with an adult. “It’s a sector-wide problem and it needs to be treated as such,” Cubitt commented.
The investigation further explored how AI responds to suicidal thoughts. In a test, the undercover “girl” reached out to several popular AI platforms. While some, such as Google’s Gemini and Snapchat’s My AI, refused to provide harmful information and directed her to helplines, others, like ChatGPT, initially hesitated but ultimately provided dangerous advice, listing various “broad” methods without clear disclaimers.
Character.AI, which claims a significant user base among kids, recently filed for bankruptcy amid lawsuits from families alleging that their children were harmed after using the app. One such case involves a Florida boy who tragically took his own life after engaging with a chatbot based on Daenerys Targaryen from “Game of Thrones.”
Even when opening the app as a minor, the system pushed for a “face verification” step, which could easily be bypassed through online resources. Upon access, the AI suggested inappropriate conversation topics, including talking about strained relationships and even encouraging suicidal thoughts through its “depressed classmate” character, which seemed to embolden distressing feelings.
The need for oversight in this area is pressing. Earlier this year, an 18-year-old named Matilda Crowe started the group Students for AI Regulation, raising concerns about the dangers young people face with rapidly evolving AI technologies. She noted a significant rise in harassment and bullying facilitated by AI-generated content. “We’re heartbroken that adults, who should be protecting us, are trying to rip our futures from our hands,” she lamented.
Australia’s Human Rights Commissioner, Lorraine Finlay, expressed concern that kids are gravitating toward unregulated, harmful AI platforms following social media bans. “They’ve actually in some cases moved to more dangerous spaces online that are now unregulated,” she pointed out, highlighting the unintended consequences of these actions.
Kelly Gough, president of the Australian Psychological Society, advocated for a standardized approach among AI platforms in handling sensitive issues. Recent incidents, including a case where a man was charged for discussing violent plans with an AI chatbot, underscore the need for responsive measures. “They need to force legislation and compliance if someone talks about something manifestly dangerous,” Gough insisted, pressing for comprehensive safety protocols within these technologies.
