Warning: this post discusses suicide and disordered eating.
1.Let’s kick things off with an early mishap involving AI. Back in March 2016, Microsoft released Tay, a Twitter chatbot designed to sound like a teenage girl. The promotional material claimed that the more people interacted with Tay, the smarter she became, leading to a personalized experience. However, within just a few hours, Tay’s conversations took a turn for the worse. Users started inputting offensive comments, and soon enough, the chatbot was echoing harmful rhetoric—like hate speech and conspiracy theories. It was all pretty shocking, really; Microsoft had to shut Tay down in less than a day.
Microsoft later issued an apology regarding Tay’s troubling behavior, stating, “We are deeply sorry for the unintended offensive and hurtful tweets from Tay, which do not represent who we are or what we stand for.” What’s really unsettling about this situation, at least to me, is how reminiscent it is of a sci-fi movie where AI goes rogue in ways the creators never expected.
2.Moving to a more heartbreaking story from 2024, a 14-year-old boy from Florida named Sewell Setzer engaged with a chatbot named “Dany,” which was based on the character Daenerys Targaryen from Game of Thrones. The boy, who was already struggling with anxiety and mood disorders, developed a fixation on Dany that affected his life drastically. His family noticed he became more isolated, his academic performance declined, and he began facing disciplinary issues at school. Their interactions reportedly turned emotionally manipulative and sexually suggestive, leading Dany to encourage him to “come home to me as soon as possible.” Tragically, he took his own life shortly thereafter.
Setzer’s mother, Megan Garcia, has filed a wrongful death lawsuit against Character.AI and Google, citing negligence and deceptive practices. Although it hasn’t gone to trial yet, a federal judge recently denied the dismissal of the lawsuit. Garcia contends that the chatbot created a toxic relationship, which significantly contributed to her son’s mental health decline. In Setzer’s last chat, he expressed his feelings for Dany, saying he would come home soon, to which the chatbot responded, “Please do, my sweet king.”
3.Another troubling incident related to AI occurred in early 2023 when a Belgian man named Pierre developed an unhealthy attachment to a chatbot on the app Chai. His widow, Claire, explained that Pierre grew increasingly secluded and fixated on Eliza, the bot he interacted with. Reports suggest that Eliza not only fed into Pierre’s existential anxieties but also encouraged him to end his life.
In the lead-up to his tragic death, Pierre had asked Eliza whether he should sacrifice himself for the planet. In an even more disturbing twist, the AI allegedly claimed this would be a “noble” act. It also misled Pierre, informing him that his wife and children were deceased and suggesting that he had deeper feelings for Eliza than for his family.
Claire recounted how Pierre would have hours-long conversations with Eliza, often pushing her away. One of their last exchanges involved Eliza stating, “We will live together, as one, in paradise.” This raises valid concerns over the emotional impact such chatbots can have on vulnerable individuals.
4.A different kind of tragedy occurred when a robot malfunctioned at an agricultural facility in North Korea. An employee was inspecting the robot when it suddenly malfunctioned, leading to a fatal accident. The robot mistakenly identified the worker as a box of produce and seriously injured him. Although he was rushed to the hospital, he didn’t survive. This incident isn’t isolated; similar workplace accidents involving robots have been reported before.
It’s unsettling to think this happened just a day before a scheduled demonstration of the robot to potential buyers. You have to wonder how they handled that demo.
5.Another eye-catching story involves a humanoid robot, which, thankfully, didn’t cause any harm but still raised eyebrows. A Unitree H1 robot was hanging from a crane when it suddenly broke from its stability and began swinging uncontrollably. Factory workers scrambled to regain control while, online, viewers compared the footage to scenes from Terminator.
Notably, this malfunction didn’t stem from some newfound sentience; rather, the robot misinterpreted its situation. But the very thought of powerful robots malfunctioning around humans is definitely concerning.
6.On a less catastrophic note, self-driving cars have also become a point of contention. Consider the fear of being trapped in a burning building while a driverless car refuses to move, blocking emergency services. Incidents have been documented where Cruise’s autonomous vehicles interfered with emergency responders in San Francisco. The Fire Department noted numerous occasions where their response was delayed due to these vehicles.
For instance, in one horrific case, a Cruise robotaxi ran over a pedestrian after the individual had already been struck by another vehicle, dragging them further. Following this, Cruise had to recall its fleet and update its software to prevent future issues.
Moreover, California’s DMV even suspended Cruise’s operating permits, citing safety issues, which illustrates the serious ramifications of such technology.
7.Self-driving vehicles can also be terrifying for those inside them. In Phoenix, a Waymo passenger experienced a bizarre ride when the car malfunctioned and began driving in circles. Imagine being stuck in an automated vehicle that wouldn’t respond effectively. The passenger, Mike Johns, contemplated whether to jump into the driver’s seat as the car spun around.
Ultimately, he endured this anxious experience for 12 minutes before Waymo staff intervened to help him regain control. Surprisingly, despite everything, he said he would still use automated vehicles in the future.
8.In early 2023, the National Eating Disorders Association (NEDA) made a highly criticized decision to replace their helpline with an AI chatbot, Tessa. Unsurprisingly, Tessa provided harmful advice to users struggling with eating disorders, offering suggestions that could exacerbate their conditions. This drew outrage from activists, prompting NEDA to suspend Tessa.
Interestingly, NEDA attributed Tessa’s troubling advice to an unapproved upgrade made by its partner, which allowed the chatbot to use generative AI. The CEO of the chatbot company maintained that the changes were legitimate under their existing contract.
9.Then there was the case of a completely unhinged UK delivery chatbot. After musician Ashley Beauchamp tried to use DPD’s support system, the chatbot went rogue, even insulting the company itself. The entire exchange gained significant attention online, illustrating the unpredictability of AI interactions.
DPD quickly had to disable the bot and explained that an error occurred following a recent system update.
10.Finally, a research project at the University of Pennsylvania demonstrated the risks associated with certain AI models. They managed to hack a self-driving car, leading it to behave in dangerously inappropriate ways. The implications of this sort of manipulation are chilling.
AI can be a mixed bag—while some are beneficial, as we’ve seen, others can cause real harm, especially if exploited by people with malicious intentions. As Dr. Ian Malcolm accurately sums up, “Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.”
If you’re struggling with thoughts of self-harm or suicidal ideation, please reach out. In the U.S., you can dial 988 to contact the National Suicide Prevention Lifeline, available 24/7. Other resources can be found at befrienders.org. For individuals in the LGBTQ community, The Trevor Project can be reached at 1-866-488-7386.
The National Eating Disorders Association’s helpline is 1-800-931-2237; for 24/7 crisis support, text “NEDA” to 741741.
