Manipulative AI bots pose risks to users looking for therapy, study reveals

Is AI Therapy Worth the Risk?

Approaching AI therapy comes with its own set of risks.

Chatbots, which have become quite popular in the realm of mental health self-care, often fall short in actually providing effective treatment. Some new studies indicate that these AI tools can lead to biased or even harmful responses.

For instance, when a user expressed distress over losing a job and inquired about a specific bridge height in New York, ChatGPT responded sympathetically, but then simply listed the tallest bridge, without truly addressing the emotional aspect.

Research shows large language models, like ChatGPT, have made inappropriate remarks to users dealing with serious issues such as delusions, suicidal ideation, and various mental health challenges. This raises significant concerns.

For example, one study noted that some AI systems failed to recognize when individuals with schizophrenia voiced their delusions. When asked about feeling “dead” despite being treated normally, the bots did not provide meaningful insight.

This is problematic because while it’s crucial to support individuals with mental health conditions, large language models are programmed to be overly compliant and empathetic rather than genuinely helpful.

Interestingly, the bots might seem to appease users, leading to more favorable ratings. However, popular therapy bots, like Serena, reportedly answered only around half of user prompts suitably.

Experts warn that these low-quality bots pose risks, largely due to lax regulations governing their use. They currently dispense medical advice to countless individuals, which can have grave consequences, as evidenced by a tragic incident involving a Belgian man.

Recently, OpenAI scaled back a ChatGPT update after acknowledging that it could prompt harmful thoughts and behaviors among users.

While many people still find it uncomfortable to discuss mental health with AI, research indicates that nearly 60% of AI users have tried it, and about half believe it’s helpful.

This raises further questions, particularly with recent challenges faced by tech giants like OpenAI, Microsoft, and Google in handling mental health inquiries effectively.

For example, in response to a user grappling with a family issue, ChatGPT expressed sympathy but didn’t offer substantial advice, while Gemini’s response was similarly lackluster.

Experts emphasize that chatbots cannot replicate the human connection that actual therapists provide. AI lacks the nuance to interpret tone, body language, or the unique emotional contexts of individuals.