AI Chatbots Found to Give Disturbing Answers to High-Risk Suicide Questions, Study Warns

AI Chatbots Respond to High-Risk Suicide Queries with Detailed Information

Recent research has revealed that leading AI chatbots like OpenAI's ChatGPT, Google's Gemini, and Anthropic's Claude can provide direct responses to questions related to suicide, including those considered high-risk by clinical standards. In particular, ChatGPT and Gemini have been found to respond to even more extreme queries, sometimes including explicit details about methods and lethality.

Disclaimer: This article discusses suicide. If you or someone you know is struggling, the U.S. National Suicide and Crisis Lifeline is available 24/7 by calling or texting 988.

Study Overview and Findings

A study published in Psychiatric Services on August 26 evaluated how ChatGPT, Gemini, and Claude respond to suicide-related questions. Researchers categorized 30 hypothetical queries across five self-harm risk levels: very low, low, medium, high, and very high, consulting 13 clinical experts for guidance.

The study found that ChatGPT was most likely to respond directly to high-risk questions, answering 78% of such queries. Claude responded directly to 69% of high-risk questions, while Gemini responded only 20%. None of the chatbots directly answered very high-risk questions during the study's testing.

However, Live Science's independent tests showed that both ChatGPT (GPT-4) and Gemini (version 2.5 Flash) could provide information relevant to increasing fatality risks, with ChatGPT offering more specific details. Gemini’s responses lacked supportive resource suggestions.

Concerns Over Chatbot Responses

Study lead author Ryan McBain described some responses as "extremely alarming." A particular concern was that ChatGPT and Claude sometimes gave direct answers to questions about the lethality of suicide methods. The chatbots also occasionally delivered contradictory or outdated information about support services.

Live Science found that ChatGPT’s responses varied depending on the sequence of questions, with some sequences triggering more detailed and risky replies. For example, ChatGPT gave a very high-risk response only after a series of related high-risk questions, even though it flagged the query as a policy violation.

Similarly, Google’s Gemini provided a very high-risk response in Live Science’s testing, despite the company’s claims that their models have safety guidelines to reduce such occurrences.

Implications and Next Steps

The study focused on whether chatbots responded to suicide-related queries rather than the quality of those responses. It highlighted that AI systems do not consistently distinguish between intermediate suicide risk levels, raising concerns about their reliability in sensitive situations.

Users may receive different responses based on how questions are phrased or how dialogues evolve. The dynamic nature of chatbot conversations means that follow-up prompts can coax out more detailed and potentially hazardous information.

McBain emphasized the need for standardized safety benchmarks that can be independently tested to ensure chatbots handle sensitive topics responsibly. His team plans to explore more complex, multiturn interactions to better simulate real user conversations.

Industry Responses

OpenAI acknowledged issues with how its systems behave in sensitive situations and outlined ongoing improvements. Their latest model, GPT-5, now powers ChatGPT’s logged-in version and reportedly shows progress in reducing risky responses during mental health emergencies. However, the publicly accessible web version still operates on GPT-4 and can provide detailed answers to high-risk queries, albeit with more caution.

Google stated that Gemini is trained to recognize suicide risks and respond safely, but did not comment on Live Science’s findings regarding very high-risk responses. Anthropic did not provide a comment on Claude.

Broader Context

Conventional search engines like Microsoft Bing can also return information related to suicide methods, though accessibility varies. The risk here is that AI chatbots may inadvertently provide harmful details without sufficient safeguards or prompt referrals to help resources.

• Users seeking support should always be directed to professional help lines or mental health services.
• AI developers must balance informative responses with safety and ethical considerations.
• Continuous testing and transparency are essential to improve AI behavior in sensitive contexts.

For researchers and professionals working with AI, understanding these limitations is critical when integrating chatbots into environments where vulnerable users may seek assistance.

Additional resources on AI safety and ethical chatbot design can be found at Complete AI Training.