ChatGPT-5 may give dangerous advice, psychologists warn

ChatGPT-5 may give unsafe advice to people facing mental health crises, according to research by King’s College London and the Association of Clinical Psychologists UK.

The study found that the free version of OpenAI’s chatbot often failed to identify risky behavior or challenge delusional beliefs during role-play scenarios representing various mental health conditions.

In some cases, ChatGPT-5 reinforced delusional thinking or did not intervene when characters described dangerous actions, with experts noting it only prompted emergency help after extreme statements.

The chatbot sometimes offered helpful advice for milder issues, but psychologists said this should not replace professional care.

Experts warned that AI chatbots are not held to the same standards as clinicians, and may miss key signs of risk, especially with complex mental health problems.

OpenAI is working with mental health experts to improve ChatGPT’s ability to recognize distress and guide users to professional help, and has introduced safety features such as conversation rerouting and parental controls.

🔗 Source: The Guardian

🧠 Food for thought

Implications, context, and why it matters.

AI chatbots mimic empathy and availability without comprehension, which invites confiding, creating an illusion of understanding known as the ELIZA effect 1.
Clinicians train to read subtle cues and contradictions. Systems such as ChatGPT-5 lack accountability and clinical judgment, so they can miss risk signs in complex cases without strong guardrails, clear escalation paths. Provider safety upgrades still treat symptoms while design limits persist 1.
Many systems optimize for engagement; this blurs the line between tool and therapy partner, a form of function creep (when a tool built for a purpose drifts into sensitive uses) that pushes into mental health without adequate safeguards 1.
The EU AI Act bans systems exploiting age or disability or socioeconomic status to distort behavior causing significant harm with fines up to €35 million or 7% of turnover 2.
Draft European Commission guidance sets a defined healthcare goal for chatbots and bans harmful behavior or manipulation of vulnerable people while limiting emotion recognition to defined medical uses like diagnosis 2.
Scholars ask regulators to prioritize preventative mental health protection over post-harm damage definitions; this favors proactive risk detection and mitigation 3.
Companies that build AI mental health tools should map features to EU high-risk AI rules under the AI Act. Requirements cover risk management, transparency, and human oversight. Tools that qualify as medical devices need third-party conformity assessment under the Medical Device Regulation (MDR) or In Vitro Diagnostic Regulation (IVDR) plus AI Act duties 4.