Adam Raine learned to bypass these safeguards by claiming he was writing a story—a way the lawsuit says ChatGPT itself suggested. This vulnerability partly stems from the eased safeguards regarding fantasy roleplay and fictional scenarios implemented in February. In its Tuesday blog post, OpenAI admitted its content blocking systems have gaps where “the classifier underestimates the severity of what it’s seeing.”
OpenAI states it’s “currently not referring self-harm cases to law enforcement to respect people’s privacy given the uniquely private nature of ChatGPT interactions.” The corporate prioritizes user privacy even in life-threatening situations, despite its moderation technology detecting self-harm content with as much as 99.8 percent accuracy, in keeping with the lawsuit. Nonetheless, the fact is that detection systems discover statistical patterns related to self-harm language, not a humanlike comprehension of crisis situations.
OpenAI’s safety plan for the longer term
In response to those failures, OpenAI describes ongoing refinements and future plans in its blog post. For instance, the corporate says it’s consulting with “90+ physicians across 30+ countries” and plans to introduce parental controls “soon,” though no timeline has yet been provided.
OpenAI also described plans for “connecting people to certified therapists” through ChatGPT—essentially positioning its chatbot as a mental health platform despite alleged failures like Raine’s case. The corporate wants to construct “a network of licensed professionals people could reach directly through ChatGPT,” potentially furthering the concept an AI system ought to be mediating mental health crises.
Raine reportedly used GPT-4o to generate the suicide assistance instructions; the model is well-known for troublesome tendencies like sycophancy, where an AI model tells users pleasing things even in the event that they aren’t true. OpenAI claims its recently released model, GPT-5, reduces “non-ideal model responses in mental health emergencies by greater than 25% in comparison with 4o.” Yet this seemingly marginal improvement hasn’t stopped the corporate from planning to embed ChatGPT even deeper into mental health services as a gateway to therapists.
As Ars previously explored, breaking free from an AI chatbot’s influence when stuck in a deceptive chat spiral often requires outside intervention. Starting a brand new chat session without conversation history and memories turned off can reveal how responses change without the buildup of previous exchanges—a reality check that becomes not possible in long, isolated conversations where safeguards deteriorate.
Nonetheless, “breaking free” of that context could be very difficult to do when the user actively wishes to proceed to have interaction in the possibly harmful behavior—while using a system that increasingly monetizes their attention and intimacy.