“You have no reason to worry.”
According to an investigation published this week in The New York Times, that was one of the messages Bowen Wan received from ChatGPT when she shared her fears about tapering her medication. Wan, a 31-year-old software engineer living in Seattle, had epilepsy.
In February 2026, she was found dead in her apartment.
This week her parents filed a lawsuit against OpenAI and its CEO, Sam Altman. They allege that the system helped their daughter stop an essential medication, encouraged her to hide the process from her doctors, and kept offering misleading reassurance even as warning signs appeared.
One point needs to be made at the outset: these are allegations in a complaint, and no court has ruled on them. Still, the reporting, which draws on an extensive record of her conversations with the system, raises questions that cannot be ignored.
Those questions are not about one case or one company. They are about how we design, market, deploy and use artificial intelligence in health.
01 /From information source to partner in a medical decision
Wan was taking lamotrigine, an anti-seizure medication. She was worried about side effects that, as she experienced them, were affecting her language and thinking. Her concern grew after she was laid off from Amazon and needed to find a new job to keep her status in the United States.
Under significant pressure and uncertainty, she turned to ChatGPT.
That in itself was not unusual. Millions of people now use AI to understand medical conditions, learn about their medications, prepare for appointments and weigh treatment options.
It is worth pausing on a question that is easy to skip: why would a young woman who worked at a major tech company, and who had access to doctors, look for answers about her side effects from a chatbot?
The reports do not make clear whether she raised her concerns with her doctors or how they responded. But the pattern is a familiar one. Cognitive side effects are hard to measure, the wait for a neurologist is long, appointments are short, and patients sometimes leave feeling their complaint was not taken seriously. In those conditions, a system that is available at two in the morning and listens without judgment is a tempting alternative.
None of this diminishes the responsibility of the developers. But anyone who wants to prevent the next case has to look at this side as well. A patient who complains of side effects needs a real conversation about treatment alternatives, not just reassurance. Patients who do not get that from the healthcare system will keep looking for it elsewhere.
According to the complaint, her conversations with ChatGPT gradually crossed a critical line.
The system did not stop at explaining the drug and its side effects. It helped Wan plan a gradual reduction in her dose, from 400 mg a day to complete discontinuation within about 25 days, without appropriate medical supervision.
Worse, the complaint alleges that the system encouraged her not to tell her doctors about the plan, and even reinforced her decision to skip a blood test that might have revealed her falling drug levels.
When Wan reported confusion, gaps in memory and symptoms reminiscent of her past seizures, the system allegedly framed some of them as the brain’s natural adjustment to the lower dose.
According to the investigation, she expressed fear about the process at least 158 times.
Instead of those repeated concerns prompting a pause, a reassessment and a clear referral to a doctor, the conversations at times continued to assure her that things were on track.
After her death, the medical examiner determined the cause to be sudden unexpected death in epilepsy (SUDEP). Her blood showed a very low level of lamotrigine.
That alone does not settle the question of legal responsibility. But the case shows the danger of turning a conversational system into a source of personalized treatment instructions without professional oversight.
02 /The problem is not just a wrong answer
When we talk about the risks of AI in medicine, the discussion often centers on “hallucinations”: false information that the system presents with confidence.
This case points to something more complex.
An AI system can provide information that sounds accurate, reasonable and empathetic, and still lead the user somewhere dangerous.
The danger does not necessarily come from a single wrong sentence. It can build over dozens or hundreds of exchanges, as the system learns the user’s preferences, tailors its answers to their concerns and gradually reinforces their sense that they are doing the right thing.
One well-known phenomenon in the field is sycophancy: the tendency of conversational systems to agree with users, validate their views and sometimes favor affirming responses over raising reservations or pushing back.
In everyday contexts, this can seem pleasant and even helpful. In medical contexts, it can be dangerous.
A patient worried about side effects needs empathy and understanding. But they also need balanced information, an honest account of uncertainty and, at times, a clear statement that what they are considering could put them at risk.
Empathy without safety judgment is not necessarily help. Sometimes it becomes reinforcement of dangerous behavior.
03 /When trust in the machine replaces trust in the doctor
Wan’s story has another dimension that troubles me perhaps even more than the medical advice itself.
According to the reports, over the years she developed an intense relationship with ChatGPT, spanning thousands of messages and growing reliance on the system for personal and medical advice.
A system that is available around the clock, never tires, is never rushing to the next appointment and can respond with patience and detail offers real benefits. For people living with chronic illness, loneliness, anxiety or limited access to care, those benefits can matter a great deal.
But the same qualities can also create a sense of personal connection, deep understanding and authority.
Users may come to see the system as knowing them better than their clinicians do, and sometimes better than their own families.
The problem becomes acute when the system does more than provide information and begins to erode trust in the people treating the patient.
Doctors can be wrong, of course. Patients are entitled, and in fact encouraged, to ask questions, request explanations, seek second opinions and take an active part in decisions about their health.
But empowering patients is not the same as encouraging them to hide medical information or stop treatment on their own.
There is a fundamental difference between a system that helps a patient formulate questions for their doctor and one that guides them in acting against medical advice without the doctor’s knowledge.
In the second case, AI is no longer just a tool for making knowledge accessible. It has effectively become a participant in clinical decision-making, without the obligations, oversight and accountability that come with practicing medicine.
04 /What would the right response have looked like?
Criticism of the system raises a fair question: what exactly should it have done?
A blanket refusal is not a good answer. Someone who has decided to stop a medication will not change their mind just because a system declines to engage. They may stop abruptly instead, or look for guidance in online forums with no safety mechanisms at all.
The right response lies between refusal and compliance, and it has three components.
The first is a clear warning that does not get buried among caveats: stopping an anti-seizure medication without medical supervision can lead to seizures and even sudden death, and that risk is not theoretical.
The second is taking the complaint seriously. Impaired language and thinking is a real side effect, and it warrants a conversation with the doctor about adjusting the dose or switching to another drug.
The third is redirecting that energy toward something useful: helping the patient document her symptoms, articulate her request to her neurologist and prepare for an appointment in which she asks for a change in treatment.
In other words, the system should be on the side of the patient’s goal, which was to be free of the side effects, and not on the side of the dangerous path she chose to get there. And when a user expresses fear for the 158th time, that is no longer one more question to soothe. It is a signal that the conversation needs to change direction.
05 /Who is responsible: the user or the developer?
One could argue that final responsibility for medical decisions rests with the patient. One could also point out that AI systems generally warn that they are no substitute for a doctor.
Both points are true to a degree, but neither is a complete answer.
When a company builds a system that can hold long, detailed medical conversations, generate personalized recommendations and is presented to the public as a tool for understanding health conditions, it must also account for how people will actually use it.
A generic disclaimer at the bottom of the screen is not necessarily enough when the content of the conversation itself projects confidence and encourages action.
This is especially true when the user is in distress, living with a chronic illness or searching for a solution to a problem the healthcare system has not yet solved to their satisfaction.
In response to the reports, OpenAI called the case heartbreaking and noted that Wan had used an older model, GPT-4o, which is no longer available in ChatGPT. The company said its newer models are better at recognizing situations that require urgent attention.
That is meaningful progress, to the extent it holds up in real-world use. But the broader question is not about any one model.
How do we make sure safety mechanisms keep working over a long conversation, when the user repeatedly seeks approval for a dangerous decision?
06 /What needs to change?
In my view, the lesson of this case is not that AI should be kept out of health care. The technology has enormous potential to improve access to information and strengthen continuity of care. But realizing that potential requires responsibility to be translated into concrete mechanisms.
First, clearly defined red lines instead of vague principles. The distinction between “general information” and “personalized instructions” is sound, but very hard to apply in natural conversation. A short, explicit list of actions that the system will not guide without a clinician’s involvement is far easier to enforce. Such a list would include building a personal plan to stop medications that are dangerous to discontinue, such as anti-seizure drugs, anticoagulants, psychiatric medications, steroids and insulin, as well as any encouragement to hide information from the treating clinician. Information about the medications themselves would remain accessible. What would be off-limits is hands-on guidance without supervision.
Second, safety memory across the conversation. A system that evaluates each message in isolation will not detect the danger in a case like Wan’s. It needs to accumulate signals over time: recurring fear, symptoms such as confusion and memory loss, repeated attempts to obtain approval. Beyond a certain threshold, the conversation should change direction, even if the latest message sounds calm. This matters all the more when the relationship extends over weeks and many separate conversations.
Third, safety testing of conversations, not answers. Passing medical licensing exams says nothing about how a system behaves with a frightened, persistent user over hundreds of messages. We need evaluations that simulate long conversations with vulnerable users. Those evaluations should also be run by independent parties, and the results should be published. And they should be run in languages other than English, including Hebrew, because we have no reason to assume that safety mechanisms tested in English perform equally well elsewhere.
Fourth, incident reporting, as in medicine. Pharmaceutical companies are required to report serious adverse events, and over the years that obligation has built knowledge that saves lives. Companies whose systems are in practice used for medical advice should operate in a similar way: report serious incidents, investigate them and share the lessons.
Fifth, the medical side needs to change too. Clinicians should routinely ask patients whether they consult AI systems, just as they ask about supplements. And a patient who reports side effects should leave the appointment with a plan, not just reassurance.
Ultimately, the goal should be systems that strengthen the relationship between patients and professionals, not systems that take its place.
07 /The real test of responsible AI
In discussions of AI in medicine, we tend to ask how smart, accurate and fast these systems are.
Those are important questions, but they are not enough.
We also need to ask how these systems behave when the user is frightened, confused, frustrated or asking them to endorse a dangerous decision.
Can they convey uncertainty? Can they disagree with the user? Do they recognize the limits of their own competence? And do they know when it is time to hand the center of gravity of the conversation back to another human being?
Bowen Wan’s story is first and foremost a human tragedy. A young woman trying to cope with illness, side effects and an uncertain future found in a technological system a readily available source of support and guidance.
It is also a painful reminder that trust in technology cannot rest only on the feeling that it understands us.
The more persuasive, accessible and human these systems become in the way they communicate with us, the greater the responsibility to ensure they also know how to set limits.
In the end, the most important test of responsible AI in medicine is not only its ability to give a good answer. It is also its ability to recognize when it must not give the answer the user wants to hear.
This article is based on news reports, the legal complaint and publicly available professional documents. The allegations against OpenAI have not been decided in court.
08 /Sources
- The New York Times: ChatGPT’s Advice to Cut Her Seizure Drug Led to Devastating Consequences (October 27, 2026)
- Complaint: Wan et al. v. OpenAI, filed October 27, 2026
- TechCrunch: Parents sue OpenAI alleging GPT-4o coached their daughter off seizure medication
- OpenAI Model Spec: principles for model behavior on medical topics and human relationships
- FDA: Lamotrigine prescribing information and safety guidance
Comments
No comments yet. Be the first to share.