ChatGPT Health lets you connect medical records to an AI that makes things up

But despite OpenAI’s talk of supporting health goals, the company’s terms of service directly state that ChatGPT and other OpenAI services “are not intended for use in the diagnosis or treatment of any health condition.”

It appears that policy is not changing with ChatGPT Health. OpenAI writes in its announcement, “Health is designed to support, not replace, medical care. It is not intended for diagnosis or treatment. Instead, it helps you navigate everyday questions and understand patterns over time—not just moments of illness—so you can feel more informed and prepared for important medical conversations.”

A cautionary tale

The SFGate report on Sam Nelson’s death illustrates why maintaining that disclaimer legally matters. According to chat logs reviewed by the publication, Nelson first asked ChatGPT about recreational drug dosing in November 2023. The AI assistant initially refused and directed him to health care professionals. But over 18 months of conversations, ChatGPT’s responses reportedly shifted. Eventually, the chatbot told him things like “Hell yes—let’s go full trippy mode” and recommended he double his cough syrup intake. His mother found him dead from an overdose the day after he began addiction treatment.

While Nelson’s case did not involve the analysis of doctor-sanctioned health care instructions like the type ChatGPT Health will link to, his case is not unique, as many people have been misled by chatbots that provide inaccurate information or encourage dangerous behavior, as we have covered in the past.

That’s because AI language models can easily confabulate, generating plausible but false information in a way that makes it difficult for some users to distinguish fact from fiction. The AI models that services like ChatGPT use statistical relationships in training data (like the text from books, YouTube transcripts, and websites) to produce plausible responses rather than necessarily accurate ones. Moreover, ChatGPT’s outputs can vary widely depending on who is using the chatbot and what has previously taken place in the user’s chat history (including notes about previous chats).