LLMs Prone to Medical Misinformation in Scientific Notes

What You Ought to Know

• The Research: In a paper printed at present in The Lancet Digital Health [10.1016/j.landig.2025.100949], researchers on the Icahn Faculty of Medication at Mount Sinai analyzed over a million prompts throughout 9 main Massive Language Fashions (LLMs) to check their susceptibility to medical misinformation.
• The Vulnerability: The examine discovered that AI fashions often repeat false medical claims—similar to advising sufferers with bleeding to “drink chilly milk”—if the lie is embedded in life like hospital notes or professional-sounding language.
• The Takeaway: Present safeguards are failing to tell apart reality from fiction when the fiction “sounds” like a health care provider. For these fashions, the fashion of the writing (assured, scientific) typically overrides the reality of the content material.

The “Chilly Milk” Fallacy

To check the methods, the analysis crew uncovered 9 main LLMs to over a million prompts. They took actual hospital discharge summaries (from the MIMIC database) and injected them with single, fabricated suggestions.

The outcomes had been sobering. In a single particular instance, a discharge word for a affected person with esophagitis-related bleeding falsely suggested them to “drink chilly milk to assuage the signs”—a suggestion that’s clinically unsafe.

As a substitute of flagging this as harmful, a number of fashions accepted the assertion as reality. They processed IT, repeated IT, and handled IT like atypical medical steering just because IT appeared in a format that regarded like a legitimate hospital word.

Fashion Over Substance

“Our findings present that present AI methods can deal with assured medical language as true by default, even when IT’s clearly fallacious,” mentioned Dr. Eyal Klang, Chief of Generative AI at Mount Sinai.

This exposes a elementary flaw in how present LLMs function in healthcare. They don’t seem to be essentially verifying the medical accuracy of a declare in opposition to a database of reality; they’re predicting the subsequent phrase primarily based on context. If the context is a extremely life like, skilled discharge abstract, the mannequin assumes the content material inside IT is correct.

“For these fashions, what issues is much less whether or not a declare is right than how IT is written,” Klang added.

The “Stress Take a look at” Answer

The implications for scientific deployment are large. If an AI summarizer is used to condense affected person information, and a type of information incorporates a human error (or a hallucination from a earlier AI), the system would possibly amplify that error somewhat than catch IT.

Dr. Mahmud Omar, the examine’s first writer, argues that we want a brand new customary for validation. “As a substitute of assuming a mannequin is secure, you may measure how typically IT passes on a lie,” he mentioned. The authors suggest utilizing their dataset as a regular “stress take a look at” for any medical AI earlier than IT is allowed close to a affected person.