Making Medical AI Smarter and Safer
A conversation with Qingyu Chen, PhD, about what medical artificial intelligence learns, what it memorises, and what it takes to use it responsibly.
Qingyu Chen, PhD, and his team set out to study how artificial intelligence language models are adapted for medicine and found that what these models memorise can be both useful and risky. A model may retain valuable medical knowledge, but in a controlled study using real hospital records, the same fine-tuning – the added training that adapts a model to a specific task – that improved diagnostic performance also made it more likely to reproduce material it had seen during training, including sensitive patient information.
The study, published recently in Nature Communications, reflects a question at the center of Chen’s research: How can medical AI become not only more capable but also more reliable and safer? The study was led by its first author, Anran Li, PhD, who conducted the research as a postdoctoral researcher in Yale’s Department of Biomedical Informatics and Data Science.
Chen is an assistant professor of biomedical informatics and data science at Yale School of Medicine, with a secondary appointment in ophthalmology. He leads research on the accuracy and reasoning of medical language models and on multimodal AI-assisted disease diagnosis, which draws on both text and medical images.
The following conversation with Chen discusses what medical AI learns, what it memorises, and what it takes to use it responsibly.
What is your lab’s research focus?
Our lab does two things that are usually treated as separate: We build medical AI, and we study where it fails.
On the building side, we work with two main kinds of information medicine runs on. We develop models that read clinical records and the medical literature, models that analyse medical images to help diagnose disease and predict its course, and systems that combine both, so an AI can weigh a patient’s written history alongside their scans, the way a physician would.
But a model that performs well on a test is not the same as a model you can trust with a patient. So, we also study how these systems fail. They can state falsehoods with complete confidence. They can reach a right answer through faulty reasoning. And, as our recent work shows, they can memorise sensitive information from the data they were trained on.
Our goal is to develop medical AI that is genuinely useful, understand where it breaks down, and produce the evidence needed to know when it can – and cannot – be trusted.
Why work across both text and images?
Because medicine is inherently multimodal. A patient cannot be understood through a single paragraph or a single image. Clinical decisions often require bringing together a patient’s history, laboratory results, medical notes and imaging findings.
Our work spans both sides of that. On the language side, we develop methods to help computers understand clinical records and biomedical literature. On the imaging side, much of our research focuses on medical images and specialties that depend heavily on them—ophthalmology in particular, where we work on diagnosing eye disease and predicting how it will progress. This is also why I hold a secondary appointment in ophthalmology.
What excites us most is bringing these areas together: systems that can integrate text, images and other clinical information to form a more complete picture of a patient and better support medical decision-making.
Progress here requires more than developing new models. One of the biggest barriers is the limited availability of medical data that is large enough, reliable enough and free for researchers to share.
To help address this, we recently developed MedPMC, a system that has assembled 11 million medical images paired with their accompanying text, drawn from research literature that is openly licensed for reuse—and that is designed to keep growing as new research is published. We’ve made the data, the tools used to assemble it, the tests for measuring performance and the resulting models publicly available so that other institutions can develop, evaluate, reproduce and adapt these systems rather than starting from scratch.
Your team recently published a study in Nature Communications on how AI models ‘memorise’ medical data. What does memorisation mean here?
Memorisation means that a model can recall or reproduce content it encountered during training. If a model has been trained on clinical guidelines, it may reproduce part of a guideline when you give it the opening of that passage. If it has had additional training on a set of medical exam questions, it may produce an answer choice that appeared in that training data – even after we removed that choice from the question.
That is different from simply saying that a model performs well. When a model answers a question correctly, there are several possible explanations. It may have acquired genuine medical knowledge. It may have learned a pattern of reasoning it can apply to problems it hasn’t seen before. Or it may simply recognise the question and reproduce something it previously saw during training. If all we check is whether the final answer is correct, we cannot tell these apart.
So, our goal was to look beyond accuracy and ask a different set of questions: How often does memorisation occur? What types of content get memorised? How much can a model reproduce? Does what it memorised earlier survive further training? And what does all of this mean for using these systems in medicine?
What did you find?
We looked at the main stages a general-purpose model goes through on its way to becoming a medical one.
First, we examined models that had already undergone continued pretraining, in which a general-purpose model is trained further on large collections of medical text, including biomedical literature, clinical guidelines and clinical notes. Second, we evaluated models that had been fine-tuned on the standard question-and-answer datasets that the field uses to test medical models. Third, we conducted a privacy-protected, HIPAA-compliant study using more than 13 000 medical records to fine-tune models for disease diagnosis.
This was done in an isolated and secure computing environment. These records had already been collected in the course of care; the study did not recruit patients or change anyone’s treatment.
Across those settings, we examined both general-purpose models and models trained on medical data, 10 different datasets containing hundreds of thousands of records and thousands of model responses that we manually reviewed.
The patterns differed depending on the training stage. Continued pretraining was more likely to produce long, word-for-word matches to source documents. Fine-tuning produced less long-form copying in some settings, but more memorisation tied to the specific task the model was trained on. For example, after fine-tuning on medical question-and-answer datasets, models reproduced roughly 14% to 21% of the answer choices that had been removed from the question the model was shown.
We also found that memorisation was persistent. Depending on the setting, as much as 87% of what a model memorised during continued pretraining was still present after it had been fine-tuned on a new medical task. Fine-tuning does not necessarily erase what a model previously memorised. It may preserve that content while adding new memorisation specific to the task it was just trained on.
What did the clinical case study show?
The clinical case study showed both the potential benefit and the risk of adapting these models to real-world medical data. Fine-tuning improved diagnostic performance; for one model, the correct diagnosis came up as its first choice 54.8% of the time, up from 48.6%. In some specialties, the gains were larger than that – more than 10 percentage points in areas such as cardiology and nephrology, which deal with heart and kidney conditions.
At the same time, the study showed a real privacy risk. In a controlled test conducted in a secure research setting, we found that the model could sometimes reproduce sensitive information from the records used to train it. This was not something that would happen during patient care, but it shows that privacy risks should be evaluated before models trained on clinical data are shared or deployed.
Is memorisation always harmful?
No. One of the central findings of our study is that memorisation in medicine is not a single behaviour.
We identified three broad types. The first is beneficial memorisation. A model may accurately retain biomedical concepts, clinical guidelines, the medical literature it has read or specific medical knowledge tied to its task. That kind of memory may support factual accuracy and help the model perform medical tasks more effectively.
The second is uninformative memorisation. Models sometimes reproduce document disclaimers, section headings, formatting instructions or other boilerplate language. This adds little medical value and may indicate that the model is learning surface-level patterns rather than deeper medical understanding.
The third is harmful memorisation. This includes reproducing quirks specific to a particular dataset, word-for-word passages from patient notes, protected health information or other sensitive patient content. This form of memorisation may create privacy risks and may also indicate that the model is leaning too heavily on its training data rather than generalising to new cases.
The important question, then, is not simply whether a model memorises; it is what the model memorises, why it memorises it, and whether that memory supports or undermines the medical use it is intended for.
Did anything about the way memorisation develops surprise you?
One thing that stood out: Memorisation did not only show up late, after a model had been trained too long. It began early.
As we trained the models, we tracked their progress and compared three things: how much they were memorising, how well they were learning by the usual measure, and how accurate their diagnoses were. Memorisation began increasing relatively early, even while the standard measures still showed the model improving, and before its diagnoses had reached their peak accuracy.
That means traditional indicators researchers watch during training – such as whether the model keeps improving on held-out data, or the point at which they would normally stop training – are useful but may not be sufficient on their own. A model can appear to be learning effectively according to standard performance measures while simultaneously increasing its retention of training-specific content.
We also found two other patterns. Larger models and longer inputs were generally associated with more memorisation. By contrast, changing common generation settings such as temperature – which controls how varied the model’s answers are – had relatively limited effects. This suggests that memorisation is fundamentally connected to how a model is trained and what data it is exposed to, rather than being something that can simply be solved by adjusting how the model generates its answers after training.
What do you most want readers to take away from this work?
Adapting an AI model to medicine does not simply make it “more medical.” It changes what the model knows, what it remembers and what it may reproduce.
Some of that memory is valuable. We want models to retain accurate medical knowledge and clinical guidance. But we do not want them to rely on meaningless repetition, echo back the answers to test questions or expose sensitive information from patient records.
Trustworthy medical AI therefore requires more than measuring whether a model gets the answer right. We need to understand how it got there, what it retained from training, and whether it will stay safe and reliable when used in a new setting.
Source: Yale School of Medicine
