Scammers now use generative AI technologies to create deepfakes of physicians to sell products, extract sensitive data, and more. MD-MBA candidate Tejas S Athni, PhD analyses the types of these scams in “Classifying Physician Deepfakes: The Many Faces of AI-Powered Medical Deception“, asking: “How do they deceive and mislead audiences, and to what ends? And what are the potential clinical impacts?”
Athni identifies four ways in which deepfakes are used so far to exploit online users – commercial endorsement, engagement farming, medical misinformation, and phishing – then notes the potential risk for deepfakes to facilitate fraudulent professional communications (like interviews, presentations, and lectures) and to directly sabotage their subjects’ reputations.
Where Nursing Can be Automated, And Where It Can’t
Can – and should – bedside nursing tasks be performed by robots? Health writer and ICU nurse Jenna Congdon reviews the data in “The Robot at the Bedside: How Robotics Could Change the Future of Nursing Care”. Though one study found that a majority of bedside nursing tasks could technically be performed by robotic systems, another argued that research in laboratory settings can’t capture the complexity and flexibility of real life clinical settings, pointing to a need for human involvement. But though autonomous robots aren’t yet ready (or wanted) for most hands-on tasks, some robotics may be valuable for home care support, auxiliary nursing support, and patient independence; aiming “to reduce repetitive physical tasks, decrease time spent on nonclinical activities, and allow nurses to focus on assessment, education, and relationship building”, writes Congdon.
Millions of South Africans still struggle to access consistent, quality primary healthcare. Language barriers, distance from facilities and a shortage of trained health workers are just a few of the factors impacting the standard of care. Tshikululu Social Investments and Dr Sophia AI are addressing this gap through a new partnership to advance health equity across South Africa.
Dr Sophia AI is an Australian-developed, clinically governed decision support platform that helps clinics deliver consistent, high-quality care at the point of consultation. Through this partnership, small and medium healthcare practices in South Africa will gain access to AI-assisted clinical support that strengthens the care they provide. As these practices grow, more communities gain reliable access to primary healthcare.
The technology is intended to support — not replace — healthcare professionals. It guides structured assessments, helps identify potential risks and drug interactions, and supports clinical documentation, while clinicians remain responsible for reviewing and approving recommendations. The platform is built around explainability, traceability and clinician oversight, with an audit trail designed to support clinical governance and accountability.
A pilot is already under way with nurses working in community-based primary healthcare settings in the Western Cape. This is this first phase in testing how Dr Sophia AI can strengthen healthcare systems, and small businesses, in under-resourced communities.
The pilot will focus on localising Dr Sophia AI for South African community healthcare settings. This includes embedding South Africa’s Essential Medicines Lists and relevant clinical guidelines, refining the platform for local care pathways and testing its language capabilities in the context of how patients and healthcare professionals communicate in practice. The pilot will also gauge how the platform can grow revenue streams in small businesses by streamlining operations. Insights from the pilot will be used to continually refine the platform for the local context and inform its potential wider deployment.
Alongside the pilot, Tshikululu and Dr Sophia AI are designing a funding model that could complement traditional grant funding by linking commercial growth to social impact. The concept would see a percentage of future revenue generated through Dr Sophia AI’s commercial deployment directed towards a grant fund. This fund will be channelled towards initiatives that expand access to quality healthcare and support SMME growth, with Tshikululu bringing its experience in social investment and fund management to the model.
Dr Sophia AI complies with American FDA Clinical Decision Support Software requirements and is registered in Australia as Software as a Medical Device, Class I, Clinical Decision Support Software; providing an internationally recognised regulatory foundation for its use as clinical decision-support technology. Its South African deployment will equally align with applicable POPIA, SAHPRA and HPCSA requirements.
“Where you live, the language you speak or the availability of healthcare professionals in your community should not determine the quality of healthcare you can access,” says Leanne Emery-Hunter, CEO of Tshikululu Social Investments. “This pilot is about taking an established AI-assisted clinical platform and embedding it in the South African context — our clinical guidelines, our languages and the realities of frontline healthcare delivery.”
The partners are looking to engage funders, healthcare organisations, corporates and other potential partners interested in helping to shape and participate in the model. The ambition is to explore whether commercial adoption of the technology can create an ongoing source of funding that, in turn, supports grant-funded healthcare initiatives in underserved communities.
“Healthcare is deeply local, but the challenge of delivering consistent, high-quality care is global,” says Julia McCann, GM, Dr Sophia AI. “Our ambition is to build technology that can support healthcare professionals across very different health systems, languages and communities. South Africa is an important opportunity to demonstrate how a global platform can be meaningfully localised — combining international clinical capability with the guidelines, languages and realities of care on the ground.”
The partnership reflects a broader approach to social investment. It brings together technology, commercial and social-sector partners to develop new approaches to persistent social challenges and explore funding mechanisms that can complement traditional grant-making.
“This is the kind of social innovation we want to enable,” says Emery-Hunter. “It brings together technology, frontline expertise and a new approach to funding around a very practical question: how do we get better healthcare closer to people who have historically had the least access to it? We believe that will require collaboration, continuous learning and new ways of thinking about how social impact is funded.”
Tshikululu and Dr Sophia AI welcome conversations with organisations interested in partnering on the next phase of the initiative and helping to shape how the model could contribute to more equitable access to quality healthcare.
A new study at The University of Texas at Arlington could help substantially reduce the amount of radiation heart patients receive during imaging procedures without sacrificing diagnostic accuracy.
The study, led by Mingwu Jin (professor of physics), will focus on computed tomography angiography (CTA), an advanced imaging test that noninvasively assesses coronary artery disease. CTA combines a CT scan with iodine contrast dye injected into a patient’s vein to create 3D images of blood vessels and surrounding tissues. Doctors can use CTA to detect aneurysms, narrowed or damaged blood vessels, blood clots, tumours and other conditions.
The project is funded by a three-year, $579 745 grant from the National Heart, Lung, and Blood Institute. The project is titled “Deep leaRning-bAsed reConstruction (DRAC) for Multi-Phase Cardiac CT Angiography.”
Multi-phase coronary CTA (MP-CCTA) captures images at multiple points during the heartbeat, providing doctors with more information and helping reduce blurring caused by the heart’s movement. The drawback is that collecting images across multiple phases exposes patients to more radiation than a single-phase scan. Dr Jin’s research aims to substantially reduce that radiation exposure while preserving the quality and diagnostic value of the images.
“Lowering the radiation dose in medical imaging has been vigorously pursued due to imaging prevalence and potential cancer risk of ionising radiation,” Jin said. “Although notable progress has been made by developing advanced reconstruction methods, the dose reduction potential is far from its limit.”
CT scans use reconstruction methods to turn raw data collected during a scan into images doctors can interpret. One advanced approach is model-based iterative reconstruction (MBIR). Another is 4D reconstruction, which adds the dimension of time to standard 3D imaging. In cardiac imaging, this technique can capture clear views of fast-moving heart valves and coronary arteries while reducing motion artefacts.
Jin said that recent breakthroughs in deep learning provide an unprecedented opportunity to substantially lower radiation doses by combining MBIR with information learned from a large database of high-quality MP-CCTA images. The resulting 4D DRAC method would reconstruct a patient’s images using not only data from that individual scan, but also information learned from a much larger collection of images.
Jin said the project could substantially lower the radiation dose needed for MP-CCTA, potentially improving its use in detecting the narrowing of coronary arteries and monitoring the progression of high-risk plaque. He said the 4D DRAC methods could also be applied to other dynamic CT scans.
To test the hypothesis, Jin and his team will:
Build a synthetic patient library using computational human models based on retrospectively collected MP-CCTA data from real patients.
Develop 3D and 4D deep-learning methods that combine CT imaging physics with features learned from the high-quality patient data.
Determine the lowest radiation dose that can be used without compromising diagnostic accuracy. The team will evaluate diagnostic accuracy using a computer model designed to mimic how a radiologist assesses an image, then validate the results through a small-scale study involving human observers.
The Mayo Clinic in Rochester, Minnesota, will provide MP-CCTA patient data for the study.
“This award reflects the high quality of Dr Jin’s research and the strength of his proposed ideas,” said Alex Weiss (professor of physics and former department chair). “I know a great deal of work goes into preparing a successful proposal of this kind. This is a significant achievement for Dr Jin and a major benefit to our department.”
Graduate students Noor Rasel and Darnisha Langlais will work with Jin on the project. The grant will also allow Jin to hire undergraduate students in the summers, providing them with research opportunities to apply AI techniques to medical imaging.
A conversation with Qingyu Chen, PhD, about what medical artificial intelligence learns, what it memorises, and what it takes to use it responsibly.
Qingyu Chen, PhD, and his team set out to study how artificial intelligence language models are adapted for medicine and found that what these models memorise can be both useful and risky. A model may retain valuable medical knowledge, but in a controlled study using real hospital records, the same fine-tuning – the added training that adapts a model to a specific task – that improved diagnostic performance also made it more likely to reproduce material it had seen during training, including sensitive patient information.
The study, published recently in Nature Communications, reflects a question at the center of Chen’s research: How can medical AI become not only more capable but also more reliable and safer? The study was led by its first author, Anran Li, PhD, who conducted the research as a postdoctoral researcher in Yale’s Department of Biomedical Informatics and Data Science.
Chen is an assistant professor of biomedical informatics and data science at Yale School of Medicine, with a secondary appointment in ophthalmology. He leads research on the accuracy and reasoning of medical language models and on multimodal AI-assisted disease diagnosis, which draws on both text and medical images.
The following conversation with Chen discusses what medical AI learns, what it memorises, and what it takes to use it responsibly.
What is your lab’s research focus?
Our lab does two things that are usually treated as separate: We build medical AI, and we study where it fails.
On the building side, we work with two main kinds of information medicine runs on. We develop models that read clinical records and the medical literature, models that analyse medical images to help diagnose disease and predict its course, and systems that combine both, so an AI can weigh a patient’s written history alongside their scans, the way a physician would.
But a model that performs well on a test is not the same as a model you can trust with a patient. So, we also study how these systems fail. They can state falsehoods with complete confidence. They can reach a right answer through faulty reasoning. And, as our recent work shows, they can memorise sensitive information from the data they were trained on.
Our goal is to develop medical AI that is genuinely useful, understand where it breaks down, and produce the evidence needed to know when it can – and cannot – be trusted.
Why work across both text and images?
Because medicine is inherently multimodal. A patient cannot be understood through a single paragraph or a single image. Clinical decisions often require bringing together a patient’s history, laboratory results, medical notes and imaging findings.
Our work spans both sides of that. On the language side, we develop methods to help computers understand clinical records and biomedical literature. On the imaging side, much of our research focuses on medical images and specialties that depend heavily on them—ophthalmology in particular, where we work on diagnosing eye disease and predicting how it will progress. This is also why I hold a secondary appointment in ophthalmology.
Progress here requires more than developing new models. One of the biggest barriers is the limited availability of medical data that is large enough, reliable enough and free for researchers to share.
To help address this, we recently developed MedPMC, a system that has assembled 11 million medical images paired with their accompanying text, drawn from research literature that is openly licensed for reuse—and that is designed to keep growing as new research is published. We’ve made the data, the tools used to assemble it, the tests for measuring performance and the resulting models publicly available so that other institutions can develop, evaluate, reproduce and adapt these systems rather than starting from scratch.
Your team recently published a study in Nature Communications on how AI models ‘memorise’ medical data. What does memorisation mean here?
Memorisation means that a model can recall or reproduce content it encountered during training. If a model has been trained on clinical guidelines, it may reproduce part of a guideline when you give it the opening of that passage. If it has had additional training on a set of medical exam questions, it may produce an answer choice that appeared in that training data – even after we removed that choice from the question.
That is different from simply saying that a model performs well. When a model answers a question correctly, there are several possible explanations. It may have acquired genuine medical knowledge. It may have learned a pattern of reasoning it can apply to problems it hasn’t seen before. Or it may simply recognise the question and reproduce something it previously saw during training. If all we check is whether the final answer is correct, we cannot tell these apart.
So, our goal was to look beyond accuracy and ask a different set of questions: How often does memorisation occur? What types of content get memorised? How much can a model reproduce? Does what it memorised earlier survive further training? And what does all of this mean for using these systems in medicine?
What did you find?
We looked at the main stages a general-purpose model goes through on its way to becoming a medical one.
First, we examined models that had already undergone continued pretraining, in which a general-purpose model is trained further on large collections of medical text, including biomedical literature, clinical guidelines and clinical notes. Second, we evaluated models that had been fine-tuned on the standard question-and-answer datasets that the field uses to test medical models. Third, we conducted a privacy-protected, HIPAA-compliant study using more than 13 000 medical records to fine-tune models for disease diagnosis.
This was done in an isolated and secure computing environment. These records had already been collected in the course of care; the study did not recruit patients or change anyone’s treatment.
Across those settings, we examined both general-purpose models and models trained on medical data, 10 different datasets containing hundreds of thousands of records and thousands of model responses that we manually reviewed.
The patterns differed depending on the training stage. Continued pretraining was more likely to produce long, word-for-word matches to source documents. Fine-tuning produced less long-form copying in some settings, but more memorisation tied to the specific task the model was trained on. For example, after fine-tuning on medical question-and-answer datasets, models reproduced roughly 14% to 21% of the answer choices that had been removed from the question the model was shown.
We also found that memorisation was persistent. Depending on the setting, as much as 87% of what a model memorised during continued pretraining was still present after it had been fine-tuned on a new medical task. Fine-tuning does not necessarily erase what a model previously memorised. It may preserve that content while adding new memorisation specific to the task it was just trained on.
What did the clinical case study show?
The clinical case study showed both the potential benefit and the risk of adapting these models to real-world medical data. Fine-tuning improved diagnostic performance; for one model, the correct diagnosis came up as its first choice 54.8% of the time, up from 48.6%. In some specialties, the gains were larger than that – more than 10 percentage points in areas such as cardiology and nephrology, which deal with heart and kidney conditions.
At the same time, the study showed a real privacy risk. In a controlled test conducted in a secure research setting, we found that the model could sometimes reproduce sensitive information from the records used to train it. This was not something that would happen during patient care, but it shows that privacy risks should be evaluated before models trained on clinical data are shared or deployed.
Is memorisation always harmful?
No. One of the central findings of our study is that memorisation in medicine is not a single behaviour.
We identified three broad types. The first is beneficial memorisation. A model may accurately retain biomedical concepts, clinical guidelines, the medical literature it has read or specific medical knowledge tied to its task. That kind of memory may support factual accuracy and help the model perform medical tasks more effectively.
The second is uninformative memorisation. Models sometimes reproduce document disclaimers, section headings, formatting instructions or other boilerplate language. This adds little medical value and may indicate that the model is learning surface-level patterns rather than deeper medical understanding.
The third is harmful memorisation. This includes reproducing quirks specific to a particular dataset, word-for-word passages from patient notes, protected health information or other sensitive patient content. This form of memorisation may create privacy risks and may also indicate that the model is leaning too heavily on its training data rather than generalising to new cases.
The important question, then, is not simply whether a model memorises; it is what the model memorises, why it memorises it, and whether that memory supports or undermines the medical use it is intended for.
Did anything about the way memorisation develops surprise you?
One thing that stood out: Memorisation did not only show up late, after a model had been trained too long. It began early.
As we trained the models, we tracked their progress and compared three things: how much they were memorising, how well they were learning by the usual measure, and how accurate their diagnoses were. Memorisation began increasing relatively early, even while the standard measures still showed the model improving, and before its diagnoses had reached their peak accuracy.
That means traditional indicators researchers watch during training – such as whether the model keeps improving on held-out data, or the point at which they would normally stop training – are useful but may not be sufficient on their own. A model can appear to be learning effectively according to standard performance measures while simultaneously increasing its retention of training-specific content.
We also found two other patterns. Larger models and longer inputs were generally associated with more memorisation. By contrast, changing common generation settings such as temperature – which controls how varied the model’s answers are – had relatively limited effects. This suggests that memorisation is fundamentally connected to how a model is trained and what data it is exposed to, rather than being something that can simply be solved by adjusting how the model generates its answers after training.
What do you most want readers to take away from this work?
Adapting an AI model to medicine does not simply make it “more medical.” It changes what the model knows, what it remembers and what it may reproduce.
Some of that memory is valuable. We want models to retain accurate medical knowledge and clinical guidance. But we do not want them to rely on meaningless repetition, echo back the answers to test questions or expose sensitive information from patient records.
Trustworthy medical AI therefore requires more than measuring whether a model gets the answer right. We need to understand how it got there, what it retained from training, and whether it will stay safe and reliable when used in a new setting.
A tool developed using data on more than 3.5 million adults in Sweden can identify individuals at high risk of hip fracture using registry data, with no in-person assessment
A machine-learning tool built from Swedish national health registry data can predict hip fracture risk with high accuracy and no in-person assessment, and identifies far more at-risk individuals than current clinical screening practices, according to a study published August 27thin the open access journal PLOS Medicine by Kristian Axelsson and Mattias Lorentzon of the University of Gothenburg, Sweden, and colleagues.
Hip fractures are associated with substantial disability, illness, and death in older adults, but existing risk prediction tools typically require in-person patient assessment, including measurements like body mass index and lifestyle information, making large-scale screening difficult.
Researchers analysed nationwide registry data from 3 542 647 individuals aged 50 and older in Sweden, following them for up to ten years. During the study period, 142 327 of the participants sustained a hip fracture. Using more than 100 000 variables drawn from diagnoses, medications, procedures, and demographic and socioeconomic data, the research team developed and tested a deep-learning approach called FRACTURE-ML.
When tested on data from a separate group of people, not included in the original model development, FRACTURE-ML showed good discrimination of people who went on to fracture their hip from those who didn’t with an area under the curve (AUC) of 0.89 one year ahead, and only slightly worse with AUC 0.85 when predicting five years ahead. A simplified version using just 35 variables performed nearly as well. Compared with the current screening methods used in Swedish clinical practice, FRACTURE-ML identified nearly seven times more people at risk of hip fracture within two years (sensitivity 0.84 versus 0.12), with only a modest reduction in specificity (0.79 versus 0.98).
Because the model relies solely on registry data, it lacks information on lifestyle factors such as smoking and alcohol use, which may also affect fracture risk. The authors note that validation in other countries and studies testing real-world implementation are still needed.
“The findings show that it is possible to predict hip fracture risk at the population level without direct patient interaction,” lead author Kristian Axelsson says. “This approach could help target preventive measures more efficiently and potentially reduce the number of hip fractures.”
Mattias Lorentzon adds, “FRACTURE-ML accurately identified people at high risk of hip fracture using routinely collected healthcare and population data, without requiring an in-person clinical assessment. This could make large-scale screening more efficient and help preventive care reach people before a hip fracture occurs.”
“Hip fractures have serious consequences for independence, health and survival. A tool that can identify high-risk individuals directly from existing data could support earlier intervention and potentially reduce the burden of hip fractures across the population,” the authors say.
“One important finding was that a reduced model using only 35 predictors performed nearly as well as the much larger machine-learning model. This suggests that strong predictive performance may be achievable with a comparatively practical and interpretable tool.”
“By using information already available in national registers, FRACTURE-ML could help shift hip-fracture care from reacting after an injury to preventing the injury in the first place.”
“Machine learning performed very well, but carefully developed traditional statistical models achieved similar accuracy. The key advance may therefore be less about a particular algorithm and more about making better use of comprehensive, routinely collected data.”
Of 1357 devices authorised by the US FDA, only 3 were evaluated on clinical effectiveness
Growth of FDA-cleared AI/ML-enabled medical devices from 1995 to December 2025. Of 1,357 cleared devices, only 34 were linked to registered clinical trials and only 3 were evaluated for patient-centred outcomes. (Fig 1 of the article.), Credit: Abulibdeh R, Cajas Ordóñez SA, Celi LA, Gorijavolu R, Izath N, Markussen Lunde T, 2026, PLOS Digital Health, CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
A new analysis shows that, of 1357 artificial intelligence (AI)-based medical devices authorised by the US Food and Drug Administration (FDA) for use in patient care, only three had been tested on whether they actually improve patients’ health. Rawan Abulibdeh of the University of Toronto, Canada, and colleagues present these findings in the open access journal PLOS Digital Health on August 19, 2026.
New AI devices increasingly inform clinical care, such as systems that aid surgical planning, calculate cardiovascular risks, and guide interpretation of mammograms and other imaging. In order to be authorised for use in the US, AI devices typically only need to show “substantial equivalence” to an existing authorised device, and developers are not required to demonstrate whether new AI devices help people live healthier lives – with benefits shared equitably across diverse subgroups.
To deepen understanding of this topic, Abulibdeh and colleagues investigated how all 1357 AI devices authorised by the FDA as of December 5, 2025, had been evaluated in patients prior to authorization.
They found that only 34 of the devices had been included in registered clinical trials, with results posted for 12 and peer-reviewed manuscripts published for 12. Only 3 devices had been tested on patient-centred outcomes, such as death rates, strokes, hospitalizations, and quality of life. Most studies were conducted in highly resourced healthcare systems, and most excluded key patient subgroups, such as pregnant women, adults over 75, and non-English speakers.
The researchers suggest that structural barriers such as financial incentives and logistical challenges discourage developers from testing AI devices on patient outcomes, resulting in greater emphasis on speedy development than on rigor. They discuss how this framework could allow new tools to amplify existing disparities in healthcare and how it could lead to patients in low- and middle-income countries becoming inadvertent test populations for under-studied AI devices, as many countries rely on higher-income countries’ authorisation decisions.
On the basis of their findings, the researchers conclude that existing policies for AI medical device authorization should be redesigned. They propose a novel, three-phase framework that includes demonstration of effectiveness across diverse patient subgroups and healthcare settings.
The authors add: “We expected the evidence base to be thin, but not this thin. Out of 1357 AI devices the FDA has cleared for use in patient care, only three have been tested on whether patients actually live longer or better. Clearance tells you a device resembles something already on the market. It does not tell you it helps anyone.”
Trial did not show statistically significant difference for patient outcomes but helped clinicians improve quality of notes and recommendations
A large real-world clinical trial has found that a generative AI-powered support tool used to support frontline clinicians was safe and improved the quality of clinical decision-making but did not significantly change short-term patient outcomes.
The study, published today in Nature Medicineis one of the first randomised controlled trials worldwide to test whether generative AI can improve patient-level outcomes, rather than just clinician performance or simulated cases.
What this study shows is that AI can be integrated safely into real clinical workflows, without undermining patient trust or clinician autonomy – which is a critical foundation for any future impact.
Alastair Denniston, Chair of Regulatory Science and Innovation
Clinicians were randomly assigned to use an electronic medical record system with or without an integrated AI consult tool that provided real-time diagnostic and treatment suggestions. The AI system, known as ‘AI Consult’, was a large language model–based clinical decision support tool embedded directly within the existing electronic medical record system.
During consultations, the tool worked in the background by:
Analysing information entered by the clinician into the medical record
Generating context‑specific diagnostic and treatment suggestions, aligned with Kenyan national clinical guidelines
Flagging potential concerns using a simple colour‑coded alert system (green, yellow or red)
Clinicians retained full autonomy; they were not required to follow the AI’s advice, and retained responsibility for all diagnosis, prescribing and referral decisions. The AI interface was not visible to patients, helping preserve normal patient–clinician interaction.
Senior author Professor Bilal Mateen, Honorary Professor of Machine Learning for Health at the University of Birmingham, and Chief AI Officer at PATH, said: “This is one of the first studies to rigorously ask the hardest question about AI in healthcare: whether it actually improves outcomes for patients.
“What we found is reassuring but also sobering. The technology appears safe and clearly improves aspects of clinical decision-making, but translating those gains into measurable patient benefit is much more challenging, particularly in everyday primary care.”
Serious outcomes such as hospitalisation or death are rare in primary care, meaning extremely large studies – potentially involving more than 100 000 patients – would be needed to detect modest effects.
Professor Alastair Denniston, co-author, Professor of Regulatory Science and Innovation at the University of Birmingham and lead for health data research at the NIHR Biomedical Research Centre: Birmingham, said: “A large part of primary care is to deal with common conditions, including those that are self-limiting, where many patients require low levels of healthcare intervention. In that context, even meaningful improvements in clinical reasoning may only result in small changes in patient outcomes that are very difficult to measure.
“What this study shows is that AI can be integrated safely into real clinical workflows, without undermining patient trust or clinician autonomy – which is a critical foundation for any future impact.”
Findings: safety, quality and costs
Researchers found no statistically significant difference in treatment failure within 14 days between patients seen with AI-supported care and those receiving standard care (2.2% vs 2.0%). The study found no evidence of harm, with similar rates of hospitalisation and death in both groups.
While the AI tool did not produce measurable improvements in short-term patient outcomes, it significantly improved the quality of clinical documentation and treatment planning, as assessed by an independent panel of experienced clinicians who were blinded to whether AI had been used.
Patient satisfaction was the same in both groups, suggesting that AI support did not alter patients’ experience of care.
The study also found that, although overall antibiotic prescribing rates were similar, antibiotic‑related costs were lower in the AI‑supported group, due to more cost-conscious prescribing choices.
Although the trial was conducted in Kenya, the researchers emphasise that the findings have global relevance, including for high-income health systems.
Professor Richard Riley, Professor of Biostatistics at the University of Birmingham and senior author, said: “Robust trials like this are so important to establish the real impact of using AI in practice. They help set realistic expectations of what AI can actually contribute within existing care pathways, and helps guide where future investment and research effort should be focused. Generalisability of our findings to higher-income settings, where baseline standards of care are already high, needs to be evaluated.”
In experiments in which physicians made decisions about treating hypothetical patients, the physicians tended to trust incorrect advice presented as being generated by artificial intelligence (AI), even after given the opportunity to notice that patient recovery data contradicted the recommendations. Aranzazu Vinas of the University of the Basque Country, Spain, and colleagues present these findings in the open-access journal PLOS Digital Health.
AI systems can help physicians categorise patients according to their different care needs, such as whether a patient is more or less likely to benefit from a certain treatment. Since these systems are not perfect, they are meant to be used as suggestions, with potential errors caught and corrected by physicians.
Prior research has shown that, in general, people struggle to notice and correct mistakes made by AI. To explore how this challenge may extend to physicians, Vinas and colleagues analysed data from 223 physicians who anonymously participated in online experiments.
The physicians were asked to imagine they had the option to treat patients for a rare disease using a not-yet-proven treatment still under development. They were told that an AI system had identified which patients were more or less likely to benefit from the treatment. The physicians then chose which patients to treat, and after being presented with data on patient recovery, rated their perceptions of how reliable the AI was.
Crucially, the actual effectiveness of the hypothetical treatment did not align with the AI recommendations. In one experiment, the treatment was equally moderately effective for all patients, and in a second experiment, it was equally ineffective for all.
However, in both experiments, the physicians tended to rate the AI system as reliable and apparently did not use the patient recovery data to conclude that the AI recommendations were incorrect. In the second experiment, the physicians did not realise that the treatment was entirely ineffective.
These findings highlight potential challenges for incorporating AI-based classification into healthcare. Future research could build on this study, such as by developing and testing strategies and protocols that could increase human critical thinking and detection of AI errors, in order to maximize the benefits of the human-AI collaboration while minimising potential errors.”
Lead author Aranzazu Vinas notes: ” In both experiments, physicians mostly trusted the AI’s classifications and had trouble learning from the feedback. Furthermore, in the second experiment, professionals did not notice that the treatment was completely ineffective.”
Co-author Helena Matute adds, “People tend to say that there is always a human controlling the algorithm, but our experiments show that doctors (as well as anyone else) have problems in learning from the available evidence when it contradicts the suggestions of an algorithm.”
Co-author Fernando Blanco summarizes: “It is important to investigate the errors that humans (including doctors) make when working with algorithms, in order to learn how to minimize the problems that arise from them.”
Errors and irrelevant details mean physicians may spend more time editing AI-drafted responses than it would take to write them, a large study of an online patient portal shows
Photo by National Cancer Institute on Unsplash
Artificial intelligence is spreading rapidly in health care, with the goal of streamlining critical but onerous clerical tasks such as note-taking and charting so that physicians and nurses can devote more time to patients. But even when AI can free up doctors to correspond with patients, it may fall short in helping them do it by introducing errors and extraneous details into their messages, according to a new Dartmouth study presented at the 2026 Annual Meeting of the Association for Computational Linguistics and published in the conference proceedings.
The result is that physicians may spend more time editing responses than it would’ve taken to write them, the researchers report.
“We find that AIcan sound like a doctor but not think like one,” says Sarah Preum, an assistant professor of computer science and the study’s co-corresponding author with Parker Seegmiller, a graduate researcher in Preum’s PersistLab at Dartmouth.
The researchers conducted the first large-scale study of an online patient portal that uses AI to draft responses from physicians to patients. The team developed a tool that compares AI-generated replies to a dataset of real responses they developed with health care professionals from Dartmouth Health.
They then analysed 146 000 conversations between 10 105 patients and their primary care physicians at the large rural health system, with data anonymised.
The researchers also used their tool to evaluate physician responses drafted by Claude, Gemini, and ChatGPT, as well as the three smaller commercial platforms, Llama, Aloe, and Qwen.
“We find that AI can sound like a doctor but not think like one.”
Sarah Preum, corresponding author and assistant professor of computer science
The team reports that AI-generated answers frequently misalign with what clinicians would actually write. This includes automated responses that are too long, don’t ask follow-up questions, and use irrelevant or inaccurate medical details.
“There are smaller studies that say, ‘Oh, AI is amazing,’ but we realised there is a gap in the existing literature of a large-scale evaluation of this technology,” Preum says. “We didn’t just want to measure a platform’s accuracy, but whether it actually helps with the workload, which in this case is measured by how much editing the physician is doing.”
For example, the portal’s AI suggested telling a 32-year-old woman who is taking an acid reflux drug and was concerned about constant nausea that the medication might take some adjustment in diet. A physician replaced that by asking if there’s any chance she was pregnant.
Even little changes can add up over hundreds or thousands of messages, Preum says. “You don’t want to integrate large language models into the workflow and just shift the bottleneck so that doctors are devoting their cognitive energy to playing AI janitor and fixing mistakes,” Preum says. “But if we’re not careful, that’s a likely outcome.”
The researchers show, however, that adapting AI to how individual physicians communicate can improve accuracy by 33% and reduce editing by 26%.
“If message generation is really efficient and high quality, if it asks the right things, then it really has potential to improve efficiency,” says co-author Tim Burdick, an associate professor of community and family medicine in Dartmouth’s Geisel School of Medicine and a family medicine physician at Dartmouth Health.
“I don’t foresee a time when the portal can respond to a patient without a clinician editing it first. But as we make the models better, we’ll be able to address portal messages much more quickly and with less mental energy,” Burdick says.
The study shows that there are such things as “good” AI responses and provides a framework for implementing them into patient-physician portals, Preum says. These platforms are increasingly common among large health care systems and often customized, she says.
“That took us a long time to figure out, but if you’re trying to measure how effective this technology is, you need to define what a good response is,” she says. “We can only improve what we can measure and objectively evaluate.”
The researchers created a technique called TADPOLE (Thematic Agentic Direct Preference Optimization for Learning Enhancement) that trains AI platforms using the hybrid model they constructed from physician- and AI-generated responses.
They plugged TADPOLE into the six commercial LLMs and found that drafted responses better matched physicians’ standards for precision and information quality. “That could save a busy clinician an hour or two of work a day,” Burdick says.
Doctors and nurses today are inundated with messages from patients and caregivers who can write them online anytime, he says. An ongoing project between Burdick and the Preum Lab called PortalPal aims to streamline patient portals using AI, including by automating some steps in following up with patients to get more information.
“We’re still nowhere near the point of having clinicians removed from the workflow.”
Tim Burdick, co-author and associate professor of community and family medicine
Physicians who Burdick works with say that AI-generated drafts save about 25% of their time on shorter messages. “It’s easier to make small edits to an LLM-generated message than it is to write it from scratch,” he says. But longer drafts can include information that is not correct or accurate.
“If you have to edit 75% of the message, you may be spending more time and energy on making changes than if you were to just write it from scratch,” Burdick says. “I would guess we need to get to where the physician is editing less than 30% of the content before it has substantial benefit.”
One advantage of AI’s verbosity is that it tends to be more empathetic and thorough than physicians crunched for time, the researchers find. For example, AI is more likely to tell a patient experiencing an upset stomach that it’s sorry to hear they’re feeling nauseated.
This means AI could be used to help “nudge” doctors to show more understanding and care for the patient’s situation, or answer patient’s questions more effectively so that patients feel more heard, Preum says. The team produced sample responses such as showing empathy by praising patients for following a treatment plan (“You’ve been doing a great job with your tapering.”) or planning for changes in symptoms (“If you’re feeling dizzy, please call triage.”).
The researchers also find that 65% of all the portal messages they studied came from people over 55, with patients over 65 generating 24% of all messages. These figures suggest that patient portals in general should be designed to accommodate older people, Preum says.
Future work will study how much actual time doctors spend editing automated drafts. The team also plans to evaluate their training model TADPOLE from the user perspective, studying if and how it lightens a physician’s workload, and how doctors and patients rate its performance.
“This is one of the first studies that uses real patient portal messages to establish a generative AI model. In that regard, it’s innovative and shows us that this is not a simple task,” Burdick says. “We’re still nowhere near the point of having clinicians removed from the workflow.”
Mayo Clinic researchers and collaborators have shown that an artificial intelligence (AI) tool can analyse routine pathology slides to help clinicians classify meningiomas, the most common primary brain tumour in adults, and better understand a patient’s risk of tumour recurrence.
The study, published in The Lancet Digital Health, demonstrates that deep learning models can support the extraction of molecular and prognostic information from standard haematoxylin and eosin, or H&E, slides – the same type of tissue images already used in routine clinical care. These insights are typically obtained through DNA methylation profiling, an advanced genetic test which provides valuable diagnostic and prognostic information but can be costly, time-consuming and is unavailable in many hospitals.
“This is one of the many studies where we can harness the strength of digital pathology by capturing the last two decades of genomic and molecular knowledge into AI algorithms,” says Gelareh Zadeh, MD, PhD, chair of the Department of Neurologic Surgery at Mayo Clinic in Rochester and Chief Medical Officer for Mayo Clinic Platform.
Making advanced tumor insights more accessible
Meningiomas can vary widely in behaviour. Some grow slowly and may never return after treatment, while others are more aggressive and more likely to recur. Understanding that risk is critical for patients and care teams deciding whether additional treatment, such as radiation therapy, may be needed after surgery.
Molecular testing can help identify which tumours are more likely to recur and which may respond differently to treatment. But these tests require specialized technology and expertise, limiting access for many patients.
Using tissue samples, pathology images and clinical data from 672 patients, researchers developed and tested AI models designed to help identify patterns linked to a tumour’s biology. Drawing on multiple de-identified datasets, including data resources from Mayo Clinic Platform, the models supported classification of meningioma subtypes and recurrence risk prediction using standard pathology slides that are already part of routine patient care.
The findings suggest that, with further validation, AI-based tools could one day help clinicians obtain more detailed tumour information to inform patient care, without requiring every patient to undergo advanced genetic testing.
Helping guide treatment decisions
For patients with meningiomas, recurrence risk can influence follow-up care, imaging frequency and whether radiation therapy should be considered. The study found that AI-based predictions remained useful even after accounting for traditional clinical factors such as tumour grade, the extent to which surgery was able to remove the tumour and patient age.
Researchers also found that the AI models could identify patterns of tumour heterogeneity – differences within the same tumour – that may help explain why some tumours behave more aggressively or respond differently to treatment.
The researchers note that additional prospective studies are needed before the AI models can be used routinely in clinical care. Still, they say the findings lay the groundwork for more accessible, personalised care for patients with meningiomas – and potentially for similar AI approaches in other cancers.
As with any clinical decision-support tool, the researchers emphasise that these models would require rigorous evaluation, validation and ongoing physician oversight before being considered for routine care. “The aim is to make these algorithms readily and simply accessible for use globally, improving patient care across many healthcare settings,” says Dr Zadeh.
For a complete list of authors, disclosures and funding, review the publication.
