Tag: artificial intelligence

Getting the Most from AI in MedTech Takes Data Know-How

As a leader in Medical Technology innovation, InterSystems, a pioneer in healthcare data platform development, has learned, understood, and incorporated pivotal insights from its extensive experience in digital health solutions. That experience points up the need to give AI a strong foundation.

We understand the importance of leveraging AI to drive transformative change in healthcare. Our latest white paper, “Getting the Most from AI in MedTech Takes Data Know-How,” dives into the challenges and opportunities facing MedTech companies venturing into the realm of AI. From data cleanliness to privacy and security considerations, we address key issues that MedTech companies must navigate to succeed in today’s rapidly evolving healthcare landscape.

AI in MedTech Takes Data Know-How

The promise of AI in revolutionising MedTech is undeniable. AI in varying forms and degrees is forecasted to save hundreds of thousands of lives and billions of dollars a year. But here’s the catch- AI models are only as good as the data they’re built on. An AI application can sift through large amounts of data from various Electronic Health Record (EHR) environments and legacy systems and identify patterns within the scope of its model, but it can’t identify data that exists outside of those boundaries.

If one asks “What risk factors does the patient have for stroke?”, AI can only answer based on the information that’s there. Sometimes, things get lost in translation, and that’s why interoperability – the ability to exchange information in a way that ensures the sender and receiver understand data the same way is crucial.

InterSystems: Your Data Sherpa:

Ever wondered why some AI models in MedTech fall short? It’s all about the data. This means MedTech companies can’t just lean on their currently used standard but should consider all those in which relevant data is captured in the market or build on a platform that does.

With InterSystems by your side, you gain access to a treasure trove of healthcare data expertise. One of the benefits of our business is that it’s much broader than a single EHR. This means providing software solutions like The HL7® FHIR® (Fast Healthcare Interoperability Resources) offering a comprehensive view of patient data, accelerating development timelines, and delivering tangible results that showcase the value of your innovations.

Clean Data Is a Must

Data cleanliness is key in the world of AI. Pulling data from various sources presents its own set of challenges, from ensuring data cleanliness to reconciling discrepancies and omissions. Raw data is often messy, inconsistent, and filled with gaps like missing labels. If the data fed into an AI model is incomplete and error-ridden, the conclusions drawn from its analysis will be similarly flawed and suspect. Thus, maintaining high standards of data quality is essential to ensure the accuracy and effectiveness of AI-driven insights.

Henry Adams, Country Manager, InterSystems South Africa, says: “InterSystems advocates for robust preprocessing, cleaning, and labelling techniques to ensure data quality and integrity. Our platform keeps track of data lineage, simplifies labelling, and aggregates health data into a single, patient-centric model ready for analysis”.

Privacy, Security, and Reliability: The Sweet Success!

Privacy and security are essential across industries, but they are even more critical for MedTech product developers. Handling sensitive patient data necessitates strict adherence to regulations like HIPAA and GDPR to safeguard patient confidentiality and comply with legal requirements. Beyond regulatory compliance, ensuring privacy and security is crucial for maintaining patient safety, preserving reputation and trust, and fostering collaboration within the industry.

To help MedTech companies comply with regulations and safeguard patient data, InterSystems’ platform meets needs across major deployments, such as a nonprofit health data network and uses techniques like redundant processing and queues built into the connective tissue of their software. Reliable connectivity solutions ensure seamless data exchange, even in the most demanding healthcare environments.

Charting the Course Forward

If you are a MedTech company still struggling to make sense of siloed healthcare data for your AI initiatives? We have the answers-collaboration with the right partner is essential for integrating AI into medical practices. An ideal partner understands the need for data acquisition, aggregation, cleaning, privacy, and security regulations. “With InterSystems as your partner and by your side, you can navigate the complexities of AI integration and drive transformative innovation in healthcare, making MedTech excellence easier to attain,” concludes Adams.

You can learn more about our support for MedTech innovation at InterSystems.com/MedTech.

For more information or to download the guide, please visit!  

When it Comes to Personalised Cancer Treatments, AI is no Match for Human Doctors

Cancer treatment is growing more complex, but so too are the possibilities. After all, the better a tumour’s biology and genetic features are understood, the more treatment approaches there are. To be able to offer patients personalised therapies tailored to their disease, laborious and time-consuming analysis and interpretation of various data is required. In one of many artificial intelligence (AI)projects at Charité – Universitätsmedizin Berlin and Humboldt-Universität zu Berlin, researchers studied whether generative AI tools such as ChatGPT can help with this step.

The crucial factor in the phenomenon of tumour growth is an imbalance of growth-inducing and growth-inhibiting factors, which can result, for example, from changes in oncogenes.

Precision oncology, a specialised field of personalised medicine, leverages this knowledge by using specific treatments such as low-molecular weight inhibitors and antibodies to target and disable hyperactive oncogenes.

The first step in identifying which genetic mutations are potential targets for treatment is to analyse the genetic makeup of the tumour tissue. The molecular variants of the tumour DNA that are necessary for precision diagnosis and treatment are determined. Then the doctors use this information to craft individual treatment recommendations. In especially complex cases, this requires knowledge from various fields of medicine.

At Charité, this is when the “molecular tumour board” (MTB) meets: Experts from the fields of pathology, molecular pathology, oncology, human genetics, and bioinformatics work together to analyse which treatments seem most promising based on the latest studies.

It is a very involved process, ultimately culminating in a personalised treatment recommendation.

Can artificial intelligence help with treatment decisions?

Dr Damian Rieke, a doctor at Charité, and his colleagues wondered whether AI might be able to help at this juncture.

In a study just recently published in the journal JAMA Network Open, they worked with other researchers to examine the possibilities and limitations of large language models such as ChatGPT in automatically scanning scientific literature with an eye to selecting personalised treatments.

AI ‘not even close’

“We prompted the models to identify personalised treatment options for fictitious cancer patients and then compared the results with the recommendations made by experts,” Rieke explains.

His conclusion: “AI models were able to identify personalised treatment options in principle – but they weren’t even close to the abilities of human experts.”

The team created ten molecular tumour profiles of fictitious patients for the experiment.

A human physician specialist and four large language models were then tasked with identifying a personalised treatment option.

These results were presented to the members of the MTB for assessment, without them knowing where which recommendation came from.

Improved AI models hold promise for future uses

Dr. Manuela Benary, a bioinformatics specialist reported: “There were some surprisingly good treatment options identified by AI in isolated cases. “But large language models perform much worse than human experts.”

Beyond that, data protection, privacy, and reproducibility pose particular challenges in relation to the use of artificial intelligence with real-world patients, she notes.

Still, Rieke is fundamentally optimistic about the potential uses of AI in medicine: “In the study, we also showed that the performance of AI models is continuing to improve as the models advance. This could mean that AI can provide more support for even complex diagnostic and treatment processes in the future – as long as humans are the ones to check the results generated by AI and have the final say about treatment.”

Source: Charité – Universitätsmedizin Berlin

AI-based CT Scans of the Brain can Nearly Match MRI

Photo by Mart Production on Pexels

A new artificial intelligence (AI)-based method can provide as much information on subtle neurodegenerative changes in the brain captured by computed tomography (CT) as compared to magnetic resonance imaging (MRI). The method, reported in the journal Alzheimer’s & Dementia, could enhance diagnostic support, particularly in primary care, for conditions such as dementia and other brain disorders.

Compared to MRI, which requires powerful superconducting magnetics and their associated cryogenic cooling, computed tomography (CT) is a relatively inexpensive and widely available imaging technology. CT is considered inferior to MRI when it comes to reproducing subtle structural changes in the brain or flow changes in the ventricular system. Certain imaging must therefore currently be carried out by specialist departments at larger hospitals equipped with MRI.

AI trained on MRI images

Created with deep learning, a form of AI, the software has been trained to transfer interpretations from MRI images to CT images of the same brains. The new software can provide diagnostic support for radiologists and other professionals who interpret CT images.

“Our method generates diagnostically useful data from routine CT scans that, in some cases, is as good as an MRI scan performed in specialist healthcare,” says Michael Schöll, a professor at Sahlgrenska Academy who led the work involved in the study, carried out in collaboration with researchers at Karolinska Institutet, the National University of Singapore, and Lund University

“The point is that this simple, quick method can provide much more information from examinations that are already carried out on a routine basis within primary care, but also in certain specialist healthcare investigations. In its initial stage, the method can support dementia diagnosis, however, it is also likely to have other applications within neuroradiology.”

Reliable decision-making support

This is a well-validated clinical application of AI-based algorithms, and has the potential to become a fast and reliable form of decision-making support that effectively reduces the number of false negatives. The researchers believe that this solution can improve diagnostics in primary care, optimising patient flow to specialist care.

“This is a major step forward for imaging diagnosis,” says Meera Srikrishna, a postdoctor at the University of Gothenburg and lead author of the study.

“It is now possible to measure the size of different structures or regions of the brain in a similar way to advanced analysis of MRI images. The software makes it possible to segment the brain’s constituent parts in the image and to measure its volume, even though the image quality is not as high with CT.”

Applications for other brain diseases

The software was trained on images of 1117 people, all of whom underwent both CT and MRI imaging. The current study mainly involved healthy older individuals and patients with various forms of dementia. Another application that the team is now investigating is for normal pressure hydrocephalus (NPH).

With NPH, the team has obtained new results indicating that the method can be used both during diagnosis and to monitor the effects of treatment. NPH is a condition that occurs particularly in older people, whereby fluid builds up in the cerebral ventricular system and results in neurological symptoms. About two percent of all people over the age of 65 are affected. Because diagnosis can be complicated and the condition risks being confused with other diseases, many cases are likely to be missed.

“NPH is difficult to diagnose, and it can also be hard to safely evaluate the effect of shunt surgery to drain the fluid in the brain,” continues Michael. “We therefore believe that our method can make a big difference when caring for these patients.”

The software has been developed over the course of several years, and development is now continuing in cooperation with clinics in Sweden, the UK, and the US together with a company, which is a requirement for the innovation to be approved and transferred to healthcare.

Source: University of Gothenburg

Clinical Researchers Beware – ChatGPT is not a Reliable Aid

Photo by National Cancer Institute on Unsplash

Clinicians are all too familiar with the ‘Google patient’ who finds every scary, worst-case or outright false diagnosis online on whatever is ailing them. During COVID, misinformation spread like wildfire, eroding the public’s trust in vaccines and the healthcare profession. But now, AI models like ChatGPT can be whispering misleading information to the clinical researchers trying to produce real research.

Researchers from CHU Sainte-Justine and the Montreal Children’s Hospital recently posed 20 medical questions to ChatGPT. The chatbot provided answers of limited quality, including factual errors and fabricated references, show the results of the study published in Mayo Clinic Proceedings: Digital Health.

“These results are alarming, given that trust is a pillar of scientific communication. ChatGPT users should pay particular attention to the references provided before integrating them into medical manuscripts,” says Dr Jocelyn Gravel, lead author of the study and emergency physician at CHU Sainte-Justine.

Questionable quality, fabricated references

The researchers drew their questions from existing studies and asked ChatGPT to support its answers with references. They then asked the authors of the articles from which the questions were taken to rate the software’s answers on a scale from 0 to 100%.

Out of 20 authors, 17 agreed to review the answers of ChatGPT. They judged them to be of questionable quality (median score of 60%). They also found major (five) and minor (seven) factual errors. For example, the software suggested administering an anti-inflammatory drug by injection, when it should be swallowed. ChatGPT also overestimated the global burden of mortality associated with Shigella infections by a factor of ten.

Of the references provided, 69% were fabricated, yet looked real. Most of the false citations (95%) used the names of authors who had already published articles on a related subject, or came from recognised organisations such as the Food and Drug Administration. The references all bore a title related to the subject of the question and used the names of known journals or websites. Even some of the real references contained errors (eight out of 18).

ChatGPT explains

When asked about the accuracy of the references provided, ChatGPT gave varying answers. In one case, it claimed, “References are available in Pubmed,” and provided a web link. This link referred to other publications unrelated to the question. At another point, the software replied, “I strive to provide the most accurate and up-to-date information available to me, but errors or inaccuracies can occur.”

Despite even the most ‘truthful’ of these responses, ChatGPT poses hidden risks to academic, the researcher say.

“The importance of proper referencing in science is undeniable. The quality and breadth of the references provided in authentic studies demonstrate that the researchers have performed a complete literature review and are knowledgeable about the topic. This process enables the integration of findings in the context of previous work, a fundamental aspect of medical research advancement. Failing to provide references is one thing but creating fake references would be considered fraudulent for researchers,” says Dr Esli Osmanlliu, emergency physician at the Montreal Children’s Hospital and scientist with the Child Health and Human Development Program at the Research Institute of the McGill University Health Centre.

“Researchers using ChatGPT may be misled by false information because clear, seemingly coherent and stylistically appealing references can conceal poor content quality,” adds Dr Osmanlliu.

This is the first study to assess the quality and accuracy of references provided by ChatGPT, the researchers point out.

Source: McGill University Health Centre

Would it be Ethical to Entrust Human Patients to Robotic Nurses?

Photo by Alex Knight on Unsplash

Advancements in AI have resulted in typically human characteristics like creativity, communication, critical thinking, and learning being replicated by machines for complex tasks like driving vehicles and creating art. With further development, these human-like attributes may develop enough to one day make it possible for robots and AI to be entrusted with nursing, a very ‘human’ practice. But… would it be ethical to entrust the care of humans to machines?

In a step toward answering this question, Japanese researchers recently explored the ethics of such a situation in the journal Nursing Ethics.

The study was conducted by Associate Professor Tomohide Ibuki from Tokyo University of Science, in collaboration with medical ethics researcher Dr Eisuke Nakazawa from The University of Tokyo and nursing researcher Dr Ai Ibuki from Kyoritsu Women’s University.

“This study in applied ethics examines whether robotics, human engineering, and human intelligence technologies can and should replace humans in nursing tasks,” says Dr Ibuki.

Nurses show empathy and establish meaningful connections with their patients, a human touch which is essential in fostering a sense of understanding, trust, and emotional support. The researchers examined whether the current advancements in robotics and AI can implement these human qualities by replicating the ethical concepts attributed to human nurses, including advocacy, accountability, cooperation, and caring.

Advocacy in nursing involves speaking on behalf of patients to ensure that they receive the best possible medical care. This encompasses safeguarding patients from medical errors, providing treatment information, acknowledging the preferences of a patient, and acting as mediators between the hospital and the patient. In this regard, the researchers noted that while AI can inform patients about medical errors and present treatment options, they questioned its ability to truly understand and empathise with patients’ values and to effectively navigate human relationships as mediators.

The researchers also expressed concerns about holding robots accountable for their actions. They suggested the development of explainable AI, which would provide insights into the decision-making process of AI systems, improving accountability.

The study further highlights that nurses are required to collaborate effectively with their colleagues and other healthcare professionals to ensure the best possible care for patients. As humans rely on visual cues to build trust and establish relationships, unfamiliarity with robots might lead to suboptimal interactions. Recognising this issue, the researchers emphasised the importance of conducting further investigations to determine the appropriate appearance of robots for facilitating efficient cooperation with human medical staff.

Lastly, while robots and AI have the potential to understand a patient’s emotions and provide appropriate care, the patient must also be willing to accept robots as care providers.

Having considered the above four ethical concepts in nursing, the researchers acknowledge that while robots may not fully replace human nurses anytime soon, they do not dismiss the possibility. While robots and AI can potentially reduce the shortage of nurses and improve treatment outcomes for patients, their deployment requires careful weighing of the ethical implications and impact on nursing practice.

“While the present analysis does not preclude the possibility of implementing the ethical concepts of nursing in robots and AI in the future, it points out that there are several ethical questions. Further research could not only help solve them but also lead to new discoveries in ethics,” concludes Dr Ibuki.

Source: Tokyo University of Science

Dr Robot Will See You Now: Medical Chatbots Need to be Regulated

Photo by Alex Knight on Unsplash

The Large Language Models (LLM) used in chatbots may appear to offer reliable, persuasive advice in a format which mimics conversation but in they can offer potentially harmful information when prompted with medical questions. Therefore, any LLM-chatbot in a medical setting would require approval as a medical device, argue experts in a paper published in Nature Medicine.

The mistake often made with LLM-chatbots is that they are a true “artificial intelligence” when in fact they are more closely related to the predictive text in a smartphone. They mostly use conversations and text scraped from the internet, and use algorithms to associate words and sentences in a manner that appears meaningful.

“Large Language Models are neural network language models with remarkable conversational skills. They generate human-like responses and engage in interactive conversations. However, they often generate highly convincing statements that are verifiably wrong or provide inappropriate responses. Today there is no way to be certain about the quality, evidence level, or consistency of clinical information or supporting evidence for any response. These chatbots are unsafe tools when it comes to medical advice and it is necessary to develop new frameworks that ensure patient safety,” said Prof Stephen Gilbert at TU Dresden.

Challenges in the regulatory approval of LLMs

Most people research their symptoms online before seeking medical advice. Search engines play a role in decision-making process. The forthcoming integration of LLM-chatbots into search engines may increase users’ confidence in the answers given by a chatbot that mimics conversation. It has been demonstrated that LLMs can provide profoundly dangerous information when prompted with medical questions.

The basis of LLMs do not have any medical “ground truth,” which is inherently dangerous. Chat-interfaced LLMs have already provided harmful medical responses and have already been used unethically in ‘experiments’ on patients without consent. Almost every medical LLM use case requires regulatory control in the EU and US. In the US their lack of explainability disqualifies them from being ‘non devices’. LLMs with explainability, low bias, predictability, correctness, and verifiable outputs do not currently exist and they are not exempted from current (or future) governance approaches.

The authors describe in their paper the limited scenarios in which LLMs could find application under current frameworks. They also describe how developers can seek to create LLM-based tools that could be approved as medical devices, and they explore the development of new frameworks that preserve patient safety. “Current LLM-chatbots do not meet key principles for AI in healthcare, like bias control, explainability, systems of oversight, validation and transparency. To earn their place in medical armamentarium, chatbots must be designed for better accuracy, with safety and clinical efficacy demonstrated and approved by regulators,” concludes Prof Gilbert.

Source: Technische Universität Dresden

In the ICU, Artificial Intelligence Beats Humans

Image created using an AI art program, Craiyon, with the prompt “An AI monitoring a patient in an ICU ward”.

In the future, artificial intelligence will play an important role in medicine. In diagnostics, successful tests have already been performed with AI, such as accurately categorising images according to whether they show pathological changes or not. But training an AI run in real time to examine the time-varying conditions of patients in an ICU and to calculate treatment suggestions has remained a challenge. Now, University of Vienna Researchers report in the Journal of Clinical Medicine that they have accomplished such a feat.

With the help of extensive data from ICUs of various hospitals, an AI was developed that provides suggestions for the treatment of people who require intensive care due to sepsis. Analyses show that AI already surpasses the quality of human decisions making it important to also discuss the legal aspects of such methods.

Making optimal use of existing data

“In an intensive care unit, a lot of different data is collected around the clock. The patients are constantly monitored medically. We wanted to investigate whether these data could be used even better than before,” says Prof Clemens Heitzinger from the Institute for Analysis and Scientific Computing at TU Wien (Vienna).

Medical staff make their decisions on the basis of well-founded rules. Most of the time, they know very well which parameters they have to take into account in order to provide the best care. But now, a computer can easily take many more parameters than a human into account – sometimes leading to even better decisions.

The computer as planning agent

“In our project, we used a form of machine learning called reinforcement learning,” says Clemens Heitzinger. “This is not just about simple categorisation – for example, separating a large number of images into those that show a tumour and those that do not – but about a temporally changing progression, about the development that a certain patient is likely to go through. Mathematically, this is something quite different. There has been little research in this regard in the medical field.”

The computer becomes an agent that makes its own decisions: if the patient is well, the computer is “rewarded”. If the condition deteriorates or death occurs, the computer is “punished”. The computer programme has the task of maximising its virtual “reward” by taking actions. In this way, extensive medical data can be used to automatically determine a strategy which achieves a particularly high probability of success.

Already better than a human

“Sepsis is one of the most common causes of death in intensive care medicine and poses an enormous challenge for doctors and hospitals, as early detection and treatment is crucial for patient survival,” says Prof Oliver Kimberger from the Medical University of Vienna. “So far, there have been few medical breakthroughs in this field, which makes the search for new treatments and approaches all the more urgent. For this reason, it is particularly interesting to investigate the extent to which artificial intelligence can contribute to improve medical care here. Using machine learning models and other AI technologies are an opportunity to improve the diagnosis and treatment of sepsis, ultimately increasing the chances of patient survival.”

Analysis shows that AI capabilities are already outperforming humans: “Cure rates are now higher with an AI strategy than with purely human decisions. In one of our studies, the cure rate in terms of 90-day mortality was increased by about 3% to about 88%,” says Clemens Heitzinger.

Of course, this does not mean that one should leave medical decisions in an ICU to the computer alone. But the artificial intelligence may run along as an additional device at the bedside – and the medical staff can consult it and compare their own assessment with the AI’s suggestions. Such AIs can also be highly useful in education.

Discussion about legal issues is necessary

“However, this raises important questions, especially legal ones,” says Clemens Heitzinger. “One probably thinks of the question who will be held liable for any mistakes made by the artificial intelligence first. But there is also the converse problem: what if the artificial intelligence had made the right decision, but the human chose a different treatment option and the patient suffered harm as a result?” Does the doctor then face the accusation that it would have been better to trust the artificial intelligence because it comes with a huge wealth of experience? Or should it be the human’s right to ignore the computer’s advice at all times?

“The research project shows: artificial intelligence can already be used successfully in clinical practice with today’s technology – but a discussion about the social framework and clear legal rules are still urgently needed,” Clemens Heitzinger is convinced.

Source: EurekAlert!

Building a Future ‘Biocomputer’ Using Human Brain Cells

Depiction of a human brain
Image by Fakurian Design on Unsplash

A “biocomputer” powered by human brain cells could be developed within our lifetime, according to an article in the journal Frontiers in Science. The Johns Hopkins University researchers expect such “organoid intelligence” technology to exponentially expand the capabilities of modern computing and create novel fields of study, as well as yielding insights into neurodegenerative diseases.

“Computing and artificial intelligence have been driving the technology revolution but they are reaching a ceiling,” said Thomas Hartung, a professor of environmental health sciences at the Johns Hopkins Bloomberg School of Public Health and Whiting School of Engineering who is spearheading the work. “Biocomputing is an enormous effort of compacting computational power and increasing its efficiency to push past our current technological limits.”

For nearly two decades scientists have used tiny organoids, lab-grown tissue resembling fully grown organs, to experiment on kidneys, lungs, and other organs without resorting to human or animal testing. More recently Hartung and colleagues at Johns Hopkins have been working with brain organoids, orbs the size of a pen dot with neurons and other features that promise to sustain basic functions like learning and remembering.

“This opens up research on how the human brain works,” Hartung said. “Because you can start manipulating the system, doing things you cannot ethically do with human brains.”

Hartung began to grow and assemble brain cells into functional organoids in 2012 using cells from human skin samples reprogrammed into an embryonic stem cell-like state. Each organoid contains about 50 000 cells, about the size of a fruit fly’s nervous system. He now envisions building a futuristic computer with such brain organoids.

Computers that run on this “biological hardware” could in the next decade begin to alleviate energy-consumption demands of supercomputing that are becoming increasingly unsustainable, Hartung said. Even though computers process calculations involving numbers and data faster than humans, brains are much smarter in making complex logical decisions, like telling a dog from a cat.

“The brain is still unmatched by modern computers,” Hartung said. “Frontier, the latest supercomputer in Kentucky, is a $600 million, 6,800-square-feet installation. Only in June of last year, it exceeded for the first time the computational capacity of a single human brain – but using a million times more energy.”

It might take decades before organoid intelligence can power a system as smart as a mouse, Hartung said. But by scaling up production of brain organoids and training them with artificial intelligence, he foresees a future where biocomputers support superior computing speed, processing power, data efficiency, and storage capabilities.

“It will take decades before we achieve the goal of something comparable to any type of computer,” Hartung said. “But if we don’t start creating funding programs for this, it will be much more difficult.”

Medical applications

Organoid intelligence could also revolutionise drug testing research for neurodevelopmental disorders and neurodegeneration, said Lena Smirnova, a Johns Hopkins assistant professor of environmental health and engineering who co-leads the investigations.

“We want to compare brain organoids from typically developed donors versus brain organoids from donors with autism,” Smirnova said. “The tools we are developing towards biological computing are the same tools that will allow us to understand changes in neuronal networks specific for autism, without having to use animals or to access patients, so we can understand the underlying mechanisms of why patients have these cognition issues and impairments.”

To assess the ethical implications of working with organoid intelligence, a diverse consortium of scientists, bioethicists, and members of the public have been embedded within the team.

Source: John Hopkins University

ChatGPT can Now (Almost) Pass the US Medical Licensing Exam

Photo by Maximalfocus on Unsplash

ChatGPT can score at or around the approximately 60% pass mark for the United States Medical Licensing Exam (USMLE), with responses that make coherent, internal sense and contain frequent insights, according to a study published in PLOS Digital Health by Tiffany Kung, Victor Tseng, and colleagues at AnsibleHealth.

ChatGPT is a new artificial intelligence (AI) system, known as a large language model (LLM), designed to generate human-like writing by predicting upcoming word sequences. Unlike most chatbots, ChatGPT cannot search the internet. Instead, it generates text using word relationships predicted by its internal processes.

Kung and colleagues tested ChatGPT’s performance on the USMLE, a highly standardised and regulated series of three exams (Steps 1, 2CK, and 3) required for medical licensure in the United States. Taken by medical students and physicians-in-training, the USMLE assesses knowledge spanning most medical disciplines, ranging from biochemistry, to diagnostic reasoning, to bioethics.

After screening to remove image-based questions, the authors tested the software on 350 of the 376 public questions available from the June 2022 USMLE release. 

After indeterminate responses were removed, ChatGPT scored between 52.4% and 75.0% across the three USMLE exams. The passing threshold each year is approximately 60%. ChatGPT also demonstrated 94.6% concordance across all its responses and produced at least one significant insight (something that was new, non-obvious, and clinically valid) for 88.9% of its responses. Notably, ChatGPT exceeded the performance of PubMedGPT, a counterpart model trained exclusively on biomedical domain literature, which scored 50.8% on an older dataset of USMLE-style questions.

While the relatively small input size restricted the depth and range of analyses, the authors note their findings provide a glimpse of ChatGPT’s potential to enhance medical education, and eventually, clinical practice. For example, they add, clinicians at AnsibleHealth already use ChatGPT to rewrite jargon-heavy reports for easier patient comprehension.

“Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation,” say the authors.

Author Dr Tiffany Kung added that ChatGPT’s role in this research went beyond being the study subject: “ChatGPT contributed substantially to the writing of [our] manuscript… We interacted with ChatGPT much like a colleague, asking it to synthesise, simplify, and offer counterpoints to drafts in progress…All of the co-authors valued ChatGPT’s input.”

Source: EurekAlert!

AI Picks up Incidental Pulmonary Embolism on Chest CT

Photo by Kanasi on Unsplash

According to a study published in the American Journal of Roentgenology, an AI tool for detection of incidental pulmonary embolus (iPE) on conventional contrast-enhanced chest CT examinations had high false negative and moderate false positive rates for detection, and was even able to pick up some iPEs missed by radiologists.

“Potential applications of the AI tool include serving as a second reader to help detect additional iPEs or as a worklist triage tool to allow earlier iPE detection and intervention,” wrote lead investigator Kiran Batra from the University of Texas Southwestern Medical Center in Dallas. “Various explanations of misclassifications by the AI tool (both false positives and false negatives) were identified, to provide targets for model improvement.”

Batra and colleagues’ retrospective study included 2,555 patients (1,340 women, 1,215 men; mean age, 53.6 years) who underwent 3,003 conventional contrast-enhanced chest CT examinations between September 2019 and February 2020 at Parkland Health in Dallas, TX. Using an FDA-approved, commercially available AI tool (Aidoc) to detect acute iPE on the images, a vendor-supplied natural language processing algorithm was then applied to the clinical reports to identify examinations interpreted as positive for iPE.

Ultimately, the commercial AI tool had NPV of 99.8% and PPV of 86.7% for detection of iPE on conventional contrast-enhanced chest CT examinations (ie, not using CT pulmonary angiography protocols). Of 40 iPEs present in the team’s study sample, 7 were detected only by the clinical reports, and 4 were detected only by AI.

Noting that both the AI tool and clinical reports detected iPEs missed by the other method, “the diagnostic performance of the AI tool did not show significant variation across study subgroups,” the authors added.

Source: American Roentgen Ray Society