AI Breast Cancer Diagnoses Need a Reliability Rating
Weighting for Uncertainty Helps Pitt Team Factor for Individual Patients, not just Populations
While AI offers the ability to improve human experts’ ability to diagnose and treat diseases like breast cancer, well-publicised errors across the broader AI field are cause for concern. A team from the University of Pittsburgh has used PSC’s flagship Bridges-2 supercomputer to design a deep neural network (DNN) that takes uncertainty into account in diagnosing breast scan images. By providing a gauge of how accurate the prediction is likely to be, their AI assesses confidence that the answer is right for an individual patient, not just on average in a population.
Why it’s important
It’s a time of uncertainty for the future of AI. On the one hand, AI now powers so much of the good things we take for granted, including online shopping, transportation and road navigation, and fraud prevention. On the other, we have AI-written books on AI spouting made-up quotes and facts, autodriving cars not recognising pedestrians, and AI “helpers” spinning off into racist behaviour.
AI clearly has the ability to make our lives better. But most of us agree that we need to be more cautious about its reliability – giving us right answers doesn’t help if it also gives us wrong answers with equal confidence.
“I’m trying to develop effective AI models or methodology that can make AI model development easier, make breast cancer diagnosis more accurate, [and] make this model, make medical imaging AI models in general, more trustworthy and thus more clinically useful.”
Shandong Wu, University of Pittsburgh
Shandong Wu, a professor of radiology at the University of Pittsburgh School of Medicine, has been studying the use of AI to help radiologists identify breast cancer earlier and more effectively, as well as to identify patients’ likely future treatment needs. But he’s also been thinking about that question of reliability – of how much confidence we can and should put in its output.
Jiren Li, a PhD student working with Wu and with another radiology faculty member, Dooman Arefan, a research assistant professor at Pitt, wondered whether a breast-cancer diagnosis AI could be designed from the beginning to take uncertainty into account. That way, it could provide valid estimates of confidence along with its predictions. To implement their AI, he turned to PSC’s NSF-funded Bridges-2 system, via an allocation from ACCESS, the NSF’s network of supercomputing sites.
How PSC helped
The problem, the Pitt scientists realised, is that it’s straightforward to design a DNN-based AI that gives you correct answers most of the time – even almost all of the time. DNNs, created to teach themselves from a training data set in which the “right answers” have been labelled by humans, are really good at telling you what the answer is likely to be across a population of patients. But they’re not so good at telling you how reliable a diagnosis is for an individual patient. And when you’re talking about an AI that guides treatment of a life-threatening disease, even a few wrong diagnoses are catastrophic for the patients they affect.
The team approached the problem with what they’ve called uncertainty-weighted learning, or UWL. In this approach, the DNN architecture was designed to provide both a diagnosis and an associated “uncertainty score” from analyzing breast mammogram images. The uncertainty score becomes part of the DNN’s learning process, guiding the model’s learning to more effectively deal with those uncertain samples during training.
Creating the UWL tool posed serious computational challenges. DNN learning works best on graphics-processing units (GPUs), which feature many computational paths rather than pure speed at computing a single task. Because breast mammograms are large, high-resolution images, the GPUs would need access to large, shared memory as they deconstructed and learned from each training image. That would prevent slow-downs from having to go back and forth to data storage during the computations. In addition, large data storage would also be needed to host the full library of imaging data for AI model training and testing.
“Bridges-2 is critical for the success of these studies, because as you can imagine, this is all using large-scale, high-resolution, heterogeneous radiology imaging data, and in order to build this AI model, we need GPUs, memory, and larger storage to make this possible. Bridges-2 has been the primary source of computational resources for my lab for many years, supporting us to be able to store our data there, build multiple AI models, and evaluate these models… So, it’s really enabled us to develop innovative AI models and store large medical datasets.”
Shandong Wu, University of Pittsburgh
Bridges-2 fit the bill, with 47 GPU nodes containing 384 late-model GPUs. Better, each node has between 128 and 2000 gigabytes of RAM per node compare that with 64 GB total in a high-end laptop. PSC’s active project data storage system, Ocean, was also up to the task, with 15 petabytes of usable data capacity at high retrieval speeds. Again, that hot laptop might have as much as 2 terabytes of data storage, or 7500 times less.
The results of the Pitt group’s first analysis using several different UWL-based DNNs were encouraging. One of the DNNs in particular provided good assessments of the uncertainty of its answer, cuing doctors and their patients when a given diagnosis might be less trustworthy at the level of a specific patient. Better, it did this without reducing its accuracy at the population level. The team reported their results at the annual meeting of the Society of Photo-Optical Instrumentation Engineers (SPIE) in February, 2026 at Vancouver, Canada.
The Pitt scientists’ work has applications beyond the already important goal of making breast cancer diagnosis more accurate and trustworthy. The approach can be applied to just about any critical use of AI where the uncertainty as well as the prediction are important to know. Future directions include extending these uncertainty-based approaches to AI systems that integrate images and other types of medical records data, and enabling better human–AI collaborations, in which the decision-making can be tailored according to the measured model confidence. These advances could help create more trustworthy and clinically useful AI systems that improve both diagnostic accuracy and personalised patient care.
Source: University of Pittsburgh
