Defective Detective: We Need Explainable Machine Learning in Patient Care

Discovering a medical diagnosis and formulating a treatment plan is like a detective solving a case. Imagine if Sherlock Holmes or your favorite Criminal Minds character claimed they identified the killer, but their explanation is, “I don’t really know why, but I analyzed tons of databases and I saw this pattern. Just trust me – I tend to get these right.” This explanation would never hold up in court, so why should we accept this kind of approach from AI in medicine? Patient care has long been a human task, but the artificial intelligence boom has sparked research into their intersection. From pharmaceutical development to patient history analysis, researchers are working to harness the power of AI to transform medicine.

Artificial intelligence (AI) is the development of computer systems that can perform tasks that traditionally require human intelligence. By analyzing vast amounts of data, including patient records, research papers, medical images, and genetic information, AI is well suited to recognize patterns and make informed decisions, providing valuable support to healthcare professionals in making accurate diagnoses and effective treatment plans.

When making a diagnosis, doctors must consider numerous clues. Dr. Jennifer Barker, a professor of pediatric endocrinology at the University of Colorado School of Medicine, explains that the process involves evaluating the patient’s history, symptoms, physical exam, and laboratory tests, “What’s really important is being able to collate all of the information and decide what fits, and if the information doesn’t fit, does it matter?” Luckily, AI can examine thousands of variables in a fraction of the time a human could.

While finding disease indications is straightforward, crafting treatment plans is deeply personal. Dr. Barker explains that she spends much of her time developing a plan alongside a patient, “It’s really important to know who the patient is and how they want to approach treatment.” While an algorithm can solve a medical mystery, even the most advanced AIs won’t be able to empathize with a patient on a human level. With AI as a sidekick, doctors could increase accuracy and efficiency, freeing up time for increased patient interaction and medical access. But the keyword is sidekick.

The Problem

As AI becomes more advanced, even the model’s developers have trouble understanding how a system arrives at a conclusion, earning AI systems the nickname “black boxes.” Dr. Barker notes that physicians have trusted risk-predicting algorithms to help guide the diagnosis process for decades, but there is cause for concern.

Machine learning models reflect human biases when ‘trained’ on unrepresentative data or past cases echoing the structural racism that has long infested healthcare. A troubling example is a 2019 study published in Science, which exposed bias in a clinical risk algorithm used on an estimated 200 million people annually. The model targets patients with chronic illnesses for enrollment in increased care programs.

In theory, this could work. There is a strong correlation between healthcare spending and patient sickness: sicker patients require more care, generating higher medical costs. However, when comparing healthcare spending between Black and White patients at a given level of sickness, Black patients generate, on average, $1801 less per year. Given this gap, the research found that Black patients had to be much sicker than their White counterparts to be admitted into the program.

AI Reflects Structural Racism in Healthcare

Numerous factors could explain this gap.

• Black patients with White primary care providers receive fewer preventative care recommendations than those with Black providers, leading to consistent undertreatment.
• Patients could face barriers like transportation, jobs, and childcare.
• Black patients generally have reduced trust in health care, dating back to the Tuskegee experiments.

No matter the discrepancy’s source, developers encoded existing biases into the algorithm, leaving Black patients with delayed care.

Risk algorithms aren’t the only issue. Researchers at MIT found that X-ray analysis models underdiagnosed chest pathology in underserved patient populations. Many algorithms are not required to be reviewed by any regulatory body, and the FDA only recommends that developers test their products for potential racial or ethnic bias. So who gets to set the arbitrary threshold of harm allowed before a model enters the market? The predominantly white male data scientists and developers?

But we can fix these models! Researchers found that using the number of chronic illnesses as an alternative to health care spending, reduced bias by 84%, doubling the number of Black patients receiving care. The improved algorithm outperformed individual doctors in enrolling patients without bias.

Explainable AI is Part of the Solution

To err is human. Dr. Barker mentions that doctors’ mistakes are usually caught before harm is done. But when errors are serious, accountability processes range from hospital oversight to national regulation. Black box AI isn’t traceable or transparent, making it challenging for developers, regulators, and users to hold these systems accountable for mistakes or biases. Enter explainable AI! Explainable AI (aka XAI) is a major research focus for industry leaders like IBM and Google and is applicable in fields from medicine to finance to criminal justice. XAI helps users understand a system’s decision-making at every step, from a black box to a glass box, increasing accountability and trust in these systems.

Today’s black-box learning process compared with tomorrow’s explainable model and explanation interface

XAI works to make sense of automated decisions in a variety of ways. For example, after an AI recognizes a cat (or diagnoses lung cancer), the system would explain why it made that decision. This could mean providing decision trees, written descriptions of a model’s thought process, or regions of an image that contributed most to a prediction.

Figure 1. Image classified as “cat” because of cat’s ears and nose.

Achieving full transparency in AI models can be challenging, as there might be trade-offs between model complexity, accuracy, and interpretability. With continued research and investment, physicians can trust these systems, interpret their decisions, and recognize where bias may stem from. Additionally, policymakers must respond to the rapid development of AI with stricter standards for transparency and guidelines for the amount of harm and error allowed before a system can impact the public. Research in XAI is still developing and it could be part of the solution for more accurate and fair AI systems.

Now, after the XAI detective solves a medical mystery, it can craft transparent explanations to convince stakeholders, just like a detective presenting evidence in court.

References

1. (2000). To Err Is Human. doi:10.17226/9728

2. (N.d.). Retrieved from https://www.darpa.mil/program/explainable-artificial-intelligence

3. Alsan, M., Garrick, O., & Graziani, G. (2019). Does diversity matter for health? experimental evidence from Oakland. American Economic Review, 109(12), 4071–4111. doi:10.1257/aer.20181446

4. Ibm. (2024). What is explainable AI (XAI)? Retrieved from https://www.ibm.com/topics/explainable-ai

5. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019a). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. doi:10.1126/science.aax2342

6. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019b). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. doi:10.1126/science.aax2342

7. Schwendicke, F., Samek, W., & Krois, J. (2020). Artificial Intelligence in dentistry: Chances and challenges. Journal of Dental Research, 99(7), 769–774. doi:10.1177/0022034520915714

8. Seyyed-Kalantari, L., Zhang, H., McDermott, M. B., Chen, I. Y., & Ghassemi, M. (2021). Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 27(12), 2176–2182. doi:10.1038/s41591-021-01595-0

9. There’s more to AI bias than biased data, NIST report highlights. (2022). Retrieved from https://www.nist.gov/news-events/news/2022/03/theres-more-ai-bias-biased-data-nist-report-highlights