Interpretability of an FDA-authorized AI/ML sepsis diagnostic tool improved by SHAP values
Year: 2026
Type: article
Abstract: Objectives: To assess the interpretability and acceptance of Shapley values for making artificial intelligence/machine learning (AI/ML) tools more transparent, interpretable, and useful to clinicians. Materials and Methods: Structured assessments were conducted with 30 clinicians (15 providers; 15 nurses; 8 assessments per clinician) to evaluate their ability to understand interventional Shapley Additive exPlanations (SHAP) values, a type of Shapley value that provides individualized variable importance scores and ascertain their perspective on SHAP value utility for the use of an AI/ML sepsis diagnostic. Participants were shown the diagnostic interface for real clinical scenarios with de-identified patient data with and without SHAP values. The primary outcomes were clinician ability to correctly interpret SHAP values and clinician self-reported improvement in their understanding of how the AI/ML algorithm produced its result. Results: Participants correctly interpreted SHAP values in 235 of 240 assessments (98%; CI, 95%-99%) and reported SHAP values improved their understanding of how the algorithm produced its result in every case (240/240; 100%; CI, 99%-100%). Participants were unanimous (30/30) in preferring the interface with SHAP values over the interface without. Discussion: Clinician participants strongly preferred the device interface with SHAP values, were unanimous in reporting SHAP values improved their understanding of the AI/ML diagnostic, and scored nearly perfectly when asked to interpret SHAP values. Conclusion: These results suggest health care providers value transparency into AI/ML algorithms designed for clinical use, and that Shapley values are a useful approach to providing that transparency, which in turn may improve tool adoption and clinical utility.
Source: JAMIA Open
Authors: Gregory L Watson, Grace Staples, Robin Carver, Akhil Bhargava, Carlos López-Espina, Lee Schmalz, Farhan Ali, Peter S Antkowiak, Saleem Azad, Ramona Berghea, Lavneet Chawla, Matthew Crisp, Alon Dagan, Francisco Davila, Hugo Davila, Carmen DeMarco, Amanda Doodlesack, Aimee Espinosa, Neil S Evans, Clinton Ezekiel, Andrew Friederich, Falgun Gosai, Alexandra Halalau, Karthik Iyer, Max S Kravitz, Niko Kurtzman, John H Lee, Nicholas Maddens, Roneil Malkani, Stockton Mayer, Vikram Oke, Ashok V Palagiri, Roshni Patel, Lekshminarayan Raghavakurup, Samuel Raouf, Eric Reseland, Farid Sadaka, Deesha Sarma, Scott Smith, Tatyana Shvilkina, Matthew D Sims, Sahib Singh, Bryan A Stenson, Anwaruddin Syed, Muleta Tafa, Kurian Thomas, Sihai Dave Zhao, Ruoqing Zhu, Rashid Bashir, Bobby Reddy, Nathan I Shapiro
Institutions: Prenosis (United States), Sinai Hospital, LifeBridge Health, Beth Israel Deaconess Medical Center, Sisters of Mercy Health System, Beaumont Hospital, Royal Oak, Corewell Health William Beaumont University Hospital, OSF Saint Francis Medical Center, Harvard University, Emory University, Jesse Brown VA Medical Center, Mercy Hospital, University of Illinois Urbana-Champaign
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,762
Funder: National Institutes of Health
OpenAlex ID: W7131635698