Work

Addressing the “elephant in the room” of AI clinical decision support through organisation-level regulation

Year: 2022

Type: article

Abstract: Consider the following proprietary artificial intelligence (AI) algorithm products: (1) continual monitoring to predict likelihood of acute kidney injury (Dascena Previse, Dascena, USA); (2) predicting significant events for patients on intensive care (CLEWICU, CLEW Medical, Israel);(3) an early warning system for acute inpatient deterioration (Wave Clinical Platform, Excel Medical, USA); and (4) using electronic health record (EHR) data to predict likelihood of sepsis (Epic Sepsis Model, Epic Systems Corporation, USA).These algorithms provide early signals of potentially treatable events using real-time clinical data.However, the first three are considered software as a medical device (SaMD) under oversight of the US Food & Drug Administration (FDA) [1-3].In contrast, the last has undergone no visible regulatory scrutiny [4] and demonstrates minimal data or algorithmic transparency [5], yet is actively used in hundreds of hospitals in the United States that employ the Epic EHR [6].In 2021, an independent evaluation of this sepsis model demonstrated poor performance (relative to vendor reported metrics), failing to identify 67% of patients with sepsis, with a positive predictive value of 12% and substantial alert burden for clinicians [7].Other technology vendors [8-10] and healthcare providers [11,12], are also known for hosting development and operationalisation of proprietary algorithmic clinical decision support (CDS).It is likely that many AI implementations fly under the radar.The elephant then, sitting next to the FDA, is the different consideration given to algorithmic devices for market, and proprietary algorithms developed within existing EHR (traditionally outside of FDA scope [13]).With increasing appearance of CDS, the 21st Century Cures Act 2016 introduced statutory SaMD definitions, such that a non-device CDS is defined by provision of recommendations where clinicians can review the basis for predictions.This could arguably be applied to many algorithms classified as SaMD, and proposed 2019 guidance clarified that CDS must only "recommend" (rather than "drive") decisions, while creating no intention that "the healthcare provider rely primarily on any of such recommendations to make a clinical diagnosis or treatment decision. .." [14].This distinction remains imprecise.

Source: PLOS Digital Health

DOI: 10.1371/journal.pdig.0000111

Authors: Joe Zhang, Heather Mattie, Haris Shuaib, Tamishta Hensman, James T. Teo, Leo Anthony Celi