A retrospective analysis comparing AIDoc and RAPIDAI in the detection of large vessel occlusions
Year: 2025
Type: article
Study design: Observational Study
Abstract: Introduction Stroke is defined as an acute and focal neurological dysfunction that occurs secondary to cerebrovascular dysfunction. In 2021, stroke was the fifth leading cause of death in the United States (US), thus signifying it serious and negative burden.[1] Strokes can either be classified into ischemic or hemorrhagic. Ischemic strokes account for about 87% of all strokes in the US and occur when blood flow to the brain is reduced due to vessel occlusion. Large vessel occlusions (LVOs), which account for roughly 30% of ischemic strokes, are characterized by an occlusion of the vertebral, basilar, carotid, or proximal middle cerebral artery (segment M1). LVOs are associated with an increased risk of death and decreased positive outcomes compared to other strokes due to a larger ischemic area.[2] Time is of essence when it comes to diagnosing and treating strokes, as patients who receive quicker treatment have better outcomes;[3] therefore, finding the most effective and quickest way to diagnose ischemic strokes is of great importance. Artificial intelligence (AI) has become a popular topic in the medical community, with the belief that AI can make physicians function more quickly and efficiently. One area of health care where AI has shown to be particularly useful is in diagnosing ischemic strokes, a leading cause of death and disability worldwide.[4,5] AIDoc is a technological company that has gained US Food and Drug Administration (FDA) approval for flagging LVOs in head computed tomography angiography (CTA) scans.[4] RAPIDAI is another AI software that has been FDA approved for analysis of CTA scans for LVOs. Herein, we present a single-center retrospective study comparing these two methods of diagnosing LVOs. In addition, we conducted a systematic review with the intent to find accurate sensitivity and specificity values of these two AI methods in diagnosing LVOs. The systematic review was performed due to inconsistent sensitivities and specificities being reported in the literature.[6–10] The decision to compare these two AI modalities over other AI tools, such as Brainomix and Viz.ai, was made due to RAPIDAI and AIDoc being the only two AI methods our institution currently uses in helping diagnose ischemic strokes. In addition, RAPIDAI is one of the most popular AI modalities currently in practice, as there are multiple studies assessing its effectiveness and comparing it against other AI modalities.[8,9,11–13] Materials and Methods Study design and patient selection This study was approved by the Institutional Review Board (STUDY00001538) on August 29, 2024. Patients were recruited from November 1, 2023, to April 1, 2024, at our institution. This was conducted in accordance with the Declaration of Helsinki.[14] The inclusion criteria for this study included patients ≥18 years of age that had an acute LVO identified on imaging and went for mechanical thrombectomy. Exclusion criteria included patients who were transferred to our hospital after vascular imaging at a referring hospital. This excluded three cases. A total of 50 patients met the criteria. Analysis AI detections of AIDoc and RAPIDAI software for the presence of LVO in these cases were compared to the independent read of the CTA images by a board-certified vascular neurologist and board-certified neuroradiologist. The absence of non-LVO cases within the dataset precluded regression analyses, and the proportions of truly positive AI detections between the two software were compared using Z-test for population proportions. The significance level was set at 0.05. Clinical trial registry This work is a retrospective analytical study. No clinical trials were involved. Results Figure 1 expresses the comparison between AIDoc and RAPIDAI in detection of acute LVOs. There were 49 observations in the AIDoc group (one case excluded due to nonavailability of AIDoc detection information), out of which 38 were detected as truly positive (38/49 = 0.78). RAPIDAI was able to detect 37 out of the 50 LVO cases as truly positive (37/50 = 0.74). The Z value for the comparison was 0.41, correlating to a P value of 0.68. The result was not significant when comparing true positive (TP) detection rates between AIDoc and RAPIDAI for the given dataset.Figure 1: Bar graph expressing the comparison between AIDoc and RAPIDAI in detection of large vessel occlusions. AI: Artificial intelligence, MCA: Middle cerebral artery, ACA: Anterior cerebral artery, PCA: Posterior cerebral artery, ICA: Internal carotid arteryDiscussion Advancements in AI have the potential to revolutionize health care and decrease the risk of missing time-sensitive findings; however, AI technologies must undergo thorough testing and evaluation to evaluate for limitations, as inadequate AI performance can hinder radiologists and put patients at risk.[15] To the authors’ best knowledge, no study has directly compared the effectiveness of AIDoc versus RAPIDAI in detection of LVOs. Our findings show that AIDoc and RAPIDAI had no significant differences when assessing their efficacy and that the false negative (FN) rate was 22% and 26%, respectively, meaning physicians need to be cognizant when using these applications. The authors also conducted a systematic review to assess the sensitivity and specificity of these two AI modalities in diagnosing acute LVOs. The inclusion criteria for our review consisted of all cases in the literature that reported TP, true negative, false positive (FP), and FN values when AIDoc and RAPIDAI were used. An electronic literature search was performed by two different authors (M. B. and C. S.). Databases searched for corresponding articles included PubMed, Google Scholar, and Open Access Journal sources. Key phrases searched included: “artificial intelligence and ischemic stroke detection,” “AIDoc and strokes,” “RAPIDAI and strokes,” “sensitivity and specificity of AIDOC,” “sensitivity and specificity of RAPIDAI.” The references of eligible studies were also manually screened to identify more potential suitable studies. The search was limited to publications written in English. Our search yielded 13 articles, not including the present study, for RAPIDAI [Figure 2] and 0 articles for AIDoc. The results of our systematic review for RAPIDAI are expressed in Table 1.Figure 2: Details of search results displayed on PRISMA flow diagramTable 1: Results of systematic review that examined the sensitivity and specificity in detection of acute large vessel occlusions by RAPIDAIResults from the systematic review showed that the sensitivity and specificity for RAPIDAI were 90.5% and 85.7%, respectively. The authors were unable to find any relevant articles in the literature regarding AIDoc use, thus making the present retrospective study the first reported sensitivity and specificity findings for AIDoc. The three most cited articles from our systematic review all reported a sensitivity value of >95%,[11,19,20] meaning our review does not coincide with the findings from these articles. The need for quick, efficient, and reliable ways to detect acute ischemic strokes with AI use results from multiple inconveniences that are currently present in the realm of stroke care: strokes are common, hospitals are busy, neuroradiologists are scarce worldwide, and delays in time results in brain loss.[4] In addition, early ischemic changes on noncontrast computed tomography can be complex and difficult to diagnose;[4,23] however, despite the advancement in AI technological development, current AI algorithms are not fully equipped to take over the practice of a neuroradiologist in diagnosing strokes, as seen in the present study. Previous studies have shown neuroradiologists can detect LVOs with approximately 89%–98% sensitivity and 95%–98% specificity.[20,24,25] Both AIDoc and RAPIDAI had a sensitivity of <80% in our retrospective study, and our systematic review yielded a sensitivity of 90.5% and specificity of 85.7% for RAPIDAI, thus meaning AIDoc and RAPIDAI are currently less reliable than neuroradiologists and that these algorithms need to be further optimized to improve their overall performance. While this manuscript discusses novel and impactful findings in the realm of an important topic, limitations exist. The biggest limitation was that it was a single-center study with a small number of subjects, thus leading to a decrease in variability of the population studied. Another limitation is that the rate of FPs was not assessed. To account for these, future studies should be multicenter and consist of a larger subject number. Future studies assessing the sensitivity and specificity of AIDoc should also be conducted as there are currently no studies in the literature reporting these findings. Another limitation is that we only compared two AI modalities. This was due to our institution currently only using these two AI methods for LVO detection. Future studies should also compare the effectiveness of other AI modalities, such as Brainomix and Viz.ai, as this would allow us to see what AI method might be most useful in helping diagnose a LVO. In 2022, Schlossman et al. compared RAPIDAI and CINA for LVO detection and found RAPIDAI to have a higher sensitivity, while CINA-LVO had a higher accuracy and specificity.[17] A more recent retrospective comparative analysis compared RAPIDAI and Viz.ai and found that Viz.ai had a higher specificity than RAPIDAI for LVO detection.[8] More comparative studies are needed to validate these findings and arrive at a definitive conclusion as to which AI imaging modality is most superior. Conclusion We retrospectively analyzed the efficiency of AIDoc versus RAPIDAI in diagnosing LVOs and performed a systematic review to assess the sensitivity and specificity of these two AI algorithms. We did not find any significant difference in detection of LVOs. However, the presence of a significant number of FNs among both AI algorithms suggests that physicians should be cautious when using these algorithms. Our systematic review highlighted the importance of more studies being needed for AIDoc to accurately assess its efficiency in diagnosing acute LVOs while also showing that both AIDoc and RAPIDAI are less reliable than neuroradiologists in acute LVO detection. Author contributions MB drafted the original manuscript and prepared Table 1; CS helped draft original manuscript and conducted the literature search for the systematic review; MA helped with the literature search for the systematic review and helped draft and revises the introduction and methods section; JS helped with the literature search of the systematic review and the sensitivity and specificity calculations for the table; OA helped originate the idea of the manuscript, reviewed the images of all the cases, and helped draft the methods section; HC (Dr. Cuellar) helped originate the idea of the manuscript, helped draft the discussions section of the manuscript, and edited the final copy; HC (Dr. Chokhawala) helped originate the idea of the manuscript, reviewed the images of all the cases, helped draft the conclusion section of the manuscript, and edited the final copy. Ethical policy and institutional review board statement This study was approved by the Institutional Review Board at LSU Health Shreveport (STUDY00001538) on August 29th, 2024 and was conducted in accordance with the Declaration of Helsinki. Patients were recruited from November 1st, 2023 to April 1st, 2024 at our institution. No consent from patients was needed for this study. Data availability statement All data generated and/or analyzed during this study are included in this published article. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.
Source: Brain Circulation
DOI: 10.4103/bc.bc_5_25
Authors: Mary Bankston, Christopher Stevens, Muhammad Abubakar Ayub, Joshua Strobel, Octavio Arevalo, Hugo Cuellar, Himanshu Chokhawala
Language: English
Subfield: Epidemiology
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W4410912935