Work

Unpacking the Unreliability of Interrater Reliability of Pediatric Auscultation

Year: 2025

Type: article

Abstract: Although lower respiratory tract infections (LRTIs) remain the leading causes of morbidity and mortality among children aged 2 to 59 months globally,1 there is no unified approach to the diagnosis of LRTIs. Nevertheless, the correct diagnosis of children with LRTI is crucial as it informs treatment (eg, antibiotics for community-acquired pneumonia with suspected bacterial etiology vs supportive care for bronchiolitis). Moreover, mounting evidence suggests growing rates of antimicrobial resistance, which may be due, in part, to widespread use of antibiotics.2,3 Additionally, there is evidence that LRTIs among children may be caused predominately by viruses,4,5 although current recommendations support the use of antibiotics for clinically diagnosed community-acquired pneumonia.6–8 Thus, even more critical than the long-asked question “Does this child with respiratory symptoms have pneumonia?”9 is the question “Does this child with a suspected LRTI need antibiotics?”The definitive etiologic diagnosis of LRTI can only be made through lung biopsy, which is highly invasive and rarely needed.6 As a result, although not recommended by several national and international groups, chest radiography is often used in the assessment of children with LRTI.6–8,10 Chest radiography is used in as many as 80% of pediatric emergency department encounters for children with respiratory symptoms to assess for the presence of LRTI that may require antibiotics.11 However, chest radiography has been referred to as a “bronze standard” due to its suboptimal interrater reliability among radiologists and its inability to differentiate between viral and bacterial etiologies of conditions like community-acquired pneumonia.12,13 Point-of-care lung ultrasonography (PLUS) use is expanding in pediatric emergency departments, yet studies assessing its test characteristics have compared PLUS with chest radiography, leaving its accuracy to diagnose LRTI subject to the same scrutiny as the “bronze reference standard.”14,15 However, it is worth noting that ultrasonography may detect smaller consolidations than those visible on chest radiography.15 Instead of routine radiography or PLUS use, the Infectious Diseases Society of America (IDSA), the British Thoracic Society, and the World Health Organization encourage a clinical approach to diagnose LRTI, with a reliance on the assessment of tachypnea, chest indrawing, and on the presence or absence of wheezing.6–8,10In this issue of Hospital Pediatrics, Ramgopal et al report the results of an important study that calls into question the interrater reliability of a key part of the clinical assessment, which is the identification of both normal and abnormal breath sounds among children.16 Ramgopal et al conducted a cross-sectional, online survey study to assess the interrater reliability of pediatric emergency medicine clinicians’ impression of 15 normal and abnormal breath sounds using audio recordings previously obtained using a digital stethoscope. This study included 106 respondents from 6 academic pediatric emergency departments in the United States. Somewhat disappointingly, the authors report that only normal breath sounds had sufficient interrater reliability based on an a priori determined cutoff of the lower limit of the 95% CI for Fleiss’ kappa greater than 0.40. Ramgopal et al reported that other auscultatory findings (ie, wheezing, rhonchi, stridor, and crackles) used to diagnose common conditions such as asthma, bronchiolitis, and community-acquired pneumonia had unacceptable interrater reliability.The poor interrater reliability described in the study by Ramgopal et al contrasts with previous findings from Florin et al, which showed moderate reliability for wheezing (meeting the same ≥0.40 Fleiss’ kappa CI threshold).17 A key difference between these 2 studies was that most assessors in the Florin study (128 clinicians at a single pediatric emergency department in the United States) knew the radiographic findings when auscultating, which may have inflated reliability through confirmation bias. Moreover, clinicians identified sounds independently, so interrater reliability may have reflected shared general recognition of pathology rather than consistency in describing auscultatory features.17 Moreover, the results from a smaller study including 20 pediatric patients with bronchiolitis suggest fair agreement (Fleiss’ kappa 0.21–0.40) for auscultatory findings between attending physicians and pediatric residents.18 The suboptimal interrater reliability of auscultatory findings reported by Ramgopal et al is important for several reasons. First, several approaches to risk differentiation for children with LRTIs incorporate auscultatory findings for identifying children at risk for severe outcomes related to LRTI.19 Second, current recommendations from the IDSA recommend the diagnosis of community-acquired pneumonia be made based on clinical findings, including auscultation for children without severe disease or evidence of hypoxemia.6 Thus, the results in the study by Ramgopal et al call into question the ability of clinicians to reliably risk differentiate or even diagnose community-acquired pneumonia based on auscultation.This study by Ramgopal et al has many strengths worth noting. First, they had an exceptionally high response rate (>80%) and they addressed a very important question that builds on, and expands, prior work on the interrater reliability of breath sounds among clinicians. Moreover, the multicenter nature of this study helps supersede inherent limitations of previous single-center studies that have assessed interrater reliability of auscultatory findings. Ramgopal et al rigorously identified common breath sounds using a publicly available repository of breath sounds. Perhaps the biggest strength of this study is that it draws attention to the need to reassess how we routinely assess breath sounds in children with suspected LRTI.This study also presents several opportunities for lessons learned to better inform future studies. First, using breath sounds that are available online may be a suboptimal approach to assessing interrater reliability. As the authors note, it may rob clinicians of important clinical cues that may affect their ultimate ability to correctly diagnose LRTIs (or upper respiratory tract infection in the context of the evaluation of stridor). Additionally, the results of this study are not an assessment of “auscultation findings in practice”; they are an assessment of “breath sounds in audio clips without clinical context.” This is crucial as clinicians should ideally assess whole patients, paying attention to clues like seasonality, patient age, underlying medical history (eg, presence of asthma), and so on. Thus, given the abysmal interrater reliability of most breath sounds demonstrated in this study, future studies assessing agreement of diagnostic auscultatory findings may avoid the use of isolated recordings of breath sounds and, instead, practically assess actual signs and symptoms in addition to auscultation in order to simulate real patient care, which is ultimately what matters most.Auscultation has long been a key part of the clinical assessment. The findings from Ramgopal et al may not necessarily suggest that we abandon auscultation. Instead, we should determine better ways to study the reliability of auscultation and to describe and classify the clinicians’ understanding of breath sounds. To determine reliability in the clinical context, future studies on auscultation should avoid the use of breath sounds without broader clinical context. Although it is too early to assess their accuracy, artificial intelligence–enhanced devices are being developed to automate the task of interpretation of breath sounds, which may enhance interrater reliability of auscultation in the future. But again, such artificial intelligence–informed auscultation may be best used with broader clinical context and not in isolation. Lastly, given that the physical examination is one of the most cost- and time-efficient diagnostic tools we have, de-emphasizing physical examination findings and subsequently overusing other modalities may result in higher costs without improved diagnosis of LRTI for children.

Source: Hospital Pediatrics

DOI: 10.1542/hpeds.2025-008588

Authors: Chris A. Rees, Jacques D. du Toit, Kristina Keitel