Development and Preliminary Evaluation of the Patient–Artificial Intelligence Rating Instrument
Year: 2026
Type: preprint
Study design: Observational Study
Abstract: Importance Patients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment. Objective To develop and preliminarily evaluate a patient instrument assessing the quality of patient–AI health conversations. Design Cross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews. Setting An anonymous online survey. Participants A convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers. Exposures Participants’ own prior health-related conversations with AI tools of their choosing. Main Outcomes and Measures The Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach α, with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item–total correlations, inter-item structure, and between- and within-participant variation. Results Fourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach α was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item–total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach α. The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09). Conclusions and Relevance In this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use. Key Points Question Can a patient instrument measure the quality of patients’ health-related conversations with artificial intelligence (AI)? Findings In this cross-sectional study, 14 participants provided 46 ratings of their health-related AI conversations using a 12-item scale and a perceived-harm item (the Patient AI Rating (PAIR) instrument). The patient metric showed good internal consistency, and scores varied both by raters and across conversations rated by the same participant. Meaning The PAIR instrument offers a preliminary approach to incorporate patient perspective and experience into AI evaluation – a measure largely missing from current AI assessment; larger studies are needed to establish validity and generalizability.
Source: medRxiv
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7223222110