Work

Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis

Year: 2025

Type: conference-paper

Abstract: Qualitative analysis is typically limited to small datasets because it is time-intensive.Moreover, a second human rater is required to ensure reliable findings.Artificial intelligence tools may replace human raters if we demonstrate high reliability compared to human ratings.We investigated the inter-rater reliability of state-of-theart Large Language Models (LLMs), ChatGPT-4o and ChatGPT-4.5-preview, in rating audio transcripts coded manually.We explored prompts and hyperparameters to optimize model performance.The participants were 14 undergraduate student groups from a university in the midwestern United States who discussed problemsolving strategies for a project.We prompted an LLM to replicate manual coding, and calculated Cohen's Kappa for inter-rater reliability.After optimizing model hyperparameters and prompts, the results showed substantial agreement (κ > 0.6) for three themes and moderate agreement on one.Our findings demonstrate the potential of GPT-4o and GPT-4.5 for efficient, scalable qualitative analysis in physics education and identify their limitations in rating domain-general constructs.

Source: Physics Education Research Conference (PERC)

DOI: 10.1119/perc.2025.pr.borse

Authors: Nikhil Sanjay Borse, Ravishankar Chatta Subramaniam, N. Sanjay Rebello

Institution: Purdue University West Lafayette

Language: English

Topic: Computational and Text Analysis Methods

Subfield: General Social Sciences

Field: Social Sciences

Domain: Social Sciences

SDG: Quality education

Open Access status: hybrid

Funder: National Science Foundation