Work

Assessing a combined human coding and natural language processing method for qualitative analysis in physics education research

Year: 2025

Type: conference-paper

Abstract: As artificial intelligence becomes increasingly integrated into education research, it is critical to assess the capabilities and limitations of such tools in complex disciplinary contexts.This study explores the effectiveness of natural language processing (NLP) for topic identification using the BERTopic package to analyze undergraduate quantum mechanics syllabi.By comparing AI-generated topics with human-coded themes from a corpus of syllabi from 50 US institutions, we find that while NLP reliably identifies broad and structural elements of syllabi (e.g., policies, textbook references), it is limited in its ability to capture the nuanced and context-dependent content typical of upper-level physics instruction.The AI-derived topics were often underrepresented and diverged from expert human interpretation, particularly in areas requiring deep disciplinary knowledge.Our findings support the use of NLP as a complementary tool for theme generation and cross-validation, but emphasize the continued necessity of expert human analysis for rigorous and meaningful educational research.

Source: Physics Education Research Conference (PERC)

DOI: 10.1119/perc.2025.pr.buzzell

Authors: Alexis Buzzell, Ramón S. Barthelemy, Timothy J. Atherton

Institutions: University of Utah, Tufts University

Language: English

Topic: Computational and Text Analysis Methods

Subfield: General Social Sciences

Field: Social Sciences

Domain: Social Sciences

SDG: Quality education

Open Access status: hybrid

OpenAlex ID: W4415600140