Work

Costs and Benefits of AI-Enabled Topic Modeling in P-20 Research: The Case of School Improvement Plans

Year: 2025

Type: conference-paper

Abstract: As generative AI tools become increasingly integrated into educational research workflows, large language models (LLMs) have shown substantial promise in automating complex tasks such as topic modeling.This paper presents a user study that evaluates AI-enabled topic modeling (AITM) within the domain of P-20 education research.We investigate the benefits and trade-offs of integrating LLMs into expert document analysis through a case study of school improvement plans, comparing four analytical conditions.Our analysis focuses on three dimensions: (1) the marginal financial and environmental costs of AITM, (2) the impact of LLM assistance on annotation time, and (3) the influence of AI suggestions on topic identification.The results show that LLM increases efficiency and decreases financial cost, but potentially introduce anchoring bias that awareness prompts alone fail to mitigate. 1

Source: Workshop on Innovative Use of NLP for Building Educational Applications

DOI: 10.18653/v1/2025.bea-1.34

Authors: Syeda Sabrina Akter, Seth Hunter, David Woo, Antonios Anastasopoulos

Institutions: George Mason University, University of Utah, Athena Research and Innovation Center In Information Communication & Knowledge Technologies

Language: English

Topic: Computational and Text Analysis Methods

Subfield: General Social Sciences

Field: Social Sciences

Domain: Social Sciences

SDG: Quality education

Open Access status: hybrid

Funders: National Science Foundation, U.S. Department of Defense