AI-Assisted Sentiment Analysis of Student Voice: A Human-Validated Comparison of Lexicon and Transformer-Based Tools
Year: 2026
Type: article
Abstract: Open-ended survey responses offer rich insights into the student experience, yet analyzing this qualitative data at scale is often time-consuming and labor-intensive. This study evaluates whether off-the-shelf sentiment analysis tools can validly classify such data for institutional assessment. Analyzing 1,220 free-text responses from undergraduate students at a private liberal arts university, we compared the performance of four tools: two lexicon-based (TextBlob and VADER) and two transformer-based (BERT-nlptown and RoBERTa-cardiffnlp) against manual coding. Evaluation metrics included accuracy, macro F1 score, and Cohen’s kappa (κ). While BERT-nlptown performed best, it achieved only moderate agreement with human coding (κ = .42). Furthermore, lexicon-based approaches misclassified roughly 20% of responses due to limited vocabulary coverage. Notably, all tools struggled to accurately classify negative, neutral and mixed sentiments. We recommend conducting lexicon-coverage diagnostics, few-shot prompting, domain-specific finetuning, and human-in-the-loop oversight to ensure valid, reliable, and trustworthy institutional assessment.
DOI: 10.61669/001c.172638
Authors: Wei You, Kevork Horissian
Institution: Bucknell University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7222878592
FWCI:
Cites:
Cited by:
Related to: