Get started
Work

Sentiment Analysis of Government Mobile App Reviews: Validating Zero-Shot Classification with a Large Language Model

Year: 2026
Type: preprint
Abstract: User reviews of mobile applications provide a source of citizen feedback on digital public services, yet their automated analysis in non-English settings is constrained by the limited availability of domain-adapted classifiers. This study develops and evaluates a documented procedure for three-class sentiment classification of Czech reviews of government mobile applications using a large language model without task-specific fine-tuning. The corpus comprised 10,198 reviews of six applications. An adjudicated human reference was created for a stratified random sample of 500 reviews, and LLM performance was compared with a majority-class baseline, two lexicon-based methods, and two readily deployable encoder-based transfer models. The validation included five repeated runs under identical settings and an analysis of classification errors. Across five GPT-5.6 Sol validation runs, mean accuracy was 93.0%, mean macro-F1 was 0.827, and mean Cohen's kappa was 0.857. Macro-F1 ranged from 0.814 to 0.845, while agreement among the five runs was high (Fleiss' kappa = 0.918). The neutral class was the most challenging, with most disagreements occurring at its boundaries with negative and positive sentiment. In the full corpus, 65.9% of reviews were classified as negative, 7.9% as neutral, and 26.2% as positive. Textual sentiment was strongly associated with star ratings but was not identical to them. These findings provide corpus-specific evidence that a standardized LLM procedure can help analyze short-form non-english citizen feedback. They also underscore the need for human reference labels, stability checks, and precise documentation of the model, prompt, settings, and execution workflow.
DOI: 10.21203/rs.3.rs-11156504/v1
Language: English
Open Access status: green
OpenAlex ID: W7224407245
Cites: 44
Cited by:
Related to: 10