llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R
Year: 2026
Type: preprint
Abstract: Text classification has become a central measurement problem in psychology and the social sciences, and the arrival of large language models (LLMs) multiplied the number of available strategies, from contextual encoders and embedding services to prompted generative models and, more recently, general-purpose classification models such as Jev, which return probabilities for typed questions about a text. The present paper introduces the R package llmClassificR, which brings these strategies together in a single, consistent interface, and shows how they can be combined with network psychometrics, complexity science, and modern uncertainty quantification. The package implements (a) unsupervised classification of texts with exploratory graph analysis applied to the embeddings of local encoder models, decoder LLMs, and embedding APIs, including a prime-weight encoding that combines the networks estimated from every transformer layer into a single multiplex representation; (b) supervised classification heads on frozen embeddings, evaluated by cross-validation; (c) prompted classification with generative LLMs, with shuffled batching, label validation, strict JSON Schema outputs, ensembles, and self-consistency; (d) retrieval-augmented, automated, and human-guided topic discovery; (e) Jev-based classification with Choice, Noul, and Score questions; (f) do-it-yourself Jev-like classifiers built from label descriptions and local or API embeddings; and (g) calibration, temperature scaling, Platt scaling, selective classification, and conformal prediction sets. The tutorial uses synthetic texts generated by LLMs, descriptions of emotional situations labeled by the people who wrote them (EmpatheticDialogues), and Reddit comments labeled by several raters (GoEmotions), which makes it possible to show how methods that look nearly perfect on synthetic data behave on real text, how the reliability of the labels limits the accuracy that any classifier can demonstrate, and how model uncertainty relates to human disagreement.
Author: Hudson Golino
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225100604