Work

Beyond the Mean: Evaluative Heterogeneity, Teaching Context, and the Evidential Meaning of Student Evaluations

Year: 2026
Type: preprint
Abstract: Student evaluations of teaching (SETs) remain widely used in consequential institutional decisions, yet their interpretation typically relies on mean scores that obscure differences in student agreement and the contexts in which evaluations are produced. This study examines evaluation level and evaluative heterogeneity as distinct but complementary properties of SET evidence and investigates how each is structured across instructors, teaching contexts, and time. We analyze four years of university-wide administrative SET data using cross-classified and within-between models, complemented by context-adjusted institutional classifications. Evaluation level showed substantially greater instructor-associated clustering than heterogeneity, whereas heterogeneity was predominantly occasion-specific. Teaching conditions were differently associated with the two properties, and demographic associations were context dependent rather than uniform. Contextual adjustment rarely changed institutional classifications but mattered primarily near the decision boundary, where similar mean scores could rest on different evidential profiles. These findings show that a SET mean does not fully characterize the evidence supporting it. Response volume, evaluative heterogeneity, teaching context, and longitudinal stability can qualify the evidential weight attached to mean ratings, particularly when SETs inform consequential institutional decisions.
Language: English
Subfield: Education
Open Access status: green
OpenAlex ID: W7236431679
Cites:
Cited by:
Related to: 10