Separating Probability Calibration from Selective Reliability in Transformer-based Stress Detection under Domain Shift
Year: 2026
Type: article
Abstract: Probability calibration does not establish selective reliability in stress detection.This study jointly audits calibration, selective prediction, training variability, domain shift, and semantic stability on Dreaddit, controlling data leakage and accounting for post clustering.BERT, RoBERTa, and DeBERTa are evaluated across five paired training seeds using temperature scaling and label smoothing of strength 0.10, yielding 30 fits and 45 evaluation conditions.Both interventions reduce Brier score, negative log likelihood, and expected calibration error in every backbone and seed pair.Temperature scaling preserves F1 and area under the risk coverage curve in all 15 pairs.Label smoothing increases this area for RoBERTa across all seeds, while BERT and DeBERTa vary in direction.Archived domain holdout and topic stability analyses provide diagnostics conditional on fitted models.This integrated audit distinguishes findings consistent across retraining from conclusions conditional on fitted models, separating training variability from clustered test uncertainty.Generalization remains limited to Dreaddit.
Authors: Rizaldi(Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Visualization, Writing – original draft, Writing – review & editing), Kusrini(Conceptualization, Methodology, Supervision, Validation, Writing – review & editing), Ema Utami(Conceptualization, Methodology, Supervision, Validation, Writing – review & editing), I Made Artha Agastya(Conceptualization, Methodology, Supervision, Validation, Writing – review & editing)
Institution: Universitas Amikom Yogyakarta
Language: English
Topic: Mental Health via Writing
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7235609075
