Year: 2026
Type: article
Abstract: Extremist content spreads quickly on social media, so moderation needs classifiers that are accurate and whose confidence can be trusted. We study the Domain-Aware Transformer Ensemble (DATE): a BERTweet-base classifier (pre-trained on 850M English tweets), contextual-substitution and random-edit augmentation applied only inside each training fold, and a soft-voting ensemble of ten fold models. We evaluate on the Social Media Extremism Detection dataset (2,250 training posts, 749 labelled test posts) with a leakage-controlled protocol and report three kinds of evidence separately. (i) Cross-validation: single DATE models reach 0.822 accuracy (±0.027) on held-out folds. (ii) Held-out test posts: the ten-model ensemble reaches 0.843 accuracy (95% CI 0.814–0.868), macro-F1 0.744 and ROC-AUC 0.863, and its expected calibration error is 0.030 against 0.064 for single models (paired bootstrap difference \(-0.029\) , CI \(-0.046\) to \(-0.010\) ); over five training seeds, the ensemble is better calibrated than its members in every run (mean ECE 0.037 against 0.061). (iii) Competition: the submitted DATE ensemble scored 0.8507 on the private leaderboard of the Kaggle challenge ( \(\approx\) 375 posts), tied for first place with two other teams among 164 teams; this margin is within sampling noise. Controlled ablations repeated over five seeds show that augmentation, class weights and the cosine schedule do not improve accuracy at full data size, although augmentation helps when training data is scarce; BERTweet beats RoBERTa-base in all five seeds (1.9 points on the test posts), and a TF-IDF baseline reaches comparable test accuracy. The study is limited to one English dataset whose annotation details (number of annotators, agreement) are not reported, and the test set has a different class balance from the training set.
Source: Scientific Reports
Institutions: Islamia University of Bahawalpur, Bahauddin Zakariya University, Yeungnam University, Universidad Europea del Atlántico, Universidad Internacional Iberoamericana, Fundación Universitaria Internacional de Colombia, Universidad Internacional Iberoamericana, Universidade Internacional do Cuanza, Universidad de la Romana
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7234255339
