MMLU-Pro Under Admissible Interface Perturbations: A Three-Family Stress Test with Prediction-Space Canonicalization
Year: 2026
Type: article
Abstract: Corrected version — September 2026. This version corrects the treatment of Granite choice_shuffle prediction indices. The deterministic shuffle mapping was reconstructed from the frozen source logic to recover original option identity. All nine condition-level accuracy values and within-family condition orderings remain unchanged. Seven prediction-level summary metrics are revised, and the cross-family interpretation of choice_shuffle is narrowed because Granite used a different item-wise permutation realization from Qwen and SmolLM. No new model inference run was required for this correction. This technical note presents a controlled three-family stress test on MMLU-Pro under baseline, choice_shuffle, and label_remap conditions on a locked subset of 140 items spanning 14 categories. Under the corrected analysis, MMLU-Pro remains locally usable in the tested setup while showing condition-sensitive evaluation behavior. The choice_shuffle comparison across model families should be interpreted as a comparison under the same perturbation class, not as an identical item-wise perturbation across all three families.
Author: Tavella, Danilo
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7147310351
FWCI:
Cites:
Cited by:
Related to: