Year: 2026
Type: conference-paper
Abstract: We present FLUKE (framework for linguistically-driven and task-agnostic robustness evaluation), a framework for assessing model robustness through systematic minimal variations of test data.FLUKE introduces controlled variations across linguistic levels -from orthography to dialect and style -and leverages large language models (LLMs) with human validation to generate modifications.We demonstrate FLUKE's utility by evaluating both fine-tuned models and LLMs across six diverse NLP tasks (four classification and two generation tasks), and reveal that (1) the impact of linguistic variations is highly task-dependent, with some tests being critical for certain tasks but irrelevant for others; (2) LLMs still exhibit significant brittleness to certain linguistic variations, with reasoning LLMs surprisingly showing less robustness on some tasks compared to base models, and scaling improving robustness only for surface-level modifications; (3) models are overall more brittle to natural, fluent modifications such as syntax or style changes (and especially to negation), compared to corruption-style tests such as letter flipping; (4) the ability of a model to use a linguistic feature in generation does not correlate to its robustness to this feature on downstream tasks.These findings highlight the importance of systematic robustness testing for understanding model behaviors.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140094739