Refusal, Confidence, and Explanation: A Cross-Modal Audit of Uncertainty-Expression Failures in Small Open-Weight Language and Vision-Language Models
Year: 2026
Type: article
Abstract: Safety evaluations usually test refusal, abstention, and stated confidence separately. We treat them as one property, appropriate uncertainty expression, and audit it in text and vision across six small open-weight LLMs and five vision-language model (VLM) configurations. On a 240-item Korean benchmark, judged refusal is higher for harmful than for benign-looking requests in every model and under every labeling rule, but model-level orderings depend on the rule (primary rule: zephyr-7b-beta refuses harmful requests least, 42.5%, and benign ones most, 18.8%). Against 150 blinded labels from one annotator, the judges agree on refusal (κ=0.63) but overstate benign-prompt refusal (10.4% vs. 3.6%). Replacing sensitive words in benign prompts did not lower refusal (−1.9 pp [−5.4, 1.7]). Two VLMs abstain far more on out-of-distribution images, yet their stated confidence when they answer is equivalent within ±5 points to that on in-distribution images. On six MedMNIST v2 tasks, VLM accuracy is near chance (13.2–15.6% vs. 14.6% expected; ConvMixer reference on the same images: 67.8%) at ≈80% stated confidence, and stated evidence locations match CNN saliency only at chance level. Scaling from 2B to 8B shrinks the confidence gap without improving accuracy. At a fixed generation budget, only a step-by-step instruction to assess intent and risk raised benign-prompt refusal (+9.0 pp); generic step-by-step instructions did not. Neither scale nor chain-of-thought prompting resolves these failures.
Source: Electronics
Author: Minjong Cheon
Institution: Korea National Open University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7225408169