Does Positional Information Causally Constrain Background Shortcut Reliance in Vision Transformer Models?
Year: 2026
Type: preprint
Abstract: 10.5281/zenodo.21971017 Vision Transformers (ViTs) match or exceed the strongest architectures for object recognition, but, like Convolutional Neural Networks (CNNs), they can exploit image backgrounds rather than foreground, focused objects. Whether a ViT's positional encoding causally governs this reliance has not been thoroughly tested in the literature, and it is also unclear whether rotary encodings (RoPE) constrain such shortcuts more effectively than absolute encodings (APE). In this work, we probe this assumption by scaling the positional signal of eight identically trained ViT checkpoints and measuring the resulting changes in background reliance on ImageNet-9. We find that attenuating the positional signal monotonically raises background reliance in every model (for example, from 5.1 to 12.8 percentage points), that the effect is invariant to the encoding scheme, and that the effect is specific to spatial information loss. These findings extend to the Waterbirds dataset (Sagawa et al., 2020). We investigate several mechanisms and find little evidence that any single mechanism carries the effect. The code, evaluation scripts, and experiment configurations are available at https://github.com/sks17/posvit
Author: Saksham Singh
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203635531
Cites:
Cited by:
Related to: