Work

Attention Under Attack: Analog Noise Effects and Mechanistic Vulnerabilities in Transformer Models

Year: 2026

Type: conference-paper

Abstract: Analog in-memory computing (AIMC) offers substantial efficiency gains for transformer inference but introduces hardware-induced noise that can distort attention behavior.Prior studies primarily focus on AIMC evaluations for vision tasks and CNN-based models.They largely overlook how hardware-induced noise perturbs internal attention dynamics in NLP models.In this work, we present the first fine-grained analysis of analog vulnerability in pretrained transformers, examining projection submodules, attention heads, and layerwise dynamics across multiple NLP tasks.Results show that query (Q), key (K), and value (V) projections are the most sensitive components, while feed-forward layers remain comparatively robust.Also, analog noise yields depth-dependent degradation in higher layers, leading to scattered attention and disrupted token routing.This pre-deployment analysis mitigates potential resource misuse before physical deployment and offers practical guidance for designing noise-resilient analog NLP transformers.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-short.21

Authors: Mafizur Rahman, Lijun Qian

Institution: Prairie View A&M University

Language: English

Topic: Advanced Memory and Neural Computing

Subfield: Electrical and Electronic Engineering

Field: Engineering

Domain: Physical Sciences

Open Access status: hybrid

Funder: Army Research Office

Awards: G3294896131, G7452299184