Prescreening Layer: Lightweight Key Modulation for Attention Mechanisms
Year: 2026
Type: preprint
Abstract: Attention mechanisms have become ubiquitous in modern deep learning architectures, yet their default formulation treats all key dimensions uniformly across sequencepositions. We introduce the Prescreening Layer (PSL),a lightweight, learnable module that dynamically modulates key tensors prior to query–key interaction. PSL aggregates temporal information via mean pooling, passesthe result through a small two-layer MLP with a sigmoidgate, and applies the resulting modulation as a residual addition to the original keys. Our experiments ona controlled transformer setup show that a minimal PSLconfiguration (hidden dimension 4) improves final lossby 4.9% over the baseline with only 0.58% additionalparameters. Larger PSL configurations degrade performance, suggesting that highly compressed prescreeningis sufficient and beneficial. We provide detailed analysisof gate statistics, gradient flow, and computational overhead
Author: Nar, Furkan
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7143014808
Cites:
Cited by:
Related to: