WiP: Beyond Tokenization: Structure-Aware Attention and Hybrid Backbone for Wireless Foundation Models
Year: 2026
Type: conference-paper
Abstract: Plume showed that protocol-aware tokenization lets a compact 140M-parameter model rival frontier LLMs on 802.11 trace understanding [6]; Plume-2 showed that architecture and scale move the needle, yet tokenization still dominates [7]. This work asks the next question: does the training architecture itself matter? We study two architecture-level levers that do not touch the tokenizer. First, a structure-aware attention bias A = QKT + b, where b injects packet-trace geometry from the existing separator tokens. Second, a hybrid Mamba/attention backbone that interleaves linear-time state-space layers with a few dense global attention layers [4, 5]. On five real-world failure categories, the five configurations span 96.1–98.6% Top-1 accuracy. Mamba and the reduced-attention hybrid achieve 1.7× and 1.5× the Transformer's forward-pass throughput while trailing its accuracy by 2.1 and 0.6 points, respectively. In our preliminary single-run results, the structure bias improves Top-1 by 0.4 points on the Transformer and 0.8 points on the hybrid. The biased hybrid reaches 98.4% Top-1 at 13.8k tok/s, slightly exceeding the Transformer baseline's 98.2% while providing 1.48× its throughput. These results suggest that architecture is primarily an efficiency lever and that packet-relative bias can recover the hybrid's accuracy.
Authors: Swadhin Pradhan, Shazal Irshad, Jerome Henry
Institutions: Cisco Systems (United States), Cisco Systems (China)
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7223098683