Masked Spikformer: Gaussian based and Random Spike Masking for Energy-Efficient Spiking Transformers
Year: 2025
Type: conference-paper
Abstract: Spiking Neural Networks (SNNs) are increasingly explored for their energy efficiency and biological plausibility, offering a compelling alternative to conventional artificial neural networks in neuromorphic applications. However, even Spik-former a fully spiking adaptation of the Transformer model, can exhibit significant computational redundancy due to excessive spike activity, resulting in non-negligible energy consumption. In this paper, we introduce Masked Spikformer, a unified framework for regulating temporal spike activity in spiking Transformers through three complementary masking strategies: Random Spike Masking (RSM), Gaussian-Based Spike Masking (GSM), and Gaussian-Based Spike Weighting (GSW). These approaches encompass both binary masking and continuous, learnable temporal weighting. Our method is integrated into fully spiking architectures and applied consistently during both training and inference. Experimental results on neuromorphic benchmarks (CIFARIO-DVS and DVS128 Gesture) show that the proposed masking strategies significantly reduce energy consumption while maintaining high classification performance. Notably, the best accuracy on DVS128 Gesture is achieved by RSM with 70% masking, while GSW(i), the inverse Gaussian weighting variant, attains the highest accuracy on CIFARIO-DVS. RSM also provides the lowest energy consumption across both datasets, highlighting the effectiveness of temporal sparsity for energy-efficient spiking Transformers.
Language: English
Subfield: Electrical and Electronic Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4416157882
FWCI:
Cites:
Cited by:
Related to: