A self-supervised approach to malware classification using hybrid VAE-Swin Transformer models
Year: 2026
Type: article
Abstract: Malware detection and classification are critical in cybersecurity, where accurate identification of malware families enhances threat response and mitigation. Despite significant advances in malware image-based classification, challenges such as limited capture of local and global contextual dependencies, loss of spatial information, and high computational complexity continue to hinder robust malware family classification, particularly for diverse and imbalanced datasets. This paper introduces MalVEIT, a hybrid deep Variational Autoencoder (VAE) using an attention mechanism in the bottleneck of the model for malware classification under a self-supervised learning paradigm. Malware binaries are transformed into image representations, where the selected RGB formulation is adopted based on comparative evaluation against grayscale and other channel-construction alternatives. The VAE encoder then compresses these patches into a compact latent representation parameterized by learned mean and variance. Within the bottleneck layer of the VAE, shifted-window based self-attention mechanism is implemented using Swin Transformers for hierarchical feature extraction, enabling effective capture of both local structural patterns and global contextual dependencies. The refined features are sampled and reconstructed through a VAE decoder enforcing Kullback–Leibler divergence regularization to maintain a smooth distribution. The aggregated feature maps are then passed to a fully connected layer for malware family classification. The proposed model is evaluated on six benchmarked datasets, viz., Malimg, Malevis, Big-2015, Blended, Packed PE, and Obfuscation. It consistently outperforms state-of-the-art methods, demonstrating robust multi-family malware classification across diverse datasets with data imbalance and outliers.
Authors: Anisha Mahato, Rana Majumdar, Swarup Kr Ghosh
Language: English
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,670
OpenAlex ID: W7212447906