Work

Early Prediction of Dynamic Sparsity in Large Language Models

Year: 2025

Type: conference-paper

Abstract: Large language models are powerful but computationally very expensive.We investigate dynamic sparsity in attention mechanisms, using the OPT model as a case study.We explore the dynamic nature of redundancy in attention heads and analyze which components of the model provide sufficient information to predict sparsity effectively.Our findings highlight the norm of attention outputs as a reliable criterion for ranking head importance.We systematically evaluate embeddings across layers and time steps, showing that dynamic sparsity predictions can be achieved early in the model pipeline with minimal loss in accuracy.By elucidating the mechanisms underlying dynamic sparsity, this work lays a foundation for more efficient and scalable transformer models.

Source: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN)

DOI: 10.14428/esann/2025.es2025-97

Authors: Reza Sedghi, Amit Kumar Pal, Anand Subramoney, David Kappel

Institutions: Ruhr University Bochum, University of London, Royal Holloway University of London, Bielefeld University

Language: English

Topic: Advanced Neural Network Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: Bundesministerium für Bildung und Forschung, Gauss Centre for Supercomputing

Award: Verbundprojekt: Energiesparende verteilte Sensorsysteme für maschinelles Sehen - EVENTS -