An Effective Metaheuristic-Based Structured Pruning Method for Transformer in Short-Term Load Forecasting
Year: 2024
Type: conference-paper
Abstract: The transformer has good predictive performance for short-term load forecasting, but its computational cost is higher than traditional machine learning models, such as random forests.Structured pruning techniques reduce computational load and model size, enabling transformer deployment in resource-constrained environments.This paper proposes a structured pruning framework for the transformer that decomposes the transformer into multiple substructures.It evaluates the importance of each substructure unit and employs a metaheuristic algorithm to find the optimal pruning rate for each substructure.Unlike traditional weight-based importance evaluation methods, This paper employs limited training epochs to learn additional parameters and identify the most critical units.Experimental results show that, compared to other standard methods available for metaheuristic-based pruning, the proposed approach achieves better performance while maintaining high pruning rates in short-term load forecasting tasks.Using a metaheuristic algorithm, floating-point operations and parameter count of the transformer are reduced by over 88% with slight performance improvements.
Authors: Luo-Fan Wu, Gagan Vishwakarma, Chun-Wei Tsai
Institution: National Sun Yat-sen University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Award: G5323464858
OpenAlex ID: W4415168068