Year: 2025
Type: article
Abstract: This study explored using tensor decomposition techniques, specifically canonical polyadic decomposition (CPD) and Tucker decomposition, for optimizing transformer-based large language models (LLMs).By focusing on feed-forward layers and the final fully connected layer, we evaluated the impact of these decompositions on memory efficiency, computational performance, and overall model behavior.Experiments on the NSMC and Tiny Shakespeare datasets were used to analyze the trade-offs between compression, training time, and accuracy for different rank settings.The results reveal that Tucker decomposition provides a balanced approach to compression and computational efficiency, while CPD offers competitive performance but with higher computational costs.This work provides valuable insights into deploying tensor decomposition techniques for optimizing LLMs in resource-constrained environments.
Authors: Kyeung-joo Yoon, Jong-Lark Kim
Institution: Sogang University
Language: English
Subfield: Computational Mathematics
Field: Mathematics
Domain: Physical Sciences
Open Access status: diamond
Awards: G1251031942, G30685149
OpenAlex ID: W4412506342