Work

Variable Projection algorithms: Theoretical insights and a novel approach for problems with large residual

Year: 2025

Type: article

Abstract: This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification , image processing , and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman’s VP algorithm can achieve a similar convergence rate as the Golub & Pereyra’s form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, inspired by these theoretical insights, we design a refined VP algorithm, termed VPLR, to address separable nonlinear optimization problems with large residual. This algorithm enhances convergence performance by addressing the coupling relationship between parameters in separable models and continually refining the approximated Hessian matrix to counteract the influence of large residual. The effectiveness of this refined algorithm is corroborated through numerical experiments.

Source: Automatica

DOI: 10.1016/j.automatica.2025.112300

Authors: Guangyong Chen, Peng Xue, Min Gan, Jing Chen, Wenzhong Guo, C.L. Philip Chen

Institutions: Fuzhou University, Qingdao University, Jiangnan University, South China University of Technology

Language: English

Topic: Statistical and numerical algorithms

Subfield: Applied Mathematics

Field: Mathematics

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $3,760

Funders: National Natural Science Foundation of China, Natural Science Foundation of Fujian Province