Optimizing Variational Physics-Informed Neural Networks Using Least Squares
Year: 2025
Type: article
Abstract: Variational Physics-Informed Neural Networks often suffer from poor convergence when using stochastic gradient-descent-based optimizers. By introducing a least squares solver for the weights of the last layer of the neural network, we improve the convergence of the loss during training in most practical scenarios. This work analyzes the computational cost of the resulting hybrid least-squares/gradient-descent optimizer and explains how to implement it efficiently. In particular, we show that a traditional implementation based on backward-mode automatic differentiation leads to a prohibitively expensive algorithm. To remedy this, we propose using either forward-mode automatic differentiation or an ultraweak-type scheme that avoids the differentiation of trial functions in the discrete weak formulation. The proposed alternatives are up to one hundred times faster than the traditional one, recovering a computational cost-per-iteration similar to that of a conventional gradient-descent-based optimizer alone. To support our analysis, we derive computational estimates and conduct numerical experiments in one- and two-dimensional problems.
Language: English
Subfield: Statistical and Nonlinear Physics
Field: Physics and Astronomy
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,170
Funders: Ministerio de Ciencia, Innovación y Universidades, Consejería de Transformación Económica, Industria, Conocimiento y Universidades, European Commission, Eusko Jaurlaritza, Euskal Herriko Unibertsitatea, Ministerio de Ciencia e Innovación, Basque Center for Applied Mathematics, Agencia Nacional de Investigación y Desarrollo, HORIZON EUROPE Framework Programme, Agencia Estatal de Investigación, Fondo Nacional de Desarrollo Científico y Tecnológico, H2020 Marie Skłodowska-Curie Actions, HORIZON EUROPE Marie Sklodowska-Curie Actions, NextGenerationEU
Awards: G1480809515, G1612158077, G1633674249, G2262748287, G3170287466, G3248909841, G3480869486, G3607574089, G4042783231, G4241082588, G542055026, G5451035857, G55232499, G6294024929, Real-time inversion using self-explainable deep learning driven by expert knowledge, G7436716198, G7748812462, G7829631415, G8305580143, G8471817460, G8939903727, STABILIZED DISCRETIZATIONS VIA RESIDUAL MINIMIZATION ON DISCRETE DUAL NORMS
OpenAlex ID: W4408072216