Reusing Polynomial Solvers for Implicit Neural Networks
Year: 2026
Type: preprint
Abstract: Preprint; not peer reviewed. Computing gradients in implicit neural networks requires repeated linear solves. A polynomial solver can reuse its coefficients, but changes in the input and network weights make it difficult to ensure that the same polynomial remains accurate. We study this problem for tanh equilibrium layers with explicit weight constraints. A change of variables places all backward systems within one common numerical-range region, so the polynomial can be constructed once and reused with error bounds from existing matrix-function theory. We evaluate the solver in small training tasks and on 108 fixed systems, including 72 systems extracted from trained models. On those 72 systems, at relative residual tolerance 1e-5, the polynomial gives a 2.56–3.16-fold speedup over restarted GMRES when construction cost is spread over 1000 solves, while matching BiCGStab in runtime; the training tasks show similar task metrics across backward solvers. These CPU results support polynomial reuse when matrix products are inexpensive, while leaving its benefit in larger neural-network training to be established.
Author: Wu, JiaBao
Institution: Shenzhen University
Language: English
Topic: Matrix Theory and Algorithms
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225188418