Block Majorization-Minimization with Diminishing Radius for Constrained Nonsmooth Nonconvex Optimization
Year: 2025
Type: article
Abstract: Block majorization-minimization (BMM) is a simple iterative algorithm for constrained nonconvex optimization that sequentially minimizes majorizing surrogates of the objective function in each block while the others are held fixed. BMM entails a large class of optimization algorithms such as block coordinate descent and its proximal-point variant, expectation-minimization, and block projected gradient descent. We first establish that for general constrained nonsmooth nonconvex optimization, BMM with [Formula: see text]-strongly convex and [Formula: see text]-smooth surrogates can produce an [Formula: see text]-approximate first-order optimal point within [Formula: see text] iterations and asymptotically converges to the set of first-order optimal points. Next, we show that BMM combined with a trust-region method with diminishing radius has an improved complexity of [Formula: see text], independent of the inverse strong convexity parameter [Formula: see text], allowing improved theoretical and practical performance with “flat” surrogates. Our results hold robustly even when the convex subproblems are solved inexactly as long as the optimality gaps are summable. Central to our analysis is a novel continuous first-order optimality measure, by which we bound the worst-case suboptimality in each iteration by the first-order improvement the algorithm makes. We apply our general framework to obtain new results on various algorithms such as the celebrated multiplicative update algorithm for nonnegative matrix factorization by Lee and Seung, regularized nonnegative tensor decomposition, and the classical block projected gradient descent algorithm. Lastly, we numerically demonstrate that the additional use of diminishing radius can improve the convergence rate of BMM in many instances.
Source: SIAM Journal on Optimization
DOI: 10.1137/23m1604515
Authors: Hanbaek Lyu, Yuchen Li
Institution: University of Wisconsin–Madison
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
OpenAlex ID: W4409805047
