Work

Analysing heavy-tail properties of stochastic gradient descent by means of stochastic recurrence equations

Year: 2026

Type: article

Abstract: In recent works on the theory of machine learning, it has been observed that heavy tail properties of stochastic gradient descent (SGD) can be studied in the probabilistic framework of stochastic recursions. In particular, Gürbüzbalaban et al. (2021) considered a setup corresponding to linear regression for which iterations of SGD can be modelled by a multivariate affine stochastic recursion upper X Subscript n Baseline equals upper A Subscript n Baseline upper X Subscript n minus 1 Baseline plus upper B Subscript n X n = A n X n − 1 + B n $X_n=A_nX_{n-1}+B_n$ for independent and identically distributed pairs left parenthesis upper A Subscript n Baseline comma upper B Subscript n Baseline right parenthesis ( A n , B n ) $(A_n,B_n)$ , where upper A Subscript n A n $A_n$ is a random symmetric matrix and upper B Subscript n B n $B_n$ is a random vector. However, their approach is not completely correct and, in the present paper, the problem is put into the right framework by applying the theory of irreducible-proximal matrices.

Source: Journal of Applied Probability

DOI: 10.1017/jpr.2025.10036

Authors: Ewa Damek, Sebastian Mentemeier

Institutions: University of Wrocław, University of Hildesheim

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $3,550

Funder: Deutsche Forschungsgemeinschaft