Work

A line-search based SGD algorithm with Adaptive Importance Sampling

Year: 2025

Type: article

Abstract: Stochastic Gradient methods are widely used in the field of supervised learning associated with big data. In this context, importance sampling-based algorithms have been proposed to minimize the variance of the stochastic gradient by introducing practical strategies to approximate the optimal sampling distribution, which is otherwise only theoretically accessible. In this paper, we propose a scheme that combines stochastic gradient descent with adaptive importance sampling with automatic step-size selection based on a stochastic Armijo-type line-search. This approach makes the method robust to the choice of the initial step-size, which would otherwise require a tuning phase that is computationally expensive or even impractical in certain big data scenarios. Moreover, we introduce different mini-batch variants to foster the practical acceleration of the original scheme. Finally, numerical experiments are presented on real datasets to validate the proposed method in the context of supervised classification problems.

Source: Journal of Computational and Applied Mathematics

DOI: 10.1016/j.cam.2025.117120

Authors: Filippo Camellini, Serena Crisci, Anna De Magistris, Giorgia Franchini

Institutions: University of Modena and Reggio Emilia, University of Campania "Luigi Vanvitelli"

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $3,250