Work

The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function

Year: 2025

Type: conference-paper

Abstract: Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation.By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate.Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves.We also present results on how the learning rate affects the overall behavior of the network and its equilibria, showing different learning regimes and critical values of the learning rate.

Source: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN)

DOI: 10.14428/esann/2025.es2025-94

Authors: Otavio Citton, Frederieke Richert, Michael Biehl

Institution: University of Groningen

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funder: Nederlandse Organisatie voor Wetenschappelijk Onderzoek

Awards: G5093044862, The Role of the Activation Function in Feedforward Learning Systems (RAFFLES)