Work

Hermite polynomials facilitating on-line learning analysis of layered neural networks with arbitrary activation function

Year: 2025

Type: article

Abstract: Following the standard statistical mechanics methods we analyze the training by online stochastic gradient descent of two-layer neural networks in a student–teacher scenario. We focus on understanding the role that different activations play, in particular mismatches between the student and the teacher, in these learning scenarios. By expanding the activation functions in the Hermite polynomial basis, we are able to effectively approximate the relevant integrals with much less computational effort than naive numerical integration. Moreover, we also extend the framework to study scenarios of concept drift and weight decay also with arbitrary activation functions. All these extensions comprise relevant advances in the field, allowing us to obtain analytical results for more realistic scenarios.

Source: Neurocomputing

DOI: 10.1016/j.neucom.2025.131328

Authors: Otavio Citton, Frederieke Richert, Michael Biehl, Michiel Straat

Institutions: University of Groningen, University of Birmingham, Bielefeld University

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $2,670

Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek, Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen