Year: 2024
Type: article
Abstract: We have formulated a family of machine learning problems as the time evolution of parametric probabilistic models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics, memorized information (M-info) and learned information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and the parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.
Source: Entropy
DOI: 10.3390/e26020112
Author: Shervin S. Parsi
Institutions: The Graduate Center, CUNY, City University of New York
Language: English
Subfield: Statistical and Nonlinear Physics
Field: Physics and Astronomy
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,955
Awards: G6969216415, G9000242396
OpenAlex ID: W4391262396