Grokking in the Ising Model
Year: 2025
Type: preprint
Abstract: Delayed generalization, termed grokking, in a machine learning calculation occurs when the increase in test accuracy is delayed relative to the training accuracy. This paper examines grokking in the context of a dense neural network trained to classify 2D Ising model configurations into 4 equally spaced energy regions in the presence of weight decay. Partially with the aid of novel PCA-based network layer analysis techniques, the observed behavior is interpreted as a transition from a connected network to a group of sparse subnetworks in which the number of active weights in each layer decreases monotonically with depth. This architecture reduces classification errors resulting from a multiplicity of paths. The final network layers, as in a convolutional neural network, sequentially identify global features of the input classes, which enables generalization to previously unseen patterns.
Source: arXiv (Cornell University)
Authors: Hutchison, Karolina, Yevick, David
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7103187239
Cites:
Cited by:
Related to: