Work

Sparse and Structured Hopfield Networks

Year: 2024

Type: preprint

Abstract: Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2402.13725

Authors: Santos, Saul, Niculae, Vlad, McNamee, Daniel, Martins, Andre F. T.

Institutions: University of Lisbon, Instituto de Telecomunicações, Instituto Superior Técnico, University of Amsterdam, Champalimaud Foundation

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Nederlandse Organisatie voor Wetenschappelijk Onderzoek, HORIZON EUROPE European Research Council

Award: DEep COgnition Learning for LAnguage GEneration

OpenAlex ID: W4392118265

Cites:

Cited by: 1

Related to: 10