Year: 2015
Type: preprint
Abstract: Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of~\citet{mnih2007three}. The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by~\citet{mikolov2013efficient} and many subsequent papers. Experimental support is provided for the generative model assumptions, the most important of which is that latent word vectors are fairly uniformly dispersed in space.
Source: arXiv (Cornell University)
Institution: Princeton University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: AF: Medium: Towards Provable Bounds for Machine Learning, Collaborative Research: Understanding, Coping with, and Benefiting from Intractibility., AF: Small: Linear Algebra++ and applications to machine learning, AF: Small: Expansion, Unique Games, and Efficient Algorithms, Statistical and Computational Tradeoffs in High Dimensional Learning, G4353487559, G8876996369
OpenAlex ID: W2949165393