Get started
Work

You Shall Know a Word by the Company It Keeps: A Narrative Review of Word Embeddings from Harris to fastText

Year: 2026
Type: review
Abstract: Word embeddings---the vectors whose geometry encodes the words' meanings---moved from Harris's distributional hypothesis through latent semantic analysis, Latent Dirichlet allocation, and the neural probabilistic language models to word2vec's skip-gram, Levy and Goldberg's implicit factorization, GloVe's co-occurrence vectors, and fastText's subwords. This article presents a narrative review of that arc's canonical line: Harris's 1954 Distributional Structure, Deerwester and colleagues's 1990 indexing by LSA, Landauer and Dumais's 1997 Plato's problem, Blei, Ng, and Jordan's 2003 LDA, Bengio and colleagues's 2003 neural language model, Collobert and Weston's 2008 unified architecture, Mikolov and colleagues's 2010 recurrent language model, Mikolov and colleagues's 2013 efficient representations, Mikolov and colleagues's 2013 compositionality, Pennington, Socher, and Manning's 2014 GloVe, Levy and Goldberg's 2014 implicit matrix factorization, and Bojanowski and colleagues's 2017 subword enrichment. The review is organized around three themes: the distributional hypothesis, in which the company's structure, the LSA's singular values, the Plato's solution, and the LDA's topics made meaning findable in the co-occurrence; the neural turn, in which the language models' predictions and the word2vec's efficiency made the embeddings neural and cheap; and theory and extensions, in which the implicit factorization's equivalence, the GloVe's global counts, and the subwords' morphology completed the embeddings' grammar. It is concluded that word embeddings are the meaning's geometry---and that their arc is the distributional hypothesis's vindication at industrial scale.
DOI: 10.5281/zenodo.22181929
Language: English
Open Access status: green
OpenAlex ID: W7204705288
FWCI:
Cites:
Cited by:
Related to: 10