Linear transformations of semantic spaces for word-sense discrimination and collocation compositionality grading
Year: 2015
Type: other
Abstract: Latent Semantic Analysis (LSA) and Word Space are two semantic models derived from the \nvector space model of distributional semantics that have been used successfully in word-sense \ndisambiguation and discrimination. LSA can represent word types and word tokens in con- \ntext by means of a single matrix factorised by Singular Value Decomposition (SVD). Word \nSpace is able to represent types via word vectors and tokens through two separate kinds of \ncontext vectors: direct vectors that count first-order word co-occurrence and indirect vec- \ntors that capture second-order co-occurrence. Word Space objects are optionally reduced by \nSVD. Whilst being regarded as related, little has been discussed about the specific relation- \nship between Word Space and LSA or the benefits of one model over the other, especially with \nregard to their capability of representing word tokens. This thesis aims to address this both \ntheoretically and empirically. \nWithin the theoretical focus, the definitions of Word Space and LSA as presented in the \nliterature are studied. A formalisation of these two semantic models is presented and their \ntheoretical properties and relationships are discussed. A fundamental insight from this theor- \netical analysis is that indirect (second-order) vectors can be computed from direct (first-order) \nvectors through a linear transformation involving a matrix of word vectors (a word matrix), \nan operation that can itself be seen as a method of dimensionality reduction alternative to \nSVD. Another finding is that in their unreduced form, LSA vectors and the Word Space dir- \nect (first-order) context vectors define approximately the same objects and their difference \ncan be exactly calculated. It is also found that the SVD spaces produced by LSA and the \nWord Space word vectors are also similar and their difference, which can also be precisely \ncalculated, ultimately stems from the original difference between unreduced LSA vectors and \nWord Space direct vectors. It is also observed that the indirect ?second-order? method of \ntoken representation from Word Space is also available to LSA, in a version of the representa- \ntion that has remained largely unexplored. And given the analysis of the SVD spaces produced \nby both models, it is hypothesised that, when exploited in comparable ways, Word Space and \nLSA should perform similarly in actual word-sense disambiguation and discrimination exper- \niments. \nIn the empirical focus, performance comparisons between different configurations of LSA \nand Word Space are conducted in actual word-sense disambiguation and discrimination ex- \nperiments. It is found that some indirect configurations of LSA and Word Space do indeed \nperform similarly, but other LSA and Word Space indirect configurations as well as their dir- \nect representations perform more differently. So, whilst the two models define approximately the same spaces, their differences are large enough to impact performance. Word Space?s sim- \npler, unreduced direct (first-order) context vectors are found to offer the best overall trade off \nbetween accuracy and computational expense. Another empirical exercise involves comparis- \nons of geometric properties of Word Space?s two token vector representations aimed at testing \ntheir similarity and predicting their performance in means-based word-sense disambiguation \nand discrimination experiments. It is found that they are not geometrically similar and that \nsense vectors computed from direct vectors are more spread than those computed from indir- \nect vectors. Word-sense disambiguation and discrimination experiments performed on these \nvectors largely reflect the geometric comparisons as the more spread direct vectors perform \nbetter than indirect vectors in supervised disambiguation experiments, although in unsuper- \nvised discrimination experiments, no clear winner emerges. The role of the Word Space word \nmatrix as a dimensionality reduction operator is also explored. Instead of simply truncating \nthe word matrix, a method in which dimensions representing statistically associated word \npairs are summed and merged, called word matrix consolidation, is proposed. The method \nachieves modest but promising results comparable to SVD. Finally, the word vectors from \nWord Space are tested empirically in a task designed to grade (measure) the compositionality \n(or degree of ?literalness?) of multi-word expressions (MWEs). Cosine similarity measures \nare taken between a word vector representing the full MWE, and word vectors represent- \ning each of its individual member words in order to measure the deviation in co-occurrence \ndistribution between the MWE and its individual members. It is found that this deviation \nin co-occurrence distributions does correlate with human compositionality judgements of \nMWEs.
Author: Alfredo Maldonado
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2588783985