Work

Limitations of the entropy measure in n-gram language modelling

Year: 2022
Type: article
Abstract: The original Shannon entropy can produce deceiving, yet correct, probability estimations. Two main properties of this estimation play a role in this unexpected output: (i) the logical influence of text sequence length on probability estimation, and (ii) a possibly unwanted impact of disproportionally high probabilities in language models with lower n-gram counts. Within this contribution, the aim is to closely examine entropy and present a clear and practical overview of the assumptions of entropy, its possible limitations, and ways in which the measure can be adapted to compensate for these possible limitations.
Source: Lirias
Authors: Bauwens, Michael, Vanbrabant, Peter, Tummers, José
Language: English
Open Access status: green
OpenAlex ID: W7001780730
FWCI:
Cites:
Cited by:
Related to: 10