Mitigating the paucity-of-data problem: exploring the effect of training corpus size on classifier performance for natural language processing
Year: 2001
Type: conference-paper
Abstract: In this paper, we discuss experiments applying machine learning techniques to the task of confusion set disambiguation, using three orders of magnitude more training data than has previously been used for any disambiguation-in-string-context problem. In an attempt to determine when current learning methods will cease to benefit from additional training data, we analyze residual errors made by learners when issues of sparse data have been significantly mitigated. Finally, in the context of our results, we discuss possible directions for the empirical natural language research community.
Authors: Michele Banko, Eric Brill
Institution: Microsoft (United States)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W2005435014
