Exploring the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal
Year: 2025
Type: article
Abstract: This paper evaluates Large Language Models’ (LLMs) limitations through their masked text processing capabilities. We introduce two novel tasks: MskQA, which measures reasoning on masked question-answering datasets, and MskCal, which assesses numerical reasoning on masked arithmetic problems. Testing GPT-4o and 4o-mini reveals that LLM performance depends significantly on masking rates and semantic information availability. Our experiments demonstrate that performance decreases as semantic information is reduced, with GPT4o consistently outperforming 4o-mini, particularly in numerical reasoning tasks. The study shows that LLMs maintain reasonable accuracy with masking rates below 40%, but struggle with heavily masked computational tasks. Our findings illuminate the interaction between background knowledge and reasoning ability in masked text processing, highlighting the need for more robust methods to assess LLMs’ true comprehension capabilities.
Authors: Haru Tada Sato, Fuka Matsuzaki
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4414417539
FWCI:
Cites:
Cited by:
Related to: