Work

Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean

Year: 2025

Type: conference-paper

Abstract: We introduce the Korean Grammar Evaluation BenchMark (KoGEM), designed to assess the linguistic competence of LLMs and humans in Korean.KoGEM consists of 1.5k multiplechoice QA pairs covering five main categories and 16 subcategories.The zero-shot evaluation of 27 LLMs of various sizes and types reveals that while LLMs perform remarkably well on straightforward tasks requiring primarily definitional knowledge, they struggle with tasks that demand the integration of realworld experiential knowledge, such as phonological rules and pronunciation.Furthermore, our in-depth analysis suggests that incorporating such experiential knowledge could enhance the linguistic competence of LLMs.With Ko-GEM, we not only highlight the limitations of current LLMs in linguistic competence but also uncover hidden facets of LLMs in linguistic competence, paving the way for enhancing comprehensive language understanding.Our code and dataset are available at

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-long.492

Authors: SungHo Kim, Nayeon Kim, Taehee Jeon, SangKeun Lee

Institution: Korea University

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid