Work

Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities

Year: 2026

Type: conference-paper

Abstract: We introduce the Korean Canonical Legal Benchmark (KCL), a benchmark designed to assess language models' legal reasoning capabilities independently of domain-specific knowledge.KCL provides question-level supporting precedents, enabling a more faithful disentanglement of reasoning ability from parameterized knowledge.KCL consists of two components: (1) KCL-MCQA, multiple-choice problems of 283 questions with 1,103 aligned precedents, and (2) KCL-Essay, open-ended generation problems of 169 questions with 550 aligned precedents and 2,739 instance-level rubrics for automated evaluation.Our systematic evaluation of 30+ models shows large remaining gaps, particularly in KCL-Essay, and that reasoning-specialized models consistently outperform their general-purpose counterparts.We release all resources, including the benchmark dataset and evaluation code, at https: //github.com/lbox-kr/kcl.

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2026.eacl-short.17

Authors: Hongseok Oh, Wonseok Hwang, Kyoung-Woon On

Institution: University of Seoul

Language: English

Topic: Artificial Intelligence in Law

Subfield: Political Science and International Relations

Field: Social Sciences

Domain: Social Sciences

SDG: Peace, justice, and strong institutions

Open Access status: hybrid

Funders: National Research Foundation, National Research Foundation of Korea, Ministry of Science and ICT, South Korea