Exam questions are not consultations: on training and evaluating specialised legal language models with examination material, and an alternative evaluation design
Year: 2026
Type: preprint
Abstract: Examination material is an attractive source for training and evaluating specialised legal language models: it comes with fixed facts, reference solutions and human score distributions. I examine what a design built on such material can and cannot measure for advisory work in German tax law. First, an exam question hands over all relevant facts, whereas a consultation starts from incomplete facts; in my own games with hidden facts, 68 % of lost points were facts the adviser did not ask about. Second, the law changes over time; knowledge stored in model weights has one date, while the correct answer depends on the date of the case. My games contain both directions of this error, including one made by my own pipeline. Third, an answer key is written for one legal state and becomes outdated when the law changes. Fourth, the overlap of training and test sources and the use of model judges need explicit controls. A dated anchor of official norm texts addresses the second point without retraining: on 52 expert questions, one model found 35 % of the facts from memory and 83 % with the anchor. I propose an evaluation design based on fictitious cases with hidden facts, blind judges, control questions drawn from random norms and three metrics: decisive fact found, question asked, correct version cited. The argument concerns the design of evaluation, not the qualification of the people who build such models. Part of a series; overview: https://doi.org/10.5281/zenodo.23949487. Files: English version (authoritative) and German version (Deutsche Fassung). Not legal or tax advice.
Author: Sevriukov, Sergei
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225253330