Year: 2026
Type: conference-paper
Abstract: Retrieval-augmented generation (RAG) systems can play an important role in making law more accessible.However, large and reliable resources for training and benchmarking such systems remain scarce, especially for under-resourced languages like Dutch.To address this gap, and building on previous work (Louis et al., 2024), we introduce bLLeQA, a bilingual parallel question-answering dataset grounded in Belgian legal resources, both in French and Dutch.The dataset contains aligned questions, answers, and supporting articles in both languages, enabling evaluation of both retrieval and end-to-end RAG pipelines.Using bLLeQA, we benchmark the full RAG pipeline in a zero-shot setting, covering retrieval, citation extraction, refusal behavior, and generation quality.Our experiments show that openweight models are competitive with proprietary models in retrieval and citation extraction, but lag behind in generation quality in the RAG pipeline.Across all models, refusal capability remains weak, meaning that models do not reliably detect when the provided supporting sources are incomplete.In addition, the end-toend RAG setup still yields a substantial share of flawed responses, reaching 20% even in the best-case scenario.e m m a 3 -2 7 B Q w e n 3 .5-2 7 b G L M -4 .7 -fl a s h Q w e n 3 -3 0 B Q w e n 3 .5-3 5 b G P T -o s s -1 2 0 B Q w e n 3 .5-1 2 2 B Q w e n 3 .5-3 9 7 B M is t r a l-L a r g e D e e p s e e k -3 .2G L M -5 K im i-k 2 -t h in k in g K im i-k 2 .5 G P T -5 -n a n o G e m in i-3 .1 -fl a s h -li t e G P T -5 -m in i G e m in i-3 -fl a s h C la u d e -h a ik u -4 .5 G e m in i-3 .1 -p r o G P T -5 .4C la
Institution: University of Antwerp
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Funder: Vlaamse regering
OpenAlex ID: W7166880957