Work

法的言語モデルにおける幻覚の検出と理解:不確実性と内部シグナルに基づく複合的手法の研究

Year: 2025
Type: dissertation
Abstract: Detecting and understanding hallucinations in large language models (LLMs) is crucial, especially within high-stakes domains such as law, where the reliability of information significantly impacts decision-making processes.This thesis addresses the critical challenge of hallucination detection in legal language models by leveraging uncertainty measures and internal model signals, developing a comprehensive multi-method framework specifically designed for legal question-answering (QA) tasks.Initially, this research investigates prominent existing hallucination detection techniques, focusing on semantic uncertainty, reference-based verification (RefChecker), and internal model checks (LLM-Check).It identifies limitations such as dependency on external references and susceptibility to confident yet incorrect assertions, motivating the development of a novel detection pipeline that combines these insights.The thesis introduces an internal scoring system tailored specifically for legal QA.This system exploits semantic entropy, attention patterns, and hidden-state embeddings from within the LLMs themselves, requiring no external reference and only a single forward pass for each query.Experimental evaluations demonstrate that this method significantly outperforms baseline approaches in accuracy, precision, recall, and computational efficiency, successfully identifying subtle and confidently-stated hallucinations.Furthermore, the thesis contributes by developing and adapting the CUAD-QA dataset, converting realworld commercial contracts into a robust evaluation benchmark that rigorously tests various hallucination detection methods.Extensive experiments conducted on this dataset highlight the effectiveness of the proposed internal scoring system, emphasizing practical applicability and superior performance compared to conventional methods.Finally, this research provides insights into the behavioral patterns of hallucinations in legal QA, pinpointing linguistic and structural cues associated with model uncertainty and errors.These findings inform guidelines for designing trustworthy LLM-based legal assistants, underscoring the importance of internal consistency checks, transparent confidence scores, and effective integration of detection signals.This thesis thus offers substantial advancements in the understanding and mitigation of hallucinations in legal LLM applications, laying the groundwork for more reliable and transparent AI tools in the legal domain.
Language: English
Open Access status: green
OpenAlex ID: W7160743104
Cites: 19
Cited by:
Related to: 10