Work

Hallucination Detection Using Evidence Retrieval and Natural Language Inference

Year: 2026
Type: preprint
Abstract: Large language models (LLMs) can generate fluent responses containing unsupported, incorrect, or unverifiable factual statements. TruthLens is an evidence-based claim-verification pipeline that examines factual statements individually rather than assigning one label to an entire answer. It extracts sentence-level claims, retrieves candidate evidence using BM25 lexical retrieval and MiniLM semantic retrieval, combines rankings with Reciprocal Rank Fusion (RRF), and applies a pretrained DeBERTa-v3 Natural Language Inference (NLI) model to the highest-ranked evidence sentence. Entailment, contradiction, and neutral predictions are mapped to SUPPORTED, CONTRADICTED, and UNKNOWN. On 19,998 FEVER development examples and a restricted 27,055-sentence evidence corpus, hybrid retrieval achieved Recall@1/5/10 of 56.51%/65.02%/65.67%, outperforming the individual retrieval methods in this experiment. A frozen hybrid top-1 NLI configuration achieved 62.25% accuracy and 0.6231 macro-F1 on a separate 2,000-example held-out subset. An exploratory HaluEval experiment showed poor transfer under the current question-answering evaluation formulation. The results demonstrate a working retrieval-and-verification prototype while exposing limitations in evidence coverage, claim extraction, coreference, uncertainty handling, and cross-dataset generalization.
Language: English
Open Access status: green
OpenAlex ID: W7225876896
Cites: 9
Cited by:
Related to: 10