Work

MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models

Year: 2026

Type: conference-paper

Abstract: Large language models (LLMs) are increasingly used in the mental health domain, yet it remains unclear how well they capture related biomedical knowledge and how reliably they apply it to clinically salient structured judgments.Here, we present a knowledgegraph (KG)-grounded benchmark for assessing LLMs on mental-health entity recognition, relation judgment, and two-hop reasoning.The benchmark is derived from PrimeKG and comprises nine task families with KGsupported answers and controlled negative options.Experiments across 15 closed-and opensource LLMs reveal a persistent recognitionto-judgment gap: leading models achieve nearceiling performance on entity typing and on the small relation-typing subset, yet they still struggle with relation prediction and two-hop reasoning.Additionally, short KG-derived snippets benefit some models but degrade performance for others.Moreover, output-format reliability can substantially influence measured performance under constrained multiple-choice settings, highlighting the critical role of response validity in benchmark-based evaluation.MH-GraphBench should therefore be interpreted as evaluating agreement with a curated mentalhealth slice of PrimeKG under a constrained multiple-choice interface, rather than as a direct assessment of real-world clinical safety.

Source: Workshop on Generation, Evaluation and Metrics (GEM)

DOI: 10.18653/v1/2026.gem-main.38