Human Validation of Contextual Semantic Annotations
Year: 2026
Type: dataset
Abstract: Semantic feature norms provide interpretable representations of conceptual meaning, but traditional human ratings typically characterize concepts in isolation and therefore do not capture potential changes induced by linguistic context.SCRIBE extends a neurocognitively grounded semantic feature space to contextual settings by using large language models (LLMs) to estimate semantic properties of concepts in context.In its controlled evaluation, SCRIBE used LLM-generated contexts designed to increase or decrease individual concept–property associations, yielding systematic changes in the resulting semantic annotations.Here, we conduct a human validation of these contextual manipulations.We collected human ratings for 324 concept–property pairs across high-target, low-target, and out-of-context conditions, covering 54 semantic properties and concept–property associations spanning low, mid, and high baseline strengths.After quality control, 485 participants contributed an average of approximately seven independent ratings per annotation item.Human judgments followed the intended ordering, with ratings highest in the high-target condition, intermediate out of context, and lowest in the low-target condition; all three pairwise comparisons were statistically significant.The effect generalized broadly across the semantic feature space, with the high-target condition exceeding the out-of-context condition for all 54 properties. These results provide human validation that interpretable semantic property judgments can be systematically modulated by linguistic context, supporting the use of context-sensitive semantic annotations in SCRIBE.
Authors: Kolling, Camila, Toneva, Mariya
Institution: Max Planck Institute for Software Systems
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W7204168081