Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric
Year: 2026
Type: conference-paper
Abstract: Meaning in human language is relational and context-dependent, and it emerges through a dynamic system of signs rather than fixed relationships between words and concepts (Saussure, 1916).Relatedly, the fields of Semiotics and Hermeneutics emphasize that meaning emerges through contextually situated interpretive processes (Gadamer, 1975;Perez et al., 2026), a complexity that has historically challenged computational systems designed to process, represent, and evaluate human language (Shan, 2025).Building on these perspectives, this article advances an interdisciplinary framework for evaluating meaning in machine-generated language and introduces the Inductive Conceptual Rating (ICR) metric, a mixed-method approach, grounded in inductive content analysis and reflective thematic analysis that assesses semantic accuracy and meaning alignment in generative artificial intelligence (GenAI) outputs.The ICR metric is applied in an empirical study that compares thematic summaries generated by large language models (LLMs) with the human-generated output across five datasets (N = 50-800).Results show that although models achieve high linguistic similarity scores, they consistently unperformed relative to human outputs in capturing recurring, contextually grounded meanings.This work concludes with by discussing implications for meaning evaluation and future research recommendations.
Authors: Natalie Perez, Sreyoshi Bhaduri, Aman Chadha
Institutions: University of Hawaiʻi at Mānoa, University of Hawaii System
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7166829864