Year: 2023
Type: conference-paper
Abstract: The SMATCH metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems.However, we observe some issues in the metric that jeopardize meaningful evaluation.E.g., opaque preprocessing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds.Without upper-bounds, however, fair evaluation is not guaranteed.Furthermore, adaptions of SMATCH for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework.For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring.Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies.For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs.For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency.For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects.
Author: Juri Opitz
Institution: Heidelberg University
Language: English
Topic: Semantic Web and Ontologies
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4386566456