Collaborative reasoning framework integrating multimodal large language models and knowledge graphs for dragon and lion dance technical action semantic understanding
Year: 2026
Type: article
Abstract: Dragon and lion dance occupies an unusual position: a nationally recognised item of intangible cultural heritage that is at the same time a codified competitive sport. Judging it is correspondingly hard, because one scored element carries temporal, spatial, dynamic and cooperative meaning at once. This study couples a multimodal large language model (MLLM) with a domain-specific knowledge graph (KG) so that technical actions can be recognised, scored and explained against the rulebook. The graph encodes 100 action types, 224 technical elements, 135 scoring criteria and 83 common error patterns, all distilled from official regulations, expert interviews and historical competition records. A perception-retrieval-reasoning pipeline drives the framework: the MLLM proposes a preliminary label, that label triggers structured subgraph retrieval, and the retrieved triples then constrain multi-hop reasoning. A cross-modal anchoring module aligns visual action embeddings with KG entity representations through contrastive learning with rule-aware hard negatives, while a path verbaliser converts the winning traversal into a rule-referenced adjudication report. We evaluate on 4,659 clips split disjointly by competition, so that no routine, team or venue appears in more than one subset. Measured against skeleton-based recognisers, recent video MLLMs, graph-augmented retrieval pipelines and a dedicated action quality assessment regressor, the framework reaches 82.7% classification accuracy, 44.2 BLEU-4 and 61.5 ROUGE-L; it predicts execution scores with a Spearman correlation of 0.812 and a relative L2 distance of 0.681, identifies deduction items at 79.4% F1, and cites the correct rule clause 85.6% of the time. Ablations, hyper-parameter sweeps, efficiency profiling and degraded-video tests indicate that knowledge grounding, perceptual anchoring, structured explanation and conflict-driven re-retrieval each address a different failure mode. Two clips traced end to end, one handled correctly and one not, show that every sentence of a generated report maps onto a graph edge that a reviewer can inspect. The framework is offered as a design for adjudication support that informs a judging panel rather than replacing it; whether it transfers to sports governed by other rulebooks has yet to be tested.
Source: Scientific Reports
Authors: Ao Yang, QingYing Ge, Maobin Zhu
Institutions: Qufu Normal University, Huizhou University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7223062070