Work

Formal Modeling of Augmented Reality Maintenance-Guidance Processes Based on Multimodal Knowledge Extraction and Graph Reasoning

Year: 2026

Type: article

Abstract: Augmented reality maintenance guidance (ARMG) provides support for industrial maintenance by converting knowledge into instructions and integrating virtual information with physical equipment. However, constructing guidance processes still depends on the manual interpretation of heterogeneous maintenance materials, while inconsistent knowledge representations and missing information hinder automated modeling. This study proposes a formal modeling method that integrates multimodal knowledge extraction with graph reasoning. A Maintenance Guidance Ontology (MGO) and a six-tuple maintenance-event template are first developed to unify the semantic representation of actions, objects, resources, and related elements. Qwen3-VL is then fine-tuned to the ARMG domain through low-rank adaptation (LoRA) to extract maintenance knowledge from text and video, respectively. Finally, for missing elements in extracted knowledge, we develop an MGO-constrained semantic–temporal graph-convolutional network (MGO-STGCN). It combines relation semantics, step order, weighted neighborhood aggregation, and an ontology-constrained mask. The model predicts and completes three missing-element types: tools, spare parts and consumables, and precautions. Both extraction models outperformed the comparison methods in terms of slot-level and event-level F1 scores. MGO-STGCN achieved F1 scores of 0.799, 0.735 and 0.680 for the three completion tasks. Ablation experiments indicated that relation–temporal weighting and ontological constraints make complementary contributions to completion performance. These results demonstrate the potential of the proposed framework for unified formal representation of multimodal maintenance knowledge and completion of missing guidance information.

Source: Mathematics

DOI: 10.3390/math14214114

Authors: Hongduo Wu, Xingzheng Wei, Yifu Yan, Ziyue Guo, Dong Zhou

Institutions: National Key Laboratory of Science and Technology on Reliability and Environmental Engineering, Beihang University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $3,137

OpenAlex ID: W7225651038

FWCI:

Cites:

Cited by:

Related to: