Research on exhibit design and user experience optimization of intangible cultural heritage digital museum based on multimodal interaction
Year: 2026
Type: article
Abstract: The digital protection of intangible cultural heritage is undergoing a paradigm shift from static recording to immersive human-computer interaction. However, existing systems cannot generally perceive users' real-time cognitive states, and it is difficult to handle the high-coupling audio-visual features in intangible cultural heritage scenes. To address this challenge, this paper proposes an adaptive exhibit design framework based on a multimodal interaction network. In this paper, a novel behavior-gated trilinear attention mechanism is designed that deeply fuses three distinct input dimensions: the visual details extracted by Swin Transformer V2, the transient acoustic features captured by Wav2Vec 2.0, and the user behavior features (gaze points and gestures) acting as dynamic gating signals to optimize cross-modal semantic alignment. Experiments on the self-built ICH-MModal-2024 dataset show that the model achieves an accuracy of 94.2% in recognizing specific user interaction intents (namely, active exploration, passive browsing, and confusion), surpassing both the unimodal VideoMAE algorithm and advanced multimodal networks such as MBT and Perceiver, and maintains a single-frame inference delay of 24ms, meeting real-time rendering requirements. Furthermore, a user experience study with 60 participants showed that the system increased users' average exhibit stay time to 195s and significantly reduced cognitive load. The research results confirm that the framework not only ensures the robustness of high-concurrency engineering but also effectively enhances the immersion and depth of interaction in cultural communication and provides a new technical path for the intelligent inheritance of complex cultural heritage.
Source: Systems and Soft Computing
Author: Chuang Guo
Institution: Liuzhou Polytechnic University
Language: English
Subfield: Museology
Field: Arts and Humanities
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,870
OpenAlex ID: W7204872388