Work

Large multimodal model assisted underground tunnel damage inspection and human-machine interaction

Year: 2025

Type: article

Abstract: Artificial Intelligence is playing an increasingly important role in tunnel inspection as a core driver of the new generation of engineering. Traditional methods are difficult to directly generate human linguistic information and lack valid messages extracted from different modalities. This paper proposes Damage LMM, a multimodal damage detection model that can handle images or videos as well as text inputs, to realize fast damage identification and human-computer interaction. The visual instruction database is first created from real damage data collected using different visual sensors and captions extracted by a regional convolutional neural network. The basic language model is then fine-tuned into a specialised Damage LMM, which enhances user instructions by integrating virtual prompt injection and system messages. Finally, the enhanced prompts are processed through the tuned multimodal model to generate a detailed visual description of the damage. The performance of the method is evaluated using a real tunnel dataset, and the results show that it has better robustness and accuracy than other models in multimodal data, with an accuracy of 0.93 for the in-domain image data and a contextual correlation of 0.94. The proposed method can effectively identify tunnel defects and realize multimodal user interaction functions with a moderate number of markers and a short delay time, which will greatly help engineers to quickly obtain effective information and assess the degree of damage at the tunnel inspection site. • Innovative multimodal damage detection model that can process images/videos along with text inputs. • Specialised fine-tuning for generating detailed damage descriptions based on visual instruction database. • Achieves high accuracy and strong contextual correlation, outperforming traditional models in robustness and precision. • Enables engineers to quickly access actionable information with minimal markers and short delay times. • Realizing damage identification and enhanced human-machine interaction to improve tunnel inspection efficiency.

Source: Journal of Infrastructure Intelligence and Resilience

DOI: 10.1016/j.iintel.2025.100154

Authors: Yanzhi Qi, Zhi Ding, Yaozhi Luo

Institutions: Hangzhou City University, Zhejiang University

Language: English

Topic: Infrastructure Maintenance and Monitoring

Subfield: Civil and Structural Engineering

Field: Engineering

Domain: Physical Sciences

SDG: Industry, innovation and infrastructure

Open Access status: gold

Estimated APC paid: $1,600