Work

mLLM-CRD: multi-modal LLM based context-aware rule detection using LoRA-integrated attention for worker safety monitoring

Year: 2026

Type: article

Abstract: Large Vision-Language Models (VLMs), which jointly process visual and textual information, have demonstrated strong performance in general scene understanding; however, their application to safety-critical industrial environments remains limited by the need for precise semantic reasoning, interpretability, and computational efficiency. Conventional worker safety monitoring systems typically rely on rigid object detection pipelines, which are prone to error propagation and provide limited explanatory capability. This paper presents m LLM-CRD, a multimodal framework that formulates safety compliance assessment as an image-to-language reasoning task. Built upon Bootstrapping Language-Image Pre-training (BLIP-2), the proposed approach applies a joint parameter-efficient fine-tuning strategy using Low-Rank Adaptation (LoRA), which is integrated into both the Querying Transformer (Q-Former) and the encoder-decoder-based language model while keeping the vision encoder frozen, thereby enabling task adaptation with minimal additional parameters. To support rule-aware semantic reasoning, we construct a scenario-based question-answer (QA) dataset that encodes safety compliance conditions as structured language queries derived from industrial safety rules. Experimental results show that m LLM-CRD consistently outperforms zero-shot VLM baselines in both classification accuracy and natural language generation quality, particularly under class-imbalanced conditions. These findings suggest that unified multimodal reasoning can provide interpretable, language-based assessments of worker safety compliance, offering a practical alternative to conventional multi-stage detection pipelines in real-world industrial settings.

Source: Journal of King Saud University - Computer and Information Sciences

DOI: 10.1007/s44443-026-00572-2

Authors: Jaehoon Lim, Yeonkyu Kwak, Jieun Lee, Tae-yong Kim, Jongpil Jeong

Institution: Sungkyunkwan University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,990

Funder: Institute for Information and Communications Technology Promotion