Development of a Retrieval-Augmented Generation-Based Information Service Chatbot for Hallucination Mitigation in Critical Industrial Environments
Year: 2026
Type: conference-paper
Abstract: The integration of Large Language Models (LLMs) into enterprise information systems offers advanced capabilities; however, their deployment in safety-critical industrial environments is severely impeded by the operational risks of AI hallucinations. In domains like hazardous chemical handling, which operate on a zero-tolerance paradigm for factual inaccuracies, relying on manual document parsing from unstructured Material Safety Data Sheets (MSDS) is highly inefficient and prone to human error. This study aims to resolve this issue by developing a Retrieval-Augmented Generation (RAG)-based information service chatbot for chemical hazard identification. The methodology integrates layout-aware parsing via Docling, vector storage, cross-encoder re-ranking using FlashRank, and the nomic-embed-text embedding model. This architecture leverages the Llama 3.2 Vision model, integrated through a Telegram Bot API. Evaluation was conducted on 50 Golden Datasets through a manual binary audit performed independently by the researchers against ground-truth documents. The empirical results demonstrated a Faithfulness score of 0.98, Context Precision of 0.86, and Answer Relevance of 0.76. An in-depth failure analysis identified two primary bottlenecks: semantic gaps during the retrieval phase, and LLM synthesis failures (accounting for 30% of the validation tests) where the local model failed to construct coherent answers despite retrieving the correct documents. Operationally, the chatbot successfully reduced information access time from a 5 to 10-minute manual search to a median latency of 4.7 seconds.
Institution: Binus University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7204806267