Year: 2025
Type: dissertation
Abstract: Existing Retrieval-Augmented Generation (RAG) systems suffer from great difficulties in preserving spatial and semantic relationships amongst text, images, and tables in multimodal document processing, causing information fragmentation and lowered accuracy in comprehension. Although past studies have demonstrated single Large Language Models (LLMs) perform particularly best in certain areas GPT-4o reaching 91% vision-task accuracy, Claude 3.5 Sonnet reaching 93.1% in fact-checking, and Gemini 1.5 Pro winning at long-context processing no existing study has elaborately investigated ensemble architectures composed of multi-LLMs for preserving cross-modal relationships in document understanding systems. This study bridges the essential gap by introducing a novel multi-LLM ensemble architecture that cleverly orchestrates GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro using content-aware model selection algorithms. The system adopts a document-based knowledge graph with typed relationships to maintain cross-modal relationships, alongside hybrid retrieval processes that blend vector similarity with graph traversal for better discovery of contexts. Extensive testing involving 1500 queries and 65 varied documents across five industry applications demonstrated significant performance outcomes. Statistical significance (p < 0.001) was observed for 74.04% semantic similarity, indicating a 13.1% absolute improvement compared to single-model methods. The preservation of cross-modal relationships achieved an effectiveness rate of 89.9% across spatial, semantic, hierarchical, and sequential dimensions. Enterprise-scale deployment feasibility was confirmed by 0.65-second average response time with complete system reliability. The research contributions include: a systematic solution for ensemble coordination in multiple-LLM multimodal document understanding, a novel graph-based approach that preserves cross-modal relationships to address fragmentation challenges in state-of-the-art RAG systems, and an industrial-strength system that demonstrates successful applicability in enterprise settings. This research establishes new performance benchmarks and methodological foundations for smart model orchestration in future retrieval-augmented generation systems.
Author: Addanki, Mahesh Kumar
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204407887