Schematic Network Extraction using Multimodal Generative Models : A Performance Evaluation of GPT-4o
Year: 2025
Type: dissertation
Abstract: Extracting structural information from images is a task traditionally handled using object detection and rule-based methods. This thesis explores whether general-purpose multimodal language models, specifically GPT-4o, can interpret technical images and generate structured outputs without task-specific fine-tuning. The potential benefit of this approach lies in streamlining information retrieval for organizations handling large volumes of visually complex data, potentially reducing the need for extensive datasets typically required for training traditional computer vision systems. Technical images in this study are represented by electrical circuit diagrams. GPT-4o was evaluated on its ability to identify components, produce consistent and accurate responses, and generate netlists from circuit images. Two datasets were used for evaluation: one synthetically generated, and one manually created. Ground truth data was generated using traditional object detection and image processing techniques. Three model configurations were tested: the original GPT-4o model, a retrieval-augmented generation (RAG) version, and a labeled version where component symbols were annotated with textual labels in the image. The labeled configuration achieved the highest performance across all tasks, suggesting that embedded textual cues strongly support accurate and consistent predictions. The RAG setup, in contrast, often underperformed, possibly due to confusion introduced by loosely relevant context. Netlist generation was the most challenging task for all configurations. The results show that general-purpose multimodal models have some potential for structured visual understanding, but also clear limitations in handling complex domain-specific tasks. Overcoming these limitations will likely require methods that go beyond using the models in their current form, such as hybrid approaches, memory-based reasoning, and task-specific fine-tuning.
Author: Olofsson, Lisa
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7112842953