Year: 2025
Type: dissertation
Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various Natural Language Processing (NLP) tasks. However, their performance is highly sensitive to the input prompt used to query them. Manual prompt engineering, while effective, is often time-consuming and requires significant expertise. This thesis investigates Prompt Optimization, focusing on a closed-loop, automated approach to discover prompts that maximize LLM performance for specific tasks. We implement and evaluate a system that iteratively refines an initial question based on the LLMs generated answers compared against a known ground truth. The system utilizes an LLM (specifically, OpenAI’s gpt-4o model) for both generating answers in an extractive Question Answering (QA) task and proposing refinements to the prompt itself. While there are scores that are non-binary based and can capture semantic similarity, the optimization goal is to maximize lexical similarity metrics, primarily F1- score, between the generated answer and the ground truth. Our methodology involves an iterative loop managed by a QuestionOptimizer component, which integrates modules for answer generation (AnswerGenerator), answer evaluation (AnswerEval), including F1-score and Exact Match (EM), and prompt refinement (PromptRefiner). The system’s performance is compared against an open-loop baseline where no prompt refinement occurs. Experiments are conducted using a specific instance from a biomedical abstract context, focusing on retrieving a precise sample volume. The results demonstrate that the closed-loop optimization process significantly improves performance, consistently achieving high F1 and EM scores (often reaching 1.0) compared to the suboptimal baseline performance (F1 approx. 0.4). Analysis of the optimization iterations reveals how the system automatically generates increasingly specific and effective question phrasing. However, this accuracy improvement comes at the cost of increased computational resources (time, token usage, Application Programming Interface (API) calls). This work highlights the potential of automated, closed-loop systems for optimizing prompts for specific, accuracy-critical QA tasks, while also underscoring the efficiency trade-off’s involved.
Author: Fontes, Arild
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7055573909