Year: 2024
Type: dissertation
Abstract: Retrieval Augmented Generation (RAG) is an increasingly popular machine learning technique combining the realm of Large Language Models (LLMs) with an Embedding Model based Information Retrieval System.This enables traceable, fact-based LLM generated answers, which is critical for professional tasks.The retrieval component of a RAG-system is crucial for accuracy, and opensource embedding models provide a solid foundation for successfully implementing such systems.However, for highly domain-specific text, a steep decrease in retrieval performance is observed, reducing RAG's effectiveness across specialized landscapes.Fine-tuning open-source embedding models on domain-specific corpora could significantly enhance RAG's applicability by adapting the retrieval system to specific domains.For this purpose, various approaches to domain adaptation of dense retrieval systems were explored, aiming to optimize retrieval performance on a corpus consisting of highly technical car repair and diagnosis manuals.The investigated techniques included multiple text-splitting strategies, synthetic data generation, LLM-based data processing, hard negative mining, and several different fine-tuning methods.Evaluation was conducted on both synthetic and human-annotated data.Peak performance was reached for a system utilizing an LLM for mining of hard negatives, creating an expanded dataset used for staged fine-tuning of the embedding model with Multiple Negatives Ranking loss followed by Online Contrastive loss.Overall, fine-tuning the embedding model on synthetic data was shown to yield statistically significant improvements in performance across both synthetic and human annotated queries.Moreover, the fine-tuned model outperformed a much larger base embedding model.
Authors: Fureby, Lycke, Hansen, Filippa
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Lunds Universitet
OpenAlex ID: W7037654093