Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications
Year: 2025
Type: conference-paper
Abstract: This paper proposes a Question-Answering (QA) system for the telecom domain using 3rdGeneration Partnership Project (3GPP) technical documents. Alongside, a hybrid dataset, Telco-DPR11Telco-DPR dataset presented in this paper is available open-source [1]., which consists of a curated 3GPP corpus in a hybrid format, combining text and tables, is presented. Additionally, the dataset includes a set of synthetic question/answer pairs designed to evaluate the retrieval performance of QA systems on this type of data. The retrieval models, including the sparse model, Best Matching 25 (BM25), as well as dense models, such as Dense Passage Retriever (DPR) and Dense Hierarchical Retrieval (DHR), are evaluated and compared using top-K accuracy and Mean Reciprocal Rank (MRR). The results show that DHR, a retriever model utilising hierarchical passage selection through fine-tuning at both the document and passage levels, outperforms traditional methods in retrieving relevant technical information, achieving a Top-10 accuracy of 86.2 %. Additionally, the Retriever-Augmented Generation (RAG) technique, used in the proposed QA system, is evaluated to demonstrate the benefits of using the hybrid dataset and the DHR. The proposed QA system, using the developed RAG model and the Generative Pre-trained Transformer (GPT)-4, achieves a 14% improvement in answer accuracy, when compared to a previous benchmark on the same dataset.
Authors: T. Saraiva, M. Sousa, P. Vieira, A. Rodrigues
Institutions: Instituto de Telecomunicações, Celfinet (Portugal)
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Eurostars
OpenAlex ID: W4410228196