Evaluating Transformer Models For Document Question Answering : A Comparative Study On Multilingual And Normative Texts
Year: 2024
Type: dissertation
Abstract: Document Question Answering (DQA) systems have gained significant traction in recent years, especially with the rise of large language models (LLMs). However, the application of these models to normative documents, such as legal texts, guidelines, or emergency instructions, presents unique chal- lenges. Normative documents often contain formal, domain- specific language, making it difficult for DQA models to extract precise answers. This study evaluates the performance of various transformer-based models on a bilingual dataset of university normative documents, comprising English and Turk- ish texts. For English, models BERT, DistilBERT, ALBERT, RoBERTa, T5, and Longformer are analyzed. For Turkish, XLM-R, and fine-tuned Turkish-specific models BERTurk and PEFT mT0 are utilized to address the unique morphological challenges of the Turkish language. Evaluation metrics, in- cluding token-level accuracy, F1 score, and execution time, are used to analyze the models’ performance in extracting relevant information. This research identifies the most effective models for processing normative documents in each language and highlights the limitations of current DQA systems in handling complex, formal texts across linguistic boundaries.
Source: Doria (University of Helsinki)
Author: Yaradanakul, Zeynep
Institution: Universitat de les Illes Balears
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7115264002
