Year: 2025
Type: dissertation
Abstract: Generative Artificial Intelligence (GenAl), driven by Large Language Models (LLMs) such as GPT, has emerged as a promising tool for producing complex content. This study developed and evaluated a Retrieval-Augmented Generation (RAG) system using the Gemini model, based on scientific articles published in the Brazilian Symposium on Human Factors in Computing Systems (IHC-SBC) and the HCI International conference. The resulting model, named IAH- Chatbot, was trained on a corpus of 280 articles and assessed through both qualitative and quantitative approaches. Human evaluation involved six experts in Human-Computer Interaction (HCI), who rated the model highly for factual accuracy (mean 5.00), clarity (4.67), and relevance (4.50). The IAHChatbot showed competitive performance compared to established models such as ChatGPT, Claude, and Gemini, particularly in terms of conceptual and contextual alignment. However, it scored lower on completeness (3.67), likely due to limitations in the dataset, which lacked formal definitions and practical examples. Automated evaluation using the ragas toolkit showed varying performance depending on question type. The best results were observed in a question directly linked to a document in the training corpus (Context Recall: 1.0; Faithfulness: 1.0; Factual Correctness (F1): 0.56). Overall average scores were: Context Recall: 0.5; Faithfulness: 0.58; and F1: 0.77. The analysis indicates that the model often generated semantically correct responses even when the retrieved context was limited or not directly used. The findings suggest that the model holds strong potential as a pedagogical and research-support tool in the field of Human-Computer Interaction, especially when enhanced by curated textual data and iterative refinement. The proposed methodology is replicable and well-aligned with the needs of the Brazilian scientific context.
Author: Cerquiare, Felipe Alves
Language: Portuguese
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7155453308