Uppgiftsspecifik bedömning av stora språkmodeller : En studie för människa-robot interaktion
Year: 2025
Type: dissertation
Abstract: In recent years, there has been a rapid development in the AI. Especially in the field of deep learning, where LLMs are rapidly improving. The utilization of LLMs are increasing and LLMs are integrating themselves into human lives through various developments. Examples include the increasing research in which LLMs are integrated into robots; as well as the integration of LLM into phone voice assistants. The development of LLMs is led by multiple companies, introducing their own trained models. With the increasing number of LLMs, an important question arises: Which one is better in which tasks? And with the integration topic, another question arises: Which one is more fit to integrate into embodied agents such as robots? This thesis investigated which LLM would be suitable to integrate into a robot that interacts with humans. The comparison was evaluated through a task-specific scenario, namely a cooking scenario. The LLMs were assessed in their cooking-, evaluation- and task understanding knowledge. The evaluation is reported through mean averaging in various tasks. Through the cooking scenario, this thesis also investigated LLMs’ general knowledge and NLU capabilities. This thesis evaluated LLMs from OpenAI, Meta and Google: GPT-3.5 Turbo, GPT-4o mini, LLaMA 3.0 8B Instruct, LLaMA 3.1 8B Instruct, Gemini 1.0 Pro and Gemini 1.5 Flash. Two datasets were used to evaluate these models. The first one, the Cooking Dataset, was developed in this research. This dataset consist of five question types, totalling at 261 question. The resulting Cooking Dataset consisted of five cooking phases, including questions regarding: preparation, cooking, serving, eating and cleaning. Furthermore, it was built upon five question types, out of which the majority of them are related to typical questions one can face whilst cooking. The second dataset, called The semantic dataset, includes human descriptions to different household objects. This dataset was used to evaluate the LLMs’ capabilities to ask follow-up questions before making predictions of room and location of the referred objects. The results show that the models excel in different areas. However, three models were consistently achieving the highest scores: GPT-4o mini, Gemini 1.5 Flash and Gemini 1.0 Pro. When the models were allowed to ask follow-up questions before making a prediction, GPT-4o mini performed the best. The Gemini models on the other hand, excelled when it came to the immediate predictions instead.
Author: Shenawa, Aiman
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6997259409