Year: 2026
Type: dissertation
Abstract: Recent advances in large language models (LLMs) have enabled AI agents to interact with software systems through natural language, reducing complex multi-step graphical user interface (GUI) workflows to single natural language instructions. This thesis presents CHATiCAL, a multi-agent system designed to interface with an enterprise Calibration Management Software (CMS) via natural language. The work investigates two tool-calling paradigms: (1) a general agent leveraging retrieval-augmented generation (RAG) to dynamically select from a broad toolset, and (2) a specialized agent operating within a constrained scope using predefined tools. Both approaches employ an iterative reasoning-action loop to decompose tasks, invoke tools, and refine outputs. The thesis analyzes key design considerations for reliable agent deployment in enterprise environments, including tool selection strategies, scope definition, and architectural trade-offs between flexibility and control. It also identifies limitations in current agent-server communication structures and investigates the practical use of the Model Context Protocol (MCP) for tool invocation. Evaluation is conducted on three datasets of increasing task complexity. Performance is measured using retrieval accuracy, tool-calling accuracy, tool-trace alignment, and overall task completion. Results show that higher-capability models (GPT-4.1 and GPT-5.4-mini) perform strongly across both single- and multi-step tasks, with GPT-4.1 exhibiting greater consistency across tasks. Additionally, a lightweight model (GPT-4.1-nano) is evaluated and performs adequately on simpler tasks but degrades significantly in multi-step scenarios, highlighting the sensitivity of agent performance to underlying model capacity. Overall, the findings demonstrate the potential and usability of multi-agent LLM systems in enterprise software interaction, while highlighting key trade-offs between generality, control, and model capability.
Source: Aaltodoc (Aalto University)
Author: Ma, Ilona
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7165707250