Year: 2024
Type: dissertation
Abstract: This thesis investigates the development and evaluation of a natural language interface for Ponsse's customers' forestry operational data using Snowflake's Cortex Analyst. The work focuses on enabling non-technical users to query structured data through natural language while maintaining calculation reliability for operational decisions. The implementation utilizes Ponsse's Snowflake data platform and a semantic model that defines domain-specific terminology, business rules, and calculation patterns. \n \nThe work involved transforming a complex star schema of over a hundred views into a denormalized table, developing an incrementally refined semantic model, and creating a comprehensive testing framework. The system was evaluated using 71 domain-specific questions, categorized into three complexity levels, through a custom-built benchmarking system that enabled systematic validation of query accuracy and performance. Four configurations were tested, comparing Cortex Analyst’s default open-source models against configurations with OpenAI’s GPT-4o enabled, both with and without verified SQL examples. \n \nThe configuration with GPT-4o achieved 88.17% accuracy with an average response time of 5.6 seconds, while the default configuration reached 69.01% accuracy with an average response time of 14.1 seconds. As query complexity and semantic model size increased, GPT-4o configurations showed stability in both accuracy and response times, whereas the default open-source configurations showed significant degradation in both. Particular challenges were observed in average calculations for all configurations. \n \nThrough a parallel implementation of Vanna.AI, the work provides insights into multi-agent versus single-pass architectural approaches for natural language interfaces. The findings show that while these interfaces can improve data access, building them requires careful attention to semantic modeling, testing methods, and system design. The developed testing framework provides a foundation for future semantic model development and is adaptable to other systems using language models.
Source: Theseus (Ammattikorkeakoulujen)
Author: Heinoja, Pyry
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7021241883