Utvärdering av effektiviteten hos stora språkmodeller för att summera svenska resultaträkningar
Year: 2024
Type: dissertation
Abstract: This thesis explores the effectiveness of large language models (LLMs) in generating coherent and accurate natural language summaries from Swedish income statements. By employing various prompting strategies, this research investigates integration of a deterministic and explainable preprocessing system: one designed to optimize data relevance for LLM inputs – termed “guided zero-shot prompt”. The study systematically compares three prompting techniques: zero-shot, self-augmentation, and guided zeroshot, assessing their influence on summary accuracy and coherence. These techniques are tested in combination with three different LLMs: GPT-4, Llama 3 8B, and Llama 3 70B. Statistically significant improvements were observed with the guided zeroshot prompting approach, across all tested LLMs. In terms of aligning the generated summaries with human-created references, the guided prompt outperformed the other prompts, achieving a higher F-score for every model. The guided prompting notably reduced the inclusion of irrelevant numerical data and hallucinations, enhancing the factual precision of the summaries. In terms of LLM selection, the Llama 3 8B model performed notably worse than the other models. The results suggest that GPT-4 should be preferred if quality is prioritized over cost. These results underscore the potential for using LLMs to transform complex numerical data into more accessible narrative forms, enhancing various aspects of financial reporting and the possibility of interacting with numerical data using LLMs.
Author: Lord, Tobias
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7043625517