Between Truth and Hallucinations: Evaluation of the Performance of Large Language Model-Based AI Plugins in Website Quality Analysis
Year: 2025
Type: article
Abstract: Although large language models (LLMs) like the Generative Pre-trained Transformer (GPT) are growing increasingly popular, much remains to learn about their potential for website quality auditing. The article evaluates the performance of LLM AI plugins (GPT models) in website and web application auditing. The author built and tested two original ChatGPT-4o Plus (OpenAI) plugins: Website Quality Auditor (WQA) and WebGIS Quality Auditor (WgisQA). Their performance was cautiously and carefully analysed and compared to traditional auditing tools. The results demonstrated the limitations of the AI plugins, including their propensity for false outcomes. The general conclusion is that using AI tools without considering their characteristics may lead to the propagation of AI hallucinations in audit reports. The study fills in the research gap with the results on the capabilities and limitations of AI plugins in the context of auditing. It also suggests further directions for improvement.
Source: Applied Sciences
DOI: 10.3390/app15052292
Author: Karol Król
Institution: University of Agriculture in Krakow
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W4407765889