Systematic evaluation of ChatGPT performance in providing renewable energy information
Year: 2025
Type: article
Abstract: With the advent of attention mechanisms and the development of transformer-based architectures, a new era of large language models (LLMs) has emerged. These models have given rise to a wide range of fine-tuned applications capable of providing high-quality and informative responses. ChatGPT, a chatbot developed by OpenAI using a popular LLM called GPT 3.5, has demonstrated its capability to provide valuable information in a range of fields. The renewable energy field is a prominent field, which aims to explore alternative sources of energy that are more affordable and effective. In this study, we examine how accurate ChatGPT is when it comes to providing general, non-technical information on renewable energy compared to human experts in this field. A set of prompts was presented to ChatGPT as well as to a human expert. Responses have been collected and evaluated using a set of similarity measures. Further to this, a pre-trained Google vector has been applied to emphasize semantic similarity, and then a more sophisticated LLM competitor evaluation, Gemini, has been employed to evaluate the degree of similarity between ChatGPT responses and those of human experts. It surprisingly comes to a point where ChatGPT responses were more accurate and relevant than those of human experts’ responses on renewable energy prompts. This study concludes that ChatGPT is a promising and supportive resource for renewable-energy information, offering responses remarkably close to those of a human expert.
Source: PeerJ Computer Science
Authors: Mohammed S. Ibrahim, Ahmed J. Aljaaf, Mohammed Al-khafajiy, Ahmed Adil Nafea, Nor Samsiah Sani
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,395
Funder: Universiti Kebangsaan Malaysia
Award: G2820128283
OpenAlex ID: W7110412138