Consideration of prompts as a neglected factor in research on evaluating ChatGPT’s translation performance
Year: 2025
Type: article
Study design: Systematic Review
Abstract: The rapid development and adoption of ChatGPT have sparked increasing academic interest in its translation performance. However, little attention has been paid to the role of prompts – a key factor influencing ChatGPT’s output. This study addresses this gap by analyzing the consideration of prompts in 32 articles that evaluated ChatGPT’s translation performance, selected through the PRISMA framework. The findings reveal that prompt consideration is generally inadequate: nearly a third of the articles did not mention prompts, and most studies that did acknowledge prompts only addressed superficial aspects, such as merely mentioning the concept or specifying the prompts used, rather than treating prompts as a key variable. Additionally, prompt usage is often limited and lacks justification. The analysis further reveals a significant disciplinary disparity, as studies in computer science and information technology have demonstrated the most comprehensive approaches, surpassing those in translation and linguistics, as well as other disciplines. Based on these findings, this study proposes a series of implementation-focused recommendations to enhance prompt consideration in ChatGPT-related research, and underscores the importance of interdisciplinary collaboration to foster a more integrated research approach.
Author: Yu Zhang
Institution: Shanghai International Studies University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W4412109572