6ER-022 Analysis of the performance of generative artificial intelligence in the critical evaluation of articles
Year: 2025
Type: conference-paper
Abstract: Background and Importance Critical reading of scientific articles ensures the quality and internal validity of pharmaceutical research. Generative artificial intelligence (AI) can optimise this process by enhancing evaluation and bias detection. However, rigorous assessment is crucial to ensure its effectiveness and accuracy in this field. Aim and Objectives Evaluate if generative AI matches human critical analysis in assessing the internal validity of clinical trial-based scientific articles Material and Methods Eight articles were analysed according to the recommendations of the CASPe methodology. The answers to the questions on internal validity were compared with those provided by ChatGPT-4.0 (Turbo). Results When analysing the internal assessment questions, those with objective answers, such as whether it was a controlled clinical trial, if there was a defined question, or if the patient assignment was random, were correctly answered by the AI achieving 100% accuracy. However, in more complex and subjective questions, which required deeper analysis, such as the influence of an open-label design on the primary outcome, potential changes in sample size, or whether the populations were balanced, the AI showed limitations. In these cases, it failed in at least one trial, and regarding changes in sample size or the primary outcome, it only succeeded in two trials, demonstrating lower accuracy in areas that require interpretation. Conclusion and Relevance Critical reading of clinical trials is essential for advancing knowledge. While AI can analyse textual information, it currently lacks the expertise of human specialists. However, with further training, its analytical abilities could improve, offering a promising line of future research. References and/or Acknowledgements http://www.redcaspe.org/system/tdf/materiales/plantilla_ensayo_clinico_v1_0.pdf?file=1&type=node&id=158&force= , 3. Conflict of Interest No conflict of interest
Authors: M Rodríguez Marín, MG Urtecho Amarillo, MA Mallol Yañez
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
OpenAlex ID: W4408377157