P-361 Can we trust Cochrane reviews in infertility? An assessment of the trustworthiness of randomized controlled trials (RCTs) included in Cochrane infertility reviews
Year: 2025
Type: conference-abstract
Abstract: Study question What is the prevalence of RCTs with trustworthiness concerns and how often do conclusions of Cochrane reviews change when these trials are excluded? Summary answer Almost half of RCTs informing Cochrane reviews in infertility have trustworthiness concerns. Their exclusion leads to changes in conclusions in 22% of the relevant comparisons. What is known already Cochrane reviews are widely regarded as the most reliable source of evidence for guiding clinical care, as they summarise the results of RCTs. However, the trustworthiness of the RCTs underpinning these reviews has been the subject of debate. It is unclear to what extent untrustworthy RCTs influence the conclusions from Cochrane reviews. We assessed the trustworthiness of RCTs incorporated in Cochrane reviews about infertility, and their impact on the overall conclusions of these reviews. Study design, size, duration We searched for Cochrane reviews on infertility published between 1 March 2021 and 1 March 2024. Reviews were eligible if they addressed infertility interventions and reported on at least one comparison on clinical pregnancy. We then searched the eligible reviews for relevant RCTs manually and via Crossref, a citation-linking tool. We extracted the aggregated effect sizes of infertility interventions on clinical pregnancy reported by Cochrane reviews, and their corresponding contingency tables. Participants/materials, setting, methods Each RCT was assessed by reviewers using the Trustworthiness-in-RAndomized-Controlled-Trials (TRACT) checklist, with results verified by a senior author. The TRACT tool has seven domains: governance, author group, intervention, timeframe, drop-out rates, baseline characteristics, and outcomes. A study was not trustworthy when it scored positive on > =3 domains or authors had a retracted paper. For each comparison, we calculated the effect size for clinical pregnancy based on trustworthy RCTs, untrustworthy RCTs, and all RCTs combined. Main results and the role of chance We identified 17 Cochrane reviews reporting on 324 RCTs All reviews focused on interventions. Of 324 RCTs, 132 (40.7%) did not meet trustworthiness criteria. The areas of greatest concern regarding trustworthiness were trials reporting zero drop-outs (46.9%) and identical numbers of participants in each group (18.8%). Other significant areas of concern were studies having absent (22.5%) or retrospective registration (26, 8.0%) and authors or institutions with retracted papers or large numbers of RCTs in a short period (37, 11.4%). After excluding trials with trustworthiness concerns, 12/63 (19.0%) comparisons changed from statistically significant to non-significant, whereas 2/63 (3.2%) comparisons changed from statistically non-significant to significant. Additionally, for 6/63 (9.5%) conclusions, no forest plot could be generated, as no RCTs on this comparison met trustworthiness criteria. In 47 comparisons reporting a positive treatment effect (OR > 1), 14/47 (29.8%) changed to higher OR after excluding trials with trustworthiness concerns, 18/47 (38.3%) changed to lower OR, and 15/47 (31.9%) showed no change. In 10 comparisons reporting a negative treatment effect (OR < 1), 5/10 (50.0%) changed to higher OR after excluding trials with trustworthiness concerns, 1/10 (10.0%) changed to lower OR, and 4/10 (40.0%) showed no change. Limitations, reasons for caution Despite the assistance from the Cochrane infertility/gynaecology group, not all full articles of RCTs included in each Cochrane review were obtained. However, this represents only less than 1% of all trials. We are in the process of writing to authors of RCTs not meeting our trustworthiness criteria. Wider implications of the findings Almost half of RCTs in Cochrane reviews about Infertility treatments have trustworthiness concerns. This leads to changes in clinical recommendations. Untrustworthy RCTs affect conclusions from 22.2% of Cochrane reviews, excluding these RCTs usually makes the treatment effects less extreme. Trial registration number No
Source: Human Reproduction
Authors: C Tang, V Zhou, A Hu, Q Feng, D L Rolnik, B W Mol
Institution: Monash University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W4411748751
