Year: 2014
Type: article
Abstract: The importance of transparency in the conduct and reporting of medical research is increasingly recognized. Fueling that interest is growing evidence that the literature is biased by poor reporting practices. Prime concerns are the nonpublication of whole studies and the selective reporting of results within publications, both driven by the nature of the findings. Regardless of whether such behavior is intended to mislead, such post hoc decision-making clearly opposes key principles of research and causes bias. Nonreporting and misleading reporting do not just mislead researchers in the field, they also diminish the evidence base underpinning clinical practice and harm patients; there is a strong case that they represent scientific misconduct (1, 2). Practices that distort the evidence base are unethical. The results of multiple similar studies will vary around what we can consider to be the truth. Some studies will have promising results and others will not. Even when in truth the effect being studied is absent, for 5% of studies the result will be formally statistically significant (P < 0.05). Preferential publication of the significant studies will make the evidence for any question seem more promising than it should be. A major driver of biased reporting is the use of P values and the unfortunate related practice of labeling studies as “positive” or “negative.” For randomized controlled trials (RCTs), there is extremely strong evidence that what gets published is indeed biased in favor of statistically significant results (3, 4). That bias means that the effectiveness of drugs or other interventions is overestimated. Although the finger is often pointed at industry, academics and noncommercial funders are just as guilty (1). Further, nonsignificant findings are often given a positive spin (5). Nonpublication and selective publication are not the only problems, however. Studies that have compared published articles with study protocols or registry entries have documented frequent discrepancies. Remarkably, even the primary outcome of RCTs is frequently different in a publication from what was stated in the protocol or registry entry (3). Important discrepancies from what was planned should be explained; failure to do so casts serious doubt on the reliability of the findings. Even so, when a study has a published protocol or has been registered, any discrepancies are out in the open. Otherwise, any such changes are hidden. Further, dozens of reviews have shown that journal articles often omit key information about how the RCT was done (6), severely hampering assessment of their reliability. Comparable evidence is accumulating for other types of research, including observational and genetic epidemiology (7, 8) and animal research (9). Similar evidence exists for prognostic biomarker studies (10); almost all published prognostic studies of cancer biomarkers have statistically significant results (11). And further evidence of bias in the published literature comes from increasing cases of failure to replicate research findings (12). As yet there has been rather limited evidence about selective publication of diagnostic research (13). Studies of diagnostic test accuracy are somewhat different from RCTs and prognostic marker studies in that there is not a strong dependency on statistical significance to judge “success.” Rather, emphasis is on measures of discrimination, mainly sensitivity and specificity or area under the ROC curve. The absence of a direct analog of a “positive” study does not mean that similar biases are not a problem in the reporting of diagnostic test accuracy studies. The study of registered test accuracy studies by Korevaar et al. in this issue of Clinical Chemistry (14) seems to be the first large examination of this question. Their findings are qualitatively similar to those of several comparable reviews of RCTs: many studies remained unpublished and there were frequent discrepancies between the publication and the registry information. This new study adds to the growing body of literature that documents questionable research publication practices. The fact that these findings are not a surprise must not obscure the fact that the present situation is unacceptable. It is indeed surely shocking that so many researchers feel no need to explain serious discrepancies between two sets of public information about their study. We do not know why there are such discrepancies; research is needed in this area. Whereas some discrepancies between register and publication will be entirely reasonable, others could well be associated with bias, in particular the selective reporting from multiple outcomes or multiple analyses (e.g., variation in cut-points). It seems likely that researchers who register their study would be more methodologically aware, understand better the risks associated with selective publication, and do somewhat better studies than those who do not register. The findings of this review may thus give a rather optimistic picture of the literature, especially as currently few test accuracy studies are registered. The Helsinki Declaration states, “Researchers have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports. … Negative and inconclusive as well as positive results must be published or otherwise made publicly available. … Reports of research not in accordance with the principles of this Declaration should not be accepted for publication” (15). There is no suggestion here that these obligations apply only to clinical trials, nor should they. If registration is seen as a solution, if not the solution, for trials, then the same should apply to diagnostic and prognostic studies. Across many types of research, accumulating evidence of bias has led to increasing support for greater transparency, especially relating to registration, publication of full protocols, and adherence to reporting guidelines. None of these will solve all the problems, but certainly all will help. Research registers, notably clinicaltrials.gov, were set up primarily with clinical trials in mind. Among other things, registration “facilitate[s] transparency and completeness of the reporting of clinical trials and, ultimately, strengthen[s] the validity and value of the scientific evidence base” (16). Preregistration may make most sense for prospective studies, but it can certainly also be valuable for studies based on (re)analysis of existing datasets. Some people are concerned that preregistration and a detailed study protocols inhibit exploratory analyses. I do not agree—a protocol can simply specify what is predetermined and which aspects are to be exploratory. Study registration has received previous support for studies of biomarkers (17, 18). Journals are the unwitting conduit for the misleading (overoptimistic) nature of the published literature. Publication bias poses a major challenge as the bias arises primarily from what is not submitted for publication, not what is. What can journals do about it? I feel it is not practical to suggest, at this time, that journals require registration as a condition of publication (or indeed peer review), but they could signal that they may do so in the near future. Such action would succeed only as a policy change by many journals collectively, as happened for RCTs. Meanwhile, journals should strongly encourage prospective registration of diagnostic and prospective studies. They should ensure that authors report whether their study was registered. They should check for information on a registry when assessing submissions, and authors should be required to explain important discrepancies. If possible, journals should develop incentives for those who had preregistered their studies. And the scientific community should begin to recognize that evidence from preregistered studies is more reliable than evidence from unregistered studies. It took 20 years to implement trial registration after John Simes suggested it in 1986. Let us ensure we move much faster for studies of biomarkers.
Source: Clinical Chemistry
Author: Douglas G Altman
Institution: University of Oxford
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: bronze
Awards: G3587225636, G747464377
OpenAlex ID: W2145485635