Get started
Work

Commentary: The challenge of evaluating and improving evidence when research lacks reproducibility

Year: 2016
Type: editorial
Abstract: Epidemiologists fill an uncomfortable space between the fields of statistics and public health. Our statistical colleagues invent novel methods to increasingly decipher truth from fiction, reducing bias in our estimated effects of a treatment, often in complicated and confounded settings. Our public health colleagues must implement treatments in the context of real-world politics, to protect health programmes when investments in health are not sufficiently valued for their intrinsic worth to society. It is between these spaces that epidemiologists face the (often thankless) job of evaluating and improving evidence for or against the real-world effectiveness of public health programmes. Our task is to decipher the true benefits and risks of interventions in settings far messier than the ideal experimental settings we desire, which must be approached using a long-term lens that can seem callous to cries for urgent advocacy from public health groups. It is in this context that Jullien, Sinclair and Garner1 summarize and appraise the credibility of economic studies suggesting long-term health, education and economic benefits from mass deworming programmes affecting children in low- and middle-income countries. Jullien and colleagues find substantial evidence for bias and limited credibility for the claims cited by advocacy and fund-raising organizations.1 This finding may not be surprising but it is important to document it, as the use of biased or un-credible information has the potential to distort the allocation of resources away from alternative programmes for which long-term health, education and economic benefits are clearly established (if such programmes exist). What should epidemiologists do with such findings? Our first, most obvious task would be to consider whether there is still value in deworming programmes even if long-term health, education and economic benefits do not accrue (i.e. benefits in terms of immediate well-being, discomfort, and other aspects of health). The second task would be to evaluate whether the hypotheses proposed can in fact be tested more rigorously with less bias, possibly with newer trials or novel datasets providing long-term insights into outcomes of persons affected by previous randomized deworming programmes. But we would argue that there is a third, more broad-based dilemma hidden beneath the evaluation conducted by Jullien and colleagues, which needs further explication and commitment from epidemiologists: the problem posed by studies that are not only biased or selectively reported, but fundamentally lack transparency and reproducibility. The term ‘reproducible’ has been used in varied contexts over the years; here, we follow Goodman et al.2 to refer to three key types of reproducibility: (i) ‘methods reproducibility’, which indicates that a study has provided enough information about the experimental or computation procedures so that subsequent authors can repeat the study using the same data; (ii) ‘results reproducibility’, which indicates that a new study produces corroborating results using the same experimental methods as a previous study; and (iii) ‘inferential reproducibility’, which is perhaps the most important: the ‘drawing of qualitatively similar conclusions from either an independent replication of a study or a re-analysis of the original study’.2 Methods reproducibility depends on providing details on study design necessary to fully repeat a study. By this standard, three trials assessed by Jullien et al. may not be considered methodologically reproducible. They all failed to provide a pre-planned protocol. The Baird study3 additionally presented effect size estimates for outcome metrics based on undisclosed data, preventing basic reproduction of the computation procedures that produced these estimates. Critically, the transparent and consistent reporting of the number of analyses performed, and justification for why certain outcomes were selected and presented, are needed for methods reproducibility. The Baird study3 also failed to meet this standard, with outcomes dropped between different versions of the same article. Standards of results reproducibility and inferential reproducibility are also violated by the several studies reviewed. In two replication studies4,5 using the same data and/or analytical approach as the original Kenya trial,6 the subsequent authors found errors and noted that the previously claimed results were not observed. Rndom errors are possible, but it is difficult to understand why the results differed without adequate information on how the analyses were conducted. When study design, definitions, outcomes and analytical methods are unclear or altered across various reports, individual research findings are less likely to be true.7 If we think of research reproducibility through a Bayesian lens, the goal of repeating studies is to increase the amount of total available evidence that either supports or does not support the claims made by previous studies. If a hypothesis being tested has a low prior probability, a considerable amount of high-quality evidence is necessary to change the a priori reasoning.2 In this case, all three deworming studies appraised by Jullien et al.1 either selectively reported or failed to report some outcomes, which makes it difficult for subsequent authors to draw the same conclusions from either the original data or from newly collected data. The paradigm proposed by Goodman et al.2 provides a framework for epidemiologists who seek to produce rigorously reproducible studies. We would argue that when viewing epidemiological studies through this framework, we should not only focus on cumulative evidence instead of on individual significance tests, but should also focus on designing our own research with methods, results and inferential reproducibility in mind.
DOI: 10.1093/ije/dyw338
Language: English
Open Access status: bronze
OpenAlex ID: W2585301434
Cites: 10
Cited by: 1
Related to: 10