What Anaesthesia is doing to combat scientific misconduct and investigate data fabrication and falsification
Year: 2016
Type: editorial
Abstract: Scientific misconduct is not new and neither is there any evidence that it is on the increase, despite an apparent increase in the number of retractions over the last ten years 1. However, a recent spate of high-profile cases involving anaesthetists has firmly turned the spotlight onto our profession 2. Anaesthesia has been and remains at the forefront of this fight against individuals and groups who bring our profession in particular and science in general into disrepute 3. This issue of the journal contains an analysis of 32 randomised controlled trials published from 1993 to 2012 in nine anaesthetic journals 4. This demonstrates clearly that there is a high probability that the data in these trials are not consistent with random sampling; in other words, that the studies warrant further urgent investigation. The first publication of the use of the Carlisle Method was the investigation into randomised controlled trials published by Yoshitaka Fujii, which led to 138 retractions (to date) and Fujii's legacy 5. Dr John Carlisle then published an improvement of his methodology using Monte Carlo simulation 6. Dr Yuhji Saitoh was a co-author of Dr Fujii in 32 of the retracted papers. In 2015, he submitted a manuscript for which Fujii was not a co-author to Anaesthesia and Intensive Care, and examination of the baseline data led to a request for the raw data files and ethics approval documents, which were provided. However, the nature of these only added to the suspicion of the journal editors. Saitoh's current institutional employer and his previous one where the study was carried out were both informed that the journal editors had concerns about the homogenous distribution of the baseline data. This led to an investigation at Dr Saitoh's institution (Tokyo Women's Medical University) following an admission from Dr Saitoh that he had conducted the study without approval from the institutional ethics committee. The decision of the Disciplinary Committee is due to be published shortly. Meanwhile, further analysis of all trials published by Dr Saitoh has led to the article in this month's edition of the journal 4. So what next? Clearly, there is still a lot of work to do to validate the Carlisle Method. It helped expose Fujii, has now contributed to an investigation into Saitoh, and also identified Boldt 7 and Reuben 8 independently of other analyses. Further analysis of published work may reveal other randomised controlled trials that reported data not consistent with random sampling. However, only when the analysis leads to an investigation which concludes that scientific misconduct has taken place can it be shown to have 'worked'. How many times does it need to work for it to be validated? The answer is not clear, but each time the method is successfully applied (if the outcome of an investigation that concludes a case of scientific misconduct can indeed be called success), validation becomes more likely. By publication of the detailed methodology in this journal in 2015, Dr. Carlisle has thrown his method open to inspection and further analysis, and, as Editor-in-Chief, I welcome all interested parties to test it out themselves 9. I believe that further testing and analysis can only help with the validation of this method of data analysis. One of the most common questions I get asked is, by publishing the analysis, are we allowing people to work out the best way to fool the analysis and produce data that appear to have been produced by random sampling? Of course, this is possible, but not publishing the method is likely to lead to complaints of secrecy and having something to hide. We have nothing to hide, and hence the decision to publish the exact coding. We have decided to screen all randomised controlled trials submitted to the journal using the Carlisle Method. Any that fall foul due to suspicious data that are not consistent with random sampling will be rejected and the authors informed of the reason for rejection. The question arises what to do next. Under COPE guidance, and for many other legitimate reasons, being suspicious of submitted data is completely different from a formal investigation that can assign guilt. Such suspicion should be communicated to the author, who is given the right of reply. However, as a journal we do not have the right, according to COPE, to contact other journals to inform them of our suspicion, as it is unproven and unsubstantiated. Information should only be shared if there is no response from the author, the response is inadequate, or more than one journal is thought to be affected. This means that the author can go on to submit their work to another journal, most likely in the anaesthetic field, for further peer review. This has happened before – I know of one confirmed case where a manuscript was rejected by two journals because of concerns about the data before being published in another journal. As Editor-in-Chief, I struggle as to know what to do in such instances, torn between 'innocent until proven guilty' and preventing suspicious data being published in the public domain. We hope that by screening all submissions from now, we will not in the future publish data from a randomised trial that is not consistent with random sampling. We also hope that that other journals will follow suit and also screen submissions. Only if all journals screen randomised controlled trials before acceptance and publication is there a chance that we can stop this in its tracks. Hence, we call for other editors, statisticians, authors and readers to apply the Carlisle Method for themselves and help validate it. A number of commercial programs to check for plagiarism now exist and are in routine use, and our journal screens all submissions using iThenticate, and does not publish articles with a high CrossCheck score. Perhaps in time a similar system will exist to analyse the distribution of data such that the process becomes automated and ubiquitous. I doubt it will be foolproof – the most determined will get past any system of checks and alarms – but it will surely weed out the Fujiis. After all, our main aim is to stop the publication of fabricated or falsified data in the future, as opposed to identifying the miscreants from the past. AK is the Editor-in-Chief of Anaesthesia. No external funding or other competing interests declared.
Source: Anaesthesia
DOI: 10.1111/anae.13731
Author: A. A. Klein
Institution: Royal Papworth Hospital
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $4,970
OpenAlex ID: W2565522607