Nuts and Bolts of Well-conducted Systematic Reviews: Research Quality Matters!
Year: 2024
Type: review
Abstract: James Lind conducted the first systematic review (SR) in 1753 and published a report that sought to offer a clear and objective presentation of the available data regarding scurvy.[1] However, the increasing need to enhance the state of evidence synthesis was not given significant emphasis until the 1970s and 1980s. “Effectiveness and Efficiency: Random Reflections on Health Service,” a textbook written by Archie Cochrane in 1972, highlighted the crucial nature of randomized control trials (RCTs) in figuring out the results of medical interventions.[2] Gradually, the need to enhance research synthesis was emphasized by policymakers, scholars, and clinicians around the world, that led to the formal establishment of SRs in the late 20th century in order to systematically summarize the available evidence to aid decision-making by various stakeholders and consumers.[3] REACHING THE “BOTTOM LINE” The plethora of ever-increasing medical literature makes it difficult, if not impossible, for busy clinicians and researchers to stay up-to-date. It also poses a number of technical and practical concerns about generalizing findings across the population, such as variations in the sample size, methods, assessment tools, representation of statistical test values, and missing outcomes, along with varying strengths and limitations of the conducted research. Murad et al. proposed a new evidence-based medicine pyramid wherein SRs and meta-analyses (MAs) are viewed as a “lens” through which the quantity and quality of the existing evidence is ascertained.[4] SRs are intended to yield robust, accurate, and reproducible conclusions, allowing for the review of observational, analytical, experimental (randomized or non-randomized) studies as well as case reports, case series, and qualitative studies.[5] Although SRs appear similar to scoping reviews in terms of having explicit, transparent, peer-reviewed search strategy, a priori review protocol, and standardized data extraction forms, SRs have more focused review question, well-defined eligibility criteria, PROSPERO registration, mandatory quality assessment (QA), synthesis of findings from included studies and generation of summarized findings into MA (prevalence or incidence, etiology or risk, diagnostic accuracy or prognostic data, psychometric measures, quantitative effectiveness) by using appropriate statistical measures, or meta-synthesis (experiential or expert opinion data), or both as in mixed methods reviews.[6] SRs that synthesize the findings of different intervention trials are highly effective approaches to get to the “bottom line” on what works and what does not.[5] RATIONALE BEHIND CONDUCTING SYSTEMATIC REVIEW Doubling time for medical knowledge has reduced by more than 95% from 1950 (50 years) to 2020 (2.5 months).[7] Rapid advances in technologies in recent decades have led to an increased rate of publications and improved accessibility, leading to information explosion, with a possible risk of “pointless publications” and “inconsistencies” among studies, amplifying the need for addressing research questions through SRs and MAs. The rationale behind conducting SR has been conceptualized into a color-coded framework under three sections– 1) SR as a type of synthetic research, 2) SR as a more informed and less biased review, and 3) SR as an efficient scientific tool.[8] Although conducting a high-quality large-scale RCT is difficult, if not impossible, due to limiting factors including study design (e. g., drug), conduct of study (e. g., bias, attrition), higher costs, and consumes more time and resources; it is usually faster and less expensive to undertake an SR. If a well-conducted SR, including homogeneous studies, signifies a genuine association between two variables (e.g., a drug and therapeutic effects) with an appropriate confidence interval, it may avoid the need for future studies to test the same hypothesis.[8] REGISTRATION OF PROTOCOL To promote transparency, a detailed and well-defined protocol must be devised prior to the beginning of the review process. It must include the review question(s), search strategies, eligibility criteria (e.g., research design, participant or intervention characteristics), the definition of outcome measurements, and methods used for effect measurement, data analysis, and heterogeneity exploration. Registration of SR at conception (i.e., at the protocol stage) in specific networks, such as PROSPERO or PRISMA, and/or published as a research protocol before conducting the SR is vital.[8] Registration of protocol performs three major purposes. First, it helps to avoid unexpected duplications of review topics. Second, protocols per se could be peer-reviewed (and amended as needed) prior to the review by publishing as a distinct paper. Peer reviewers can assist reviewers in improvising the SR protocol (e.g., incorporating appropriate search strings for a sensitive search) to address a specific issue. Third, a detailed predefined protocol acts as a guideline for reviewers, reducing their dependency on ad-hoc decisions, and aids to reduce subjective bias in conducting and reporting the review. It also allows peer reviewers to evaluate for any deviation from predefined protocol. This improves the systematicity and objectivity of the review process, from both the reviewers’ and peer-reviewers perspective.[8] ADDRESSING EXTERNAL AND INTERNAL VALIDITY SR needs more than one reviewer (preferably three or more) for the purpose of screening, extracting data, and reporting the specifics of each included study with acceptable inter-rater reliability. It is vital for the reviewers to reach a consensus over a study’s screening and data extraction phases as well as how they handled conflicts.[8] Teamwork and inter-rater consensus ensure reduced subjective influence, and increased systematicity and objectivity of the review process, thus increasing the dependability of the review findings. Reviewers need to perform sensitive search with appropriate Boolean operators into multiple research databases (preferably three or more along with gray literature) to screen, summarize and integrate original knowledge, critically appraise for quality of the included studies, attain acceptable inter-rater reliability, check for publication bias, and discuss the findings along with strengths and limitations of the SRs with future implications, in order to conduct more informed and less biased SRs. This increases the external validity and explanatory power of the results, increases the precision of the estimated values through MA and the statistical power to detect genuine relationships.[8] QUALITY ASSESSMENT Quality of studies included in the SR matters as improper handling of data can lead to deceptive, ineffective, or even harmful conclusions. Similarly, errors in the research design and implementation of the trials or studies included for MA may skew the pooled estimates in favor of one intervention over another. MA can also be misused when the distinction between a patient seen in a clinic and those included in the MA are not considered.[9] Hence, QA of studies included in the SR is paramount as it aids in reducing the risk of bias (RoB) and also increases confidence in the review findings. QA, also known as quality appraisal, critical appraisal, or RoB assessment, relates to evaluating the methodological quality and rigor of the trials or studies included in an SR by appraising, judging, and documenting potential RoB. To ensure reliability and objectivity, every reviewer must independently evaluate and critically appraise included studies by using QA tools to ensure relevance, reliability, validity, and applicability of each outcome in the included studies prior to attaining consensus subsequently with co-reviewers.[10] Tools for QA vary according to the types of studies that are being included. Table 1 provides the commonly used QA tools. When studies with varying quality scores are included in SR and MA, the interpretation of the findings can be deceptive. The tools used in QA also have criteria for scoring and considering the values of the studies as high quality, medium and low. For instance, in an SR of RCTs, reviewers assess RoB in six domains of potential bias, which include selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias, as being either high, low, or some concerns.[10] A SR of QA tools reported the availability of tools for different types of studies, with most tools available for assessing RCTs.[11]Table 1: Quality assessment tools commonly used in systematic reviewsCHALLENGES AND LIMITATIONS Despite the existence and frequent implementation of standards, including the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (https://www.prisma-statement.org), a recent living SR highlighted that many articles demonstrated numerous flaws in the conduct, methods, and reporting of published SRs.[12] Such valid critiques may stem from the fact that some current SRs depart from the original philosophy and accepted reasoning underlying the fundamentals of SR.[8] This emphasizes the necessary methodological training to conduct SR and MA effectively. Notably, SRs are not equipped to handle every clinically significant issue, and its recommendations might occasionally pose pragmatic difficulty.[9] SRs may have constraints that can affect the validity and reliability of the results, such as (1) any bias in the search and eligibility criteria used in the review may add to errors or inaccuracies in conclusions,[13] (2) the possibility of missing evidence, i. e., as the volume of included studies in SRs increases, higher probability of crucial evidence being overlooked,[13] (3) the quality of the included studies can differ, influencing the total strength of the evidence,[14] (4) data availability, i. e., some studies may not give sufficient information for analysis,[15] (5) improper data handling can lead to deceptive, ineffective or even harmful conclusions,[9] and (6) measuring differences between socioeconomic categories may appear challenging when one needs to assess its impact on equity analysis in cost-effectiveness studies.[15] CONCLUSION SRs and MAs are frequently cited highest level of evidence that facilitate evidence-based guidelines and medical decision-making. Adhering to higher and stricter standards of methodological and reporting criteria is crucial to eliminate bias, ensure transparency, and improve reliability, relevance, validity, statistical capacity and explanatory power of the findings.[8] However, failure to identify and address methodological flaws in SR undermines scientific credibility.[12] Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.
Source: Journal of Psychiatry Spectrum
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W4404521944