admin
In orthopedic research, systematic literature reviews and meta-analyses are essential for analyzing evidence across studies, drawing meaningful conclusions, and providing reliable recommendations that serve many roles. They can provide a synopsis of the state of knowledge in a field, from which future research priorities can be identified; they can address questions that otherwise could not be answered by individual studies; they can identify problems in primary research that should be rectified in future studies; and they can generate or evaluate theories about how or why phenomena occur. Systematic literature reviews (SLRs) can be leveraged to generate various types of knowledge for different users or stakeholders (patients, healthcare providers, researchers, policy makers, regulators, and reimbursement payors).
SLRs can serve many roles within a regulatory context. SLRs may compile data on a specific device/intervention for a particular outcome(s), often providing evidence for the safety and effectiveness of a medical device or intervention, establishing a reliable literature-based control or performance goal. SLR data-derived traceability and integrity, including data sourcing, collection, synthesis, and interpretation, helps assess the adequacy of outcome(s) or conclusion(s).
For these research methods to be effective and trustworthy, however, researchers must follow a set of standards that ensure transparency, reproducibility, and consistency. One such standard is the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement.
The PRISMA statement is a methodology designed to improve the quality and reliability of SLRs and meta-analyses by providing a detailed framework for reporting and conducting these reviews. By adhering to the PRISMA statement, orthopedic researchers can produce more transparent and reproducible studies that contribute to the integrity and quality of generated evidence. This column explores the PRISMA statement, its key components, and significance in orthopedic research, particularly in the context of the medical device regulatory review.1-3
Why Is the PRISMA Statement Important?
SLRs and meta-analyses are fundamental orthopedic research tools. A systematic literature review seeks to synthesize all relevant studies on a particular topic or clinical question within clearly defined and predetermined boundaries to provide a comprehensive, unbiased summary of the evidence. Meta-analysis refers to a statistical technique often used in conjunction with systematic reviews, which combines data from multiple studies to provide a more precise estimate of an intervention’s or treatment’s effect.
These research methods are particularly valuable in orthopedics because they can help determine treatment efficacy, compare surgical techniques, and evaluate rehabilitation methods and other interventions. By pooling data from various studies, researchers can address questions that individual studies may not be sufficiently powered to answer. However, for the synthesis and analyses to be credible, they must follow stringent guidelines to ensure sound methodology, accurate conclusions, and reproducible findings.
When considering the validity and usefulness of systematic reviews, especially those produced or funded by a product sponsor for regulatory review purposes, regulators are primarily interested in ensuring the review adheres to the guiding methodology.
What Is the PRISMA Statement?
The PRISMA statement provides a standardized framework for reporting systematic reviews and meta-analysis, aiming to improve transparency, consistency, and quality by outlining key elements that should be included when describing the research process, methods, results, and conclusions of a systematic literature review, ultimately enhancing the reliability and interpretability of the finding while decreasing bias risk.
The PRISMA statement was first published in 2009 and later updated in 2020 to reflect evolving research practices and improve the clarity of systematic reviews and meta-analyses. This consists of a 27-item checklist that outlines the essential components researchers must address when reporting a systematic review or meta-analysis. The checklist covers various review process stages, including planning, conducting, analyzing, and reporting results. The checklist can be downloaded as a Microsoft Word document and allows the user to indicate the location where each item is addressed within the review report.
The checklist is accompanied by an “explanation and elaboration” paper, which provides a thorough discussion of each checklist element and helpful examples for its implementation.
It is worth noting here that the aforementioned guidance is applicable to the systematic review report, a separate guidance is available for the development of protocols of systematic reviews (PRISMA-P 2015 statement).
Key PRISMA Statement Components
1. Title and Abstract: The title of a systematic review or meta-analysis should clearly reflect the review’s focus, and the abstract should summarize the study’s main components. The PRISMA statement recommends the abstract include essential information such as the review’s objectives, methods, results, and conclusions. This provides readers with a clear overview of the study’s content and helps the reader quickly assess the review’s relevance.
2. Rationale: This section explains why the review is being conducted and its potential significance. It outlines the clinical question or problem the SLR aims to address. In a regulatory review context, this could involve questions about the medical device safety and effectiveness, establishing a literature-based control or performance goal, and providing a state-of-the-art review for the device type.
3. Objectives: The SLR’s objectives should be clearly stated and focused. In orthopedic research, objectives may vary depending on the review’s scope. The PICO framework (Population, Intervention, Comparator, Outcome) or one of its variants could be a helpful tool.
4. Eligibility Criteria: These criteria define which studies are included or excluded from the review based on factors such as publishing date, study design, follow-up, population, intervention, and outcome measures and definitions. Properly identified eligibility criteria are vital to ensure the review is based on high-quality studies relevant to the research question. Ambiguous criteria such as “no relevant outcome data” could be a red flag and should be avoided when possible.
5. Information Sources and Search Strategy: A comprehensive search strategy is recommended to identify all relevant studies. This includes specifying databases used, search terms, and any language or publication date restrictions. All available databases should be considered unless not applicable or duplicative (with clinical justification). Search terms should account for case sensitive differences in key words or phrases (e.g., total ankle replacement, TAR, total ankle arthroscopy, TAA, etc.). The search terms should include focused terms, outcomes measures, pathological/diagnostic details, and indexing strategies. Any predefined subgroups of interest should also be accounted for in the search strategy. Search dates for each database and search string should be recorded, and all identified citations stored in a citation manager prior to screening.
6. Study Selection: This section documents how studies were selected for the review by the designated reviews. The PRISMA statement recommends multiple reviewers be involved in the screening and study selection process to minimize bias. As an example, two independent reviewers assess studies identified for inclusion, with any disagreement between reviewers adjudicated by an independent third reviewer. Reasons for exclusion should be clearly documented and indicate agreement across the reviewers. Study selection is often a time-consuming and resource intensive step. Various artificial intelligence (AI)-powered platforms are available to streamline the screening process.
7. Data Extraction: This involves collecting key information from the included studies, such as sample size, intervention details, outcomes, and results. It is suggested that data extraction be conducted independently by multiple reviewers to reduce errors and inconsistencies. All data relevant to the SLR’s goal should be extracted. For data embedded in graphs or figures, various software tools are available.
8. Risk of Bias Assessment: Assessing bias risk in included studies is a critical part of the systematic review process. The PRISMA statement encourages the use of validated tools to evaluate bias risk in the included studies, which can help identify potential threats to the validity of the review’s findings (Cochrane Methods ROBINS-I tool). In orthopedic research, factors such as study design, sample size, and funding sources can introduce biases that may affect the results’ interpretation. Bias risk should be assessed for each individual study included in the review. Studies identified as having a “high” bias risk could be excluded from a sensitivity analysis when synthesizing data.
9. Synthesis of Results: Synthesis involves summarizing the studies’ findings and discussing the device or intervention’s overall effect. In meta-analysis, this step involves statistical analysis to combine results from multiple studies. The PRISMA statement recommends using appropriate statistical techniques such as random-effects models or fixed-effects models to ensure the synthesis accurately reflects the data.
10. Subgroup Analyses and Sensitivity Analyses: Subgroup analyses explore whether certain factors (age, gender, comorbidities, etc.) influence outcomes. Sensitivity analyses help assess results’ robustness by testing how they change when certain assumptions or parameters are altered. In orthopedic reviews, these analyses are valuable for understanding how different patient populations may respond to medical devices or interventions.
11. Results and Discussion:The results section presents key findings of the review or meta-analysis, including effect sizes, confidence intervals, and statistical significance. The discussion section interprets these results in the context of existing evidence, discusses the implications for clinical practice, and highlights any limitations of the review. If the SLR’s purpose is to define a performance goal, those calculations and any statistical assumptions can be detailed in this step.
12. Funding and Conflicts of Interest: The PRISMA statement recommends that authors disclose any funding sources or conflicts of interest that may affect the review’s objectivity. Transparency in this regard is critical for maintaining the research’s credibility. In orthopedic studies, funding sources could include medical device manufacturers, academic institutions, or other key stakeholders that may have a vested interest in the review’s outcomes.
Benefits of Adhering to PRISMA Guidelines
Adherence to the PRISMA methodology offers several key benefits:
1. Increased transparency: By following the PRISMA statement, researchers make their methods and findings more transparent, allowing others (including regulators) to replicate and verify their work. This transparency is essential for establishing trust in the results and ensuring that conclusions are based on solid evidence.
2. Higher quality reviews: The structured approach provided by the PRISMA statement ensures that reviews are thorough and methodologically sound. This leads to higher-quality research that can guide clinical decision-making, inform policy, and contribute to improved patient outcomes. Higher quality reviews are at less risk of errors or bias that cast doubt on the conclusions.
3. Reduced bias: One of the PRISMA statement’s key focuses is minimizing bias at every stage of the review process. This is particularly important when the systemic review is performed or funded by a key stakeholder.
4. Enhanced credibility: Systematic reviews and meta-analyses that follow the PRISMA methodology are more likely to be accepted by peer-reviewed journals and trusted by regulators.
Concluding Recommendations
The PRISMA methodology is a useful framework for ensuring the rigor, transparency, and quality of systematic reviews and meta-analyses in orthopedic research. By following these guidelines, researchers can produce reliable, high-quality reviews that contribute to evidence-based practice and improve patient outcomes. As the field of orthopedics continues to evolve, adhering to PRISMA standards will help ensure the evidence base remains robust, trustworthy, and applicable to decision-making. Although typically more arduous and costly—largely due to the number of reviewers and additional components—SLRs performed properly and in adherence with the PRISMA statement increase confidence in the results and conclusions. Reviewer selection is an important consideration for stakeholders. For best results, reviewers should be knowledgeable on the topic or device class, ideally with prior experience performing SLRs. If an SLR is a key component of a regulatory submission, it is advisable to solicit regulator feedback on the protocol before performing the full SLR. As noted previously, a separate guidance (PRISMA-P 2015 statement) is available to guide protocol development.
References
1 The PRISMA Statement: A Guideline for Systematic Reviews and Meta-Analyses (https://www.prisma-statement.org/)
2 https://www.bmj.com/content/372/bmj.n71
3 Swartz MK. The PRISMA statement: a guideline for systematic reviews and meta-analyses. J Pediatr Health Care. 2011 Jan-Feb;25(1):1-2
MORE FROM MCRA: Cybersecurity Risks in Robotic Surgical Systems
Tim Sutton is a regulatory affairs associate at MCRA, supporting and advising clients in completing various U.S. regulatory submissions, specifically for orthopedic and spine devices. Sutton has led and supported various systematic literature reviews aligning with the PRISMA statement on behalf of client companies for use in U.S. regulatory submissions. Sutton attended Clemson University and earned a bachelor’s degree in biomedical engineering.
