Section 2.2: Knowledge Synthesis

[ Table of Contents ]

Jennifer Tetzlaff, Ottawa Hospital Research Institute
Andrea C. Tricco, PhD, Ottawa Hospital Research Institute
David Moher, PhD, Ottawa Hospital Research Institute

Definitions*

  • Knowledge syntheses (KS) consist of a clearly formulated question and use systematic and explicit methods to identify, select, critically appraise, and interpret data from relevant research
  • A meta-analysis is a statistical technique used to quantitatively integrate the results of included studies in a KS
  • A KS does not necessarily include a meta-analysis

* From the Cochrane Collaboration

Knowledge Synthesis

  • KS is used to interpret individual study results within the context of global evidence
  • KS can be used as a starting point for practice guidelines and new primary research (e.g. trials)
  • KS bridges the gap between research and decision-making

Types of Evidence

  • Many groups worldwide conduct KS and methods involved usually depends on the question(s) being considered
    • Questions regarding the effectiveness of interventions will usually include quantitative evidence (e.g. odds ratio for a particular drug versus placebo)
    • Contextual questions about why an intervention worked in a particular context will usually include qualitative evidence

General Methods of a KS

  • Incorporating qualitative evidence into KS can be:
    • Challenging
    • Difficult to locate qualitative evidence
    • Difficult to integrate qualitative evidence with quantitative evidence
    • Methods are just emerging
  • As such, the focus will be on general methods applicable to most KS

Overview

  • Assembling the review team
  • Formulating the question, protocol, and eligibility criteria
  • Identifying relevant studies
  • Selecting studies
  • Risk of bias assessment
  • Data extraction
  • Data analysis
  • Presenting results
  • Interpreting results

The Review Team

  • Determined by the question
  • May include the following people:
    • Clinical experts with knowledge of the topic
    • Methodologists who know the KS process
    • Librarian to help locate relevant studies
    • Researchers who conducted studies on the topic
    • Funder or commissioning agency for context
    • Statistician if meta-analysis is being considered
    • End-users (e.g. policy makers, patients) to increase relevance and uptake

Formulating the Question

  • Most important step because it guides the KS process
  • PICO (S or T): Population, Intervention, Comparators, Outcome, and (Study design or Time period) facilitates question development
  • May not fit all KS (e.g. Intervention sometimes replaced by Exposure) but still useful template to consider

Formulating the Protocol

  • Pre-specifies the review process
  • Important because it decreases post-hoc changes to methods and selective outcome reporting
  • Elements include primary versus secondary outcomes, search methods, appraisal of the literature, and data abstraction
  • Any changes to the protocol should be transparently reported in the review write-up

Formulating the Eligibility Criteria

  • Should extend from the question
  • Based on PICO (S or T)
  • Consider language of publication
  • Consider publication status (e.g. published vs. unpublished material)
  • Needs to be thoroughly considered, properly defined, and transparently reported

Identifying Relevant Studies

  • Based on the question and PICO (S or T)
  • MEDLINE, EMBASE, and The Cochrane Library electronic databases are commonly used for health-related research
  • At least 2 relevant databases should be searched
  • Advisable that a librarian guides this process
  • Should search for unpublished and difficult to locate (i.e. grey) literature (e.g. trial registries, public health agency websites)

Selecting Studies

  • Based on the eligibility criteria
  • 2 stages: broad screening of titles and abstracts and stricter screening of potentially relevant full-text articles
  • 2 independent reviewers should screen at all levels to ensure relevant studies aren't missed
  • Agreement between reviewers can be assessed using the kappa statistic

Risk of Bias Assessment

  • Many assessment tools have been developed to asses the risk of bias for different study designs
  • Only reporting a summary score is not advisable; the assessments for each criterion from the quality appraisal should be reported for each study
  • Sensitivity analyses on risk of bias can be conducted versus excluding studies based on risk of bias

Data Extraction

  • Primary outcomes should be differentiated from secondary outcomes
  • Data extraction form(s) should be developed a priori and pilot-tested to increase reliability
  • Potential errors are decreased if more than one reviewer independently extracts the data
  • Authors of included studies should be contacted for missing or unclear information

Data Analysis

  • Depends on the question and type of data collected
  • All KS must have a narrative synthesis of results and risk of bias
  • Standard effect measures (e.g. odds ratios, hazard ratio) may be used
  • Meta-analysis may not be possible or advisable if outcomes were assessed inconsistently and clinical, methodological or statistical heterogeneity is observed

Presenting Results

  • Screening process should be described in the text and/or presented as a flow-chart
  • Characteristics of included studies should be described in the text and/or a table (e.g. participant populations, interventions)
  • Results of risk of bias assessments should be presented in a table and/or text
  • Quantitative data should be presented as summary data (e.g. effect estimates with confidence intervals for each study) and may be presented for each outcome in a table or in a forest plot figure
  • Qualitative data can be presented visually (e.g. conceptual framework)

Interpreting Results

  • Discuss risk of bias, strength, and applicability of the evidence for each outcome
  • Relevance of the results should be considered for key stakeholders to increase applicability
  • Qualitative evidence can help explain how the intervention worked and whether it will work in a different setting
  • Should consider study and KS limitations

Disseminating Results

  • Most common form of dissemination is publication in peer-reviewed journals
    • Open access journals will increase dissemination
  • Targeted dissemination via media for the public, brief reports for health care providers, policy makers and consumers, and decision-aids for patients

Uptake of Results

Future Research

  • Increasing the uptake of KS
  • How best to update KS
  • Comparability between different types of KS (e.g. rapid reviews versus conventional reviews)
  • How to prioritize KS topics

Contact Details

Acknowledgments

  • We thank CIHR for their funding and Dr Sharon Straus for her funding and useful feedback on our paper
  • ACT: CIHR (Frederick Banting and Charles Best Canada Graduate Scholarship and a Rising Star Award from the Institute of Health Services and Policy Research)
  • DM: University of Ottawa Research Chair
Date modified: