example of systematic review in research

Doing a Systematic Review: A Student’s Guide

Saul McLeod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul McLeod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

What is Systematic Review?

A systematic review is a comprehensive, structured analysis of existing research on a specific topic. It uses predefined criteria to identify, evaluate, and synthesize relevant studies, aiming to provide an unbiased summary of the current evidence.

The explicit and systematic approach of a systematic review distinguishes it from traditional reviews and commentaries.

How systematic reviews differ from narrative reviews:

• Goals: Narrative reviews provide a summary or overview of a topic, while systematic reviews answer a focused review question.
• Sources of Literature: Narrative reviews often use a non-exhaustive and unstated body of literature, which can lead to publication bias. Systematic reviews consider a list of databases, grey literature, and other sources.
• Selection Criteria: Narrative reviews usually use subjective or no selection criteria, which can lead to selection bias. Systematic reviews have a clear and explicit selection process.
• Appraisal of Study Quality: Narrative reviews vary in their evaluation of study quality. Systematic reviews use standard checklists for a rigorous appraisal of study quality.

Systematic reviews are time-intensive and need a research team with multiple skills and contributions. There are some cases where systematic reviews are unable to meet the necessary objectives of the review question.

In these cases, scoping reviews (which are sometimes called scoping exercises/scoping studies) may be more useful to consider.

Scoping reviews are different from systematic reviews because they may not include a mandatory critical appraisal of the included studies or synthesize the findings from individual studies.

Assessing The Need For A Systematic Review

When assessing the need for a systematic review, one must first check if any existing or ongoing reviews already exist and determine if a new review is justified.

Scoping reviews frequently serve as preliminary steps before conducting full systematic reviews. They help assess the available literature’s breadth, identify key concepts, and determine the feasibility of a more comprehensive review.

This initial exploration guides researchers in refining their approach for subsequent in-depth analyses.

This process should begin by searching relevant databases.

Resources to consider searching include:

• NICE : National Institute for Health and Clinical Excellence
• Campbell Library of Systematic Reviews for reviews in education, crime and justice, and social welfare
• EPPI : Evidence for Policy and Practice Information Centre, particularly their database of systematic and non-systematic reviews of public health interventions (DoPHER)
• MEDLINE : Primarily covers the medical domain, making it a primary resource for systematic reviews concerning healthcare interventions
• PsycINFO : For research in psychology, psychiatry, behavioral sciences, and social sciences
• Cochrane Library (specifically CDSR) : Focuses on systematic reviews of health care interventions, providing regularly updated and critically appraised reviews

If an existing review addressing the question of interest is found, its quality should be assessed to determine its suitability for guiding policy and practice.

If a high-quality, relevant review is located, but its completion date is some time ago, updating the review might be warranted.

Assessing current relevance is vital, especially in rapidly evolving research fields. Collaboration with the original research team might be beneficial during the update process, as they could provide access to their data.

If the review is deemed to be of adequate quality and remains relevant, undertaking another systematic review may not be necessary.

When a new systematic review or an update is deemed necessary, the subsequent step involves establishing a review team and potentially an advisory group, who will then develop the review protocol.

How To Conduct A Systematic Review

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is a reporting guideline designed to improve the transparency and completeness of systematic review reporting.

PRISMA was created to tackle the issue of inadequate reporting often found in systematic reviews:

• Checklist : PRISMA features a 27-item checklist covering all aspects of a systematic review, from the rationale and objectives to the synthesis of findings and discussion of limitations. Each checklist item is accompanied by detailed reporting recommendations in an Explanation and Elaboration document .
• Flow Diagram : PRISMA also includes a flow diagram to visually represent the study selection process, offering a clear, standardized way to illustrate how researchers arrived at the final set of included studies.

Step 1: write a research protocol

A protocol in the context of systematic reviews is a detailed plan that outlines the methodology to be employed throughout the review process.

The protocol serves as a roadmap, guiding researchers through each stage of the review in a transparent and replicable manner.

This document should provide specific details about every stage of the research process, including the methodology for identifying, selecting, and analyzing relevant studies.

For example, the protocol should specify search strategies for relevant studies, including whether the search will encompass unpublished works.

The protocol should be created before beginning the research process to ensure transparency and reproducibility.

This pre-determined plan ensures that decisions made during the review are objective and free from bias, as they are based on pre-established criteria.

Protocol modifications are sometimes necessary during systematic reviews. While adhering to the protocol is crucial for minimizing bias, there are instances where modifications are justified. For instance, a deeper understanding of the research question that emerges from examining primary research might necessitate changes to the protocol.

Systematic reviews should be registered at inception (at the protocol stage) for these reasons:

• To help avoid unplanned duplication
• To enable the comparison of reported review methods with what was planned in the protocol

This registration prevents duplication (research waste) and makes the process easy when the full systematic review is sent for publication.

PROSPERO is an international database of prospectively registered systematic reviews in health and social care. Non-Cochrane protocols should be registered on PROSPERO.

Research Protocol

Rasika Jayasekara, Nicholas Procter. The effects of cognitive behaviour therapy for major depression in older adults: a systematic review. PROSPERO 2012 CRD42012003151 Available from:  https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42012003151

Review question

How effective is CBT compared with other interventions, placebo or standard treatment in achieving relapse prevention and improving mental status for older adults with major depression?

The search strategy aims to find both published and unpublished studies and publications. The search will be limited to English language papers published from 2002 to 2012.

A three-step search strategy will be developed using MeSH terminology and keywords to ensure that all materials relevant to the review are captured.

An initial limited search of MEDLINE and CINAHL will be undertaken followed by an analysis of the text words contained in the title and abstract, and of the index terms used to describe the article. A second search using all identified keywords and index terms will then be undertaken.

Thirdly, the reference list of all identified reports and articles will be searched for additional studies.

The databases to be searched included:

• Cochrane Central Register of Controlled Trials
• Controlled Trials
• Current Contents

The search for unpublished studies will include:

• Digital Dissertations (Proquest)
• Conference Proceedings

Experts in the field will be contacted for ongoing and unpublished trials. Experts will be identified through journal publications.

Types of study to be included

All randomised controlled trials (RCTs) assessing the effectiveness of CBT as a treatment for older adults with major depression when compared to standard care, specific medication, other therapies and no intervention will be considered.

In the absence of RCTs, other research designs such as quasi-experimental studies, case-controlled studies and cohort studies will be examined. However, descriptive studies and expert opinion will be excluded.

Condition or domain being studied

Major depression is diagnosed according to DSM IV or ICD 10 criteria.

Where trials fail to employ diagnostic criteria, the severity of depression will be described by the use of standardised rating scales, including the Hamilton Depression Rating Scale, Montgomery and Asberg Rating Scale and the Geriatric Depression Rating Scale.

The trials including participants with an explicit diagnosis of dementia or Parkinson’s disease and other mental illnesses will be excluded.

The review will include trials conducted in primary, secondary, community, nursing homes and in-patient settings.

Participants/population

The review will include trials in which patients are described as elderly, geriatric, or older adults, or in which all patients will be aged 55 or over (many North American trials of older adult populations use a cut-off of 55 years).

The review will include trials with subjects of either sex. Where possible, participants will be categorised as community or long term care residents.

Intervention(s), exposure(s)

The review will focus on interventions designed to assess the effects of CBT for older adults with major depression.

The label cognitive behavioural therapy has been applied to a variety of interventions and, accordingly, it is difficult to provide a single, unambiguous definition.

In order to be classified as CBT the intervention must clearly demonstrate the following components:

• the intervention involves the recipient establishing links between their thoughts, feelings and actions with respect to the target symptom;
• the intervention involves the correction of the person’s misperceptions, irrational beliefs and reasoning biases related to the target symptom.
• – the recipient monitoring his or her own thoughts, feelings and behaviours with respect to the target symptom; and
• – the promotion of alternative ways of coping with the target symptom.

In addition, all therapies that do not meet these criteria (or that provide insufficient information) but are labelled as ‘CBT’ or ‘Cognitive Therapy’ will be included as ‘less well defined’ CBT.

Comparator(s)/control

other interventions, placebo or standard treatment

Main outcome(s)

Primary outcomes

• Depression level as assessed by Hamilton Depression Rating Scale, Montgomery or Asberg Rating Scale or the Geriatric Depression Rating Scale.
• Relapse (as defined in the individual studies)
• Death (sudden, unexpected death or suicide).
• Psychological well being (as defined in the individual studies)

Measures of effect

The review will categorise outcomes into those measured in the shorter term (within 12 weeks of the onset of therapy), medium term (within 13 to 26 weeks of the onset of therapy) and longer term (over 26 weeks since the onset of therapy).

Additional outcome(s)

Secondary outcomes

• Mental state
• Quality of life
• Social functioning
• Hospital readmission
• Unexpected or unwanted effect (adverse effects), such as anxiety, depression and dependence on the relationship with the therapist

Data extraction (selection and coding)

Data will be extracted from papers included in the review using JBI-MAStARI. In this stage, any relevant studies will be extracted in relation to their population, interventions, study methods and outcomes.

Where data are missing or unclear, authors will be contacted to obtain information.

Risk of bias (quality) assessment

All papers selected for retrieval will be assessed by two independent reviewers for methodological validity prior to inclusion in the review.

Since the review will evaluate the experimental studies only, The Joanna Briggs Institute Meta Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI) will be used to evaluate each study’s methodological validity.

If there is a disagreement between the two reviewers, there will be a discussion with the third reviewer to solve the dissimilarity.

Strategy for data synthesis

Where possible quantitative research study results will be pooled in statistical meta-analysis using Review Manager Software from the Cochrane Collaboration.

Odds ratio (for categorical outcome data) or standardised mean differences (for continuous data) and their 95% confidence intervals will be calculated for each study.

Heterogeneity will be assessed using the standard Chi-square. Where statistical pooling is not possible the findings will be presented in narrative form.

Step 2: formulate a research question 

Developing a focused research question is crucial for a systematic review, as it underpins every stage of the review process.

The question defines the review’s nature and scope, guides the identification of relevant studies, and shapes the data extraction and synthesis processes.

It’s essential that the research question is answerable and clearly stated in the review protocol, ensuring that the review’s boundaries are well-defined.

A narrow question may limit the number of relevant studies and generalizability, while a broad question can make it challenging to reach specific conclusions.

PICO Framework

The PICO framework is a model for creating focused clinical research questions. The acronym PICO stands for:

• P opulation/Patient/Problem: This element defines the specific group of people the research question pertains to.
• I ntervention: This is the treatment, test, or exposure being considered for the population.
• C omparison: This is the alternative intervention or control group against which the intervention is being compared.
• O utcome: This element specifies the results or effects of the interventions being investigated

Using the PICO format when designing research helps to minimize bias because the questions and methods of the review are formulated before reviewing any literature.

The PICO elements are also helpful in defining the inclusion criteria used to select sources for the systematic review.

The PICO framework is commonly employed in systematic reviews that primarily analyze data from randomized controlled trials .

Not every element of PICO is required for every research question. For instance, it is not always necessary to have a comparison

Types of questions that can be answered using PICO:

“In patients with a recent acute stroke (less than 6 weeks) with reduced mobility ( P ), is any specific physiotherapy approach ( I ) more beneficial than no physiotherapy ( C ) at improving independence in activities of daily living and gait speed ( O )?

“For women who have experienced domestic violence ( P ), how effective are advocacy programmes ( I ) compared to other treatments ( C ) on improving the quality of life ( O )?”

Etiology/Harm

Are women with a history of pelvic inflammatory disease (PID) ( P ) at higher risk for gynecological cancers ( O ) than women with no history of PID ( C )?

Among asymptomatic adults at low risk of colon cancer ( P ), is fecal immunochemical testing (FIT) ( I ) as sensitive and specific for diagnosing colon cancer ( O ) as colonoscopy ( C )?

Among adults with pneumonia ( P ), do those with chronic kidney disease (CKD) ( I ) have a higher mortality rate ( O ) than those without CKD ( C )?

Alternative Frameworks

• PICOCS : This framework, used in public health research, adds a “ C ontext” element to the PICO framework. This is useful for examining how the environment or setting in which an intervention is delivered might influence its effectiveness.
• PICOC : This framework expands on PICO by incorporating “ C osts” as an element of the research question. It is particularly relevant to research questions involving economic evaluations of interventions.
• ECLIPSE : E xpectations, C lient group, L ocation, I mpact, P rofessionals involved, S ervice, and E valuation. It is a mnemonic device designed to aid in searching for health policy and management information.
• PEO : This acronym, standing for P atient, E xposure, and O utcome, is a variation of PICO used when the research question focuses on the relationship between exposure to a risk factor and a specific outcome.
• PIRD : This acronym stands for P opulation, I ndex Test, R eference Test, and D iagnosis of Interest, guiding research questions that focus on evaluating the diagnostic accuracy of a particular test.
• PFO : This acronym, representing P opulation, P rognostic F actors, and O utcome, is tailored for research questions that aim to investigate the relationship between specific prognostic factors and a particular health outcome.
• SDMO : This framework, which stands for S tudies, D ata, M ethods, and O utcomes, assists in structuring research questions focused on methodological aspects of research, examining the impact of different research methods or designs on the quality of research findings.

Step 3: Search Strategy

PRISMA  (Preferred Reporting Items for Systematic reviews and Meta-Analyses) provide appropriate guidance for reporting quantitative literature searches.

Present the full search strategies for all databases, registers and websites, including any filters and limits used. PRISMA 2020 Checklist

A search strategy is a comprehensive and reproducible plan for identifying all relevant research studies that address a specific research question.

This systematic approach to searching helps minimize bias and distinguishes systematic reviews from other types of literature reviews.

It’s important to be transparent about the search strategy and document all decisions for auditability. The goal is to identify all potentially relevant studies for consideration.

Here’s a breakdown of a search strategy:

Search String Construction

It is recommended to consult topic experts on the review team and advisory board in order to create as complete a list of search terms as possible for each concept.

To retrieve the most relevant results, a search string is used. This string is made up of:

• Keywords:  Search terms should be relevant to the subject areas of the research question and should be identified for all components of the research question (e.g., Population, Intervention, Comparator, and Outcomes – PICO). Using relevant keywords helps minimize irrelevant search returns. Sources such as dictionaries, textbooks, and published articles can help identify appropriate keywords.
• Synonyms: These are words or phrases with similar meanings to the keywords, as authors may use different terms to describe the same concepts. Including synonyms helps cover variations in terminology and increases the chances of finding all relevant studies. For example, a drug intervention may be referred to by its generic name or by one of its several proprietary names.
• Truncation symbols : These broaden the search by capturing variations of a keyword. They function by locating every word that begins with a specific root. For example, if a user was researching interventions for smoking, they might use a truncation symbol to search for “smok*” to retrieve records with the words “smoke,” “smoker,” “smoking,” or “smokes.” This can save time and effort by eliminating the need to input every variation of a word into a database.
• Boolean operators: The use of Boolean operators (AND/OR/NEAR/NOT) helps to combine these terms effectively, ensuring that the search strategy is both sensitive and specific. For instance, using “AND” narrows the search to include only results containing both terms, while “OR” expands it to include results containing either term.

Information Sources

The primary goal is to find all published and unpublished studies that meet the predefined criteria of the research question. This includes considering various sources beyond typical databases

Information sources for systematic reviews can include a wide range of resources like scholarly databases, unpublished literature, conference papers, books, and even expert consultations.

Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. PRISMA 2020 Checklist

An exhaustive, systematic search strategy is developed with the assistance of an expert librarian.

• Electronic Databases : Searches should include seven key databases: CINAHL, Medline, APA PsycArticles, Psychology and Behavioral Sciences Collection, APA PsycInfo, SocINDEX with Full Text, and Web of Science: Core Collections.
• Grey Literature : In addition to databases, forensic or ‘expansive’ searches can be conducted. This includes: grey literature database searches (e.g.  OpenGrey , WorldCat , Ethos ),  conference proceedings, unpublished reports, theses  , clinical trial databases , searches by names of authors of relevant publications. Independent research bodies may also be good sources of material, e.g. Centre for Research in Ethnic Relations , Joseph Rowntree Foundation , Carers UK .
• Citation Searching : Reference lists often lead to highly cited and influential papers in the field, providing valuable context and background information for the review.
• Handsearching : Manually searching through specific journals or conference proceedings page-by-page is another way to ensure all relevant studies are captured, particularly those not yet indexed in databases.
• Contacting Experts : Reaching out to researchers or experts in the field can provide access to unpublished data or ongoing research not yet publicly available.

It is important to note that this may not be an exhaustive list of all potential databases.

A systematic computerized search was performed for publications that appeared between 1974 and 2018 in English language journals. Four databases were searched including PsychINFO, Embase, OVOID MEDLINE, and AMED. The databases were searched with combinations of search terms relating to attachment (“attachment” OR “working model” OR “safe haven” OR “secure base” OR “felt security”) AND romantic couples (“dyad” OR “couple” OR “spous” OR “partner” OR “romantic” OR “wife” OR “husband” OR “close relationship” OR “interpersonal” OR “intimate” OR “mari”) AND social support (“support prov” OR “caregiving” OR “support giv” OR “social support” OR “enacted support” OR “support received” OR “receiv* support” OR “prov support” OR “dyadic coping” OR “interpersonal coping” OR “collaborative coping” OR “help‐seeking” OR “emotional support” OR “tangible support” OR “instrumental support” OR “perceived support” OR “responsive” OR “buffer” OR “partner support” OR “Support avail*” OR “available support”). The reference lists of the retrieved studies were checked to find other relevant publications, which were not identified in the computerized database searches.

Inclusion Criteria

Specify the inclusion and exclusion criteria for the review. PRISMA 2020 Checklist

Before beginning the literature search, researchers should establish clear eligibility criteria for study inclusion.

Inclusion criteria are used to select studies for a systematic review and should be based on the study’s research method and PICO elements.

To maintain transparency and minimize bias, eligibility criteria for study inclusion should be established a priori. Ideally, researchers should aim to include only high-quality randomized controlled trials that adhere to the intention-to-treat principle.

The selection of studies should not be arbitrary, and the rationale behind inclusion and exclusion criteria should be clearly articulated in the research protocol.

When specifying the inclusion and exclusion criteria, consider the following aspects:

• Intervention Characteristics: Researchers might decide that, in order to be included in the review, an intervention must have specific characteristics. They might require the intervention to last for a certain length of time, or they might determine that only interventions with a specific theoretical basis are appropriate for their review.
• Population Characteristics: A systematic review might focus on the effects of an intervention for a specific population. For instance, researchers might choose to focus on studies that included only nurses or physicians.
• Outcome Measures: Researchers might choose to include only studies that used outcome measures that met a specific standard.
• Age of Participants: If a systematic review is examining the effects of a treatment or intervention for children, the authors of the review will likely choose to exclude any studies that did not include children in the target age range.
• Diagnostic Status of Participants: Researchers conducting a systematic review of treatments for anxiety will likely exclude any studies where the participants were not diagnosed with an anxiety disorder.
• Study Design: Researchers might determine that only studies that used a particular research design, such as a randomized controlled trial, will be included in the review.
• Control Group: In a systematic review of an intervention, researchers might choose to include only studies that included certain types of control groups, such as a waiting list control or another type of intervention.
• Publication status : Decide whether only published studies will be included or if unpublished works, such as dissertations or conference proceedings, will also be considered.

Studies that met the following criteria were included: (a) empirical studies of couples (of any gender) who are in a committed romantic relationship, whether married or not; (b) measurement of the association between adult attachment and support in the context of this relationship; (c) the article was a full report published in English; and (d) the articles were reports of empirical studies published in peer‐reviewed journals, dissertations, review papers, and conference presentations.

Iterative Process

The iterative nature of developing a search strategy for systematic reviews stems from the need to refine and adapt the search process based on the information encountered at each stage.

A single attempt rarely yields the perfect final strategy. Instead, it is an evolving process involving a series of test searches, analysis of results, and discussions among the review team.

Here’s how the iterative process unfolds:

• Initial Strategy Formulation: Based on the research question, the team develops a preliminary search strategy, including identifying relevant keywords, synonyms, databases, and search limits.
• Test Searches and Refinement: The initial search strategy is then tested on chosen databases. The results are reviewed for relevance, and the search strategy is refined accordingly. This might involve adding or modifying keywords, adjusting Boolean operators, or reconsidering the databases used.
• Discussions and Iteration: The search results and proposed refinements are discussed within the review team. The team collaboratively decides on the best modifications to improve the search’s comprehensiveness and relevance.
• Repeating the Cycle: This cycle of test searches, analysis, discussions, and refinements is repeated until the team is satisfied with the strategy’s ability to capture all relevant studies while minimizing irrelevant results.

The iterative nature of developing a search strategy is crucial for ensuring that the systematic review is comprehensive and unbiased.

By constantly refining the search strategy based on the results and feedback, researchers can be more confident that they have identified all relevant studies.

This iterative process ensures that the applied search strategy is sensitive enough to capture all relevant studies while maintaining a manageable scope.

Throughout this process, meticulous documentation of the search strategy, including any modifications, is crucial for transparency and future replication of the systematic review.

Step 4: Search the Literature

Conduct a systematic search of the literature using clearly defined search terms and databases.

Applying the search strategy involves entering the constructed search strings into the respective databases’ search interfaces. These search strings, crafted using Boolean operators, truncation symbols, wildcards, and database-specific syntax, aim to retrieve all potentially relevant studies addressing the research question.

The researcher, during this stage, interacts with the database’s features to refine the search and manage the retrieved results.

This might involve employing search filters provided by the database to focus on specific study designs, publication types, or other relevant parameters.

Applying the search strategy is not merely a mechanical process of inputting terms; it demands a thorough understanding of database functionalities and a discerning eye to adjust the search based on the nature of retrieved results.

Step 5: screening and selecting research articles

Once the search strategy is finalized, it is applied to the selected databases, yielding a set of search results.

These search results are then screened against pre-defined inclusion criteria to determine their eligibility for inclusion in the review.

The goal is to identify studies that are both relevant to the research question and of sufficient quality to contribute to a meaningful synthesis.

Studies meeting the inclusion criteria are usually saved into electronic databases, such as Endnote or Mendeley , and include title, authors, date and publication journal along with an abstract (if available).

Study Selection

Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process. PRISMA 2020 Checklist

The selection process in a systematic review involves multiple reviewers to ensure rigor and reliability.

To minimize bias and enhance the reliability of the study selection process, it is recommended that at least two reviewers independently assess the eligibility of each study. This independent assessment helps reduce the impact of individual biases or errors in judgment.

• Initial screening of titles and abstracts: After applying a strategy to search the literature, the next step involves screening the titles and abstracts of the identified articles against the predefined inclusion and exclusion criteria. During this initial screening, reviewers aim to identify potentially relevant studies while excluding those clearly outside the scope of the review. It is crucial to prioritize over-inclusion at this stage, meaning that reviewers should err on the side of keeping studies even if there is uncertainty about their relevance. This cautious approach helps minimize the risk of inadvertently excluding potentially valuable studies.
• Retrieving and assessing full texts: For studies which a definitive decision cannot be made based on the title and abstract alone, reviewers need to obtain the full text of the articles for a comprehensive assessment against the predefined inclusion and exclusion criteria. This stage involves meticulously reviewing the full text of each potentially relevant study to determine its eligibility definitively.
• Resolution of disagreements : In cases of disagreement between reviewers regarding a study’s eligibility, a predefined strategy involving consensus-building discussions or arbitration by a third reviewer should be in place to reach a final decision. This collaborative approach ensures a fair and impartial selection process, further strengthening the review’s reliability.

First, the search results from separate databases were combined, and any duplicates were removed. The lead author (S. M.) and a postgraduate researcher (F. N.) applied the described inclusion criteria in a standardized manner. First, both the titles and abstracts of the articles were evaluated for relevance. If, on the basis of the title and/or abstract, the study looked likely to meet inclusion criteria hard copies of the manuscripts were obtained. If there was doubt about the suitability of an article, then the manuscript was included in the next step. The remaining articles were obtained for full‐text review, and the method and results sections were read to examine whether the article fitted the inclusion criteria. If there was doubt about the suitability of the manuscripts during this phase, then this article was discussed with another author (C. H.). Finally, the reference lists of the eligible articles were checked for additional relevant articles not identified during the computerized search. For the selected articles (n = 43), the results regarding the relationship between attachment and support were included in this review (see Figure 1, for PRISMA flowchart).

PRISMA Flowchart

The PRISMA flowchart is a visual representation of the study selection process within a systematic review.

The flowchart illustrates the step-by-step process of screening, filtering, and selecting studies based on predefined inclusion and exclusion criteria.

The flowchart visually depicts the following stages:

• Identification: The initial number of titles and abstracts identified through database searches.
• Screening: The screening process, based on titles and abstracts.
• Eligibility: Full-text copies of the remaining records are retrieved and assessed for eligibility.
• Inclusion: Applying the predefined inclusion criteria resulted in the inclusion of publications that met all the criteria for the review.
• Exclusion: The flowchart details the reasons for excluding the remaining records.

This systematic and transparent approach, as visualized in the PRISMA flowchart, ensures a robust and unbiased selection process, enhancing the reliability of the systematic review’s findings.

The flowchart serves as a visual record of the decisions made during the study selection process, allowing readers to assess the rigor and comprehensiveness of the review.

• How to fill a PRISMA flow diagram

Step 6: Criticallay Appraising the Quality of Included Studies

Quality assessment provides a measure of the strength of the evidence presented in a review.

High-quality studies with rigorous methodologies contribute to a more robust and reliable evidence base, increasing confidence in the review’s conclusions.

Conversely, including low-quality studies with methodological weaknesses can undermine the review’s findings and potentially lead to inaccurate recommendations.

To judge the quality of studies included in a systematic review, standardized instruments, such as checklists and scales, are commonly used. These tools help to ensure a transparent and reproducible assessment process.

The choice of tool should be justified and aligned with the study design and the level of detail required. Using quality scores alone is discouraged; instead, individual aspects of methodological quality should be considered.

Here are some specific tools mentioned in the sources:

• Jadad score
• Cochrane Risk of Bias tool
• Cochrane Effective Practice and Organisation of Care (EPOC) Group Risk of Bias Tool
• Quality Assessment of Diagnostic Accuracy Studies (QUADAS)
• Newcastle – Ottawa Quality Assessment Scale for case-control and cohort studies
• EPHPP Assessment Tool
• Critical Appraisal Skills Programme (CASP) Appraisal Checklist
• Cochrane Public Health Group (CPHG)

The quality of the study was not an inclusion criterion; however, a study quality check was carried out. Two independent reviewers (S. M. and C. H.) rated studies that met the inclusion criteria to determine the strength of the evidence. The Effective Public Health Practice Project Quality Assessment Tool for Quantitative Studies was adapted to assess the methodological quality of each study (Thomas, Ciliska, Dobbins, & Micucci, 2004). The tool was adjusted to include domains relevant to the method of each study. For example, blinding was removed for nonexperimental studies. Following recommendations by Thomas et al. (2004) each domain was rated as either weak (3 points), moderate (2 points), or strong (1 point). The mean score across questions was used as an indicator of overall quality, and studies were assigned an overall quality rating of strong (1.00–1.50), moderate (1.51–2.50),

Evidence Tables

Aspects of the appraisal of studies included in the review should be recorded as evidence tables (NICE 2009): simple text tables where the design and scope of studies are summarised.

The reader of the review can use the evidence tables to check the details, and assess the credibility and generalisability of findings, of particular studies.

Critical appraisal of the quality of included studies may be combined with data extraction tables.

Step 7: extracting data from studies

To effectively extract data from studies that meet your systematic review’s inclusion criteria, you should follow a structured process that ensures accuracy, consistency, and minimizes bias.

1. Develop a data extraction form:

• Design a standardized form (paper or electronic) to guide the data extraction process : This form should be tailored to your specific review question and the types of studies included.
• Pilot test the form : Test the form on a small sample of included studies (e.g., 3-5). Assess for clarity, completeness, and usability. Refine the form based on feedback and initial experiences.
• Reliability : Ensure all team members understand how to use the form consistently.

2. Extract the data:

• General Information: This includes basic bibliographic details (journal, title, author, volume, page numbers), study objective as stated by the authors, study design, and funding source.
• Study Characteristics: Capture details about the study population (demographics, inclusion/exclusion criteria, recruitment procedures), interventions (description, delivery methods), and comparators (description if applicable).
• Outcome Data: Record the results of the intervention and how they were measured, including specific statistics used. Clearly define all outcomes for which data are being extracted.
• Risk of Bias Assessment: Document the methods used to assess the quality of the included studies and any potential sources of bias. This might involve using standardized checklists or scales.
• Additional Information: Depending on your review, you may need to extract data on other variables like adverse effects, economic evaluations, or specific methodological details.

3. Dual independent review:

• Ensure that at least two reviewers independently extract data from each study using the standardized form. Cross-check extracted data for accuracy to minimize bias and helps identify any discrepancies.
• Have a predefined strategy for resolving disagreements: This might involve discussion, consensus, or arbitration by a third reviewer.
• Record the reasons for excluding any studies during the data extraction phase. :This enhances the transparency and reproducibility of your review.
• If necessary, contact study authors to obtain missing or clarify unclear information : This is particularly important for data critical to your review’s outcomes.
• Clearly document your entire data extraction process, including any challenges encountered and decisions made. This enhances the transparency and rigor of your systematic review.

By following these steps, you can effectively extract data from studies that meet your inclusion criteria, forming a solid foundation for the analysis and synthesis phases of your systematic review.

Step 8: synthesize the extracted data

The key element of a systematic review is the synthesis: that is the process that brings together the findings from the set of included studies in order to draw conclusions based on the body of evidence.

Data synthesis in a systematic review involves collating, combining, and summarizing findings from the included studies.

This process aims to provide a reliable and comprehensive answer to the review question by considering the strength of the evidence, examining the consistency of observed effects, and investigating any inconsistencies.

The data synthesis will be presented in the results section of the systematic review.

• Develop a clear text narrative that explains the key findings
• Use a logical heading structure to guide readers through your results synthesis
• Ensure your text narrative addresses the review’s research questions
• Use tables to summarise findings (can be same table as data extraction)

Identifying patterns, trends, and differences across studies

Narrative synthesis uses a textual approach to analyze relationships within and between studies to provide an overall assessment of the evidence’s robustness. All systematic reviews should incorporate elements of narrative synthesis, such as tables and text.

Remember, the goal of a narrative synthesis is to go beyond simply summarizing individual studies. You’re aiming to create a new understanding by integrating and interpreting the available evidence in a systematic and transparent way.

Organize your data:

• Group studies by themes, interventions, or outcomes
• Create summary tables to display key information across studies
• Use visual aids like concept maps to show relationships between studies

Describe the studies:

• Summarize the characteristics of included studies (e.g., designs, sample sizes, settings)
• Highlight similarities and differences across studies
• Discuss the overall quality of the evidence

Develop a preliminary synthesis:

• Start by describing the results of individual studies
• Group similar findings together
• Identify overarching themes or trends

Explore relationships:

• Look for patterns in the data
• Identify factors that might explain differences in results across studies
• Consider how study characteristics relate to outcomes

Address contradictions:

• Consider differences in study populations, interventions, or contexts
• Look at methodological differences that might explain discrepancies
• Consider the implications of inconsistent results
• Don’t ignore conflicting findings
• Discuss possible reasons for contradictions

Avoid vote counting:

• Don’t simply tally positive versus negative results
• Instead, consider the strength and quality of evidence for each finding

Assess the robustness of the synthesis:

• Reflect on the strength of evidence for each finding
• Consider how gaps or limitations in the primary studies affect your conclusions
• Discuss any potential biases in the synthesis process

Step 9: discussion section and conclusion

Summarize key findings:.

• Summarize key findings in relation to your research questions
• Highlight main themes or patterns across studies
• Explain the nuances and complexities in the evidence
• Discuss the overall strength and consistency of the evidence
• This provides a clear takeaway message for readers

Consider study quality and context:

• Assess whether higher quality studies tend to show different results
• Examine if findings differ based on study setting or participant characteristics
• This helps readers weigh the relative importance of conflicting findings

Discuss implications:

• For practice: How might professionals apply these findings?
• For policy: What policy changes might be supported by the evidence?
• Consider both positive and negative implications
• This helps translate your findings into real-world applications

Identify gaps and future research:

• Point out areas where evidence is lacking or inconsistent
• Suggest specific research questions or study designs to address these gaps
• This helps guide future research efforts in the field

State strengths and limitations:

• Discuss the strengths of your review (e.g., comprehensive search, rigorous methodology)
• Acknowledge limitations (e.g., language restrictions, potential for publication bias)
• This balanced approach demonstrates critical thinking and helps readers interpret your findings

Minimizing Bias

To reduce bias in a systematic review, it is crucial to establish a systematic and transparent review process that minimizes bias at every stage. Sources provide insights into strategies and methods to achieve this goal.

• Protocol development and publication: Developing a comprehensive protocol before starting the review is essential. Publishing the protocol in repositories like PROSPERO or Cochrane Library promotes transparency and helps avoid deviations from the planned approach, thereby minimizing the risk of bias.
• Transparent reporting: Adhering to reporting guidelines, such as PRISMA, ensures that all essential aspects of the review are adequately documented, increasing the reader’s confidence in the transparency and completeness of systematic review reporting.
• Dual independent review: Employing two or more reviewers independently at multiple stages of the review process (study selection, data extraction, quality assessment) minimizes bias. Any disagreements between reviewers should be resolved through discussion or by consulting a third reviewer. This approach reduces the impact of individual reviewers’ subjective interpretations or errors.
• Rigorous quality assessment: Assessing the methodological quality of included studies is crucial for minimizing bias in the review findings. Using standardized critical appraisal tools and checklists helps identify potential biases within individual studies, such as selection bias, performance bias, attrition bias, and detection bias.
• Searching beyond published literature: Explore sources of “grey literature” such as conference proceedings, unpublished reports, theses, and ongoing clinical trial databases.
• Contacting experts in the field : Researchers can reach out to authors and investigators to inquire about unpublished or ongoing studies
• Considering language bias : Expanding the search to include studies published in languages other than English can help reduce language bias, although this may increase the complexity and cost of the review.

Reading List

• Galante, J., Galante, I., Bekkers, M. J., & Gallacher, J. (2014). Effect of kindness-based meditation on health and well-being: a systematic review and meta-analysis .  Journal of consulting and clinical psychology ,  82 (6), 1101.
• Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses .  Psychological bulletin ,  143 (6), 565.
• Murray, J., Farrington, D. P., & Sekol, I. (2012). Children’s antisocial behavior, mental health, drug use, and educational performance after parental incarceration: a systematic review and meta-analysis .  Psychological bulletin ,  138 (2), 175.
• Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017). A systematic review of personality trait change through intervention .  Psychological bulletin ,  143 (2), 117.
• Chu, C., Buchman-Schmitt, J. M., Stanley, I. H., Hom, M. A., Tucker, R. P., Hagan, C. R., … & Joiner Jr, T. E. (2017). The interpersonal theory of suicide: A systematic review and meta-analysis of a decade of cross-national research.   Psychological bulletin ,  143 (12), 1313.
• McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review .  Journal of clinical psychology ,  76 (1), 59-101.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 18 July 2024.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2024, July 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 9 December 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

Study Design 101: Systematic Review

• Case Report
• Case Control Study
• Cohort Study
• Randomized Controlled Trial
• Practice Guideline
• Systematic Review
• Meta-Analysis
• Helpful Formulas
• Finding Specific Study Types

A document often written by a panel that provides a comprehensive review of all relevant studies on a particular clinical or health-related topic/question. The systematic review is created after reviewing and combining all the information from both published and unpublished studies (focusing on clinical trials of similar treatments) and then summarizing the findings.

• Exhaustive review of the current literature and other sources (unpublished studies, ongoing research)
• Less costly to review prior studies than to create a new study
• Less time required than conducting a new study
• Results can be generalized and extrapolated into the general population more broadly than individual studies
• More reliable and accurate than individual studies
• Considered an evidence-based resource

Disadvantages

• Very time-consuming
• May not be easy to combine studies

Design pitfalls to look out for

Studies included in systematic reviews may be of varying study designs, but should collectively be studying the same outcome.

Is each study included in the review studying the same variables?

Some reviews may group and analyze studies by variables such as age and gender; factors that were not allocated to participants.

Do the analyses in the systematic review fit the variables being studied in the original studies?

Fictitious Example

Does the regular wearing of ultraviolet-blocking sunscreen prevent melanoma? An exhaustive literature search was conducted, resulting in 54 studies on sunscreen and melanoma. Each study was then evaluated to determine whether the study focused specifically on ultraviolet-blocking sunscreen and melanoma prevention; 30 of the 54 studies were retained. The thirty studies were reviewed and showed a strong positive relationship between daily wearing of sunscreen and a reduced diagnosis of melanoma.

Real-life Examples

Yang, J., Chen, J., Yang, M., Yu, S., Ying, L., Liu, G., ... Liang, F. (2018). Acupuncture for hypertension. The Cochrane Database of Systematic Reviews, 11 (11), CD008821. https://doi.org/10.1002/14651858.CD008821.pub2

This systematic review analyzed twenty-two randomized controlled trials to determine whether acupuncture is a safe and effective way to lower blood pressure in adults with primary hypertension. Due to the low quality of evidence in these studies and lack of blinding, it is not possible to link any short-term decrease in blood pressure to the use of acupuncture. Additional research is needed to determine if there is an effect due to acupuncture that lasts at least seven days.

Parker, H.W. and Vadiveloo, M.K. (2019). Diet quality of vegetarian diets compared with nonvegetarian diets: a systematic review. Nutrition Reviews , https://doi.org/10.1093/nutrit/nuy067

This systematic review was interested in comparing the diet quality of vegetarian and non-vegetarian diets. Twelve studies were included. Vegetarians more closely met recommendations for total fruit, whole grains, seafood and plant protein, and sodium intake. In nine of the twelve studies, vegetarians had higher overall diet quality compared to non-vegetarians. These findings may explain better health outcomes in vegetarians, but additional research is needed to remove any possible confounding variables.

Related Terms

Cochrane Database of Systematic Reviews

A highly-regarded database of systematic reviews prepared by The Cochrane Collaboration , an international group of individuals and institutions who review and analyze the published literature.

Exclusion Criteria

The set of conditions that characterize some individuals which result in being excluded in the study (i.e. other health conditions, taking specific medications, etc.). Since systematic reviews seek to include all relevant studies, exclusion criteria are not generally utilized in this situation.

Inclusion Criteria

The set of conditions that studies must meet to be included in the review (or for individual studies - the set of conditions that participants must meet to be included in the study; often comprises age, gender, disease type and status, etc.).

Now test yourself!

1. Systematic Reviews are similar to Meta-Analyses, except they do not include a statistical analysis quantitatively combining all the studies.

a) True b) False

2. The panels writing Systematic Reviews may include which of the following publication types in their review?

a) Published studies b) Unpublished studies c) Cohort studies d) Randomized Controlled Trials e) All of the above

Evidence Pyramid - Navigation

• Meta- Analysis
• Case Reports
• << Previous: Practice Guideline
• Next: Meta-Analysis >>

• Last Updated: Sep 25, 2023 10:59 AM
• URL: https://guides.himmelfarb.gwu.edu/studydesign101

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2962
• [email protected]
• https://himmelfarb.gwu.edu

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Systematically Reviewing the Literature: Building the Evidence for Health Care Quality

Suzanne austin boren , phd, david moxley , mlis.

• Author information
• Copyright and License information

Contact: [email protected]

Corresponding author.

There are important research and non-research reasons to systematically review the literature. This article describes a step-by-step process to systematically review the literature along with links to key resources. An example of a graduate program using systematic literature reviews to link research and quality improvement practices is also provided.

Introduction

Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review. It is important to systematically review the literature when one would like to justify the need for a study, to update personal knowledge and practice, to evaluate current practices, to develop and update guidelines for practice, and to develop work related policies. 1 A systematic review draws upon the best health services research principles and methods to address: What is the state of the evidence on the selected topic? The systematic process enables others to reproduce the methods and to make a rational determination of whether to accept the results of the review. An abundance of articles on systematic reviews exist focusing on different aspects of systematic reviews. 2 – 9 The purpose of this article is to describe a step by step process of systematically reviewing the health care literature and provide links to key resources.

Systematic Review Process: Six Key Steps

Six key steps to systematically review the literature are outlined in Table 1 and discussed here.

Systematic Review Steps

1. Formulate the Question and Refine the Topic

When preparing a topic to conduct a systematic review, it is important to ask at the outset, “What exactly am I looking for?” Hopefully it seems like an obvious step, but explicitly writing a one or two sentence statement of the topic before you begin to search is often overlooked. It is important for several reasons; in particular because, although we usually think we know what we are searching for, in truth our mental image of a topic is often quite fuzzy. The act of writing something concise and intelligible to a reader, even if you are the only one who will read it, clarifies your thoughts and can inspire you to ask key questions. In addition, in subsequent steps of the review process, when you begin to develop a strategy for searching the literature, your topic statement is the ready raw material from which you can extract the key concepts and terminology for your strategies. The medical and related health literature is massive, so the more precise and specific your understanding of your information need, the better your results will be when you search.

2. Search, Retrieve, and Select Relevant Articles

The retrieval tools chosen to search the literature should be determined by the purpose of the search. Questions to ask include: For what and by whom will the information be used? A topical expert or a novice? Am I looking for a simple fact? A comprehensive overview on the topic? Exploration of a new topic? A systematic review? For the purpose of a systematic review of journal research in the area of health care, PubMed or Medline is the most appropriate retrieval tool to start with, however other databases may be useful ( Table 2 ). In particular, Google Scholar allows one to search the same set of articles as PubMed/MEDLINE, in addition to some from other disciplines, but it lacks a number of key advanced search features that a skilled searcher can exploit in PubMed/MEDLINE.

Examples of Electronic Bibliographic Databases Specific to Health Care

Note: These databases may be available through university or hospital library systems.

An effective way to search the literature is to break the topic into different “building blocks.” The building blocks approach is the most systematic and works the best in periodical databases such as PubMed/MEDLINE. The “blocks” in a “building blocks” strategy consist of the key concepts in the search topic. For example, let’s say we are interested in researching about mobile phone-based interventions for monitoring of patient status or disease management. We could break the topic into the following concepts or blocks: 1. Mobile phones, 2. patient monitoring, and 3. Disease management. Gather synonyms and related terms to represent each concept and match to available subject headings in databases that offer them. Organize the resulting concepts into individual queries. Run the queries and examine your results to find relevant items and suggest query modifications to improve your results. Revise and re-run your strategy based on your observations. Repeat this process until you are satisfied or further modifications produce no improvements. For example in Medline, these terms would be used in this search and combined as follows: cellular phone AND (ambulatory monitoring OR disease management), where each of the key word phrases is an official subject heading in the MEDLINE vocabulary. Keep detailed notes on the literature search, as it will need to be reported in the methods section of the systematic review paper. Careful noting of search strategies also allows you to revisit a topic in the future and confidently replicate the same results, with the addition of those subsequently published on your topic.

3. Assess Quality

There is no consensus on the best way to assess study quality. Many quality assessment tools include issues such as: appropriateness of study design to the research objective, risk of bias, generalizability, statistical issues, quality of the intervention, and quality of reporting. Reporting guidelines for most literature types are available at the EQUATOR Network website ( http://www.equator-network.org/ ). These guidelines are a useful starting point; however they should not be used for assessing study quality.

4. Extract Data and Information

Extract information from each eligible article into a standardized format to permit the findings to be summarized. This will involve building one or more tables. When making tables each row should represent an article and each column a variable. Not all of the information that is extracted into the tables will end up in the paper. All of the information that is extracted from the eligible articles will help you obtain an overview of the topic, however you will want to reserve the use of tables in the literature review paper for the more complex information. All tables should be introduced and discussed in the narrative of the literature review. An example of an evidence summary table is presented in Table 3 .

Example of an evidence summary table

Notes: BP = blood pressure, HbA1c = Hemoglobin A1c, Hypo = hypoglycemic, I = Internet, NS = not significant, PDA = personal digital assistant, QOL = quality of life, SMBG = self-monitored blood glucose, SMS = short message service, V = voice

5. Analyze and Synthesize Data and information

The findings from individual studies are analyzed and synthesized so that the overall effectiveness of the intervention can be determined. It should also be observed at this time if the effect of an intervention is comparable in different studies, participants, and settings.

6. Write the Systematic Review

The PRISMA 12 and ENTREQ 13 checklists can be useful resources when writing a systematic review. These uniform reporting tools focus on how to write coherent and comprehensive reviews that facilitate readers and reviewers in evaluating the relative strengths and weaknesses. A systematic literature review has the same structure as an original research article:

TITLE : The systematic review title should indicate the content. The title should reflect the research question, however it should be a statement and not a question. The research question and the title should have similar key words.

STRUCTURED ABSTRACT: The structured abstract recaps the background, methods, results and conclusion in usually 250 words or less.

INTRODUCTION: The introduction summarizes the topic or problem and specifies the practical significance for the systematic review. The first paragraph or two of the paper should capture the attention of the reader. It might be dramatic, statistical, or descriptive, but above all, it should be interesting and very relevant to the research question. The topic or problem is linked with earlier research through previous attempts to solve the problem. Gaps in the literature regarding research and practice should also be noted. The final sentence of the introduction should clearly state the purpose of the systematic review.

METHODS: The methods provide a specification of the study protocol with enough information so that others can reproduce the results. It is important to include information on the:

Eligibility criteria for studies: Who are the patients or subjects? What are the study characteristics, interventions, and outcomes? Were there language restrictions?

Literature search: What databases were searched? Which key search terms were used? Which years were searched?

Study selection: What was the study selection method? Was the title screened first, followed by the abstract, and finally the full text of the article?

Data extraction: What data and information will be extracted from the articles?

Data analysis: What are the statistical methods for handling any quantitative data?

RESULTS: The results should also be well-organized. One way to approach the results is to include information on the:

Search results: What are the numbers of articles identified, excluded, and ultimately eligible?

Study characteristics: What are the type and number of subjects? What are the methodological features of the studies?

Study quality score: What is the overall quality of included studies? Does the quality of the included studies affect the outcome of the results?

Results of the study: What are the overall results and outcomes? Could the literature be divided into themes or categories?

DISCUSSION: The discussion begins with a nonnumeric summary of the results. Next, gaps in the literature as well as limitations of the included articles are discussed with respect to the impact that they have on the reliability of the results. The final paragraph provides conclusions as well as implications for future research and current practice. For example, questions for future research on this topic are revealed, as well as whether or not practice should change as a result of the review.

REFERENCES: A complete bibliographical list of all journal articles, reports, books, and other media referred to in the systematic review should be included at the end of the paper. Referencing software can facilitate the compilation of citations and is useful in terms of ensuring the reference list is accurate and complete.

The following resources may be helpful when writing a systematic review:

CEBM: Centre for Evidence-based Medicine. Dedicated to the practice, teaching and dissemination of high quality evidence based medicine to improve health care Available at: http://www.cebm.net/ .

CITING MEDICINE: The National Library of Medicine Style Guide for Authors, Editors, and Publishers. This resource provides guidance in compiling, revising, formatting, and setting reference standards. Available at http://www.ncbi.nlm.nih.gov/books/NBK7265/ .

EQUATOR NETWORK: Enhancing the QUAlity and Transparency Of health Research. The EQUATOR Network promotes the transparent and accurate reporting of research studies. Available at: http://www.equator-network.org/ .

ICMJE RECOMMENDATIONS: International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. The ICJME recommendations are followed by a large number of journals. Available at: http://www.icmje.org/about-icmje/faqs/icmje-recommendations/ .

PRISMA STATEMENT: Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Authors can utilize the PRISMA Statement checklist to improve the reporting of systematic reviews and meta-analyses. Available at: http://prisma-statement.org .

THE COCHRANE COLLABORATION: A reliable source for making evidence generated through research useful for informing decisions about health. Available at: http://www.cochrane.org/ .

Examples of Systematic Reviews To Link Research and Quality Improvement

Over the past 17 years more than 300 learners, including physicians, nurses, and health administrators have completed a course as part of a Master of Health Administration or a Master of Science in Health Informatics degree at the University of Missouri. An objective of the course is to educate health informatics and health administration professionals about how to utilize a systematic, scientific, and evidence-based approach to literature searching, appraisal, and synthesis. Learners in the course conduct a systematic review of the literature on a health care topic of their choosing that could suggest quality improvement in their organization. Students select topics that make sense in terms of their core educational competencies and are related to their work. The categories of topics include public health, leadership, information management, health information technology, electronic medical records, telehealth, patient/clinician safety, treatment/screening evaluation cost/finance, human resources, planning and marketing, supply chain, education/training, policies and regulations, access, and satisfaction. Some learners have published their systematic literature reviews 14 – 15 . Qualitative comments from the students indicate that the course is well received and the skills learned in the course are applicable to a variety of health care settings.

Undertaking a literature review includes identification of a topic of interest, searching and retrieving the appropriate literature, assessing quality, extracting data and information, analyzing and synthesizing the findings, and writing a report. A structured step-by-step approach facilitates the development of a complete and informed literature review.

Suzanne Austin Boren, PhD, MHA, (above) is Associate Professor and Director of Academic Programs, and David Moxley, MLIS, is Clinical Instructor and Associate Director of Executive Programs. Both are in the Department of Health Management and Informatics at the University of Missouri School of Medicine.

None reported.

• 1. Polit DF, Beck CT. Nursing Research: principles and methods. 9th edition. Lippincott, Williams & Wilkins; Philadelphia: 2011. [ Google Scholar ]
• 2. Bruce J, Mollison J. Reviewing the literature: adopting a systematic approach. Journal of Family Planning and Reproductive Health Care. 2004;30(1) doi: 10.1783/147118904322701901. [ DOI ] [ PubMed ] [ Google Scholar ]
• 3. Cronin P, Ryan F, Coughlin M. Undertaking a literature review: A step-by-step approach. British Journal of Nursing. 2008;17(1):38–43. doi: 10.12968/bjon.2008.17.1.28059. [ DOI ] [ PubMed ] [ Google Scholar ]
• 4. Crowther DM. A clinician’s guide to systematic reviews. Nutr Clin Pract. 2013;28:459–462. doi: 10.1177/0884533613490742. [ DOI ] [ PubMed ] [ Google Scholar ]
• 5. Hasse SC. Systematic reviews and meta-analysis. Plast Reconstr Surg. 2011;127:955–966. doi: 10.1097/PRS.0b013e318200afa9. [ DOI ] [ PubMed ] [ Google Scholar ]
• 6. Mandrekar JN, Mandreker SJ. Systematic reviews and meta analysis of published studies: An over view and best practices. J Thorac Oncol. 2011;6(8):1301–1303. doi: 10.1097/JTO.0b013e31822461b0. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 7. Ng KH, Peh WC. Writing a systematic review. Singapore Med J. 2010 May;51(5):362–6. [ PubMed ] [ Google Scholar ]
• 8. Price B. Guidance on conducting a literature search reviewing mixed literature. Nursing Standard. 2009;23(24):43–49. doi: 10.7748/ns2009.02.23.24.43.c6829. [ DOI ] [ PubMed ] [ Google Scholar ]
• 9. Engberg S. Systematic reviews and meta-analysis: Studies of studies. J Wound Ostomy Continence Nurs. 2008;35(3):258–265. doi: 10.1097/01.WON.0000319122.76112.23. [ DOI ] [ PubMed ] [ Google Scholar ]
• 10. Benhamou PY, Melki V, Boizel R, et al. One-year efficacy and safety of Web-based follow-up using cellular phone in type 1 diabetic patients under insulin pump therapy: the PumpNet study. Diabetes & Metabolism. 2007;33(3):220–6. doi: 10.1016/j.diabet.2007.01.002. [ DOI ] [ PubMed ] [ Google Scholar ]
• 11. Marquez Contreras E, de la Figuera von Wichmann M, Gil Guillen V, Ylla-Catala A, Figueras M, Balana M, Naval J. Effective news of an inter vention to provide information to patients with hypertension as short text messages and reminder sent to their mobile phone [Spanish] Aten Primaria. 2004;34(8):399–405. doi: 10.1016/S0212-6567(04)78922-2. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 12. Moher D, Liberati A, Tetzlaff J, Altman DG The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med. 2009;151(4):264–269. W64. doi: 10.7326/0003-4819-151-4-200908180-00135. [ DOI ] [ PubMed ] [ Google Scholar ]
• 13. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):181. doi: 10.1186/1471-2288-12-181. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 14. Hart MD. Informatics competency and development within the US nursing population workforce. CIN: Computers, Informatics, Nursing. 2008;26(6):320–329. doi: 10.1097/01.NCN.0000336462.94939.4c. [ DOI ] [ PubMed ] [ Google Scholar ]
• 15. Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory, primary care setting: A systematic review of the literature. Informatics in Primary Care. 2008 Jun;16(2):79–91. doi: 10.14236/jhi.v16i2.679. [ DOI ] [ PubMed ] [ Google Scholar ]
• PDF (510.6 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

Systematic Reviews

What is a systematic review.

• Steps of a Systematic Review
• Systematic Review Types
• Literature Reviews

Page Contents

Key characteristics of a systematic review, systematic reviews are not..., systematic reviews vs. narrative/literature reviews, standards and guidelines, appraising a systematic review, systematic review examples and further reading, medical library director.

Health Sciences Librarian

A systematic review is a ‘study about studies’ which focuses on a specific research question that it must try to answer by using explicit, transparent and reproducible methods to identify, critically appraise and synthesize results of similar studies to generate evidence-based findings. ( From Stephanie Roth’s Introduction to Systematic Reviews Course. June 2020 ).  

Identify : review teams develop search strategies for multiple research databases and resources, and screen their findings against pre-specified eligibility criteria. , appraise : once all the evidence has been located and screened, review teams evaluate the quality of the evidence, assessing to understand where more research may be needed. , synthesize : finally, the review team brings together the different pieces of evidence, identifying trends, issues, or patterns in the data. the rigorous systematic review methodology makes these reviews a great tool for clinicians, educators, and stakeholders involved in decision-making or best practice recommendations. .

• Time-intensive: A typical systematic review can take 12-18 months to complete. 
• Collaboration-intensive: The systematic review’s methodology requires multiple researchers to screen and evaluate evidence. A team of at least three is required. 
• Protocol-based: Before beginning a systematic review, the team will pre-define their research question, expected outcomes, search strategy, and inclusion/exclusion criteria in a protocol. This helps mitigate potential bias or data manipulation.  
• Research-question based: Systematic reviews are rooted in clear, focused questions. Many reviews will use the PICO framework or something similar to build their question. 
• Exhaustive: The goal of a systematic review is to locate all of the available evidence for your question. 
• Exclusive to medicine or healthcare: Systematic reviews are commissioned or developed for many different disciplines interested in making evidence-informed decisions. 
• Exclusive to quantitative research: While many systematic reviews analyze studies that created quantitative data, qualitative and mixed methods reviews do exist and follow the systematic review methodology. 
• The only “good” review type: Different types of reviews are helpful depending on your context and question. Meet with a librarian to discuss if a systematic review approach is right for your team. 
• Exhausting (hopefully!): While systematic reviews can be time-intensive, the guiding methodology and opportunity for collaboration is invigorating to many researchers at all career levels. 

*For more information how to conduct a narrative literature review see: https://medlib.belmont.edu/LitReviews  

One of the nice things about systematic reviews is that there are well-established standards and guidelines for this research methodology.  

JBI Manual for Evidence Synthesis  

• guides authors who wish to conduct systematic and scoping reviews following JBI methodologies 
• Each chapter is devoted to the synthesis of different types of evidence to address different types of clinical and policy-related questions. 
• Easy to apply, step-by-step guidance 

Cochrane Handbook for Systematic Reviews of Interventions 

• The official guide that describes preparing and maintaining Cochrane systematic reviews 
• includes guidance on the standard methods applicable to every review (planning a review, searching and selecting studies, data collection, risk of bias assessment, statistical analysis, GRADE and interpreting results) 

Reporting Guidelines 

• PRISMA-Checklist and Flow Diagram  
• PRISMA-P extension for protocols 
• PRISMA-S extension for search reporting 

While systematic reviews are generally considered to be at the top of the “pyramid of evidence,” not all systematic reviews are created equal . Like any other research, they should be discussed, critiqued, and updated. Here are some tools for appraising systematic reviews: 

Critical Appraisal Skills Programme Systematic Review Checklists (CASP) 

• Use these checklists to analyze how a systematic review was conducted. 

Risk of Bias Assessment Tool for Systematic Reviews (ROBIS) 

• Use this tool to (1) assess relevance (optional), (2) identify concerns with the review process and (3) judge risk of bias for systematic reviews. 
• Grant, M.J., & Booth, A. (2009). A typology of reviews: an analysis of 14 different review types and associated methodologies . Health Information and Libraries Journal, 26: 91-108. DOI: 10.1111/j.1471-1842.2009.00848.x 
• Munn, Z., Stern, C., Aromataris, E. et al. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences . BMC Med Res Methodol 18, 5 (2018). https://doi.org/10.1186/s12874-017-0468-4 
• Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses . Annual review of psychology, 70(1), 747-770. 
• Allen, C., Walker, A. M., Premji, Z. A., Beauchemin-Turcotte, M. E., Wong, J., Soh, S., Hawboldt, G. S., Shinkaruk, K. S., & Archer, D. P. (2022). Preventing persistent postsurgical pain: A systematic review and component network meta-analysis. European journal of pain (London, England), 26(4), 771–785. https://doi.org/10.1002/ejp.1915  
• Hodder, R. K., O'Brien, K. M., Wyse, R. J., Tzelepis, F., Yoong, S., Stacey, F. G., & Wolfenden, L. (2024). Interventions for increasing fruit and vegetable consumption in children aged five years and under. The Cochrane database of systematic reviews, 9(9), CD008552. https://doi.org/10.1002/14651858.CD008552.pub8   
• Jefferies, D., McNally, S., Roberts, K., Wallace, A., Stunden, A., D'Souza, S., & Glew, P. (2018). The importance of academic literacy for undergraduate nursing students and its relationship to future professional clinical practice: A systematic review. Nurse education today, 60, 84–91. https://doi.org/10.1016/j.nedt.2017.09.020  
• Next: Steps of a Systematic Review >>
• Last Updated: Dec 6, 2024 12:52 PM
• URL: https://belmont.libguides.com/systematicreviews