What is a systematic review?
A systematic review is a method of answering a question by finding, appraising and synthesising evidence in a systematic and unbiased way.
This guide will take you through the steps involved in conducting a systematic review. Click one of the stages below to see information, tips, resources and support available for each part of the process.
*this is a rough estimate of the time needed to complete each stage, but will vary depending on the complexity of your review.
Systematic Review Service Charter
The Library provides tiered support for researchers conducting systematic reviews. Postgraduate students who conduct systematic reviews as part of their degree can request up to 3 hours of support per systematic review. Postdoctoral researchers and beyond can opt for more extensive support, depending on their requirements and our capacity. For details, please refer to the service charter.
Feedback
We are always improving our resources. If you'd like to share your feedback with us, please do so via our feedback form.
Before you start
Is this a systematic review?
You might find that a systematic review isn't the best study type for your research topic perhaps due to the scope of your question or the nature of the existing body of literature. There are many other kinds of reviews that share some of the same characteristics.
The most common question is about the difference between a standard literature review and a systematic review. The table below explains some of the key differences. A full list of review types and their characteristics can be found in Tools & resources.
Systematic review vs. Literature review
|
Systematic review |
Literature review |
Definition |
High-level overview of primary research on a focused question that identifies, selects, synthesises, and appraises all high quality research evidence relevant to that question |
Qualitatively summarises evidence on a topic using informal or subjective methods to collect and interpret studies |
Goals |
Answer a focused clinical question. Eliminate bias |
Provide summary or overview of a topic |
Question |
Clearly defined and answerable clinical question. Recommend using PICO as a guide |
Can be a general topic or a specific question |
Components |
- Prespecified eligibility criteria
- Systematic search strategy
- Assessment of the validity of findings
- Interpretation and presentation of results
- Reference list
|
- Introduction
- Methods
- Discussion
- Conclusion
- Reference list
|
Number of authors |
Three or more |
One or more |
Timeline |
Months to years. Average timeline is 18 months |
Weeks to months |
Requirements |
Thorough knowledge of topic
Perform searches of all relevant databases
Statistical analysis resources (for meta-analysis)
|
Understanding of topic
Perform searches of one or more databases
|
Value |
Connects practicing clinicians to high quality evidence. Supports evidence-based practice |
Provides summary of literature on a topic |
Most systematic reviews are currently conducted in fields like health and medicine. Increasingly, systematic reviews are being published in other disciplines such as humanities and education.
Throughout this guide, we'll be making reference to bodies like Cochrane, the Joanna Briggs Institute, and the Campbell Collaboration. These organisations are responsible for the development of many best practice frameworks and guidelines. We always recommend consulting these original sources for more specific advice.
Tips
Things you should know
- Systematic reviews take time to do properly. The estimated timeline for a Cochrane systematic review is 18 months, full time.
A team is important - not only is it essential for you to work with at least one other person when selecting studies for eligibility and extracting data, it's recommended that you consult with different experts at different stages. Helpful contacts are linked throughout this guide. They include:
- Subject matter experts (like your supervisor or colleagues) for question formulation and writing up findings
- Librarians for guidance around searching for literature
- Statistical support for data analysis and synthesis
A systematic review needs to be well documented for reporting purposes. Your team will be producing and managing a high volume of data, including:
- Different iterations of protocols
- Search logs
- Records retrieved from database searches
- Decision making using PRISMA flow charts
- Research data extracted from the evidence
Finding collaborators
- If you're an HDR student conducting a systematic review, you can work with your supervisor.
Check the Research Impact subject guide for ways to identify and contact potential collaborators, such as:
- Reach out to your network/s to identify people with common research interests.
- Use academic social networks like Mendeley and ResearchGate to find potential collaborators.
- Keep your researcher profiles (e.g. USYD Academic Profile, Google Scholar, ORCiD, and Scopus) up to date so others can easily find you.
Before meeting with your Librarian
- Make sure you know your topic - scope it out to see if there are existing reviews. Information about how to 'scope' can be found in the Define Research Question stage of this portal. Scoping searches in health can be conducted in sources like TRIP, Scopus (multidisciplinary), Web of Science (multidisciplinary), the Cochrane Library, Epistemonikos, and PROSPERO.
- Have a specific research question - further information on this is in the Define Research Question stage of this portal.
- Consider what kinds of information you’ll need to answer your question and have a literature search plan.
- Have a (draft) protocol - further information on how to do this can be found in the Develop and Register Protocol stage of this portal.
Need help?
Workshops
The Library regularly runs Systematic Review workshops to get you started with searching. To see upcoming workshops and to register, check the Library Calendar.
One-on-one support
Your first port of call for help will be your supervisor.
If you are after some one-on-one support, book an appointment with a Librarian. To make your appointment as productive as possible, we recommend reading through this guide so you know which stages of the process Librarians can assist with.
How will you conduct your analysis? If a meta-analysis is on the cards, it might be a good idea to speak to a statistician at the Sydney Informatics Hub during the planning stages. Advice is free to HDR students. Check the SIH website for an outline of their services and link to the assistance form.
References
- Aromataris, E., Munn, Z. (Editors). (2017). The Joanna Briggs Institute reviewer’s manual The Joanna Briggs Institute. Available from https://reviewersmanual.joannabriggs.org/.
- Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review: a student’s guide. London: SAGE.
- Cook, D. J. (1997). Systematic reviews: synthesis of best evidence for clinical decisions. Annals of Internal Medicine, 126(5), 376.
- Crawford, C. C., Boyd, C. C., & Jonas, W. B. (2015). Systematic reviews in practice. Alexandria, VA: Samueli Institute.
- Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108.
- Green, B. N., Johnson, C. D., & Adams, A. (2006). Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine, 5(3), 101–117.
- Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews. Los Angeles: SAGE.
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, & Welch VA (editors). (2019). Cochrane Handbook for Systematic Reviews of Interventions (version 6.0). Cochrane. Available from www.training.cochrane.org/handbook.
- Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: the evolution of a rapid review approach. Systematic Reviews, 1(1), 10.
- Morris, M., Boruff, J. T., & Gore, G. C. (2016). Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association, 104(4), 346–354.
- Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Medical Research Methodology, 18(1), 5.
- Whittemore, R., & Knafl, K. (2005). The integrative review: updated methodology. Journal of Advanced Nursing, 52(5), 546–553.
Define a research question
If a systematic review is the best way to answer your research question, what's next?
Scoping your topic
When starting a systematic review, it's a good idea to look in different databases and grey literature sources to see what evidence is available.
Conducting a scoping search allows you to put your research in context and justify its importance, identify any potential 'gaps' in your research area, find any existing systematic reviews, and see if there is enough evidence to write a systematic review.
Resources that can be used to scope a topic
Research topic vs. a review question
A 'research topic' is the area of study that you are researching, while a 'review question' is the more focused question that your systematic review aims to answer. For example, you could be studying non-therapeutic treatment of dementia but your review might focus on the question below:
Example
- How effective are music-based therapy programs for relieving anxiety and depression in patients with Alzheimer's?
If you’re stuck on focusing your research topic into a specific question, mnemonics are tools that can help you. Depending on the kind of question you’re asking, they highlight different concepts that are needed. Mnemonics can also help you build a search strategy. A list of different mnemonics can be found below, under ‘Search mnemonics’.
Types of systematic review questions
The most common type of systematic reviews are those answering efficacy questions (also known as therapy or intervention questions), where an intervention is evaluated to determine its effect on a population. These are the most common types of reviews conducted in health and medicine, but they are also common in the social sciences.
Our example is a clinical question assessing the effectiveness of an intervention. It can be broken down into PICO which is the mnemonic commonly used in these types of questions.
Example
- Population: patients with Alzheimer's
- Intervention: music-based therapy programs
- Comparison: no comparison
- Outcome: anxiety and depression
Search mnemonics
Type of review question |
Aim |
Mnemonic |
Efficacy (intervention / therapy) |
To evaluate the effectiveness of a treatment or intervention |
PICO - Population, intervention, comparison, outcomes |
Experiential |
To investigate the experience or meaning of a particular phenomenon |
PICo - Population, phenomena of interest, context |
Experiential (qualitative) |
To investigate the experience or meaning of a particular phenomenon |
PICOC - Population, intervention, comparison, outcomes, context |
Prevalence / Incidence |
To determine the prevalence and / or incidence of a certain condition |
CoCoPop - Condition, context, population |
Diagnostic test accuracy |
To determine how well a diagnostic test works in terms of its sensitivity and specificity for a particular diagnosis |
PIRD - Population, index test, reference test, diagnosis of interest |
Aetiology / Risk |
To determine the association between particular exposures / risk factors and outcomes |
PEO - Population, exposure, outcome |
Expert opinion / Policy |
To review and synthesise current expert opinion, text or policy on a certain phenomena |
PICo - Population, intervention / phenomena of interest, context |
Psychometric |
To evaluate the psychometric properties of a certain test, normally to determine the reliability and validity of a particular test |
Construct of interest / name of the measurement instrument(s), population, type of measurement instrument, measurement properties |
Prognostic |
To determine the overall prognosis for a condition, the link between specific prognostic factors and an outcome and / or prognostic / prediction models and prognostic tests |
PFO - Population, prognostic factors / models of interest, outcome |
Methodology |
To examine and investigate current research methods and potentially their impact on research quality |
SDMO - Types of studies, types of data, methods, outcome |
Need help?
Your supervisor can give you advice on focusing a research question and the systematic review process. A Librarian can help you conduct scoping searches, identify the key concepts in your review question, and create a search strategy.
References
- Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435-1443.
- Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS, and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(579).
- Munn, Z., Stern, C., Aromataris, E., Lockwood C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Medical Research Methodology, 18(1), 5. https://doi.org/10.1186/s12874-017-0468-4.
- Schardt, C., Adams, M.B., Owens, T., Keitz, S., & Fontelo, P. (2007). Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Medical Informatics and Decision Making, 7(16).
- Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Information & Libraries Journal, 19(2).
Develop and register protocol
A protocol specifies the question a systematic review is attempting to answer and the methods that will be used to answer it.
Why do you need to register a protocol?
- To help avoid duplication of effort through the public listing of reviews in progress
- To provide transparency and allow for peer review of the systematic review process
- To help readers assess the quality of the review in terms of bias, through comparison of the protocol and the published systematic review
They are published in public registers like PROSPERO. The table below shows an example of a protocol and what each section will typically describe.
Typical structure
Section of a protocol |
What it will typically describe |
Rationale |
Reasons for conducting the review in the context of what is known about the research area |
Objectives |
An explicit statement of the question the systematic review will address |
Methods |
The methodologies that will be used when conducting the review, including:
- Eligibility criteria which studies must meet to be included in the review
- Information sources that will be searched
- A draft search strategy that will be used for (at least) one database, including planned limits
- Management of data and records throughout the review process
- The selection process used in the screening and eligibility stages
- Planned methods of data extraction
- Assessing the risk of bias in individual studies
- Proposed methods of data synthesis
- Any planned assessments of meta-bias(es)
- How confidence in cumulative evidence will be assessed
|
There are many reporting standards available which outline the essential information you need to provide in a protocol. Requirements will change depending on where you will publish the protocol. Some of these standards include:
- PRISMA - a generic reporting standard for systematic reviews and meta-analyses
- PROSPERO - a public register which registers protocols and specifies essential registration fields which must be completed to describe the planned review in appropriate detail
- Joanna Briggs Institute - JBI registers protocols and specifies the components that comprise a JBI protocol in its reviewer's manual
It is often the case that reviewers may need to change their original protocol after a review has commenced. Any changes made should be reported and explained in an amendment to the protocol, as well as in the systematic review itself (Gough, Oliver, & Thomas, 2017). All registers have procedures for submitting amendments.
Inclusion and exclusion (eligibility) criteria
Inclusion and exclusion criteria help you screen papers for inclusion in your review. It's important to explicitly state these at the beginning, while you're constructing your search strategy. Some criteria include:
- Study design or characteristics - such as PICO, study design, setting, and time frame
- Report characteristics - including years considered, language, and publication status
Inclusion criteria example
- Papers published in the last 15 years
Exclusion criteria example:
- Papers focusing on other types of dementia (e.g. lewy body disease, vascular dementia, alcohol related dementia)
- Papers about other interventions (e.g. art therapy)
It's important to justify each criterion in the methodology section of your systematic review. Different reporting guidelines may require different criteria.
Tips
- In addition to searching PROSPERO for in-progress reviews on your topic, it's also a good idea to register your systematic review here if it's within scope.
- When writing a protocol, use a resource like the PRISMA-P checklist.
Need help?
For assistance with writing your protocol, the best person to ask is your supervisor or a subject expert in your faculty.
References
- Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews. Los Angeles: SAGE.
- Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
- Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1).
- National Institute for Health Research. (n.d.). What is registration? Retrieved from https://www.crd.york.ac.uk/prospero/#aboutpage.
- Patino, C.M., & Ferreira, J.C. (2018).Inclusion and exclusion criteria and why they matter. Jornal Brasileiro de Pneumologia, 44(2), 84.
- PRISMA. (n.d.). What is a protocol? Retrieved from http://prisma-statement.org/Protocols/.
- Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ, 349, g7647.
- Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense. Systematic Reviews, 1(7).
Plan search strategy and select databases
Systematic reviews aim to evaluate all relevant information about a topic to answer a question. This means you need to locate all available evidence relevant to your question. Databases provide access to different collections of resources, so you’ll need to do some work thinking about which ones will contain the information you’re looking for. If you don't search enough, you'll potentially miss out on crucial studies. However there may be practical reasons for limiting the scope of your review, such as:
- lack of resources (people, funding)
- timeframe
- lack of evidence for reviews in emerging areas
- access to language translation
Creating a comprehensive literature search
A comprehensive literature search is one that eliminates bias during the search planning stage. Here are three kinds of bias to watch out for:
- Publication bias - relevant research may be ‘unpublished’ and therefore unavailable through most databases which only index ‘published’ content. This may necessitate you searching sources outside of databases e.g. clinical trial registers.
- Language / geographic bias - there are relevant studies that may be published in other languages and other regions. Do you need to account for these in your search?
- Indexing bias - this may occur if you only search a limited number of databases or use only a few search terms.
Which databases should I search?
Depending on your review question, you need to consider searching both multidisciplinary and subject-specific databases to ensure your search is comprehensive. Multidisciplinary databases contain a high volume of literature from a diverse range of subjects. They are helpful in locating literature on your topic from disciplines you might not have considered. On the other hand, subject-specific databases contain in-depth coverage of literature which is highly specific to a field. A balance of both databases is required in your search strategy.
For a comprehensive list of databases, check the Library's list of databases.
Databases which are commonly used for searching in health and medicine include:
Medline |
A database containing medicine, nursing, dentistry, veterinary medicine, preclinical science, health administration, and health care literature. |
Scopus |
A multidisciplinary database containing health, arts, humanities, science, business, and finance journals. |
Web of Science |
A multidisciplinary database containing science, social science, arts and humanities journals. |
Embase |
The European equivalent of Medline, which contains journals in pharmacology, biomedicine and drug information. |
CENTRAL |
Cochrane Central Register of Controlled Trials. A highly concentrated source of reports of randomised and quasi-randomised controlled trials. |
PsycINFO |
A multidisiplinary database containing science, social science, arts and humanities journals. |
CINAHL |
A database containing nursing and allied health literature. |
Cochrane considers Medline (via either Ovid or PubMed), Embase, and CENTRAL to be the most important sources of evidence to search when conducting a systematic review.
Regional databases include:
- LILACS Latin American and Caribbean Health Sciences Literature
- AIM African Index Medicus
- IMSEAR Index Medicus for the South East Asian Region
- Health Collection Australian content available via Informit
For our review question we will search the following databases:
- Scopus and Web of Science – for comprehensive multidisciplinary coverage from a wide range of perspectives
- Medline – for biomedical coverage of Alzheimer’s treatment and therapy; North American focused
- Embase – for biomedical coverage of Alzheimer’s treatment and therapy; European focused
- CINAHL – for Nursing and allied health perspectives of therapy programs for alleviating anxiety and depression in patients with Alzheimer’s
- PsycInfo – for psychological perspectives of anxiety and depression in patients with Alzheimer’s and the effect of various therapies
We would need to look at grey literature as well as do some hand-searching through seminal papers.
Search filters
Search filters are tried and tested methods for finding relevant research in health databases. Filters are commonly used to limit searches to a particular study type, or to find articles on a specific subject. They can be used to enhance your search, but can also be useful for identifying additional search terms.
When do I stop searching for new studies?
When conducting a systematic review, your aim is to search enough databases to be able to state with confidence that the literature has been comprehensively searched. Indicators of a comprehensive search strategy include:
- you look in a different database and don’t retrieve any new papers
- you can identify no new data
- you use a rule to decide when to stop searching
Do I need to search for grey literature?
Grey literature refers to research material that hasn't been published commercially like a book or a journal article has. The challenge lies in locating it, as it can be either published or unpublished.
Examples of published grey literature include:
- conference papers
- theses and dissertations
- government reports
- clinical trial registers
Examples of unpublished grey literature include:
- unpublished trials
- on-going studies
- unpublished data
Depending on your subject, it might be appropriate for you to include grey literature as evidence because it provides data which otherwise wouldn't be available through journal articles. Including grey literature also helps reduce publication bias.
Grey literature can sometimes be hard to locate, but we've compiled a list of tools and resources to assist you.
Other methods of locating relevant studies for your review:
- Handsearching - you can review the reference lists of papers you have already located for more relevant studies, or choose to search specific journals which apply to your research area. Journals selected for handsearching should be identified from your database searching (Aromataris & Riitano, 2014; Booth, 2016).
- Experts - you may wish to contact experts to clarify details of a study or request data. These experts may also provide papers that weren't retrieved through your comprehensive search (Aromataris & Riitano, 2014; Booth, 2016).
- Citation chaining – identify a small number of key papers on your topic, and then find all the publications they cite and are cited by to produce a set of closely related publications (Belter, 2015).
Tips
- Use a search template to help you break down the key concepts of your search, and to document a list of synonyms and alternative search terms.
To come up with a comprehensive list of search terms, scope out your topic:
- investigating how authors describe the same concept
- think about alternative spelling and region-specific terminology (e.g. US / UK English, physiotherapy / physical therapy)
- consider colloquial terms and phrases (e.g. heart attack and myocardial infarction)
It may be helpful to see how others have put together their search:
- Use pre-tested search filters as a starting point. See the Resources section for recommendations
- Examine the search strategies used in other systematic reviews that investigate similar concepts
- Databases like Medline and the Cochrane Library (both CENTRAL and the CDSR) can be accessed using different platforms, like Ovid. It is important to specify which platform you are using in your protocol and your review.
Need help?
In most cases, a Librarian can assist with building one search strategy. To get the most out of your appointment, we recommend coming prepared!
A subject expert in your faculty is best placed to provide advice on grey literature sources.
References
- Aromataris, E., & Riitano, D. (2014). Systematic reviews. AJN, American Journal of Nursing, 114(5), 49–56.
- Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766–2777.
- Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews, 5(1), 74.
- Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association, 106(4).
- Bramer, W.M., Rethlefsen, M.L., Kleijnen, J., & Franco, O.H. (2017). Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Systematic Reviews, 6(1), 245.
- McGowan, J., Sampson, M., & Lefebvre, C. (2010). An evidence based checklist for the checklist for the peer review of electronic search strategies (PRESS EBC). Evidence Based Library & Information Practice, 5(1), 149-154.
- McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75, 40–46
- Paez, Arsenio. (2017) Grey literature: an important resource in systematic reviews. Journal of evidence-based medicine, 10(3), 233-240.
̏
Screen Results
After you've conducted your searches and exported all your results into a reference manager, you can start screening articles. During this step, reviewers select which studies are relevant based on their inclusion and exclusion criteria.
To control bias and minimise the risk of excluding relevant studies, screening is usually done by two or more independent reviewers. They talk through any disagreements in decision making over study inclusion or exclusion.
A standard workflow for this stage might be:
- Remove duplicate records from your pool of results
- Examine titles and abstracts to remove articles which obviously don't meet the inclusion criteria
- Find the full text of all potentially relevant articles
- Link together any reports which discuss the same study
- Screen full text articles and assess for inclusion in the review against the criteria outlined in your protocol.
- Make any final decisions on study inclusion
The PRISMA flow diagram visually summarises the screening process. It makes the selection process transparent and clear to the reader by reporting on decisions made at various stages of the systematic review. When you're excluding articles at the full-text stage, you need to specify the reasons for exclusion. It's important to document this thoroughly.
Tips
- While EndNote can be set up to screen articles, many researchers use collaborative tools for screening to make the process easier. The University of Sydney doesn't currently support any screening tools
- Endnote can be used to retrieve full text articles. Retrieval is more successful if the reference record contains a DOI
- If the full text of an article can't be found automatically through EndNote, you may have to retrieve it manually. If the Library doesn't have access to an article, you can request it using Resource Sharing
Need help?
For assistance with screening, contact your supervisor or a subject expert in your faculty.
References
- Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). Los Angeles: SAGE.
- Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, …Wieland LS. (2019). Chapter 4.6: Searching for and selecting studies.. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors), Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
- Li T, Higgins JPT, Deeks JJ (editors).(2019). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
- Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
- Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ, 349, g7647.
- Verhagen, A.P. (2017). The art of systematic reviews. Musculoskeletal Science and Practice, 31, iv-vi.
Data extraction and appraisal
Data extraction
Your next step is extracting the necessary data from your included studies. Not only will this help you assess the validity of the studies you're including in your review, but you're also prepping for the next stage: analysis and interpretation.
Before you extract all the data from the studies, you need to decide what data is necessary to extract. The below table is a sample of items to consider. The full table can be viewed in the Cochrane Handbook.
Source |
- Study ID (created by review author)
- Citation and contact details
|
Eligibility |
Confirm eligibility for review or state reason for exclusion |
Methods |
The methodologies that will be used when conducting the review, including:
- Study design
- Study duration
- Blinding
- Notes about bias
|
Participants |
- Diagnostic criteria
- Co-morbidities
- Setting
- Age
- Sex
- Country
|
Interventions |
Total number of intervention groups |
Outcomes |
- Outcomes and time points collected
- Outcome definition
- Unit of measurement
|
Results |
|
Miscellaneous |
Key conclusions of the study authors |
Once you've decided which items are needed, use a data extraction form as a consistent way to record the information.
Appraisal
When you're appraising the studies you've decided to include in your review, you're deciding whether you can draw conclusions about the effect of the intervention. This involves looking at external validity (did the study ask an appropriate research question?) and internal validity (did they answer the question 'correctly'?).
A key consideration to ask is, to what extent are the results of the included studies believable? You can assess this by examining the studies for various types of bias, which can lead to the study overstating or understating the true effect of the intervention.
Section 8 of the Cochrane handbook names the following forms of bias:
- Selection bias
- Performance bias
- Detection bias
- Attrition bias
- Reporting bias
- Other biases
There are a number of appraisal tools which will help you assess the studies you're including in your review.
Incomplete reporting
Incomplete reporting can affect how you assess risk of bias. What's excluded has a bearing on whether you can trust the study as a whole, or if you can only trust specific outcomes. Some ways of detecting whether a study has been reported incompletely include:
- Using study appraisal checklists
- Comparing the methods to the results
- Comparing the outcomes of a protocol (if there’s one available) to the published report
- Asking the study authors for more information
After appraisal, you might find there are studies you’d like to exclude from the review. Make sure you record the reasoning behind exclusion as it will need to be reported.
Analyse and interpret
During this stage, you'll be analysing and synthesising the data in order to come to a conclusion and answer your research question.
Your review will lend itself to different methods of analysis:
- A narrative analysis involves preparing a structured summary, comparing the data and coming up with themes to find evidence of an effect
- A statistical analysis is also known as a meta-analysis. It involves analysing the results across a number of studies in order to find the size and direction of an effect
- An integrative analysis combines both narrative and statistical analysis techniques, to reflect the diversity of the evidence included. For example, an integrative analysis could be a meta-analysis which appears alongside a narrative synthesis which provides additional information about the effect
Narrative synthesis techniques
Technique |
Description |
Advantage |
Limitation |
Text description |
Consistent annotation of studies: organised summary of same characteristics |
Keeps details of each study together |
Difficult to see themes |
Tabulation |
Create table of study elements and findings |
Compare characteristics |
Does not synthesise |
Categorise |
Group or cluster studies around particular characteristic(s) |
Visualise patterns |
May limit the ability to see concepts across all studies |
Thematic analysis |
Content analysis with a rubric to identify themes and factors |
Consistency |
Takes time to develop rubric |
Describe relationships |
Relationships between primary study findings and characteristics of study. Determine differences within and between studies |
Provides sub analysis |
May be difficult with a large number of studies |
Why should you conduct a meta-analysis as part of your systematic review?
- To increase power - the chance of detecting a real effect as statistically significant (if it exists)
- To improve precision
- To answer questions not posed by the individual studies
- To settle controversies arising from apparently conflicting studies
- Generate new hypotheses
Cochrane Handbook, Section 10.2 'Introduction to meta-analysis'
Despite these advantages, a meta-analysis should only be performed if you have a group of studies which are similar - this similarity is referred to as homogeneity. At the very least, studies should be homogeneous in terms of participants, interventions, and outcomes so the meta-analysis can draw a meaningful conclusion.
Any kind of variability among studies is referred to as heterogeneity. There are several kinds - Cochrane names:
- Clinical diversity - where participants, interventions and outcomes differ between studies
- Methodological diversity - where there is variability in study design
- Statistical heterogeneity - where the observed intervention effect differs between studies because of chance
It is important to measure and address heterogeneity in your review because it affects the extent to which you can rely on the conclusions you draw.
Need help?
Your supervisor or a subject expert in your faculty can assist with analysis and interpretation of your results.
If you're an HDR student, there is free statistical advice available through the Sydney Informatics Hub. Book an appointment to talk to a statistician by using the assistance request form.
References
- Akobeng, A.K. (2005). Understanding systematic reviews and meta-analysis. Archives of Disease in Childhood, 90(8), 845-848.
- Booth, A., Papaioannou, D., & Sutton, A. (2012). Systematic approaches to a successful literature review. London: Sage.
- Deeks, J. J., Higgins, J. P., & Altman, D. G. (2011). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, & Welch VA (editors).(2019). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane, 2019.
- Foster, M., & Jewell, S. (2017). Assembling the pieces of a systematic review: a guide for librarians. Lanham, Maryland: Rowman & Littlefield.
- Impellizzeri, F.M., & Bizzini, M. (2012). Systematic review and meta-analysis: A primer. International Journal of Sports Physical Therapy, 7(5), 493-503.
- Lee, C.H., Cook, S., Lee, J.S., & Han, B. (2016). Comparison of two meta-analysis methods: Inverse-variance-weighted average and weighted sum of z-score. Genomics & Informatics, 14(4), 173-180.
- Rice, K., Higgins, J.P.T., & Lumley, T. (2017). A re-evaluation of fixed effect(s) meta-analysis. Journal of the Royal Statistical Society, 181(1), 205-227.
- Riley, R.D., Ensor, J., Jackson, D., & Burke, D.L. (2017). Deriving percentage study weights in multi-parameter meta-analysis models: With application to meta-regression, network meta-analysis and one-stage individual participant data models. Statistical Methods in Medical Research, 27(10), 2885–2905.
- Uman, L.S. (2011). Information management for the busy practitioner: Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychology, 20(1), 57-59.
- Yannascoli, S.M., Schenker, M.L., Carey, J.L., Ahn, J., & mBaldwin, K.D. (2013). How to write a systematic review: A step-by-step guide. University of Pennsylvania Orthopaedic Journal, 23, 64-69.
Write and publish
Write
When you write up your review, not only are you reporting your findings and answering your question – you’re also reporting on the decisions made and the procedures followed to get you to that point. To make sure you don’t miss anything crucial in your write-up of the review, use the ‘Preferred Reporting Items for Systematic Reviews and Meta-Analyses’ (PRISMA) checklist.
Publish
Where you publish has a huge effect on the visibility and impact of your systematic review. It's important to be as well-informed as possible before making a decision on which journal to submit your paper to. If you've followed Cochrane, Joanna Briggs Institute, or Campbell Collaboration standards for conducting and writing a systematic review, you will already have an idea of where you want to publish.
The first hurdle is finding the right journals to publish in according to the scope of your review and your target audience. When considering who might take an interest in your research:
- Look at your reference list. Which journals do you regularly and consistently cite? Your work will likely be of interest to both the journal and its readers. Make sure you check the aims and scope in the 'information for authors' pages of each journal you'd like to publish in (e.g. Journal of Clinical Nursing).
- Use a tool like the Journal/Author Name Estimator (JANE) to retrieve a list of journals that may be interested in your systematic review, based on the similarity of your abstract to those of articles they have published. More tools for other disciplines are listed in Tools and resources.
- Identify influential journals in your field using metrics. There are a variety of metrics you can use to inform your decision. Some are more useful for certain disciplines than others. Check our Research Impact guide for more information.
How should I share my research?
When you submit your article to a journal for publishing, you often sign over copyright to the publishing journal. This affects how you are able to share it in the future, and how readers are able to access it. If it is important that clinicians and practitioners are able to access your research, it may be important to consider what sharing options a journal offers, or look at open access publication.
Need help?
For assistance with writing, contact your supervisor.
In addition to providing online modules on writing, The Learning Centre also run writing workshops.
If you have a question about copyright, contact the Copyright Team.
Enquiries about Open Access can be directed to your Librarian, however if you have a question about depositing a copy of your publication into the University repository, contact the Repository Team.
The Library runs workshops on Strategic Publishing. Check the Library Calendar to see upcoming workshops and to register.