What is a systematic review?
A systematic review is a method of answering a question by finding, appraising and synthesising evidence in a systematic and unbiased way.
This guide will take you through the steps involved in conducting a systematic review. Click one of the stages below to see information, tips, resources and support available for each part of the process.
determining if a systematic review is the right fit and an overview of the logistics in conducting a review.
scoping your topic, types of research questions and writing a focused question for your review to answer.
declaring the methodology and eligibility criteria used to determine what evidence will be used.
composing a search strategy to locate all relevant evidence to answer your question.
collecting the evidence, recording your search, and exporting to a reference manager.
determining which studies to include in the review and methods for reporting on decisions.
using tools to assess the validity of the evidence.
synthesising the results to answer your question.
writing up your review and getting it published.
*this is a rough estimate of the time needed to complete each stage, but will vary depending on the complexity of your review.
Systematic Review Service Charter
The Library provides tiered support for researchers conducting systematic reviews. Postgraduate students who conduct systematic reviews as part of their degree can request up to 3 hours of support per systematic review. Postdoctoral researchers and beyond can opt for more extensive support, depending on their requirements and our capacity. For details, please refer to the service charter.
We are always improving our resources. If you'd like to share your feedback with us, please do so via our feedback form.
Before you start
If you haven't conducted a systematic review before, you might be wondering where to start. This section takes you through a series of questions to consider before starting a systematic review.
Is my question appropriate for a Systematic Review?
The systematic review process works best for a clearly defined question with a narrow scope and a single answer. Traditionally, systematic reviews in health disciplines investigate the effectiveness of an intervention (or compare the effectiveness of two interventions), but the methodology is continually adapting to address different question types and disciplines.
A systematic review may not be the best method for your research if your question is too broad or the existing evidence is too sparse or varied. There are other review types that share some of the systematic review's characteristics, namely scoping reviews and systematic literature reviews. Explore differences and similarities of various review types in this comparative table: Types of Reviews.
Complete this online module to find out which review type is the most suitable for your research question:
In many cases, the most appropriate review type for a literature review for a PhD thesis would be a scoping review.
To learn about scoping review methodology, refer to Additional Reading section on this page.
What type of systematic review is the most appropriate for my question?
If you are certain that your question can be answered by conducting a systematic review, you need to consider what type of systematic review suits it best.
All systematic reviews share the same methodology. However, different types of questions require different considerations.
To learn what kind of systematic review is the most applicable to your question, complete this online module
Is there an existing systematic review on my topic?
Before starting work on a systematic review check if a systematic review on a similar topic has already been published (or commenced).
Use the following tools to perform a quick search for your topic (a few main keywords will do), and limit to "reviews" or "systematic reviews" (depending on the interface):
Do I have enough time to complete a systematic review?
Systematic reviews take time. The estimated timeline for a Cochrane systematic review is 18 months full time.
Doing a systematic review properly can take months, even if you're not conducting a Cochrane-style systematic review.
If you expect that time might be an issue, consider conducting a rapid review instead.
For more on rapid reviews, refer to Additional reading section on this page.
Do I have access to necessary resources?
A team is an important factor for a successful systematic review. Your team may include:
- Subject matter experts (like your supervisor or colleagues) for question formulation and writing up findings
- A librarian for guidance around literature searching
- Colleagues to become second/third reviewers
If you are planning to perform a meta-analysis, you will also need access to:
Finding collaborators can be tricky. You might consider:
- Reaching out to your network/s to identify people with common research interests
- Using academic social networks like Mendeley and ResearchGate to find potential collaborators
- Keeping your researcher profiles (e.g., USYD Academic Profile, Google Scholar, ORCiD, Scopus ID, and Web of Science Researcher ID) up to date so others can easily find you
Depending on your topic, you might want to explore professional networks to identify potential collaborators among a range of stakeholders including clinicians, patients, customers, or anyone else whose contribution to the review might be useful.
To learn what levels of library support are available for researchers conducting systematic reviews explore the SR Service Charter.
The library runs regular workshops to get you started with your literature review. To see upcoming workshops and to register, check the Library Calendar.
- Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology, 18(1), 143.
- Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015). Guidance for conducting systematic scoping reviews. International journal of evidence-based healthcare, 13(3), 141-146. doi:10.1097/xeb.0000000000000050.
- Dobbins, M. Rapid Review Guidebook. Steps for Conducting a Rapid Review. National Collaborating Centre for Methods and Tools.
- Tricco A.C., Langlois E.V., Straus S.E., editors. Rapid reviews to strengthen health policy and systems: a practical guide. Geneva: World Health Organization; 2017. Licence: CC BY-NC-SA 3.0 IGO.
Define research question
A focused research question is imperative for the success of your systematic review. This section provides tips on how to make your research question suitable for a systematic review.
Research topic vs a review question
A 'research topic' is the area of study that you are researching, while a 'review question' is the more focused question that your systematic review aims to answer. Depending on your starting point, you may arrive at a review question by taking different routes.
If you are working in an Applied Sciences field, or as a clinician, your review question might be informed by a problem or a scenario encountered in the lab or in practice.
If you are working towards your PhD, your thesis topic will be much broader than a question for a systematic review. Often, to account for the many topics covered in your thesis, you may need to consider writing a scoping review (for more, see Before you start. For example, your thesis might be about studying a range of non-therapeutic treatments for dementia. In your scoping review you will identify all existing treatments, and in your systematic review you might focus on the effectiveness of only one treatment:
Research topic and scoping review (broad)
- Non-therapeutic treatment of depression
Systematic review question (narrow)
- How effective are music-based therapy programs for relieving depression in patients with Alzheimer's?
Focussing Research Question
To help you focus your research question we suggest exploring different mnemonics. Mnemonics highlight different concepts that you need to focus on. To explore different mnemonics developed for different disciplines refer to the Additional literature section on this page.
In health and medicine, mnemonics might differ depending on the type of question you are working on, e.g., aetiology, prognostic, prevalence, etc. You can explore different mnemonics in this article
For more information on mnemonics refer to Ready for Health Research Module 2.
The most widely used mnemonic for systematic reviews is PICO – Population, Intervention, Comparator, Outcomes. PICO can be used to formulate a focused research question for systematic reviews in any field where the objective is to assess the effectiveness of an intervention.
PICO originated from evidence-based practice where it is used to help clinicians identify the most important elements of a question when analysing a patient’s condition and suggesting the best treatment.
For example, if your research topic was to explore a range of non-therapeutic treatments for depression, you could use PICO to concentrate on the specific conditions, treatments, and their outcomes:
- Population: patients with Alzheimer's experiencing depression
- Intervention: music-based therapy programs
- Comparison: no comparison
- Outcome: efficacy
You can learn about evidence-based practice and the use of PICO by exploring this EBP subject guide.
For more information on how to use PICO refer to Ready for Health Research Module 2.
Limitations of PICO
When applying PICO to your research topic, remember:
- Use PICO to focus on what already exists in your topic; do not invent concepts just to match all the letters in the mnemonic. If you do not intend to compare anything to your intervention, do not include a comparator.
- If your question is not centred on an intervention or if the concepts do not fit into PICO, your research question might be better suited to a different mnemonic.
- PICO helps you identify main concepts in your topic; it is not designed to help create a search strategy. To learn more about developing a search strategy, see Plan search strategy and select databases section.
For more information on how to translate main concepts into searchable concepts, refer to Ready for Health Research Module 2.
Develop and register protocol
Writing a protocol is an important step in minimising potential bias in the systematic review process. This section provides tips on how to prepare and register a protocol, and define eligibility criteria.
Why do I need to write a protocol?
A protocol specifies the question a systematic review will attempt to answer and the methods that will be used to answer it. Having a protocol is essential because:
- It ensures that the entire team is aware of and in agreement with the question, the methods, the timelines, and the search strategy
- It enables communication with external stakeholders when seeking their contribution
- It keeps you focused and on topic
Preparing a protocol
There are many reporting standards available which outline the essential information you need to provide in a protocol. Requirements will change depending on where you publish the protocol.
Some of these templates include:
Checklist - PRISMA-P
A useful checklist that lists every item you might need to address in a protocol. The accompanying elaboration and explanation document explains each item and provides clear examples.
Templates: Campbell Collaboration
Campbell Collaboration publish systematic reviews in social science, including areas such as crime and justice, education, nutrition, & social welfare. They provide a template for protocols which explains each section in detail.
Template text for reporting search methods - Cochrane
Contains best practice examples from the Cochrane Information Specialist community on how to write about the proposed search in Cochrane protocols. Although the examples are specific to Cochrane protocols, the language and terminology can be adapted to suit your purpose.
Systematic Review Centre for Laboratory Animal Experiemntation (SYRCLE) developes methodology for systematic reviews in animal studies and provides a template for a protocol.
Amending a protocol
Sometimes reviewers need to change their original protocol after a review has commenced. Any changes made should be reported and explained in an amendment to the protocol, as well as in the systematic review itself (Gough, Oliver, & Thomas, 2017). All registers have procedures for submitting amendments.
Your search strategy needs to be included as part of your protocol
Refer to Plan search strategy and select databases section for more details.
Why do I need to register a protocol?
You must have a protocol even if you are not planning to register it. Registering a protocol is highly recommended for the following reasons:
- To let other researchers know that you are working on a particular topic and help avoid duplication of effort
- To avoid publication bias whereby only the research that rendered positive results is published
- To help readers assess the quality of the review in terms of bias, through comparison of the protocol and the published systematic review
Registering a protocol
You have a range of platforms to choose from when deciding where to register a protocol. Many of the platforms will have specific conditions and limitations that you need to be aware of; refer to the Note section for useful tips.
An international register of systematic review protocols in the following areas:
- Health and social care
- Public health
- Crime and justice
- Health-related international development
Note: PROSPERO does not register scoping reviews and does not accept submissions from students.
Cochrane is a global network of researchers who produce Cochrane Reviews - these reviews are considered the ‘gold standard’ in systematic reviews in the field of health and medicine. Cochrane publish the protocols of Cochrane affiliated researchers through the Cochrane Database of Systematic Reviews
Note: when you register your protocol with Cochrane you commit to the Cochrane-style systematic review. Your topic would need to be approved by one of the 52 Cochrane Review Groups. You will be asked to update your review every two years.
Joanna Briggs Institute
JBI is a not-for-profit research and development centre within the Faculty of Health and Medical Sciences at the University of Adelaide, with more than 70 Collaborating Entities across the world.
Note: You can register your systematic review title with JBI, including scoping review titles, if at least one of the authors belongs to an "affiliated entity". The JBI journal JBI Evidence Synthesis charges US$1000 upon acceptance of full protocols if none of the authors of the submission are members of the JBI Collaboration.
An international research network that produces systematic reviews of the effects of social interventions, Campbell registers protocols for reviews in multiple areas of study but only if you plan on publishing your systematic review through the Campbell Collaboration. Any interested researcher can propose a review title, form an author team, and submit a proposal. There is a rigorous peer assessment process for the title and, if title is accepted, for the protocol. Refer to the Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR) for further details.
Note: Campbell will not publish protocols of reviews that are to be published elsewhere.
You can submit the review protocol as a preprint on one of the following platforms that will provide you with a time stamp indicating that the protocol was created prior to starting the review:
Sydney eScholarship (SES)
SES is the University of Sydney’s open access repository that accepts works from HDR students and academics, but not coursework students.
Open Science Framework
Here is a link to the help guides on creating a registration or preregistration. The OSF will accept protocols from students.
Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies
Eligibility (inclusion and exclusion) criteria
Eligibility criteria, also referred to as inclusion and exclusion criteria, are essential for systematic reviews. They are defined at the start of the systematic review process (a priori) to help you select the most reliable evidence as part of the screening process. Eligibility criteria determine which studies will be considered eligible for inclusion in the final analysis.
Because your eligibility criteria must not change in the process of conduction the review, define these criteria carefully and in a way that allows enough flexibility without introducing additional bias.
Eligibility criteria constitute part of the protocol. Some criteria may include:
- Population characteristics – such as population problem, age range or gender
- Study design or characteristics - such as study design, setting, or time frame
- Report characteristics - including years included, language, and publication status
Eligibility criteria may include features that you want either in or out of the studies considered relevant. You can choose the most convenient way of formulating a criterion either as an inclusion, or as an exclusion. You do not need to match each criterion with its opposite, i.e. include papers in English and exclude papers in all other languages.
Inclusion criteria examples
- Randomised controlled trials
- Papers published in the last 15 years
- Papers in English language
Exclusion criteria examples
- Papers focusing on other types of dementia (e.g. lewy body disease, vascular dementia, alcohol related dementia)
- Papers about other interventions (e.g. art therapy)
Each additional criterion introduces a potential bias into your research. It's important to justify each criterion in the methodology section of your systematic review. Different reporting guidelines may require different criteria.
Preparing a protocol requires the entire systematic review team’s effort. Make sure that you consulted subject experts such as your supervisor, as well as your Academic Liaison Librarian
for the search strategy part.
- Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews. Los Angeles: SAGE.
- Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
- Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1).
- National Institute for Health Research. (n.d.). What is registration? Retrieved from https://www.crd.york.ac.uk/prospero/#aboutpage.
- Patino, C.M., & Ferreira, J.C. (2018).Inclusion and exclusion criteria and why they matter. Jornal Brasileiro de Pneumologia, 44(2), 84.
- PRISMA. (n.d.). What is a protocol? Retrieved from http://prisma-statement.org/Protocols/.
- Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ, 349, g7647.
- Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense. Systematic Reviews, 1(7).
Plan search strategy and select databases
Systematic reviews aim to evaluate all relevant information about a topic to answer a question. This section provides tips on what constitutes a comprehensive search and which information sources you need to consider.
What constitutes a comprehensive search?
A comprehensive search has three distinct characteristics:
- It utilises multiple synonyms for each main concept
- It utilises advanced syntax
- It is performed across multiple databases
How do I plan a comprehensive search?
To construct a comprehensive search, you must:
|Identify searchable concepts in your topic
The searchable concepts might or might not correspond to a chosen mnemonic.
|Identify synonyms for each of the concepts
One way to do this is to conduct a quick scoping search in different databases (including Google Scholar) and explore the titles and abstracts of the relevant articles. You can find useful tips on how to do a scoping search in Ready for Health Research Module 1.
|Identify subject headings where appropriate
Subject headings vary between databases. You will need to identify subject headings in each database, not copy/paste your search from one database to another. You can learn about the difference between subject headings and keywords in this video and in Ready for Health Research Module 3.
|Use advanced search syntax
||Each database uses distinctive search syntax - coding that controls what fields are searched, how individual words or phrases are searched, and how you combine different concepts in your search strategy. You will need to develop a unique search strategy for each database utilising your search terms and any database-specific subject headings and syntax. For more information, see Perform Searches section.
We suggest you create a concept table to capture all the required information for a comprehensive search. Download this template to keep track of your work.
Where do I search?
Each database provides access to different collections of resources, so you’ll need to think about which databases contain the information you’re looking for. If you don't search enough sources, you can miss out on crucial studies.
Depending on your review question, you need to consider searching both multidisciplinary and subject-specific databases.
- Multidisciplinary databases contain a high volume of literature from a diverse range of subjects. They are helpful in locating literature on your topic from disciplines you might not have considered.
- Subject-specific databases contain in-depth coverage of literature which is highly specific to a particular field.
A balance of both database types is required in your search strategy.
Cochrane considers Medline (via either Ovid or PubMed), Embase, and CENTRAL to be the most important sources of evidence to search when conducting a systematic review in biomedical disciplines.
Health and Medicine Databases
To learn about the main databases in Health and Medicine, download this document
Where not to search
Can I search OVID?
Ovid is a platform that hosts many databases including Medline, Embase, Cochrane Central, and PsycINFO. These databases are available on other platforms which may use different search syntax. When writing your protocol, make sure you mention the platform on which you will search the database, e.g. Medline via Ovid., CINAHL via Ebsco, etc.
Should I search PubMed or Medline or both?
The Medline database is available through a number of platforms, including paid (Ovid, Web of Knowledge etc.) and free (PubMed). PubMed is a free database maintained by the National Center for Biotechnology Information (NCBI) in the US. Whilst PubMed indexes some journals not included in Medline, most of the content in PubMed is available via the Library’s Ovid Medline subscription. Constructing your search on Ovid Medline provides increased control over search structure, aiding consistency in your methodology. When researchers don’t have access to Medline, they can search PubMed instead. PubMed may be particularly useful for animal model-based research questions and studies in genetics.
Should I search Google Scholar?
One of the important features of a systematic review search is its reproducibility. Because Google Scholar searches are not reproducible, we advise against including Google Scholar in the list of databases for comprehensive searching.
Google Scholar can be used for scoping searches, and you can use Google when searching for grey literature.
Grey literature refers to research material that hasn't been published commercially like a book or a journal article has. The challenge lies in locating it, as it can be either published or unpublished.
Examples of published grey literature include:
- conference papers
- theses and dissertations
- government and organisation reports
- clinical trial registers
- clinical guidelines
Examples of unpublished grey literature include:
- unpublished trials
- on-going studies
- unpublished data
Depending on your subject, it might be appropriate for you to include grey literature as evidence because it provides data which otherwise wouldn't be available through journal articles. Including grey literature also helps reduce publication bias.
We've compiled a list of tools and resources to assist you. You can also find useful tips on searching Google for grey literature in this document.
Search filters are validated methods for finding relevant research in health databases. Filters are commonly used to limit searches to a particular study type, or to find articles on a specific subject. They can be used to enhance your search, limit your search results to only relevant study types, or for identifying additional search terms. It is not recommended to combine multiple study design filters – where your eligibility criteria includes multiple study designs, it is instead recommended not to use a study design filter.
Study type filters
Cochrane randomised controlled trial filters
McMaster University Hedges Project contains filters for the following databases:
Search filters (Scottish Intercollegiate Guidelines Network)
PubMed Clinical Queries
Search filters (ISSG)
CINAHL Clinical Queries
Subject area filters
CareSearch Australia: find evidence-based literature via PubMed for palliative care and related topics
Integrated care: search integrated care literature via PubMed
Aboriginal and Torres Strait Islander Health: search for literature about ATSI populations and health issues via PubMed
Search Blocks (Biomedische Informatie): add 'blocks' to your database searches to limit them by topic
SYRCLE Filter for Animal Studies: designed to limit a search to relevant animal studies
Your Academic Liaison Librarian can help you plan your search strategy. Don’t forget to bring your concept table to the appointment with your Academic Liaison Librarian.
While we can suggest some resources to help you search for grey literature, actual searching is out of scope in our support for systematic reviews
- Aromataris, E., & Riitano, D. (2014). Systematic reviews. AJN, American Journal of Nursing, 114(5), 49–56.
- Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766–2777.
- Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews, 5(1), 74.
- Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association, 106(4).
- Bramer, W.M., Rethlefsen, M.L., Kleijnen, J., & Franco, O.H. (2017). Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Systematic Reviews, 6(1), 245.
- McGowan, J., Sampson, M., & Lefebvre, C. (2010). An evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evidence Based Library & Information Practice, 5(1), 149-154.
- McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75, 40–46
- Paez, Arsenio. (2017) Grey literature: an important resource in systematic reviews. Journal of evidence-based medicine, 10(3), 233-240.
Comprehensive searching is a complex skill that is refined through deliberate practice. This section offers tips on how to execute comprehensive searches effectively and efficiently across multiple databases.
Where can I get help with searching specific databases?
Before you start searching in a database, check the following:
- Does the database have a thesaurus? (look for a “map to subject headings/suggest subject terms” function on the database search page)
- What syntax does this database use? (look for search tips in the database Help section)
- Does this database have a Save Search function?
In a database with a thesaurus, you will need to search for subject terms as well as for keywords.
Syntax are search operators which can be used to modify and improve the results of keyword searches. They'll vary according to the database. See Flinders University's database syntax guide for a comprehensive table of operators used across different databases.
To learn more about specific databases in Medicine and Health, their subject headings, syntax, and search techniques, enrol in Ready for Health Research and complete Modules 3-4.
Below you can find quick database search guides for specific databases:
Database search guides
How can I run the same search in all selected databases?
Because databases use different syntax and subject headings, you'll need to adapt your strategy to fit each database you search. See Flinders University's guide to translating searches from Medline to other databases like CINAHL, PsycINFO, and Scopus.
On Ovid, you are NOT advised to:
- Re-run a search created on one database (e.g., Medline) in the other database (e.g., Embase) due to the differences in subject headings
- Use the Ovid function to exclude Medline content from Embase - this will interfere with your number of search results, recorded later in the Prisma flowchart (see below)
- Select multiple databases and search them simultaneously
What is the right number of search results for a systematic review?
There is no magic number of search results that a systematic review search should aim for. The number of retrieved results will vary depending on the topic and the area of study. Established fields will yield more results, while new fields will yield less. A comprehensive search strategy needs to balance two objectives: being inclusive enough to capture as many relevant results as possible while keeping irrelevant results to a minimum.
If your search returns too many results, you might need to:
- Be more selective with the subject terms you are using (i.e., reconsider applying the “explode” function)
- Add more main concepts
- Reconsider using common words as keywords
If your search returns too few results, you might need to:
- Include more synonyms in both subject and keyword searches
- Consider adding less specific subject headings (broader terms) or using the “explode” function
- Consider adding more keywords
- Reduce the number of main concepts
How do I check the relevance of my search results?
A good way to check if your search is working is by creating a small sample of relevant articles (5 to 10) and checking if your search retrieved these articles. You can quickly find the relevant articles for the sample set by doing a quick search in Google or PubMed.
Search for your topic in the PubMed PubReMiner tool to check why your search might not be working. Whereas a search in PubMed retrieves relevant articles, a similar search in PubReMiner retrieves metadata assigned to the articles during the indexing process, such as author names, journal titles, and subject headings, and calculates the frequency with which the same entries appear in the records. Identify high frequency subject headings used to index the relevant articles and include them in your Medline search. Alternatively, if the articles in the same topic are indexed with a wide range of low frequency subject headings, consider including keywords that consistently appear in titles and abstracts of the relevant articles.
When do I stop searching?
When conducting a systematic review, your aim is to search enough databases to be able to state with confidence that the literature has been comprehensively searched. Indicators of saturation include:
- You look in a different database and don’t retrieve any new papers
- You can identify no new data. Click on the Sage Knowledge logo to view content
- You use a rule to decide when to stop searching
How do I document my searches?
Most systematic reviews include a PRISMA flow diagram in their methods section, along with the search strategies themselves. Record your searches in a log or a document to refer to them when writing up your methodology. You can use the PRISMA chart generator to help you build a PRISMA flow diagram for your own systematic review.
To ensure accurate reporting, keep numbers of which records were obtained from each database. Set up folders in your reference manager to help keep track of this.
You can refer to the Cochrane Handbook for tips on which elements of your search need to be documented
Other methods of locating relevant studies for your review
To ensure that you’ve located all relevant studies, you may consider supplementing your comprehensive search with one or more of the following additional types of searching:
- Handsearching - review the reference lists of papers you have already located for more relevant studies, or choose to search specific journals which apply to your research area. Journals selected for handsearching should be identified from your database searching (Aromataris & Riitano, 2014; Booth, 2016).
- Contacting authors - contact authors of conference papers or relevant studies to clarify details of a study or request more data. They may also provide papers that weren't retrieved through your comprehensive search (Aromataris & Riitano, 2014; Booth, 2016).
- Citation chaining – identify a small number of key papers on your topic, and then find all the publications they cite and are cited by to produce a set of closely related publications.
- Grey literature – searching grey literature is imperative for capturing the unpublished studies. Refer to Plan Search and Select Databases section for more detail.
Any studies that were located by means other than a reproducible comprehensive database search (including grey literature) should be recorded in a separate box in your PRISMA diagram (Additional records identified through other sources).
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, & Welch VA (editors). (2019). Cochrane Handbook for Systematic Reviews of Interventions. Version 6.0. Cochrane. Available from www.training.cochrane.org/handbook.
- Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). Los Angeles: SAGE.
- Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gøtzsche, P.C., Ioannidis, J.P.A., Clarke, M., Devereaux, P.J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine, 6(7), e1000100.
Comprehensive searches always retrieve more results than will be included in your final analysis. The screening process allows you to filter out irrelevant results by applying your inclusion and exclusion criteria. This section offers tips on efficient screening.
Which tool can be used for screening?
There are many collaborative tools for screening, both free and paid, to make the process easier. The Library doesn't currently support any screening tools.
Popular paid tools include:
Covidence - a tool for screening and data extraction. Recommended by Cochrane. Your first review can be conducted free of charge. Subscription fees apply for additional reviews.
Eppi-Reviewer - a comprehensive tool used for the searching, screening, and synthesis stages of a systematic review. Freely available to Cochrane-affiliated researchers.
Free alternatives include:
A Covidence free trial - trial allows 2 reviewers to conduct 1 review.
Rayyan QCRI - a web-based tool for collaborative screening. Learns selection patterns to assist with labelling of studies.
Abstrackr - a web-based tool for collaborative screening and labelling abstracts for relevance.
Picoportal - a systematic review platform.
More screening tools (both paid and free) can be found using the Systematic Review Toolbox - a comprehensive catalogue of tools to support systematic reviews.
How do I use EndNote for screening?
The Library supports EndNote which can be set up for managing search results for a systematic review.
Please refer to this subject guide to learn a 10-step process to use EndNote for systematic reviews.
How do I export searches from different databases?
Before you screen your search results, you will need to export them from each database. You should only start exporting when all your search strategies are finalised, and no further edits are anticipated.
You can find instructions on how to export from multiple databases on the EndNote subject guide (look under the Specific Database guides tab).
Refer to this table to learn EndNote export limits across different platforms and tips on bulk exporting.
How do I screen search results?
To control bias and minimise the risk of excluding relevant studies, screening is usually done by two or more independent reviewers. They talk through any disagreements in decision making over study inclusion or exclusion.
A standard workflow for this stage might be:
- Remove duplicate records from your pool of results
- Examine titles and abstracts to remove articles which obviously don't meet the inclusion criteria
- Find the full text of all potentially relevant articles
- Link together any reports which discuss the same study
- Screen full text articles and assess for inclusion in the review against the criteria outlined in your protocol.
- Make any final decisions on study inclusion
- Record numbers of articles included and excluded at each step in the PRISMA Flowchart
If your resources are limited, you might choose to do the first screening yourself and involve the second reviewer for the second screening when selection criteria are applied more rigorously, and for the full-text analysis.
Where can I get full text articles?
If using EndNote, you can attach most full text articles automatically using the ‘Find Full Text’ function. For the rest, check Library Search and, when you have found the full text, attach it to your article records in EndNote manually (refer to the EndNote manual for instructions). If you are confident that the library does not have access to the required article, you can request it via Resource Sharing.
How can I re-run the search?
Many publishers will request that you re-run your search if six months or more lapsed between the last search and the date of review admission for publication.
To do so, you can run your saved searches and limit them to the last year of search, and then screen results just for the last year to identify any new relevant articles, and update your PRISMA flowchart accordingly.
Refer to this document for instructions on how to use EndNote to manage references when re-running systematic review searches.
Librarians cannot help you with screening; for assistance, contact your supervisor or a subject expert in your faculty.
- Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). Los Angeles: SAGE.
- Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, …Wieland LS. (2019). Chapter 4.6: Searching for and selecting studies.. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors), Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
- Li T, Higgins JPT, Deeks JJ (editors).(2019). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
- Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
- Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ, 349, g7647.
- Verhagen, A.P. (2017). The art of systematic reviews. Musculoskeletal Science and Practice, 31, iv-vi.
Data extraction and appraisal
Data appraisal is essential for a systematic review.
This section provides tips and suggests tools for efficient data extraction and appraisal.
How do I extract data?
Your next step is extracting the necessary data from your included studies. Not only will this help you assess the validity of the studies you're including in your review but will also help you prepare for the next stage: analysis and interpretation.
Before you begin to extract data from the studies, you need to decide what data elements are required for your analysis. The below table is a sample of items to consider. The full table can be viewed in the Cochrane Handbook.
- Study ID (created by review author)
- Citation and contact details
||Confirm eligibility for review or state reason for exclusion
The methodologies that will be used when conducting the review, including:
- Study design
- Study duration
- Notes about bias
- Diagnostic criteria
||Total number of intervention groups
- Outcomes and time points collected
- Outcome definition
- Unit of measurement
||Key conclusions of the study authors
Once you've decided which items are needed, use a data extraction form as a consistent way to record the information.
How do I appraise selected studies?
When appraising each study eligible for inclusion in your review, you'll be evaluating whether you can draw conclusions about the effect of the intervention.
You can assess this by examining the studies for various types of bias, which can lead to the study overstating or understating the true effect of the intervention.
Section 8 of the Cochrane handbook names the following forms of bias:
- Selection bias
- Performance bias
- Detection bias
- Attrition bias
- Reporting bias
- Other biases
There are a number of appraisal tools which will help you assess the publications / studies you're including in your review.
How do I avoid incomplete reporting?
Incomplete reporting can affect how you assess risk of bias. What's excluded has a bearing on whether you can trust the study as a whole, or if you can only trust specific outcomes. Some ways of detecting whether a study has been reported incompletely include:
- Using study appraisal checklists
- Comparing the methods to the results
- Comparing the outcomes of a protocol (if there’s one available) to the published article
- Asking the study authors for more information
After appraisal, you might find there are further studies to exclude from the review based on your eligibility criteria. Make sure you record the reasoning behind exclusion as it will need to be reported.
Analyse and interpret
To answer your research question, you will be analysing and synthesising the extracted data. This section provides tips and suggests tools for efficient data analysis.
Your review will lend itself to different methods of analysis:
- A narrative analysis involves preparing a structured summary, comparing the data and coming up with themes to find evidence of an effect.
- A statistical analysis is also known as a meta-analysis. It involves pooling and analysing the results across a number of studies using statistical methods.
- An integrative analysis combines both narrative and statistical analysis techniques, to reflect the diversity of the evidence included. For example, an integrative analysis could be a meta-analysis which appears alongside a narrative synthesis which provides additional information about the effect.
Narrative synthesis techniques
||Consistent annotation of studies: organised summary of same characteristics
||Keeps details of each study together
||Difficult to see themes
||Create table of study elements and findings
||Does not synthesise
||Group or cluster studies around particular characteristic(s)
||May limit the ability to see concepts across all studies
||Content analysis with a rubric to identify themes and factors
||Takes time to develop rubric
||Relationships between primary study findings and characteristics of study. Determine differences within and between studies
||Provides sub analysis
||May be difficult with a large number of studies
Why should I conduct a meta-analysis as part of my systematic review?
- To increase power - the chance of detecting a real effect as statistically significant (if it exists)
- To improve precision
- To answer questions not posed by the individual studies
- To settle controversies arising from apparently conflicting studies
- Generate new hypotheses
Cochrane Handbook, Section 10.2 'Introduction to meta-analysis'
Despite these advantages, a meta-analysis should only be performed if you have a group of studies which are similar - this similarity is referred to as homogeneity. At the very least, studies should be homogeneous in terms of participants, interventions, and outcomes so the meta-analysis can draw a meaningful conclusion.
Any kind of variability among studies is referred to as heterogeneity. There are several kinds - Cochrane names:
- Clinical diversity - where participants, interventions and outcomes differ between studies
- Methodological diversity - where there is variability in study design
- Statistical heterogeneity - where the observed intervention effect differs between studies because of chance
It is important to measure and address heterogeneity in your review because it affects the extent to which you can rely on the conclusions you draw.
Qualitative data analysis software which enables you to handle rich text based information. It automates many manual tasks associated with analysis, like data classification and sorting. Available to University staff through ICT.
Your supervisor or a subject expert in your faculty can assist with analysis and interpretation of your results.
If you're an HDR student, there is free statistical advice available through the Sydney Informatics Hub. Book an appointment to talk to a statistician by using the assistance request form.
- Akobeng, A.K. (2005). Understanding systematic reviews and meta-analysis. Archives of Disease in Childhood, 90(8), 845-848.
- Booth, A., Papaioannou, D., & Sutton, A. (2012). Systematic approaches to a successful literature review. London: Sage.
- Deeks, J. J., Higgins, J. P., & Altman, D. G. (2011). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, & Welch VA (editors).(2019). Cochrane Handbook for Systematic Reviews of Interventions version 6.0. Cochrane.
- Foster, M., & Jewell, S. (2017). Assembling the pieces of a systematic review: a guide for librarians. Lanham, Maryland: Rowman & Littlefield.
- Impellizzeri, F.M., & Bizzini, M. (2012). Systematic review and meta-analysis: A primer. International Journal of Sports Physical Therapy, 7(5), 493-503.
- Lee, C.H., Cook, S., Lee, J.S., & Han, B. (2016). Comparison of two meta-analysis methods: Inverse-variance-weighted average and weighted sum of z-score. Genomics & Informatics, 14(4), 173-180.
- Rice, K., Higgins, J.P.T., & Lumley, T. (2017). A re-evaluation of fixed effect(s) meta-analysis. Journal of the Royal Statistical Society, 181(1), 205-227.
- Riley, R.D., Ensor, J., Jackson, D., & Burke, D.L. (2017). Deriving percentage study weights in multi-parameter meta-analysis models: With application to meta-regression, network meta-analysis and one-stage individual participant data models. Statistical Methods in Medical Research, 27(10), 2885–2905.
- Uman, L.S. (2011). Information management for the busy practitioner: Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychology, 20(1), 57-59.
Write and publish
This section provides tips and considerations for reporting the findings of your systematic review and deciding where to publish.
How should I write up my review?
In your write-up, you are not only reporting your findings and answering your review question – you are also reporting on the decisions made and the procedures followed that led to your findings. To ensure that your write-up is complete and transparent, use the PRISMA checklist.
Where should I publish my review?
Where you publish has a huge effect on the visibility and impact of your systematic review. Here are some tips for finding the right journals to publish in according to the scope of your review and your target audience:
- Look at your reference list. Which journals do you regularly and consistently cite? Your work will likely be of interest to both the journal and its readers
- Make sure you check the aims and scope in the 'information for authors' pages of each journal you'd like to publish in (e.g. Journal of Clinical Nursing)
- Make sure the journal’s instructions to authors include enough detail and minimum standards for conducting and reporting reviews
- Journal suggester tools generate lists of suggested journals based on the similarity of your abstract to articles these journals have published. Be mindful of the coverage limits of suggester tools provided by journal publishers
- Evaluate journals using metrics to identify those most influential in your field. There are a variety of metrics you can use to inform your decision. Some are more useful for certain disciplines than others. Check the Strategic Publishing Toolkit for more information
How should I share my research?
When you submit your article to a journal for publishing, you often sign over copyright to the publishing journal. This affects how you can share it in the future, and how readers can access it. If it is important for your research to be accessible to clinicians and practitioners, check what sharing options a journal offers (often stated in open access or editorial policies on the journal/publisher website), or investigate an open access publication.
For assistance with writing, contact your supervisor.
In addition to providing online modules on writing, The Learning Centre also run writing workshops.
If you have a question about copyright, contact the Copyright Team.
Enquiries about Open Access can be directed to your Academic Liaison Librarian, however if you have a question about depositing a copy of your publication into the University repository, contact the Repository Team.
The Library runs workshops on Strategic Publishing. Check the Library Calendar to see upcoming workshops and to register.
- Haddaway, N. R., Bethel, A., Dicks, L. V., Koricheva, J., Macura, B., Petrokofsky, G., ... Stewart, G. B. (2020). Eight problems with literature reviews and how to fix them. Nature Ecology & Evolution, 4(12), 1582-1589.