Put simply, a systematic review is a highly precise, carefully crafted review and subsequent analysis of research corresponding to a specific question. According to the CDC, "A systematic review attempts to collect and analyze all evidence that answers a specific question. The question must be clearly defined and have inclusion and exclusion criteria. A broad and thorough search of the literature is performed and a critical analysis of the search results is reported and ultimately provides a current evidence-based answer to the specific question."
Before we proceed, it is essential to know that a proper systematic review will take anywhere from 12 - 18 months to complete. That's a long time! However, it's necessary because most publishers require that systematic reviews follow precise guidelines before they will consider them for publication. While the guidelines might differ across disciplines, for the most part, they involve:
As stated previously, systematic reviews are focused on a unique research question and include a protocol as well as a review team. Literature reviews, on the other hand, are an overview of the topic gathered by an individual or smaller search team. Literature reviews state what articles have been published, who key authors or researchers are, what specific or broad questions are being asked, what methods researchers use, and what theories and hypotheses are currently being researched. They seek to build a core of research instead of delving into one specific question.
| Systematic Reviews | Literature Reviews | |
|---|---|---|
| Goal: |
Synthesize, analyze, and provide an unbiased review of a research question. |
Gather definitive and unique information on topic into one location through detailed search queries. |
| Timeframe: | 12-18 months to complete | Few weeks to several months |
| Who: | Completed by a team including reviewers, statisticians or economists, librarians or expert searchers | Librarian or expert searcher |
| Organization: | Uses PRISMA standards for process. | Can use a variety of structures, at least an introduction, body, and conclusion. |
| Search Methods: | Collaborates with a librarian or expert searcher to develop a search protocol. | Uses specific search queries using appropriate databases, controlled vocabularies (including MeSH terms), operators, syntax, and filters to gather information. |
Literature reviews are meant to review a variety of articles, case studies, reports, and other research information, not answer or ask research questions like the systematic reviews. When published, they provide a useful and necessary frame of reference in the scientific community by establishing key discussions or information about the topic. Additional resources and examples of systematic reviews and literature reviews are below.
Centers for Disease Control and Prevention Systematic Reviews Site: https://www.cdc.gov/library/researchguides/systematicreviews.html
Duke Biomedical Library Systematic Review Guide: https://guides.mclibrary.duke.edu/sysreview/home
Pollock, A., & Berge, E. (2018). How to do a systematic review. International Journal of Stroke, 13(2), 138–156. https://doi.org/10.1177/1747493017743796
Cullis, P. S., Gudlaugsdottir, K., & Andrews, J. (2017). A systematic review of the quality of conduct and reporting of systematic reviews and meta-analyses in paediatric surgery. PLoS ONE, 12(4). https://doi.org/10.1371/journal.pone.0175213: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5383307/
Fadlallah, R., El-Jardali, F., Nomier, M., Hemadi, N., Arif, K., Langlois, E. V., & Akl, E. A. (2019). Using narratives to impact health policy-making: A systematic review. Health Research Policy and Systems, 17. https://doi.org/10.1186/s12961-019-0423-4: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6402129/
Pot, M. W., van Kuppevelt, T. H., Gonzales, V. K., Buma, P., IntHout, J., de Vries, R. B. M., & Daamen, W. F. (2017). Augmented cartilage regeneration by implantation of cellular versus acellular implants after bone marrow stimulation: A systematic review and meta-analysis of animal studies. PeerJ, 5. https://doi.org/10.7717/peerj.3927: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5661456/
Thompson, W., Tonkin-Crine, S., Pavitt, S. H., McEachan, R. R. C., Douglas, G. V. A., Aggarwal, V. R., & Sandoe, J. A. T. (2019). Factors associated with antibiotic prescribing for adults with acute conditions: An umbrella review across primary care and a systematic review focusing on primary dental care. Journal of Antimicrobial Chemotherapy, 74(8), 2139–2152. https://doi.org/10.1093/jac/dkz152: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6640312/
Florida Gulf Coast University Library, Literature Review (Healthcare): https://fgcu.libguides.com/c.php?g=65107&p=419881
University of California, Santa Cruz, Literature Review: https://guides.library.ucsc.edu/write-a-literature-review
University of Toronto, Literature Review Tips: http://advice.writing.utoronto.ca/types-of-writing/literature-review/
University of Wisconsin – Madison Writing Center, Literature Review Basics: https://writing.wisc.edu/handbook/assignments/reviewofliterature/
Alturkistani, H. A., Tashkandi, F. M., & Mohammedsaleh, Z. M. (2016). Histological stains: A literature review and case study. Global Journal of Health Science, 8(3), 72–79. https://doi.org/10.5539/gjhs.v8n3p72: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4804027/
Garcia-Schinzari, N. R., & Santos, F. S. (2014). Assistance to children in palliative care in the Brazilian scientific literature. Revista Paulista de Pediatria, 32(1), 99–106. https://doi.org/10.1590/S0103-05822014000100016: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182999/
Lomoro, P., Verde, F., Zerboni, F., Simonetti, I., Borghi, C., Fachinetti, C., Natalizi, A., & Martegani, A. (2020). COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: Single-center study and comprehensive radiologic literature review. European Journal of Radiology Open, 7, 100231. https://doi.org/10.1016/j.ejro.2020.10023: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7129441/
Under the systematic and literature review umbrella, there are a range of other review types. These exist because different situations will often call for different review approaches. Sometimes, you will find a wealth of randomized controlled trials on a single subject, which will allow you the chance to pursue a systematic review (or even a meta-analysis). Other times, a research field may be in the early stages of emergence. In these cases, a scoping review might be preferable.
Scoping reviews have grown in popularity in recent years due to their focus on a more agile approach to reviewing evidence. Scoping reviews may also rely on less "quality" information than a standard systematic review, and can go beyond the analysis of clinical trials. Also, "unlike a systematic review, scoping reviews do not tend to produce and report results that have been synthesized from multiple evidence sources following a formal process of methodological appraisal to determine the quality of the evidence. Rather, scoping reviews aim to provide an overview or map of the evidence."
For more information about scoping reviews and how to conduct them, check out the following resources:
***In addition to scoping reviews, there are a wealth of review types available under the systematic review umbrella. Check out the following resources to learn about common and emerging review types, and when to leverage them:
Librarians will oversee the literature search protocol in a collaborative relationship. In order to begin the search process, the librarian needs a detailed protocol establishing the research question, criteria (inclusion or exclusion), 2-4 examples of articles fitting the criteria, and preliminary search results. In return, they may ask clarifying questions or work with you to determine critical points or terms necessary in your research before developing a search method.
The librarian will evaluate your specific research question and develop a search method and subsequent search queries based on your research requirements. This means librarians can:
Understanding the method of constructing a search query may help you develop a search protocol or initiate a preliminary search before working with the librarian. When a librarian approaches a search, they first must understand the goal or specific research question in order to develop a list of subjects and terms. Librarians and expert searchers use the following tools and methods to create intricate and relevant search queries.
For anything related to the health sciences, PubMed and PubMed Central are quality databases with rigorous articles. Other important health science databases include but are not limited to: Cochrane Library, MEDLINE Plus, and ScienceDirect Journals.
In a healthcare or medical setting, all researchers need to use exact phrasing or key words in search queries. One helpful tool is the Medical Subject Headings (or MeSH) controlled vocabulary from the National Library of Medicine (NLM). This controlled vocabulary includes approved key words and descriptions representing subjects in the life sciences. Before beginning any search query, it is important to find and use the appropriate MeSH term so the research can be included in a specific, appropriate results page.
For example, when researching post traumatic stress disorder, several researchers may label their content with terms such as “post traumatic stress disorder,” “PTSD,” or “stress disorders.” Instead, the approved term according to the MeSH controlled vocabulary is “Stress Disorders, Post-Traumatic.” If the researchers did not change their terms to this controlled term, their results would not be included in a search query.
Below is a video explaining how to search PubMed using MeSH terms:
Quotation Marks: Used to indicate an exact term or phrase.
Parentheses: Used to indicate priority of specific term or phrase.
AND: Used as a conjunction to combine two terms.
OR: Used as a conjunction to indicate two separate terms.
NOT: Used as a conjunction to include one term and exclude other associated terms.
* (star): Used as a wildcard to return any term or phrase.
For example, the following string represents the combination of two terms (using the MeSH controlled vocabulary) and would return any documents that included both terms, not one or the other:
“Brain Injuries, Traumatic” AND “Behavioral Symptoms”
Similarly, the next string would represent a more broad result related to several brain injury topics:
“Brain Injuries, Traumatic” OR “Brain Injuries, Chronic”
PubMed Central includes several useful filters such as Publication Date, Open Access, Include Embargoed Articles, and Research Funder. Other databases may include Topics/Subjects, Publication Type, Format, Language, Animal Subjects, and more.
In any search, a researcher needs to maximize the best-fitting, or relevant, information to their topic in order to successfully answer or attempt to answer their research question. By using the components listed in the Search Strategies and Tools above, a researcher can more easily and quickly find the appropriate information. The two values that determine relevancy are Precision and Recall.
Precision means the success of returning of specific, useful information in a search of the entire database. It is measured by taking the number of relevant information in a search divided by the total number of identified information in that search. High precision is when a researcher collects only the useful information out of a large number of useful and not useful documents within a database. Low precision is when a researcher finds that both useful and not useful documents have been returned in a search. This may be caused by including too broad or narrow of key terms.
For example, a search query returns 1,200 articles including correct/useful key words to the question out of 4,800 total articles available on the database. That means the specificity of that search is 48% (1,200 divided by 4,800). To increase this number, the researcher may review new key words or filters to find more precise information.
Recall means establishing the most useful documents out of the already gathered, useful documents. It is measured by taking the amount of relevant information returned in a search divided by the total amount of relevant information. High recall is when a researcher finds a large amount of useful documents out of the already gathered useful documents. Low recall is when a researcher finds only a portion of the useful documents are applicable to the study or not useful documents are included in the search results.
For example, a search query returns 150 relevant documents out of an estimated 250 total relevant documents. That means the sensitivity of those documents is 60% (150 divided by 250).
Both Precision and Recall help researchers to measure the relevance of the information to their topic. It is unrealistic to achieve 100% Precision and Recall in any search query or strategy, but researchers should attempt to retrieve the most relevant documents or information for their research.
Gray literature is any documents or literature produced but not published commercially. Academic, government, business, health, and research organizations or groups still produce materials to circulate information to their shareholders and users. Some examples of gray literature include conference pamphlets, theses and dissertations, committee research and reports, annual subject reports, peer-reviewed academic articles and literature, ongoing research, patents, and more.
Despite the challenge of finding the documents, gray literature is worth including in your systematic review. It can be very current and comprehensive and provide an overall description of evidence while reducing publication bias (Paez, 2017). Gray literature includes reports with negative or null results or reports that may have been rejected by commercial publications. Since the timeline for publication extends from a few months to over a year, some researchers may want to share the reports and continue research, meaning that their results are more quickly posted. In other situations, researchers may feel that their data isn’t rigorous enough to be published to a journal but still share their findings. These produced documents provide recent, thorough data and reduce the reliance on commercial publication.
There is not a central repository for all organizations to share their literature so it can be challenging to know where to look for these materials. Organizations also have a wide variety of audiences and formats and may choose to self-publish or to publish in specific databases or websites. However, there are several useful databases for health sciences
To begin searching for gray literature, visit the following sites:
For more information on Gray Literature:
Duke Biomedical Library Systematic Review Guide (Gray Literature tab): https://guides.mclibrary.duke.edu/sysreview/greylit
Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence-Based Medicine, 10(3), 233–240. https://doi.org/10.1111/jebm.12266. https://onlinelibrary.wiley.com/doi/full/10.1111/jebm.12266
Citation management, also known as reference management, is the process of gathering and organizing bibliographic information for a report or review. Bibliographic information is any unique identifying information about the document or resource, including author name, title, date, publishing company or journal, page numbers, etc. Each set of bibliographic information is called a reference or citation. The goal is to provide as much distinguishing information as possible in a citation so readers can find that exact document listed in the references. Researchers gather these together and place them into a reference list at the end of the report; in addition, they put in-text citations throughout their report that note the author and date, which lead readers back to the reference list for the full citation.
For a systematic review, many reference lists consist of over 50 citations! It can be challenging and confusing trying to organize and gather all these citations or references together by hand, but the solution is to use a citation management platform to automatically file, organize, and list references.
Citation management platforms are software programs that integrate with a variety of web browsers and databases. They are able to analyze and download bibliographic information to an account and retrieve a full reference list in a specific reference style. More importantly, they make it much easier to navigate and organize large amounts of citations. Most web or browser based platforms are also accessible from any computer with an account login. They are designed to be easy to use and fast.
For your systematic review, a CMP can reduce the time spent writing out citations, looking up citation style rules, alphabetizing the entries by author or organization, organizing citations by subject, and attempting to find that one specific article from a list of saved web pages from your browser. If you are working on your systematic review for over a year, chances are some of the documents or citations you find may undergo some changes. The CMP allows you to return and add, edit, or delete citations from the saved bibliographic list instead of attempting to find the error by hand.
In general, a large portion of systematic review writers use CMPs. The platform helps improve the quality of the paper through providing transparency about the quantity and quality of the citations used in the review. Through this transparent reference list, other researchers can attempt to reproduce the research by reading those citations. A major component of systematic research is peer-review. Without a reference list, researchers will not know where or how you got specific information, and they cannot attempt to confirm the quality of the systematic review.
At NAU, both Zotero and Mendeley are available for you to use. Zotero is a free to use, downloadable citation platform that includes a variety of citation format styles and uses browser extensions to capture citations. Use the Zotero Guide below to help you learn how to create an account, add or import citations, organize your work, and create bibliographies. This is the NAU Librarian recommended platform.
Setting up Zotero
To set up Zotero on your computer, you need to download and install the software on your computer. There are options available for Mac and Windows (and Linux). Another useful tip is to include the browser extension and Microsoft Word plugin to easily capture documents. You can transfer, organize, and cite documents within the software program.
For step by step help on setting up Zotero, click here to view the NAU Zotero Course.
NAU Cline Library, Zotero Research Guide: http://libraryguides.nau.edu/zotero
Finding and collecting research is only a small part of the systematic review journey. Once research has all been clustered into a single place (like Zotero), titles and abstracts will then need to be screened for relevance.
Generally, screening occurs just after the search results have been collected. Access to full text is not necessary at this stage. All a team needs is the title and abstract for each search result. To make screening easier, the following steps are recommended:
Once the first screening stage is finished, the reviewers will then work together to review the full text of all selected papers. At this stage, there may be some papers which do not meet criteria for grading. Use the following questions (Hammelfarb Health Sciences Library 2020) to determine if the full text does not meet the criteria for further inclusion:
There are critical appraisal tools available online that can help you through this process. These include (but are not limited to):
Just like in the first step of the screening process, all decisions should be recorded.
Now that you've finished screening, it's time to grade the quality of the studies selected for the review. This is one of the most time consuming steps in the process and should be performed with rigor and attention to detail.
There are several resources available to help teams make it through grading. One of the most essential is the Cochrane GRADE Handbook. The GRADE Handbook can walk a team through every step of the process; however, it is important to keep in mind that this handbook is designed primarily for teams undertaking a Cochrane Systematic Review. (If you don't know what a Cochrane Systematic Review is, you're probably not taking one on, so don't worry about it too much!)
Aside from the GRADE Handbook, the following resources might also prove useful:
Once you've finished grading, you're set to write your paper! If you have any questions on any part of this section, feel free to reach out to your PBC Librarian at: catherine.lockmiller@nau.edu.
PBC Library launched a systematic reviews pilot service in Summer 2018. As part of this service, the NAU PBC librarian provides search coordination and review processes guidelines to teams working on different review types. For now, the service is freely available to all NAU faculty at PBC currently considering embarking on a review.
For anybody who wants to learn more, the librarian requests at least one face-to-face or video-based consultation to talk about the review journey. It is also required that all teams complete the PBC Library Systematic Reviews Initial Steps Planner.
Aside from the Initial Steps Planner, the following resources are freely provided to individuals at NAU interested in beginning their systematic review journey:
If you have any questions about this service, feel free to call, email, or schedule an appointment to learn more!