Put simply, a systematic review is a highly precise, carefully crafted review and subsequent analysis of research corresponding to a specific question. According to the CDC, "A systematic review attempts to collect and analyze all evidence that answers a specific question. The question must be clearly defined and have inclusion and exclusion criteria. A broad and thorough search of the literature is performed and a critical analysis of the search results is reported and ultimately provides a current evidence-based answer to the specific question."
What are the steps to completing a systematic review?
Before we proceed, it is essential to know that a proper systematic review will take anywhere from 12 - 18 months to complete. That's a long time! However, it's necessary because most publishers require that systematic reviews follow precise guidelines before they will consider them for publication. While the guidelines might differ across disciplines, for the most part, they involve:
Organize a review team, consisting of the following:
a content expert;
at least two reviewers;
a literature search expert (generally a librarian);
a statistician (if conducting a meta-analysis);
Frame your research goals in the form of a precise question;
If applicable, consider whether open access journals are right for your needs
What is the difference between systematic and literature reviews?
As stated previously, systematic reviews are focused on a unique research question and include a protocol as well as a review team. Literature reviews, on the other hand, are an overview of the topic gathered by an individual or smaller search team. Literature reviews state what articles have been published, who key authors or researchers are, what specific or broad questions are being asked, what methods researchers use, and what theories and hypotheses are currently being researched. They seek to build a core of research instead of delving into one specific question.
The following table compares the two types of reviews:
Comparison of Systematic Reviews and Literature Reviews
Synthesize, analyze, and provide an unbiased review of a research question.
Gather definitive and unique information on topic into one location through detailed search queries.
12-18 months to complete
Few weeks to several months
Completed by a team including reviewers, statisticians or economists, librarians or expert searchers
Librarian or expert searcher
Uses PRISMA standards for process.
Can use a variety of structures, at least an introduction, body, and conclusion.
Collaborates with a librarian or expert searcher to develop a search protocol.
Uses specific search queries using appropriate databases, controlled vocabularies (including MeSH terms), operators, syntax, and filters to gather information.
Literature reviews are meant to review a variety of articles, case studies, reports, and other research information, not answer or ask research questions like the systematic reviews. When published, they provide a useful and necessary frame of reference in the scientific community by establishing key discussions or information about the topic. Additional resources and examples of systematic reviews and literature reviews are below.
Thompson, W., Tonkin-Crine, S., Pavitt, S. H., McEachan, R. R. C., Douglas, G. V. A., Aggarwal, V. R., & Sandoe, J. A. T. (2019). Factors associated with antibiotic prescribing for adults with acute conditions: An umbrella review across primary care and a systematic review focusing on primary dental care. Journal of Antimicrobial Chemotherapy, 74(8), 2139–2152. https://doi.org/10.1093/jac/dkz152: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6640312/
Under the systematic and literature review umbrella, there are a range of other review types. These exist because different situations will often call for different review approaches. Sometimes, you will find a wealth of randomized controlled trials on a single subject, which will allow you the chance to pursue a systematic review (or even a meta-analysis). Other times, a research field may be in the early stages of emergence. In these cases, a scoping review might be preferable.
Scoping reviews have grown in popularity in recent years due to their focus on a more agile approach to reviewing evidence. Scoping reviews may also rely on less "quality" information than a standard systematic review, and can go beyond the analysis of clinical trials. Also, "unlike a systematic review, scoping reviews do not tend to produce and report results that have been synthesized from multiple evidence sources following a formal process of methodological appraisal to determine the quality of the evidence. Rather, scoping reviews aim to provide an overview or map of the evidence."
For more information about scoping reviews and how to conduct them, check out the following resources:
***In addition to scoping reviews, there are a wealth of review types available under the systematic review umbrella. Check out the following resources to learn about common and emerging review types, and when to leverage them:
So how does a librarian help you find appropriate literature?
Librarians will oversee the literature search protocol in a collaborative relationship. In order to begin the search process, the librarian needs a detailed protocol establishing the research question, criteria (inclusion or exclusion), 2-4 examples of articles fitting the criteria, and preliminary search results. In return, they may ask clarifying questions or work with you to determine critical points or terms necessary in your research before developing a search method.
The librarian will evaluate your specific research question and develop a search method and subsequent search queries based on your research requirements. This means librarians can:
Find previous systematic reviews to prevent research duplication.
Develop and document broad search strategy to return precise and relevant information.
Find appropriate databases relevant to specific research question.
Identify key terms existing in healthcare databases through Medical Subject Headings (MeSH) or other appropriate controlled vocabulary.
Perform search queries in order to find detailed articles making up a comprehensive picture of current research of the question.
Gather total relevant results of search queries.
Strategic Searching and Tools
Understanding the method of constructing a search query may help you develop a search protocol or initiate a preliminary search before working with the librarian. When a librarian approaches a search, they first must understand the goal or specific research question in order to develop a list of subjects and terms. Librarians and expert searchers use the following tools and methods to create intricate and relevant search queries.
A database is a collection of organized data contained within a storage system, most commonly a software program. Each database may house different subjects or types of information. It is important to choose the most applicable databases to the research question.
For anything related to the health sciences, PubMed and PubMed Central are quality databases with rigorous articles. Other important health science databases include but are not limited to: Cochrane Library, MEDLINE Plus, and ScienceDirect Journals.
A controlled vocabulary is a defined set of structured words or phrases useful for classifying/labeling and retrieving content in a search system. The controlled vocabulary will include a hierarchy (a structure that organizes terms from broad to narrow), subject headings, preferred or variant terms, and descriptions of terms. By adhering to a controlled vocabulary, all researchers will use the same terms, making retrieving content faster and more accurate.
In a healthcare or medical setting, all researchers need to use exact phrasing or key words in search queries. One helpful tool is the Medical Subject Headings (or MeSH) controlled vocabulary from the National Library of Medicine (NLM). This controlled vocabulary includes approved key words and descriptions representing subjects in the life sciences. Before beginning any search query, it is important to find and use the appropriate MeSH term so the research can be included in a specific, appropriate results page.
For example, when researching post traumatic stress disorder, several researchers may label their content with terms such as “post traumatic stress disorder,” “PTSD,” or “stress disorders.” Instead, the approved term according to the MeSH controlled vocabulary is “Stress Disorders, Post-Traumatic.” If the researchers did not change their terms to this controlled term, their results would not be included in a search query.
Below is a video explaining how to search PubMed using MeSH terms:
A search operator is a special input command that can retrieve variations or combinations of key words, either broadening or limiting a search query. When used together, they can communicate with the data in a database to return appropriate results.
Quotation Marks: Used to indicate an exact term or phrase.
Parentheses: Used to indicate priority of specific term or phrase.
AND: Used as a conjunction to combine two terms.
OR: Used as a conjunction to indicate two separate terms.
NOT: Used as a conjunction to include one term and exclude other associated terms.
* (star): Used as a wildcard to return any term or phrase.
For example, the following string represents the combination of two terms (using the MeSH controlled vocabulary) and would return any documents that included both terms, not one or the other:
“Brain Injuries, Traumatic” AND “Behavioral Symptoms”
Similarly, the next string would represent a more broad result related to several brain injury topics:
“Brain Injuries, Traumatic” OR “Brain Injuries, Chronic”
The syntax is the way the terms, punctuation, and the order are listed in a search query. Researchers enter text (data) in precise strings, not sentences, to communicate with the data system. Entering sentences such as "What is the link between heart attacks or stroke and acid reducer medicine?" will not work within a database or search engine. Instead, the researcher isolates key terms "Myocardial Infarction," "stroke," and "omeprazole" and decides what order to search in. Most terms can be interchangeable, but the punctuation and operators must be in the correct positions.
For example, "Myocardial Infarction" "stroke" "omeprazole" doesn't automatically explain we are seeking information on either heart attacks or strokes as well as adding a drug into the mix. A better syntax for this question might be: ("Myocardial Infarction" OR "stroke") and "omeprazole" . This syntax string appropriately uses punctuation and operators to phrase the question in a way to retrieve appropriate information.
A filter in a database is a preset option to customize the retrieved documents in a search query. Filters are shown on the search page usually as check boxes or clickable attributes that include documents indexed with those specific terms. When researching, it is important to be aware of filters to remove unnecessary documents from a search query.
PubMed Central includes several useful filters such as Publication Date, Open Access, Include Embargoed Articles, and Research Funder. Other databases may include Topics/Subjects, Publication Type, Format, Language, Animal Subjects, and more.
Precision and Recall Information
In any search, a researcher needs to maximize the best-fitting, or relevant, information to their topic in order to successfully answer or attempt to answer their research question. By using the components listed in the Search Strategies and Tools above, a researcher can more easily and quickly find the appropriate information. The two values that determine relevancy are Precision and Recall.
Precision means the success of returning of specific, useful information in a search of the entire database. It is measured by taking the number of relevant information in a search divided by the total number of identified information in that search. High precision is when a researcher collects only the useful information out of a large number of useful and not useful documents within a database. Low precision is when a researcher finds that both useful and not useful documents have been returned in a search. This may be caused by including too broad or narrow of key terms.
For example, a search query returns 1,200 articles including correct/useful key words to the question out of 4,800 total articles available on the database. That means the specificity of that search is 48% (1,200 divided by 4,800). To increase this number, the researcher may review new key words or filters to find more precise information.
Recall means establishing the most useful documents out of the already gathered, useful documents. It is measured by taking the amount of relevant information returned in a search divided by the total amount of relevant information. High recall is when a researcher finds a large amount of useful documents out of the already gathered useful documents. Low recall is when a researcher finds only a portion of the useful documents are applicable to the study or not useful documents are included in the search results.
For example, a search query returns 150 relevant documents out of an estimated 250 total relevant documents. That means the sensitivity of those documents is 60% (150 divided by 250).
Both Precision and Recall help researchers to measure the relevance of the information to their topic. It is unrealistic to achieve 100% Precision and Recall in any search query or strategy, but researchers should attempt to retrieve the most relevant documents or information for their research.
What is gray literature?
Gray literature is any documents or literature produced but not published commercially. Academic, government, business, health, and research organizations or groups still produce materials to circulate information to their shareholders and users. Some examples of gray literature include conference pamphlets, theses and dissertations, committee research and reports, annual subject reports, peer-reviewed academic articles and literature, ongoing research, patents, and more.
How can it help my systematic review?
Despite the challenge of finding the documents, gray literature is worth including in your systematic review. It can be very current and comprehensive and provide an overall description of evidence while reducing publication bias (Paez, 2017). Gray literature includes reports with negative or null results or reports that may have been rejected by commercial publications. Since the timeline for publication extends from a few months to over a year, some researchers may want to share the reports and continue research, meaning that their results are more quickly posted. In other situations, researchers may feel that their data isn’t rigorous enough to be published to a journal but still share their findings. These produced documents provide recent, thorough data and reduce the reliance on commercial publication.
Where can I find gray literature?
There is not a central repository for all organizations to share their literature so it can be challenging to know where to look for these materials. Organizations also have a wide variety of audiences and formats and may choose to self-publish or to publish in specific databases or websites. However, there are several useful databases for health sciences
To begin searching for gray literature, visit the following sites:
Citation management, also known as reference management, is the process of gathering and organizing bibliographic information for a report or review. Bibliographic information is any unique identifying information about the document or resource, including author name, title, date, publishing company or journal, page numbers, etc. Each set of bibliographic information is called a reference or citation. The goal is to provide as much distinguishing information as possible in a citation so readers can find that exact document listed in the references. Researchers gather these together and place them into a reference list at the end of the report; in addition, they put in-text citations throughout their report that note the author and date, which lead readers back to the reference list for the full citation.
For a systematic review, many reference lists consist of over 50 citations! It can be challenging and confusing trying to organize and gather all these citations or references together by hand, but the solution is to use a citation management platform to automatically file, organize, and list references.
Citation Management Platforms (CMP) and Systematic Reviews
Citation management platforms are software programs that integrate with a variety of web browsers and databases. They are able to analyze and download bibliographic information to an account and retrieve a full reference list in a specific reference style. More importantly, they make it much easier to navigate and organize large amounts of citations. Most web or browser based platforms are also accessible from any computer with an account login. They are designed to be easy to use and fast.
For your systematic review, a CMP can reduce the time spent writing out citations, looking up citation style rules, alphabetizing the entries by author or organization, organizing citations by subject, and attempting to find that one specific article from a list of saved web pages from your browser. If you are working on your systematic review for over a year, chances are some of the documents or citations you find may undergo some changes. The CMP allows you to return and add, edit, or delete citations from the saved bibliographic list instead of attempting to find the error by hand.
In general, a large portion of systematic review writers use CMPs. The platform helps improve the quality of the paper through providing transparency about the quantity and quality of the citations used in the review. Through this transparent reference list, other researchers can attempt to reproduce the research by reading those citations. A major component of systematic research is peer-review. Without a reference list, researchers will not know where or how you got specific information, and they cannot attempt to confirm the quality of the systematic review.
Available Citation Management Platforms
At NAU, both Zotero and Mendeley are available for you to use. Zotero is a free to use, downloadable citation platform that includes a variety of citation format styles and uses browser extensions to capture citations. Use the Zotero Guidebelow to help you learn how to create an account, add or import citations, organize your work, and create bibliographies. This is the NAU Librarian recommended platform.
Setting up Zotero
To set up Zotero on your computer, you need to download and install the software on your computer. There are options available for Mac and Windows (and Linux). Another useful tip is to include the browser extension and Microsoft Word plugin to easily capture documents. You can transfer, organize, and cite documents within the software program.
For step by step help on setting up Zotero, click here to view the NAU Zotero Course.
Finding and collecting research is only a small part of the systematic review journey. Once research has all been clustered into a single place (like Zotero), titles and abstracts will then need to be screened for relevance.
Generally, screening occurs just after the search results have been collected. Access to full text is not necessary at this stage. All a team needs is the title and abstract for each search result. To make screening easier, the following steps are recommended:
Every team will need at least two screeners who screen every article independent of one another;
Screeners should simply indicate "Keep", "Discard", or "Unsure" for each article;
Once screening is completed, the screeners will come together with a tie breaker. In the event of a "tie", the tie breaker will decide whether a paper should be kept or discarded;
NOTE: there may be some papers which are not relevant to the review, but which might be valuable in the literature review section of the finished paper. These should be labelled as such, then kept and stored in a separate place from the articles selected for review.
REMEMBER: always record every step in the decision-making process.
Full text appraisal
Once the first screening stage is finished, the reviewers will then work together to review the full text of all selected papers. At this stage, there may be some papers which do not meet criteria for grading. Use the following questions (Hammelfarb Health Sciences Library 2020) to determine if the full text does not meet the criteria for further inclusion:
Does this study address a clearly focused question?
Did the study use valid methods to address this question?
Are the valid results of this study important?
Are these valid, important results applicable to my patient or population?
There are critical appraisal tools available online that can help you through this process. These include (but are not limited to):
Just like in the first step of the screening process, all decisions should be recorded.
Now that you've finished screening, it's time to grade the quality of the studies selected for the review. This is one of the most time consuming steps in the process and should be performed with rigor and attention to detail.
There are several resources available to help teams make it through grading. One of the most essential is the Cochrane GRADE Handbook. The GRADE Handbook can walk a team through every step of the process; however, it is important to keep in mind that this handbook is designed primarily for teams undertaking a Cochrane Systematic Review. (If you don't know what a Cochrane Systematic Review is, you're probably not taking one on, so don't worry about it too much!)
Aside from the GRADE Handbook, the following resources might also prove useful:
Once you've finished grading, you're set to write your paper! If you have any questions on any part of this section, feel free to reach out to your PBC Librarian at: firstname.lastname@example.org.
PBC Library Systematic Reviews Service
PBC Library launched a systematic reviews pilot service in Summer 2018. As part of this service, the NAU PBC librarian provides search coordination and review processes guidelines to teams working on different review types. For now, the service is freely available to all NAU faculty at PBC currently considering embarking on a review.