PBC Library

What is the difference between systematic and literature reviews?

As stated previously, systematic reviews are focused on a unique research question and include a protocol as well as a review team. Literature reviews, on the other hand, are an overview of the topic gathered by an individual or smaller search team. Literature reviews state what articles have been published, who key authors or researchers are, what specific or broad questions are being asked, what methods researchers use, and what theories and hypotheses are currently being researched. They seek to build a core of research instead of delving into one specific question.

The following table compares the two types of reviews:

Literature reviews are meant to review a variety of articles, case studies, reports, and other research information, not answer or ask research questions like the systematic reviews. When published, they provide a useful and necessary frame of reference in the scientific community by establishing key discussions or information about the topic. Additional resources and examples of systematic reviews and literature reviews are below.

Further reading

For more information on Systematic Reviews:

Centers for Disease Control and Prevention Systematic Reviews Site: https://www.cdc.gov/library/researchguides/systematicreviews.html

Duke Biomedical Library Systematic Review Guide: https://guides.mclibrary.duke.edu/sysreview/home

Pollock, A., & Berge, E. (2018). How to do a systematic review. International Journal of Stroke, 13(2), 138–156. https://doi.org/10.1177/1747493017743796

Links to Systematic Reviews:

Cullis, P. S., Gudlaugsdottir, K., & Andrews, J. (2017). A systematic review of the quality of conduct and reporting of systematic reviews and meta-analyses in paediatric surgery. PLoS ONE, 12(4). https://doi.org/10.1371/journal.pone.0175213: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5383307/

Fadlallah, R., El-Jardali, F., Nomier, M., Hemadi, N., Arif, K., Langlois, E. V., & Akl, E. A. (2019). Using narratives to impact health policy-making: A systematic review. Health Research Policy and Systems, 17. https://doi.org/10.1186/s12961-019-0423-4: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6402129/

Pot, M. W., van Kuppevelt, T. H., Gonzales, V. K., Buma, P., IntHout, J., de Vries, R. B. M., & Daamen, W. F. (2017). Augmented cartilage regeneration by implantation of cellular versus acellular implants after bone marrow stimulation: A systematic review and meta-analysis of animal studies. PeerJ, 5. https://doi.org/10.7717/peerj.3927: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5661456/

Thompson, W., Tonkin-Crine, S., Pavitt, S. H., McEachan, R. R. C., Douglas, G. V. A., Aggarwal, V. R., & Sandoe, J. A. T. (2019). Factors associated with antibiotic prescribing for adults with acute conditions: An umbrella review across primary care and a systematic review focusing on primary dental care. Journal of Antimicrobial Chemotherapy, 74(8), 2139–2152. https://doi.org/10.1093/jac/dkz152: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6640312/

For more information on Literature Reviews:

            Florida Gulf Coast University Library, Literature Review (Healthcare): https://fgcu.libguides.com/c.php?g=65107&p=419881

            University of California, Santa Cruz, Literature Review: https://guides.library.ucsc.edu/write-a-literature-review

            University of Toronto, Literature Review Tips: http://advice.writing.utoronto.ca/types-of-writing/literature-review/

University of Wisconsin – Madison Writing Center, Literature Review Basics: https://writing.wisc.edu/handbook/assignments/reviewofliterature/

Links to Literature Reviews:

Alturkistani, H. A., Tashkandi, F. M., & Mohammedsaleh, Z. M. (2016). Histological stains: A literature review and case study. Global Journal of Health Science, 8(3), 72–79. https://doi.org/10.5539/gjhs.v8n3p72: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4804027/

Garcia-Schinzari, N. R., & Santos, F. S. (2014). Assistance to children in palliative care in the Brazilian scientific literature. Revista Paulista de Pediatria, 32(1), 99–106. https://doi.org/10.1590/S0103-05822014000100016: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182999/

Lomoro, P., Verde, F., Zerboni, F., Simonetti, I., Borghi, C., Fachinetti, C., Natalizi, A., & Martegani, A. (2020). COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: Single-center study and comprehensive radiologic literature review. European Journal of Radiology Open, 7, 100231. https://doi.org/10.1016/j.ejro.2020.10023: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7129441/

Scoping reviews:

Under the systematic and literature review umbrella, there are a range of other review types. These exist because different situations will often call for different review approaches. Sometimes, you will find a wealth of randomized controlled trials on a single subject, which will allow you the chance to pursue a systematic review (or even a meta-analysis). Other times, a research field may be in the early stages of emergence. In these cases, a scoping review might be preferable.

Scoping reviews have grown in popularity in recent years due to their focus on a more agile approach to reviewing evidence. Scoping reviews may also rely on less "quality" information than a standard systematic review, and can go beyond the analysis of clinical trials. Also, "unlike a systematic review, scoping reviews do not tend to produce and report results that have been synthesized from multiple evidence sources following a formal process of methodological appraisal to determine the quality of the evidence. Rather, scoping reviews aim to provide an overview or map of the evidence." 

For more information about scoping reviews and how to conduct them, check out the following resources:

• JBI Manual for Evidence Synthesis, CH. 11.1: https://wiki.joannabriggs.org/pages/viewpage.action?pageId=3178748
• PRISMA for Scoping Reviews: http://www.prisma-statement.org/Extensions/ScopingReviews
• Cochrane Training, Scoping Reviews: https://training.cochrane.org/resource/scoping-reviews-what-they-are-and-how-you-can-do-them

***In addition to scoping reviews, there are a wealth of review types available under the systematic review umbrella. Check out the following resources to learn about common and emerging review types, and when to leverage them:

• Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91-108. doi:10.1111/j.1471-1842.2009.00848.x
• Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Medical Research Methodology. 2018;18:5. doi:10.1186/s12874-017-0468-4.

So how does a librarian help you find appropriate literature?

Librarians will oversee the literature search protocol in a collaborative relationship. In order to begin the search process, the librarian needs a detailed protocol establishing the research question, criteria (inclusion or exclusion), 2-4 examples of articles fitting the criteria, and preliminary search results. In return, they may ask clarifying questions or work with you to determine critical points or terms necessary in your research before developing a search method.

The librarian will evaluate your specific research question and develop a search method and subsequent search queries based on your research requirements. This means librarians can:

• Find previous systematic reviews to prevent research duplication.
• Develop and document broad search strategy to return precise and relevant information.
• Find appropriate databases relevant to specific research question.
• Identify key terms existing in healthcare databases through Medical Subject Headings (MeSH) or other appropriate controlled vocabulary.
• Perform search queries in order to find detailed articles making up a comprehensive picture of current research of the question.
• Gather total relevant results of search queries.

Strategic Searching and Tools

Understanding the method of constructing a search query may help you develop a search protocol or initiate a preliminary search before working with the librarian. When a librarian approaches a search, they first must understand the goal or specific research question in order to develop a list of subjects and terms. Librarians and expert searchers use the following tools and methods to create intricate and relevant search queries.

1. Databases

   • ​​A database is a collection of organized data contained within a storage system, most commonly a software program. Each database may house different subjects or types of information. It is important to choose the most applicable databases to the research question. 

   • For anything related to the health sciences, PubMed and PubMed Central are quality databases with rigorous articles. Other important health science databases include but are not limited to: Cochrane Library, MEDLINE Plus, and ScienceDirect Journals.

2. Controlled Vocabularies

   • A controlled vocabulary is a defined set of structured words or phrases useful for classifying/labeling and retrieving content in a search system. The controlled vocabulary will include a hierarchy (a structure that organizes terms from broad to narrow), subject headings, preferred or variant terms, and descriptions of terms. By adhering to a controlled vocabulary, all researchers will use the same terms, making retrieving content faster and more accurate. ​​

   • In a healthcare or medical setting, all researchers need to use exact phrasing or key words in search queries. One helpful tool is the Medical Subject Headings (or MeSH) controlled vocabulary from the National Library of Medicine (NLM). This controlled vocabulary includes approved key words and descriptions representing subjects in the life sciences. Before beginning any search query, it is important to find and use the appropriate MeSH term so the research can be included in a specific, appropriate results page.

   • For example, when researching post traumatic stress disorder, several researchers may label their content with terms such as “post traumatic stress disorder,” “PTSD,” or “stress disorders.” Instead, the approved term according to the MeSH controlled vocabulary is “Stress Disorders, Post-Traumatic.” If the researchers did not change their terms to this controlled term, their results would not be included in a search query.

   • Below is a video explaining how to search PubMed using MeSH terms:

3. Operators

   • ​​A search operator is a special input command that can retrieve variations or combinations of key words, either broadening or limiting a search query. When used together, they can communicate with the data in a database to return appropriate results.

     • Quotation Marks: Used to indicate an exact term or phrase.

     • Parentheses: Used to indicate priority of specific term or phrase.

     • AND: Used as a conjunction to combine two terms.

     • OR: Used as a conjunction to indicate two separate terms.

     • NOT: Used as a conjunction to include one term and exclude other associated terms.

     • * (star): Used as a wildcard to return any term or phrase.

   • For example, the following string represents the combination of two terms (using the MeSH controlled vocabulary) and would return any documents that included both terms, not one or the other:

     • “Brain Injuries, Traumatic” AND “Behavioral Symptoms”

   • Similarly, the next string would represent a more broad result related to several brain injury topics:

     • “Brain Injuries, Traumatic” OR “Brain Injuries, Chronic”

4. Syntax

   • ​​The syntax is the way the terms, punctuation, and the order are listed in a search query. Researchers enter text (data) in precise strings, not sentences, to communicate with the data system. Entering sentences such as "What is the link between heart attacks or stroke and acid reducer medicine?" will not work within a database or search engine. Instead, the researcher isolates key terms "Myocardial Infarction," "stroke," and "omeprazole" and decides what order to search in. Most terms can be interchangeable, but the punctuation and operators must be in the correct positions. 
   • For example, "Myocardial Infarction" "stroke" "omeprazole" doesn't automatically explain we are seeking information on either heart attacks or strokes as well as adding a drug into the mix. A better syntax for this question might be: ("Myocardial Infarction" OR "stroke") and "omeprazole" . This syntax string appropriately uses punctuation and operators to phrase the question in a way to retrieve appropriate information.

5. Filters

   • ​​A filter in a database is a preset option to customize the retrieved documents in a search query. Filters are shown on the search page usually as check boxes or clickable attributes that include documents indexed with those specific terms. When researching, it is important to be aware of filters to remove unnecessary documents from a search query.

   • PubMed Central includes several useful filters such as Publication Date, Open Access, Include Embargoed Articles, and Research Funder. Other databases may include Topics/Subjects, Publication Type, Format, Language, Animal Subjects, and more.

Precision and Recall Information

In any search, a researcher needs to maximize the best-fitting, or relevant, information to their topic in order to successfully answer or attempt to answer their research question. By using the components listed in the Search Strategies and Tools above, a researcher can more easily and quickly find the appropriate information. The two values that determine relevancy are Precision and Recall.

Precision (Specificity)

Precision means the success of returning of specific, useful information in a search of the entire database. It is measured by taking the number of relevant information in a search divided by the total number of identified information in that search. High precision is when a researcher collects only the useful information out of a large number of useful and not useful documents within a database. Low precision is when a researcher finds that both useful and not useful documents have been returned in a search. This may be caused by including too broad or narrow of key terms.

For example, a search query returns 1,200 articles including correct/useful key words to the question out of 4,800 total articles available on the database. That means the specificity of that search is 48% (1,200 divided by 4,800). To increase this number, the researcher may review new key words or filters to find more precise information.

Recall (Sensitivity)

Recall means establishing the most useful documents out of the already gathered, useful documents. It is measured by taking the amount of relevant information returned in a search divided by the total amount of relevant information. High recall is when a researcher finds a large amount of useful documents out of the already gathered useful documents. Low recall is when a researcher finds only a portion of the useful documents are applicable to the study or not useful documents are included in the search results.

For example, a search query returns 150 relevant documents out of an estimated 250 total relevant documents. That means the sensitivity of those documents is 60% (150 divided by 250).

Both Precision and Recall help researchers to measure the relevance of the information to their topic. It is unrealistic to achieve 100% Precision and Recall in any search query or strategy, but researchers should attempt to retrieve the most relevant documents or information for their research.

What is gray literature?

Gray literature is any documents or literature produced but not published commercially. Academic, government, business, health, and research organizations or groups still produce materials to circulate information to their shareholders and users. Some examples of gray literature include conference pamphlets, theses and dissertations, committee research and reports, annual subject reports, peer-reviewed academic articles and literature, ongoing research, patents, and more.

How can it help my systematic review?

Despite the challenge of finding the documents, gray literature is worth including in your systematic review. It can be very current and comprehensive and provide an overall description of evidence while reducing publication bias (Paez, 2017). Gray literature includes reports with negative or null results or reports that may have been rejected by commercial publications. Since the timeline for publication extends from a few months to over a year, some researchers may want to share the reports and continue research, meaning that their results are more quickly posted. In other situations, researchers may feel that their data isn’t rigorous enough to be published to a journal but still share their findings. These produced documents provide recent, thorough data and reduce the reliance on commercial publication.

Where can I find gray literature?

There is not a central repository for all organizations to share their literature so it can be challenging to know where to look for these materials. Organizations also have a wide variety of audiences and formats and may choose to self-publish or to publish in specific databases or websites. However, there are several useful databases for health sciences

To begin searching for gray literature, visit the following sites:

• ClinicalTrials.gov (U.S. National Library of Medicine):
  • Over 340,000 studies for documents produced in the U.S. and 215 other countries. Includes recruiting, completed, unknown, withdrawn, and terminated reports.
• PubMed Central (U.S. National Library of Medicine):
  • Over 5.5 million full text, open access documents from across the U.S. Includes a user guide, FAQ, and advanced searching.
• Google Books:
  • Over 25 million full text books from across the globe. Books out of copyright may be downloaded as PDF. Includes books from the Library Project.
• Retractionwatch.com (Funded by User Contributions):
  • Scientific community driven blog dedicated to updating information about retracted documents.
• Open Grey (French Institute of Scientific and Technical Information/National Center for Scientific Research – INIST-CNRS):
  • Over 700,000 references for science, technology, biomedical science, economics, and social science and humanities documents produced in Europe. Allows locating and exporting documents.

Further Reading

For more information on Gray Literature:

Duke Biomedical Library Systematic Review Guide (Gray Literature tab): https://guides.mclibrary.duke.edu/sysreview/greylit

Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence-Based Medicine, 10(3), 233–240. https://doi.org/10.1111/jebm.12266. https://onlinelibrary.wiley.com/doi/full/10.1111/jebm.12266

What is Citation Management?

Citation management, also known as reference management, is the process of gathering and organizing bibliographic information for a report or review. Bibliographic information is any unique identifying information about the document or resource, including author name, title, date, publishing company or journal, page numbers, etc. Each set of bibliographic information is called a reference or citation. The goal is to provide as much distinguishing information as possible in a citation so readers can find that exact document listed in the references. Researchers gather these together and place them into a reference list at the end of the report; in addition, they put in-text citations throughout their report that note the author and date, which lead readers back to the reference list for the full citation.

For a systematic review, many reference lists consist of over 50 citations! It can be challenging and confusing trying to organize and gather all these citations or references together by hand, but the solution is to use a citation management platform to automatically file, organize, and list references.

Citation Management Platforms (CMP) and Systematic Reviews

Citation management platforms are software programs that integrate with a variety of web browsers and databases. They are able to analyze and download bibliographic information to an account and retrieve a full reference list in a specific reference style. More importantly, they make it much easier to navigate and organize large amounts of citations. Most web or browser based platforms are also accessible from any computer with an account login. They are designed to be easy to use and fast.

For your systematic review, a CMP can reduce the time spent writing out citations, looking up citation style rules, alphabetizing the entries by author or organization, organizing citations by subject, and attempting to find that one specific article from a list of saved web pages from your browser. If you are working on your systematic review for over a year, chances are some of the documents or citations you find may undergo some changes. The CMP allows you to return and add, edit, or delete citations from the saved bibliographic list instead of attempting to find the error by hand.

In general, a large portion of systematic review writers use CMPs. The platform helps improve the quality of the paper through providing transparency about the quantity and quality of the citations used in the review. Through this transparent reference list, other researchers can attempt to reproduce the research by reading those citations. A major component of systematic research is peer-review. Without a reference list, researchers will not know where or how you got specific information, and they cannot attempt to confirm the quality of the systematic review.

Available Citation Management Platforms

At NAU, both Zotero and Mendeley are available for you to use. Zotero is a free to use, downloadable citation platform that includes a variety of citation format styles and uses browser extensions to capture citations. Use the Zotero Guide below to help you learn how to create an account, add or import citations, organize your work, and create bibliographies. This is the NAU Librarian recommended platform.

Setting up Zotero

To set up Zotero on your computer, you need to download and install the software on your computer. There are options available for Mac and Windows (and Linux). Another useful tip is to include the browser extension and Microsoft Word plugin to easily capture documents. You can transfer, organize, and cite documents within the software program. 

For step by step help on setting up Zotero, click here to view the NAU Zotero Course.

Screening, analyzing, and grading evidence

Finding and collecting research is only a small part of the systematic review journey. Once research has all been clustered into a single place (like Zotero), titles and abstracts will then need to be screened for relevance.

Screening

Generally, screening occurs just after the search results have been collected. Access to full text is not necessary at this stage. All a team needs is the title and abstract for each search result. To make screening easier, the following steps are recommended:

1. Every team will need at least two screeners who screen every article independent of one another;
2. Screeners should simply indicate "Keep", "Discard", or "Unsure" for each article;
3. Once screening is completed, the screeners will come together with a tie breaker. In the event of a "tie", the tie breaker will decide whether a paper should be kept or discarded;
   1. NOTE: there may be some papers which are not relevant to the review, but which might be valuable in the literature review section of the finished paper. These should be labelled as such, then kept and stored in a separate place from the articles selected for review.
4. REMEMBER: always record every step in the decision-making process. 

Full text appraisal

Once the first screening stage is finished, the reviewers will then work together to review the full text of all selected papers. At this stage, there may be some papers which do not meet criteria for grading. Use the following questions (Hammelfarb Health Sciences Library 2020) to determine if the full text does not meet the criteria for further inclusion:

• Does this study address a clearly focused question?
• Did the study use valid methods to address this question?
• Are the valid results of this study important?
• Are these valid, important results applicable to my patient or population?

There are critical appraisal tools available online that can help you through this process. These include (but are not limited to):

• CEBM critical appraisal tools: https://www.cebm.net/2014/06/critical-appraisal/
• CASP critical appraisal tools and checklists: https://casp-uk.net/casp-tools-checklists/

Just like in the first step of the screening process, all decisions should be recorded.

Grading

Now that you've finished screening, it's time to grade the quality of the studies selected for the review. This is one of the most time consuming steps in the process and should be performed with rigor and attention to detail.

There are several resources available to help teams make it through grading. One of the most essential is the Cochrane GRADE Handbook. The GRADE Handbook can walk a team through every step of the process; however, it is important to keep in mind that this handbook is designed primarily for teams undertaking a Cochrane Systematic Review. (If you don't know what a Cochrane Systematic Review is, you're probably not taking one on, so don't worry about it too much!)

Aside from the GRADE Handbook, the following resources might also prove useful:

• The GRADE Working Group: https://www.gradeworkinggroup.org/
• AGREE Appraisal Tool: https://www.agreetrust.org/
• The JADAD Scale for Reporting RCTs: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470988343.app1
• The PEDRO Scale for Rehabilitative Health Research: https://www.pedro.org.au/english/downloads/pedro-scale/

Once you've finished grading, you're set to write your paper! If you have any questions on any part of this section, feel free to reach out to your PBC Librarian at: catherine.lockmiller@nau.edu.

PBC Library Systematic Reviews Service

PBC Library launched a systematic reviews pilot service in Summer 2018. As part of this service, the NAU PBC librarian provides search coordination and review processes guidelines to teams working on different review types. For now, the service is freely available to all NAU faculty at PBC currently considering embarking on a review. 

For anybody who wants to learn more, the librarian requests at least one face-to-face or video-based consultation to talk about the review journey. It is also required that all teams complete the PBC Library Systematic Reviews Initial Steps Planner. 

Aside from the Initial Steps Planner, the following resources are freely provided to individuals at NAU interested in beginning their systematic review journey:

• PBC Library workbook for recording search results and screening data
• PBC Library handout for analyzing full texts

If you have any questions about this service, feel free to call, email, or schedule an appointment to learn more!