What is ‘risk of bias’ in a systematic review?

The last time you read a systematic literature review, you may have seen a graph that looked like traffic lights.

The “traffic lights” graph in a systematic review report shows the risk of bias for each study that was reviewed. Bias is a systematic error that can lead to the wrong outcomes and conclusions. The error can be a mistake in the design, conduct, or analysis of the study. In the graph, each row represents a study and each column represents one type of bias. The color represents the reviewer’s conclusion about the risk of each type of bias in each study. Red means a high risk of bias, yellow means an unclear risk of bias, and green means a low risk of bias.

For example, the researcher who created this example graph concluded that in the 2012 study (lead author Lane), there was a low risk of selection bias, reporting bias, and attrition bias (green), but a high risk of detection bias (red). The researcher could not arrive at a definite conclusion regarding the risk of performance bias, so the notation shows unclear risk (orange/yellow).

When you write a systematic review, this graph is simply a visual representation of your evaluation of the risk of each bias in each study you reviewed. Instead of writing all that information in many words in the text or in a table, this simple graph gives a quick picture of your conclusions using three colors. It is a useful tool for telling the reader your assessment of each study you reviewed.

Before you begin your systematic review, you should take four actions:

(1) Decide what types of bias you will evaluate in the studies that you include in the review.
(2) Decide how you will quantify the risk of bias.

(These two decisions are your procedure for assessing the risk of bias.)

(3) Write this procedure in your systematic review protocol.
(4) Write this procedure in the methods section of your systematic review report.

Taking these four actions accomplishes four goals:

(1) Creating a clear understanding of how you will evaluate each study. This is your map for assessing the risk of bias.
(2) Ensuring transparency (not hiding what you did or how you did it).
(3) Making your findings reproducible, so other researchers can repeat your analysis if they wish to check the validity of your conclusions.
(4) Writing the methods section of your systematic review report before you perform the review saves you time later. You can write the introduction section and the methods section before you perform the review, in parallel with writing your protocol.

The types of bias that are most commonly evaluated in a systematic review are:

(1) Selection bias

This can occur when the experimental groups differ in their baseline characteristics. Selection bias can be prevented by randomizing and blinding the process of allocating subjects to the experimental groups. Allocating subjects to the treatment group versus placebo group or to the new drug group versus standard care group should not be a choice of the investigator. The investigator can introduce bias by deciding to allocate subjects to groups using factors such as age or disease severity. This is wrong. The allocation must be randomized.

(2) Performance bias

This can occur when the experimental groups receive differing levels of care or when one group is exposed to factors that affect the outcomes but the other group is not. The objective of the study is to evaluate the efficacy (performance) of the intervention. However, knowing which intervention you are receiving may affect your outcome.

(3) Detection (measurement) bias

This can occur when the experimental groups are not treated equally in terms of how the outcomes are collected or how the information is verified. This can also happen when the measurements are affected by the characteristics of some participants. Detection bias can lead to overestimating or underestimating the size of the effect.

(4) Reporting bias

This can occur when the investigators report only some of the findings and not all the findings. Perhaps they report only the findings that are statistically different between the experimental groups. The investigators are distorting the results by not revealing all the findings.

(5) Attrition bias

This can occur when the numbers of subjects withdrawing from a study are not equal across the experimental groups. Withdrawals result in incomplete outcome data for the experimental group. When there are more outcome data in one group than in the other group, the findings may be unreliable and not robust enough for conclusions.

Source: The Cochrane Handbook.

⚠️ Caution: YOU can also be a source of bias!

You must be very careful not to introduce bias into your systematic review. When you plan your systematic review and write your protocol, ensure that you do not commit a systematic error in the review design or in the way you locate the studies, select the studies, or analyze the studies. The information here will help you to avoid biasing your systematic review.

Writing the results

In the results section of your report, give an overall assessment of the risk of bias in the studies you reviewed. This is a paragraph of text that complements your “traffic light” graph. This is also your opportunity to tell the reader about the overall quality of the original research studies in the field, and whether some of them have methodology flaws that make their conclusions unreliable.

In this manner, you may expose a weakness in the way that these studies were conducted. This may give you an understanding of the approach that is needed to answer the research question empirically and robustly. When you identify the gaps and limitations, and explain them in your systematic review report, you can recommend the next avenue of investigation in this research field.

Final thought

At this point, who is the expert who knows exactly what is missing from our knowledge and can design the perfect study to close the gaps in this field of research?

You! 


Dr Dean Meyer has a background in environmental science with a specialist interest in toxicology and public health. Her doctoral research work focused on molecular mechanisms of metal detoxification in an invertebrate model. Her other research interests include the mechanisms of toxicity and disease causation, and the occupational sources of xenobiotics and their physiological effects.

Dr Meyer spent 8 years working at the Centers for Disease Control and Prevention in Atlanta, and has an extensive background in the areas of laboratory safety and environmental health.

She is a certified Editor in the Life Sciences (ELS) and joined Edanz Group as an editor in 2015.


Edanz offers a range of self-study courses and premium services on understanding, designing, conducting, writing, and publishing systematic reviews.


Related Articles

Responses