Assessing Bias in a Study

When listening to a presentation or reading an article in which data is presented to support a conclusion, one must always consider alternative explanations that may threaten the validity of the conclusions. Specifically, one needs to consider whether random error, bias or confounding could have undermined the conclusions to a significant extent. Virtually all studies have potential flaws, but carefully done studies are designed and conducted in a way that minimizes these problems so that they don't have any important effect on the conclusions. However, in other studies that are conducted in difficult circumstances (e.g., a prospective cohort study in a homeless population in which one would expect difficulty maintaining follow-up) or in poorly designed studies, biases may have a major impact and produce large overestimates or underestimates of the true association. In view of this, it is always important to ask oneself:

  1. Given the conditions of the study, could bias have occurred?
  2. Is it likely that bias actually was present?
  3. If there were bias, would it bias the results toward the null or away from the null?
  4. Is the magnitude of distortion likely to be small and inconsequential or large?

Direction of Bias

  1. You know that non-differential misclassification will always bias towards the null.
  2. The effects (i.e., either overestimation or underestimation) of differential misclassification and selection bias will depend on the details, but you can figure this out by just making a simple model of the true relationship and examining the effects of changing the numbers as if there were either selction bias or differential misclassification.

The Obstacle Course

Man_Jumping_a_Hurdle.jpg Developing skill in identifying bias and predicting its potential impact on an association requires practice and experience. Here is a series of questions on bias from old exams that will give you some practice and sharpen your understanding of bias.

 

 

 

 

The Doll and Hill Case-Control Study

Read the summary in the iframe below (a summary of the classic case-control study by Sir Richard Doll and Bradford Hill in 1948), and then answer the questions beneath the frame.

 Thinking man icon. There are 4 questions related to the Doll and Hill study.

 Toggle open/close quiz question

Based the brief description of the Doll and Hill study above, it likely that selection bias influenced the study results.

 
 

 Toggle open/close quiz question

Based on the brief summary above differential misclassification of exposure status (smoking) could have biased the estimated magnitude of association.

 
 

 Toggle open/close quiz question

The summary shows that the association between smoking and lung cancer was stronger in men than in women. It is noteworthy that smoking in women was generally not socially acceptable in the 1940s.

Is the following statement true or false? Non-differential misclassification of exposure (smoking) could have biased the estimate of the magnitude of association in women.

 
 

 Toggle open/close quiz question

Is the following statement true or false? It is likely that misclassification of outcome biased the results of the Doll and Hill case-control study described above.

 
 

 

The next iFrame is a PDF summary of a study by Joann Manson and others looking at the effect of ecercise on coronary heart disease in women. Read the summary and then answer the questions beneath the iFrame.

Manson et al.: The Effect of Exercise on Coronary Heart Disease in Women

Thinking man icon. There are two questions related to the study by Manson et al.

 Toggle open/close quiz question

The purpose of the study by Manson et al. was to examine the association between physical activity and heart disease. Read the excerpts from the journal article by Manson et al., and then determine whether the following statement is true or false.

Misclassification of outcome probably introduced substantial bias.

 
 

 Toggle open/close quiz question

Is the following statement regarding the study by Manson et al. true or false?

It is likely that selection bias undermined the validity of the study's conclusions.

 
 

 


return to top | previous page