Assessing Bias in a Study
When listening to a presentation or reading an article in which data is presented to support a conclusion, one must always consider alternative explanations that may threaten the validity of the conclusions. Specifically, one needs to consider whether random error, bias or confounding could have undermined the conclusions to a significant extent. Virtually all studies have potential flaws, but carefully done studies are designed and conducted in a way that minimizes these problems so that they don't have any important effect on the conclusions. However, in other studies that are conducted in difficult circumstances (e.g., a prospective cohort study in a homeless population in which one would expect difficulty maintaining follow-up) or in poorly designed studies, biases may have a major impact and produce large overestimates or underestimates of the true association. In view of this, it is always important to ask oneself:
- Given the conditions of the study, could bias have occurred?
- Is it likely that bias actually was present?
- If there were bias, would it bias the results toward the null or away from the null?
- Is the magnitude of distortion likely to be small and inconsequential or large?
Direction of Bias
- You know that non-differential misclassification will always bias towards the null.
- The effects (i.e., either overestimation or underestimation) of differential misclassification and selection bias will depend on the details, but you can figure this out by just making a simple model of the true relationship and examining the effects of changing the numbers as if there were either selction bias or differential misclassification.
The Obstacle Course
Developing skill in identifying bias and predicting its potential impact on an association requires practice and experience. Here is a series of questions on bias from old exams that will give you some practice and sharpen your understanding of bias.
The Doll and Hill Case-Control Study
Read the summary in the iframe below (a summary of the classic case-control study by Sir Richard Doll and Bradford Hill in 1948), and then answer the questions beneath the frame.
The next iFrame is a PDF summary of a study by Joann Manson and others looking at the effect of ecercise on coronary heart disease in women. Read the summary and then answer the questions beneath the iFrame.
Manson et al.: The Effect of Exercise on Coronary Heart Disease in Women