Initial Scan
I will scan the paper first to get an overview and to identify the potential problems I might encounter. I then go back and read the paper carefully and try to answer the questions that are listed below. During my initial scan I read the Abstract entirely to get an overview. I then scan the Introduction to identify the issues and the questions being addressed. I then jump to the last several paragraphs of the Discussion to see their conclusions and the implications of the study. Next, I go back to the Methods to see how they selected subjects, how they collected data, and how they analyzed it. When doing this I actively search for potential problems that could have caused errors as a result of bias, confounding, or inadequate evaluation of random error (chance). I then scan the Results section, and I carefully study all of the Figures and Tables. I then read the remainder of the Discussion to see how the authors discuss the potential limitations of the study.
I then reread the paper and consciously try to answer the following questions.
Detailed Reading and General Questions to Consider
The key thing is to think in a structured way. An important first step is to identify the type of study, since this will frame how you think about it, and it will tip you off regarding potential pitfalls with respect to chance, bias and confounding.
Introduction
What was the primary question that the authors were trying to answer? Why were they asking this? Rationale? What was their goal?
Methods
What type of study design was used?
- Was this a logical choice, given the goals of the study?
- What are the weaknesses of this study design?
- What problems and biases might have occurred?
How were subjects identified and enrolled? How successful was enrollment?
- Could selection bias have occurred as a result of control selection bias, or differential non-participation in a case-control study?
- Did selection of controls meet the "would" criterion?
- If it was a cohort study, how complete was follow up
How carefully was the exposure of interest defined?
- How was the exposure assessed?
- What was the quality of the exposure data? ]
- Was exposure data validated?
How carefully was the outcome of interest defined?
- How was it assessed? Was it validated?
Could selection bias have affected the results?
What was the potential for information bias?
- Non-differential misclassification? Errors in recording or coding of data? General inability of subjects to remember?
- Differential misclassification? Recall bias? Interviewer bias? Recorder bias? Differential quality of data?
What were the likely confounding variables?
- Did the authors control for confounding in the design of the study, in the analysis, or both?
- Did they fail to account for any potentially important confounders? Was control of confounding adequate? Could there have been residual confounding?
- Did they perform stratified analysis? Did they use regression analysis?
Would these problems bias toward the null or away from the nul?
Results
Do the results suggest an association?
- If so, how was it assessed, and how strong was the association?
- Did the authors estimate risk ratios or risk differences?
- How precise were the measures of association? Was the sample size adequate? Did the authors report confidence intervals? p-values?
- Did the authors adequately assess random error?
Discussion
Was the interpretation appropriate?
Are the results of this study consistent with other studies in this area? If there are differences with other study findings, what could they be due to?
Conclusion
What are the public health implications of the study?