Overview of Critical Reading
How to Critcally Review an Article
The ability to read critically is a skill that requires training and practice. Once acquired, it is a life-long habit that is automatic. With practice, one becomes discerning... perhaps even skeptical. Virtually, all studies and reports have flaws. This does not mean that they have no value. Carefully done studies make a contribution to the overall body of evidence regarding the determinants of disease. How one weighs the evidence depends upon these habits of critical review.
After completing this module, the student will be able to:
A scientific article is a publication that is based on empirical evidence. It can support a hypothesis with original research, describe existing research or comment on current trends in a specific field. Types of scientific research include:
Original Research
Methods Papers
Meta-analysis
Commentary
The title gives a brief, concise indicator of what the paper is be about
The authors are the ones who worked on the study and analysis that is presented in the paper. In many or even most cases, some of the authors may not have contributed to actually writing the paper but all contributed to some aspect of the research—study design/implementation, data analysis, background research, etc. The order of the authors is also important—the first author is the one who led this specific analysis, the last or last few authors are often the senior or principal investigators.
The abstract gives an abbreviated summary of the article. Many times it will be broken down into smaller mini-sections that reflect the larger sections contained in the paper.
The introduction gives context to the research—why this research is important and what other work has been done to explore this issue—as well as a statement of the hypothesis that the researchers are investigating.
This sections explains in detail how the authors went about investigating their hypothesis. This is generally where you will find a description of the study design used. In some cases that included analysis of well-known, existing studies such as the Framingham Heart Study or the Black Women's Health Study, the study design may be briefly summarized and a citation for a complete methods paper included in the end notes. This section also includes methods of analysis, selection criteria and how data was imputed or controlled.
This sections contains the results of the methodology section.
In this section, the authors analyze their results and explain the significance (or, rarely, the insignificance) of their findings. They also may explain how their findings fit into existing literature and anything that may have surprised them about their results.
This section relates how the findings of this paper relate to the background and hypothesis put forth in the Introduction. It also oftentimes includes a call for further research and investigation.
This is a place for the authors to thank the people, institutions and funders that contributed to their research.
This is essentially a bibliography where works referenced in the paper are listed. This is a great place to find further resources on the subject you're interested in, even if the specific paper you're looking at isn't particularly helpful.
Tables are a way for investigators to display data related to their research. Table 1 is usually a summary of the study population, with Table 2 communicating the principal finding of the study. Further tables often display stratified analysis.
Figures are a way for investigators to show their results in a visual way and can take a variety of forms including histograms, pie charts, scatter charts, etc.
I will scan the paper first to get an overview and to identify the potential problems I might encounter. I then go back and read the paper carefully and try to answer the questions that are listed below. During my initial scan I read the Abstract entirely to get an overview. I then scan the Introduction to identify the issues and the questions being addressed. I then jump to the last several paragraphs of the Discussion to see their conclusions and the implications of the study. Next, I go back to the Methods to see how they selected subjects, how they collected data, and how they analyzed it. When doing this I actively search for potential problems that could have caused errors as a result of bias, confounding, or inadequate evaluation of random error (chance). I then scan the Results section, and I carefully study all of the Figures and Tables. I then read the remainder of the Discussion to see how the authors discuss the potential limitations of the study.
I then reread the paper and consciously try to answer the following questions.
The key thing is to think in a structured way. An important first step is to identify the type of study, since this will frame how you think about it, and it will tip you off regarding potential pitfalls with respect to chance, bias and confounding.
What was the primary question that the authors were trying to answer? Why were they asking this? Rationale? What was their goal?
What type of study design was used?
How were subjects identified and enrolled? How successful was enrollment?
How carefully was the exposure of interest defined?
How carefully was the outcome of interest defined?
Could selection bias have affected the results?
What was the potential for information bias?
What were the likely confounding variables?
Would these problems bias toward the null or away from the nul?
Do the results suggest an association?
Was the interpretation appropriate?
Are the results of this study consistent with other studies in this area? If there are differences with other study findings, what could they be due to?
What are the public health implications of the study?
Use common sense. A number of years ago NIH sponsored a study to determine whether patients with acute spinal injuries benefited from administration of large doses of steroids shortly after injury. A very carefully designed randomized clinical trial was done with very careful assessment of changes in sensory and motor function. In fact, the investigators were able to show that there was a statistically significant improvement in those who had received steroids. However, the differences were clinically minute. We're talking about maybe a centimeter or two more sensation in a patient who is basically a quadriplegic. So from a functional point of view the differences between the groups weren't of any consequence at all. To make matters worse, the patients who received high doses of steroids had a higher risk of pneumonia, so the steroids may have done more harm than good.
Can the conclusions of the study be generalized to other populations besides the study population? In other words, are the results externally valid?
Over the past four decades there has been evidence to suggest that people who regularly eat fish have lower rates of heart disease, and many have postulated that this is due to the fact that fats/oils in some fish are enriched with long-chain, polyunsaturated n-3 (omega-3) fatty acids. Nevertheless, there is currently controversy about whether increased dietary intake of n-3 fatty acids by eating fish or taking fish oil capsules is beneficial.
To practice the critical review strategies presented earlier in this module, we have selected four articles that provide different view-points on the issue of fish oil and cardiovascular disease. Each article is accompanied by a set of questions. All articles are available via Pub Med.
The Question:
Fish oil supplements are solid with marketing that often suggests that fish oil consumption reduces the risk of cardiovascular disease.
Should you or your parents should be encouraged to eat more fish and/or take dietary supplements containing fish oil? Your goal is to make an evidence-based recommendation. For the purpose of this discussion, please disregard the potential adverse effects of mercury in pregnant woman.
Kromhout D, Bosschieter EB, et al.: The inversion relation between fish consumption and 20-year mortality from coronary heart disease. N. Engl. J. Med. 1985;312:1205-9. Link to PDF for this article: http://www.nejm.org/doi/pdf/10.1056/NEJM198505093121901
What type of study is this? Who were the participants?
What were the primary exposures and outcomes (endpoints) of interest? How were they ascertained? Did the investigators adequately define and precisely assess these? Is it likely that misclassification of the primary exposures and outcomes affected the results of the study?
How were the subjects selected into the study? Is it possible that selection bias affected the results of the study?
How did the investigators evaluate random error? Was the statistical analysis adequate?
How did the investigators deal with potential confounding? Was this adequate?
What does Table 1 show? How would you interpret these findings? Can a risk ratio be = 0? How would you interpret the confidence intervals in the last row of the table?
What does Table 4 show? How would you interpret these findings?
What is your overall assessment of the quality of this study? Do you believe the results are internally valid? Are the results externally valid? If so, to whom?
Albert CM, Campos H, et al.: Blood levels of long-chain n-3 fatty acids and the risk of sudden death. N. Engl. J. Med. 2002;346:1113-8. Link to PDF for this article: http://www.nejm.org/doi/pdf/10.1056/NEJMoa012918
What type of study is this? Who were the participants?
What were the primary exposures and outcomes (endpoints) of interest? How were they ascertained? Did the investigators adequately define and precisely assess these?
How were the subjects selected into the study? Is it possible that selection bias affected the results of the study?
How did the investigators evaluate random error? Was the statistical analysis adequate?
How did the investigators control for confounding?
What is the purpose of Table 1? What conclusions can you draw from this table?
On page 1115 the authors state, " ...significantly lower levels of long-chain n-
3 fatty acids were found among current smokers than among former smokers or those who had never smoked ...." Is this important? Why or why not?
On page 1115 the authors also state, "In addition, the base-line blood level of long-chain n-3 fatty acids was significantly correlated with fish intake at 12 months (R2=0.24, P=0.001). What can one conclude from this?
What does Table 2 show? What conclusions would you draw from this table?
What does Table 3 show? What conclusions would you draw from this table?
What is your overall assessment of the quality of this study? Do you believe the results are internally valid? Are the results externally valid? If so, to whom?
Kromhout D, Giltay EJ: n-3 fatty acids and cardiovascular events after myocardial infarction. N Engl. J. Med. 2010;363:2015-26. Link to PDF for this article: http://www.nejm.org/doi/pdf/10.1056/NEJMoa1003603
What type of study was this?
Who comprised the study population? What were the inclusion and exclusion criteria for participation in the study?
What groups were compared?
Were the methods adequately described?
Is it likely that random error had a major effect on the conclusions drawn by the authors?
How did the investigators assess adherence to the protocol? Is it likely that misclassification of exposure affected the results?
Is it likely that selection bias affected the results?
Is it likely that misclassification of outcome affected the results?
What are your conclusions based on Table 1?
The authors state, " The two groups that received EPA-DHA were combined and compared with the two groups that did not receive EPA-DHA. Similarly, the two groups that received ALA were combined and compared with the two groups that did not receive ALA." Why did they do this?
What do the results in Table 2 indicate?
What do the findings in Table 3 suggest?
What is your overall assessment of the quality of this study? Do you believe the results are internally valid? Are the results externally valid? If so, to whom?
The Risk and Prevention Study Collaborative Group: n−3 fatty acids in patients with multiple cardiovascular risk factors. N Engl. J. Med. 2013;368:1800-8. Link to PDF for this article: http://www.nejm.org/doi/pdf/10.1056/NEJMoa1205409
(Note: In the study the authors computed a "hazard ratio," which is something that we haven't talked about in EP713. For our purposes in this exercise, you can regard the hazard ratio as a rate or risk ratio that compares risk over time.)
What type of study was this?
What were the inclusion and exclusion criteria for participation in the study?
What groups were being compared?
Were the methods adequately described?
Is it likely that random error had a major effect on the conclusions drawn by the authors?
What is the purpose of Table 1? What conclusions can you draw from Table 1?
In the section entitled " CARDIOVASCULAR RISK FACTORS AND MEDICATIONS" the authors note that, " By the end of the trial, the overall cardiovascular risk profile had improved in both groups...." Exactly what was the improvement? Why is it that the "placebo" group improved? What is the significance of this in terms of the validity of the study? In the same section the authors stated (page 1803), " Prescriptions of recommended cardiovascular drugs increased during follow-up to a similar degree in the two study groups." What is the potential importance of this, if any?
Is it likely that bias had a significant impact on the conclusions? If so, which type of bias?
Is it likely that confounding had a significant impact on the conclusions?
The authors indicate that there was no evidence of interaction of treatment with n−3 fatty acids in relation to aspirin use and statin use at baseline showed no evidence of interaction with these drugs (P = 0.34 and P = 0.28 for interaction, respectively).
What does Figure 2 show? Pay particular attention to gender effects and effects in those who had > cardiovascular risk factors vs. those who had fewer.
What were the rates of non-compliance? How did the authors deal with non-compliance? What effect would this have on the results/outcome?
What is your overall assessment of the quality of this study? Do you believe the results are internally valid? Are the results externally valid? If so, to whom?