Internal and External Validity

Instrumentation

Observed changes seen between observation points (ie. Pre-test and post-test) may be due to changes in the testing procedure. This could include changes to the content or the mode of administration and data collection.

Tips include:

• Keep an eye out for this if there are multiple observation/test points in your study
• Go for consistency. Instrumentation threats can be reduced or eliminated by making every effort to maintain consistency at each observation point. This includes the instrument (questionnaire, type of testing kit, etc.), the administrators, and the method of administration (paper, telephone, etc.)

---

Regression

The tendency of extreme pre-test scores to revert back toward the population mean, such that when individuals are selected for program participation based on extreme pretest results their posttest scores will tend to shift toward the mean score, regardless of the efficacy of the program.

Tips include:

• Avoid selecting participants based on extreme performance or scores.

---

Maturation

This is a threat that is internal to the individual participant. It is the possibility that mental or physical changes occur within the participants themselves that could account for the evaluation results. In general, the longer the time from the beginning to the end of a program the greater the maturation threat.

Tips include:

• If feasible within your evaluation questions, reducing the amount of time between the pretest and posttest can limit maturation threats.
• Should be particularly mindful of this threat when working with children, as they are going through a great deal of mental and physical changes.

---

Testing

The administration of a pretest prior to the program may convey knowledge to the participants. This particular threat can either overstate or understate your program effect.

Tips include:

• Keep an eye out for this threat whenever there is a pretest-posttest design and no comparison group to help control for the learning curve of taking the pretest.

---

History

Observed program results may be explained by events or experiences (external) that impact the individual between program participation and follow up.

Tips include:

• Evaluators should do their best to identify any external events or changes that may impact their program results (media coverage, policies, major events, etc.)
• As with maturation threats, history threats can be reduced by limiting follow up time.

---

Selection

Whenever you have a nonequivalent comparison group and an exposure group, the difference seen in their posttest scores could be due to pre-existing differences between the groups rather than the impact of the program itself. This is of particular concern when the exposure group and the comparison group are significantly different from one another in characteristics.

Tips include:

• Be alert for this potential threat if you are working with a nonequivalent comparison group.

Interactions with Selection Threats

• Selection - History: Selection differences between the participants in the intervention and the comparison groups lead to differences in exposure or impact of historical events
• Selection - Maturation: Selection differences between the participants in the intervention and the comparison groups lead to differences in maturation effects
• Selection - Instrumentation: Selection differences between the participants in the intervention and the comparison groups lead to differences in instrument scores

Internal Validity Scenarios

Below are examples of health program evaluations, each highlighting a specific threat to internal validity. For each scenario, determine the most pressing threat to internal validity. Once you have reviewed all scenarios, select Show Answers to review the correct responses.

Scenario 1: A middle school has a new afterschool program for eighth graders targeted at increasing media literacy surrounding alcohol. The goal is to make them "savvy" consumers of advertising, and reduce the impact of such ads on alcohol consumption. Unfortunately, the program can only support 20 students. The evaluator decides to administer a pretest to all 200 eighth graders in the school, and take the 10% with the lowest test scores. Instrumentation, regression, testing, maturation, or history threat?

Scenario 2: The evaluators administer the pre-test for an evaluation as a pen and paper survey, and then for the post-test decide to adapt the survey to an online version.

Scenario 3: The Heart Healthy program is a one-day seminar targeted at education surrounding healthy food choices and cooking skills to reduce risk of major cardiac events. A pretest is given to see how knowledgeable participants are regarding heart healthy foods and how best to prepare them. The day after the seminar a posttest is circulated to discern program impact.

Scenario 4: A new family planning consultation is implemented at a local community health clinic, which experiences a high percentage of women reporting unwanted pregnancies. Peer educators are used to discuss family planning and contraceptive use with women using the clinic. The evaluators follow up with the participants one year later to record contraceptive use and whether any unwanted pregnancies occurred. There is no comparison group, so evaluators are looking only at the group that received the counseling. They find that since receiving counseling the rate of birth control usage has increased significantly. However, the evaluators are aware that last year a regulation was put in place providing free birth control to women regardless of insurance status.

Scenario 5: A program targeted at promoting physical fitness and healthy eating among pre-teens was conducted in 2010 in one public middle school in Charlestown. It is now 2015 and the team is interested in following up to see the current activity level and BMI of the participants and how it varies from their pre-program scores, compared to a public middle school located in the South End.

Show Answers