Describe the Program
In order to develop your evaluation questions and determine the research design, it will be critical first to clearly define and describe the program.
Both steps, Describe the Program and Engage Stakeholders, can take place interchangeably or simultaneously. Successful completion of both of these steps prior to the evaluation planning creates a strong foundation for the evaluation.
Consider first, the key features that should be included in a complete program description.
Key Program Description Features
- Specify the program's theory of cause and effect
- Describe the program's theory of implementation
- Specify the program objectives
- Specify the program components (inputs, outputs, and any other relevant resources/features)
A detailed, holistic program description will help you to develop and narrow down your evaluation research questions.
A Program's Operational Theories
Two key features of a program that should be detailed here, as they can dictate program success or failure are program Theory of Cause and Effect and Theory of Implementation.
Theory of Cause and Effect is the underlying logic or theoretical support for the program. It explains why the program will be lead to certain outcomes.
Using If-Then Statements
It can be helpful to think of you statement as being similar to a logic or mathematical proof built with if-then statements
X → Y
If the program does X, then Y (a desirable short and/or long-term effect) will occur.
Keep in mind as you begin your program description that many of the program details you are seeking may already be included as current program components such as the program mission, strategic goals or objectives. Also, stakeholders may be able to assist you with this step, and should be consulted to ensure you are fully grasping the theoretical basis for the program.
Here you should consider both the ultimate outcome of the program and any mediators that are necessary between your program activities and your end result. For example, if by participating in a self-esteem workshop, leads to increased self-efficacy for students (mediating variable), and reduces alcohol usage.
Theory of Implementation describes the strategy for translating a hypothetical program to the real-world. It's the how, between your if and then.
A program's theory of cause and effect may be well-supported and proven, but how the program is run may dictate success or failure. A process evaluation can help with detailing whether a program was implemented as intended.
Case Study: Description of Health Bucks
Please take a moment to review the Health Bucks program description provided via the Center for Training and Research Translation (Center TRT)
Based on this information, describe each of the following:
- Health Bucks Theory of Cause and Effect
- Health Bucks Theory of Implementation
If you are interested in learning more about the program and resulting evaluation components the TRT has laid out a fairly detailed summary of the Health Bucks Program accessible via PDF.
Program Objectives and Components
Specifying the program objectives and the program components (inputs, outputs, and any other relevant resources/features) are another critical piece of developing a full program description.
Keep in mind the following components:
- Goals and objectives
- Target population(s)
- Desired outcomes, both immediate, short and long-term
- Program Activities
- Any Program Outputs
- The Program Resources/Inputs including program materials and features, as well as the people who make it possible)
- Relationship of activities and outcomes
A powerful way to explore program objectives and components is visually via a logic model. Logic models can be particularly helpful for showing relationships between the different program features and outcomes. Your stakeholders may even have a logic model from their program development process, which you can adapt for the evaluation.
While developing a logic model is by no means a required output for an evaluation, logic models can be helpful organizational tools during this step in the process. They are also nice products to share with evaluation stakeholders to confirm you are interpreting the program correctly prior to starting the evaluation.
Overview of Logic Models
Now a bit more into the nitty-gritty of logic model development. The main components of a logic model include:
Resources, Activities, Outputs, and Impact.
Next, we will dive a bit deeper into each of these pieces of the logic model in turn.
Considering what you know about Health Bucks thus far, take a few minutes to draw up a high-level logic model which includes the key components, objectives, and is true to the program's theory of cause and effect and theory of implementation. Logic Model Handout
Resources
Resources/Inputs can include:
- Humans (staff, participants, any other organizations that will need to be involved)
- Financials
- What you need to create the program activities (e.g. facilities, pen and paper, computer programs, etc.)
The more detailed you can be here the more helpful it will be on the backend.
Activities
Consider here what processes, tools, events, technology, and actions are part of the program's implementation.
These could include things such as...
- Services being offered (training, counseling, screening)
- Created Infrastructure (relationships, capacity building, etc.)
In short, how will the resources of the prior section will be utilized?. Consider again the theory of implementation—what is being done in order to meet your end goals?
Outputs
For outputs, consider what the program will ultimately deliver (amount of product or services) if the planned activities are accomplished.
It is important to consider how outputs are different from the next section of your logic model, which is outcomes.
Output should be focused solely on the quantity and carrying forward your listed activities (for example, "in the parenting training, 25 participants were given a brochure with 10 key rules for successful parenting"). In short it should describe what is actually accomplished among the activities. Outcomes, as we will discuss in the next section, are more focused on what happens as a result of the activities (quality of the program)
Outcomes
Identify what knowledge, attitudes, beliefs, behaviors, skills, or other results will come from the program. In other words, what/how will your target aduience benefit if the given outputs aer accomplished?
It is helpful to divide these outcomes up temporally into short (immediate), intermediate, and long-term.
Short-term (think immediate) often focus on attitudes and knowledge. For example/ After the information session, all participants will know their options to lower their STI risk, and will be able to demonstrate safe condom usage as indicated via successful completion of the knowledge exam
Intermediate involves early-stage behavior change or changes in attitudes. For example / Participants will believe that practicing safe sex, such as regular condom usage, will reduce their STI risk
Long-term involves sustained change targeted by the program. For example: Condom use during sexual activity will increase among participants by 50% or more.
Well-written and complete outcome statements will usually define the following five elements (Source: Fisher, Imm, Chinman & Wandersman, 2006):
- Who will change - the [people] you are training
- What will change - the knowledge, attitudes, and skills you expect to change
- By how much - how much change you think you can realistically achieve
- By when - the timeframe within you hope to see change
- How the change will be measured - the surveys, tests, interviews, or other methods you will use to measure the different changes specified
Impact
This is the overarching then of your Theory of Cause and Effect If-then statement
In short, what your program is attempting to accomplish and what benefits will be achieved for your target population.
Playing off our prior example, the impact of a "safer sex" initiative encouraging steps such as safe condom usage may hope to have some of the following impacts:
- Reduce incidence of new STIs by 30% within the overall community
- Reduce prevalence of STIs in the target population by 25%
Keep in mind that impact can mean many different things depending on the goals of your program. It could be a change in a social norm or overarching pedagogy (such as encouraging a major lifestyle change), or a particular health outcome (such as reducing number of chronic illnesses or deaths). Either way, for evaluation purposes it is often important to define outcomes and impacts which are specific and measurable.
As aforementioned, a detailed logic model can be very helpful during the evaluation process. It provides a descriptive overview of the program theory, features, and how these interact overtime to produce the desired outcomes and impact.
- Process Evaluation often focuses on Resource/Inputs, Activities and Outputs. Reviewing what was actually done and what resulting outputs were accomplished. They can see if a program was implemented with fidelity and make recommendations for how to strengthen these first sections of the logic model.
- Outcome Evaluation on the other hand, seeks to prove the relationship between Outputs, Outcomes and Impact.
- Though how far you can go on the evaluation depends on the program stage (for example, you wouldn't expect to see a 3 year impact results if your program had only been running for the past year, so you would focus instead on some of the short and intermediate outcomes as proof points).
An important consideration is to decide how detailed you would like your logic model to be. Some, such as the Health Bucks logic model are highly specific and detailed. Yet, you may find that a very basic summary will accomplish your evaluation team and key stakeholder needs.
Image Source: L Olsho, Baronberg S, Y Abel, C Austin, C Booker, J Greece, J Levin, L Staub-DeLong, DK Walker, Abrami A, Holloway K, Jernigan J, Payne G. Health Bucks Evaluation Tool Kit, CDC, 2012.
Key Program Features
Now that you are familiar with the logic model as a helpful tool for describing your program, let's review the key program features to include as well as some additional factors to consider in your program overview:
Key Program Description Features
- Specify the program's theory of cause and effect
- Describe the program's theory of implementation
- Specify the program objectives
- Specify the program components (inputs, outputs, and any other relevant resources/features)
Also important to consider...
- Stage of program development (Is it a pilot? Has it been around for several years? Etc.)
- Context
Understanding program stage and context will be helpful for considering your evaluation questions and final design.
Again, consider that stakeholders such as program coordinators may be a particularly helpful resource at this point in the process. They will have internal knowledge of the background of a program.
Describe the Program Checklist
Source: CDC manual for Health Program Evaluation
- Compile a comprehensive description of the program including the theory of cause and effect, targets, inputs, activities and short and long-term outcomes
- Identify the stage of development and context the program operates within
- Logic Model Check list (strongly recommended but may not always be needed):
- Convert theory of cause and effect, targets, inputs, activities and short and long-term outcomes into a simple, global logic model
- Elaborate the model as needed
- Develop more detailed models from the global model as needed
