Page 4 of 10

# Describe the Program

In order to develop your evaluation questions and determine the research design, it will be critical first to clearly define and describe the program.

Both steps, Describe the Program and Engage Stakeholders, can take place interchangeably or simultaneously. Successful completion of both of these steps prior to the evaluation planning creates a strong foundation for the evaluation.

Consider first, the key features that should be included in a complete program description.

Key Program Description Features

• Specify the program's theory of cause and effect
• Describe the program's theory of implementation
• Specify the program objectives
• Specify the program components (inputs, outputs, and any other relevant resources/features)

A detailed, holistic program description will help you to develop and narrow down your evaluation research questions.

## A Program's Operational Theories

Two key features of a program that should be detailed here, as they can dictate program success or failure are program Theory of Cause and Effect and Theory of Implementation.

Theory of Cause and Effect is the underlying logic or theoretical support for the program. It explains why the program will be lead to certain outcomes.

Using If-Then Statements

It can be helpful to think of you statement as being similar to a logic or mathematical proof built with if-then statements

X → Y

If the program does X, then Y (a desirable short and/or long-term effect) will occur.

Keep in mind as you begin your program description that many of the program details you are seeking may already be included as current program components such as the program mission, strategic goals or objectives. Also, stakeholders may be able to assist you with this step, and should be consulted to ensure you are fully grasping the theoretical basis for the program.

Here you should consider both the ultimate outcome of the program and any mediators that are necessary between your program activities and your end result. For example, if by participating in a self-esteem workshop, leads to increased self-efficacy for students (mediating variable), and reduces alcohol usage.

Theory of Implementation describes the strategy for translating a hypothetical program to the real-world. It's the how, between your if and then.

A program's theory of cause and effect may be well-supported and proven, but how the program is run may dictate success or failure. A process evaluation can help with detailing whether a program was implemented as intended.

Case Study: Description of Health Bucks

Please take a moment to review the Health Bucks program description provided via the Center for Training and Research Translation (Center TRT)

Based on this information, describe each of the following:

1. Health Bucks Theory of Cause and Effect
2. Health Bucks Theory of Implementation

If you are interested in learning more about the program and resulting evaluation components the TRT has laid out a fairly detailed summary of the Health Bucks Program accessible via PDF.

## Program Objectives and Components

Specifying the program objectives and the program components (inputs, outputs, and any other relevant resources/features) are another critical piece of developing a full program description.

Keep in mind the following components:

• Goals and objectives
• Target population(s)
• Desired outcomes, both immediate, short and long-term
• Program Activities
• Any Program Outputs
• The Program Resources/Inputs including program materials and features, as well as the people who make it possible)
• Relationship of activities and outcomes

A powerful way to explore program objectives and components is visually via a logic model. Logic models can be particularly helpful for showing relationships between the different program features and outcomes. Your stakeholders may even have a logic model from their program development process, which you can adapt for the evaluation.

While developing a logic model is by no means a required output for an evaluation, logic models can be helpful organizational tools during this step in the process. They are also nice products to share with evaluation stakeholders to confirm you are interpreting the program correctly prior to starting the evaluation.

## Overview of Logic Models

Now a bit more into the nitty-gritty of logic model development. The main components of a logic model include:

, , , and .

Next, we will dive a bit deeper into each of these pieces of the logic model in turn.

Considering what you know about Health Bucks thus far, take a few minutes to draw up a high-level logic model which includes the key components, objectives, and is true to the program's theory of cause and effect and theory of implementation. Logic Model Handout

Resources

Resources/Inputs can include:

• Humans (staff, participants, any other organizations that will need to be involved)
• Financials
• What you need to create the program activities (e.g. facilities, pen and paper, computer programs, etc.)

The more detailed you can be here the more helpful it will be on the backend.

Activities

Consider here what processes, tools, events, technology, and actions are part of the program's implementation.

These could include things such as...

• Services being offered (training, counseling, screening)
• Created Infrastructure (relationships, capacity building, etc.)

In short, how will the resources of the prior section will be utilized?. Consider again the theory of implementation—what is being done in order to meet your end goals?

Outputs

For outputs, consider what the program will ultimately deliver (amount of product or services) if the planned activities are accomplished.

It is important to consider how outputs are different from the next section of your logic model, which is outcomes.

Output should be focused solely on the quantity and carrying forward your listed activities (for example, "in the parenting training, 25 participants were given a brochure with 10 key rules for successful parenting"). In short it should describe what is actually accomplished among the activities. Outcomes, as we will discuss in the next section, are more focused on what happens as a result of the activities (quality of the program)

Outcomes

Identify what knowledge, attitudes, beliefs, behaviors, skills, or other results will come from the program. In other words, what/how will your target aduience benefit if the given outputs aer accomplished?

It is helpful to divide these outcomes up temporally into short (immediate), intermediate, and long-term.

Short-term (think immediate) often focus on attitudes and knowledge. For example/ After the information session, all participants will know their options to lower their STI risk, and will be able to demonstrate safe condom usage as indicated via successful completion of the knowledge exam

Intermediate involves early-stage behavior change or changes in attitudes. For example / Participants will believe that practicing safe sex, such as regular condom usage, will reduce their STI risk

Long-term involves sustained change targeted by the program. For example: Condom use during sexual activity will increase among participants by 50% or more.

Well-written and complete outcome statements will usually define the following five elements (Source: Fisher, Imm, Chinman & Wandersman, 2006):

• Who will change - the [people] you are training
• What will change - the knowledge, attitudes, and skills you expect to change
• By how much - how much change you think you can realistically achieve
• By when - the timeframe within you hope to see change
• How the change will be measured - the surveys, tests, interviews, or other methods you will use to measure the different changes specified

Impact

This is the overarching then of your Theory of Cause and Effect If-then statement

In short, what your program is attempting to accomplish and what benefits will be achieved for your target population.

Playing off our prior example, the impact of a "safer sex" initiative encouraging steps such as safe condom usage may hope to have some of the following impacts:

• Reduce incidence of new STIs by 30% within the overall community
• Reduce prevalence of STIs in the target population by 25%

Keep in mind that impact can mean many different things depending on the goals of your program. It could be a change in a social norm or overarching pedagogy (such as encouraging a major lifestyle change), or a particular health outcome (such as reducing number of chronic illnesses or deaths). Either way, for evaluation purposes it is often important to define outcomes and impacts which are specific and measurable.

As aforementioned, a detailed logic model can be very helpful during the evaluation process.   It provides a descriptive overview of the program theory, features, and how these interact overtime to produce the desired outcomes and impact.

• Process Evaluation often focuses on Resource/Inputs, Activities and Outputs.   Reviewing what was actually done and what resulting outputs were accomplished.   They can see if a program was implemented with fidelity and make recommendations for how to strengthen these first sections of the logic model.
• Outcome Evaluation on the other hand, seeks to prove the relationship between Outputs, Outcomes and Impact.
• Though how far you can go on the evaluation depends on the program stage (for example, you wouldn't expect to see a 3 year impact results if your program had only been running for the past year, so you would focus instead on some of the short and intermediate outcomes as proof points).

An important consideration is to decide how detailed you would like your logic model to be. Some, such as the Health Bucks logic model are highly specific and detailed. Yet, you may find that a very basic summary will accomplish your evaluation team and key stakeholder needs.

Image Source: L Olsho, Baronberg S, Y Abel, C Austin, C Booker, J Greece, J Levin, L Staub-DeLong, DK Walker, Abrami A, Holloway K, Jernigan J, Payne G. Health Bucks Evaluation Tool Kit, CDC, 2012.

## Key Program Features

Now that you are familiar with the logic model as a helpful tool for describing your program, let's review the key program features to include as well as some additional factors to consider in your program overview:

Key Program Description Features

• Specify the program's theory of cause and effect
• Describe the program's theory of implementation
• Specify the program objectives
• Specify the program components (inputs, outputs, and any other relevant resources/features)

Also important to consider...

• Stage of program development (Is it a pilot? Has it been around for several years? Etc.)
• Context

Understanding program stage and context will be helpful for considering your evaluation questions and final design.

Again, consider that stakeholders such as program coordinators may be a particularly helpful resource at this point in the process. They will have internal knowledge of the background of a program.

Describing the evaluation in terms of objectives and components is an important initial step to describing the program.

Outcome statements should be as specific as possible, including detail surrounding who will change, what will change, by how much, by what time point and how change will be measured.  Detail now will help with forming evaluation questions that are specific and measurable

Considering the stage and context of the program is just as important as reviewing the inputs, activities, outputs, outcomes and intended impact

A logic model will need to be heavily detailed in order to be helpful for your evaluation

Logic models once established should not be altered.

Describe the Program Checklist

Source: CDC manual for Health Program Evaluation

• Compile a comprehensive description of the program including the theory of cause and effect, targets, inputs, activities and short and long-term outcomes
• Identify the stage of development and context the program operates within
• Logic Model Check list (strongly recommended but may not always be needed):
1. Convert theory of cause and effect, targets, inputs, activities and short and long-term outcomes into a simple, global logic model
2. Elaborate the model as needed
3. Develop more detailed models from the global model as needed
list of module pages by topic
current page (page 4)

#### Resources

Any of the resources that go into a program. Consider those that are critical both at the organizational and community level.

#### Activities

How the resources are utilized within the program. Think events, tools, processes that are part of implementing (aka actually running) the program.

#### Outputs

Consider what the product of your activities will be. Include here not only the types but also the levels and targets of your activity delivery.

#### Outcomes

What specific changes will result in your target population when they engage with the program. Consider what attitudes, knowledge, and behavior...

#### Impact

Fundamental intended (or perhaps intended) change resulting from the program.