Program Evaluation for PhDs

Simulation author – Corin White, PhD

Simulation vetted by professionals in the Bay Area

Simulation Objective:

Assess the Success and Impact of a Program or Project

Associated Simulation Library:

Science Sim

Background

Program planning and evaluation are a significant component of any type of outreach effort, especially if one’s role is the director of the outreach initiative. Individuals with the role of Science Outreach Director can work in a variety of settings e.g. nonprofit, academe or government.

Focusing on evaluation or assessment part of the planning process is important, because it is critical to know if the goals and objectives of the program are being achieved. Thus, each activity or event must be planned with a mission statement lens. For example, if one goal is to train scientists in the method and practice of teaching, several program activities such as a workshop should be geared towards this mission. The knowledge gained through evaluation also aids in acquiring resources through the development of a pitch and summary of impact. Such resources are volunteers and/or collaborators, equipment and/or monies in the form of donations or grants. Therefore, the tasks explored here involve skills that are useful to develop, and have the potential to profoundly effect on the success of a project/program.

Note: The in-depth program evaluation explored here is usually done by a professional evaluator. The evaluator is not typically program staff, but may represent a stakeholder.

The Process

The process below includes categories of information and activities that are necessary to plan and evaluate the effectiveness of a program. Steps 1-5 may be represented in a logic model. A logic model or framework is a tool used for the planning, implementation and/or evaluation of a project/program.

Establish the goals and objectives of the project/program
Evaluate the availability of resources
Determine your activities
Record your anticipated outcomes and impact
Choose a method to evaluate the goals/objectives of the project/program
Run the evaluation and record data
Determine if you have evidence to support step 1
Create a report that summarizes step 6
Update the logical model and program with the appropriate changes

Resources

Next Generation Science Standards
Evaluation report examples, resources, and tips to Start to Developing a Logic Model from InformalScience.org
Sample evaluation reports from the NIH Science Education Partnership Award (SEPA)
Evaluation module from Evidence-Based Public Health Training Series from the University of Washington
Logic model templates and examples from UNC Chapel Hill
On Selecting Data Collection Methods from the CDC and Prevention.
Also, view this example of program evaluation from the viewpoint of a funding agency.

Project SMART Goals

Self Reflection Guide

Informational Interview Guide

The Exercise

Program X partners local scientists and teachers to teach natural or physical science modules in kindergarten through 6th grade classrooms. Modules must includes hands-on lessons that are active, investigative and meet the Next Generation Science Standards (NGSS). The main goals and/or objectives of Program X are as follows:

Introduce students to a diverse set of role models in science.
Create sustainable partnerships in support of science education.
Expose students and teachers to inquiry based learning.
Give scientists experience and confidence in teaching.
To increase the use of hands-on teaching strategies in the classroom.

Program X is funded for an additional two years and, you must make sure that the goals of the program are being met. The school district is especially interested in this evaluation; since, there is an opportunity to include additional schools in the upcoming academic year.

For this exercise, you are the Executive Director of the Science Education Consortium that serves several elementary school districts and your task is to evaluate Program X.

Task 1 - Critique of a Logic Model

A logic model, program matrix or logical framework is a graphical depiction of the relationships between program elements. These elements include inputs, outputs and outcomes. Logic models are used for planning, executing and properly assessing a project/program. Before starting this task, familiarize yourself with examples of logic models from informalscience.org, such as “A Start to Developing a Logic Model” and “Logic Model for Agricultural Literacy Programming“. Below is a simplified logic model for Program X containing program inputs, outputs, outcomes and evidence. The model does not includes the program goals/objectives (see “The exercise”), a summary of external factors or any assumptions.Your job is to correct the logical model and provide a rationale for your edits. Critique the model by answering the following questions:

Are the inputs adequate to generate the appropriate outputs?
Are the outputs adequate to meet the short and mid term outcome?
Are the outcomes aligned with the program goals/objectives?

After completing this task, check your corrected logic model with a sample corrected logical model provided. How do they differ? Logical models will vary depending on the author and place of employment there’s no right or wrong answer.

Task 2 - Evaluate Data for Program X

During the previous school year, Program X partnered 37 scientists with 18 teachers in 15 teams that served approximately 600 students. Evaluation data for Program X is available below:

78% of the students surveyed agreed with the following statement: “I want to be a scientist when I grow up.”
The percentage of students that viewed science as “fun” changed from 23% to 62% after Program X.
60% of the partner teachers feel as if they have access to resources that will aid them in teaching science.
34% of the partner teachers feel as if they have learned strategies for hands on teaching.
82% of the partner teachers are more likely to use inquiry based science lessons.
67% of the graduate students gained confidence in teaching.

Keeping the program goals in mind (from “The Exercise”), examine the data and answer the following questions:

Is the data useful?
What are the general trends of the data?
Did you have sufficient survey participation?
What is the biggest impact?
Are there any problems with these data?
Does the data support the goals of the program described in “The Exercise”?
If you require more data, what type of data do you need?

Write a summary that contains the information above. This summary would be presented in a meeting to your staff that implemented Program X. This task represents steps 7 and 8 of the process outlined above. Step 6 has been hypothetically completed and provided for the purpose of this simulation.

Task 3 - Prepare a Report

Using your results from Tasks 1 & 2, create a report that includes:

The description of the program and goals. These are listed in the Exercise section.
A description of the population surveyed.
The original and your updated logical model. Include a few statements about why you changed the logical model
Answers to questions from Task 2.
Your overall assessment of Program X and any new or refined goals for the program. If you propose new goals, be sure to include your rationale for those changes.

This report is what you would present to the school district or the sponsors. This task represents step 8 and 9 in the process outlined above.

Deliverables

Task 1

Edit the logical model and write a few sentences explaining your changes. This logical model would be discussed with your staff and, once finalized, become part of a grant application.

Task 2

A written summary but, you may choose to present the data in a drawing, graph, etc. as appropriate. This summary should contain all of the data and, any accordant commentary.

Task 3

A brief report. Remember, your audience is the school district, potential collaborators, and sponsors. The report should be broad, clear and concise.

Full reports can be lengthy, for the purposes of this simulation, the report should be no more than 2-3 pages and include the following:

Title
Description of the program and goals
Description of the population surveyed
The corrected logical model (deliverable from Task 1)
Evaluation data from Task 2
Your assessment of the data (deliverable from Task 2)
Final assessment and any recommendations for the program

All of these tasks can be completed in a word processor.

Additional Tasks

Develop educational resources
Evaluate project or program progress and/or outcomes
Develop program evaluation methods
Train fellow educators
Search and apply for relevant funding sources
Marketing
Establish and manage relationships with volunteers and vendors
Budget management
Write and speak to diverse audiences