Introduction
Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. It involves the use of scientific procedures to determine as systematically and objectively as possible whether a given project, program, or policy is meeting specified goals. There are three major types of program evaluation: formative evaluation, process evaluation, and summative evaluation. This essay will discuss each type in detail.
Formative Evaluation
Formative evaluation is conducted during the development and pilot testing of programs or policies. It focuses on improving the quality, effectiveness, and impact of the program or policy by gathering feedback. The main purpose of formative evaluation is to help stakeholders determine if the program is on the right track and identify areas for improvement so corrections can be made before full-scale implementation.
Some key aspects of formative evaluation include:
Helping to refine and improve program design, activities, implementation, and materials based on participant feedback. Changes are made as the program is being developed and tested out on a small scale.
Assessing whether program resources and activities are consistent with the goals and objectives that were set out. The evaluation gives feedback on what is and isn’t working to meet the goals.
Identifying contextual factors like environmental influences, available resources that could impact program effectiveness. This helps determine if the program design fits with the situation and target audience.
Pinpointing strengths and weaknesses in the program theory, implementation strategies, and activities. The evaluation helps determine what program components need altering to increase the chances of success.
Monitoring program delivery and quality as it is being piloted. Formative evaluation is used to track if activities and resources are being delivered as intended.
Allowing for mid-course corrections by collecting ongoing feedback from participants and stakeholders. Changes identified through the evaluation can be incorporated before full implementation.
Overall, the goal of formative evaluation is to provide continuous feedback for improvement so the actual program that gets implemented has the highest chances of success based on the feedback received during development and piloting. It is a beneficial type of evaluation to conduct for new or existing programs undergoing revision.
Process Evaluation
Process evaluation focuses on evaluating how a program, policy or other intervention is being implemented – monitoring the day to day operations. The main purpose is to describe the activities, document how they are delivered, and explore the strengths and weaknesses in the implementation process. Insights from a process evaluation help determine if a program can be improved for better delivery and if the intended objectives are on track to be achieved given how it has been implemented so far.
Some key aspects of process evaluation include:
Describing what actually gets done, which components are emphasized, and how closely the implementation matches the written program design and objectives.
Assessing the dosage or amount and quality of what is delivered versus what was intended. Process evaluation looks at program adherence and fidelity.
Exploring participant engagement, how well staff delivers the content, and what factors affect quality and participation levels.
Identifying useful delivery methods and strategies that worked well and those that did not.
Pinpointing challenges, barriers, facilitators in the implementation process and program delivery that could be addressed.
Monitoring program operations, resources, and infrastructure to determine if adequate supports are in place.
Documenting any contextual factors like environmental impacts on rollout and efforts to adapt the program based on these issues.
Process evaluations emphasize using both qualitative and quantitative data collection methods like observations, interviews, surveys to understand implementation strengths and limitations from various stakeholder perspectives. Insights can then be used for ongoing quality improvement or to inform replication efforts if the program is found to be effective.
Summative Evaluation
Summative evaluation is used to examine the overall effectiveness and impact of a program or intervention that has been fully implemented. It focuses on outcomes and determining if the program has met its goals and objectives. Summative evaluations are generally conducted at the end of a project cycle to guide decisions about future funding, adoption, expansion or discontinuation.
Some key aspects of summative evaluation include:
Measuring intended and unintended short, medium and long-term outcomes of the program compared to the stated goals using quantitative metrics wherever possible. Both positive and negative effects are studied.
Assessing the overall effectiveness, efficiency and cost-effectiveness of the program design and implementation strategies in producing the targeted outcomes.
Determining if the program theory and logic model holds true – i.e. whether the activities led to the expected outcomes.
Evaluating participant satisfaction, how well needs were addressed, and the strength of impact including for different subgroups. Qualitative data on experiences is collected.
Identifying overall lessons learned and best practices that made the program successful or encounters challenges to sustainability, adoption or scale up.
Making recommendations to stakeholders about continuation, expansion, adaptation or termination based on rigorous analysis of outcomes achieved versus resources invested.
Summative evaluation uses an objective lens to provide a performance assessment and accountability report to funders, organizations and the community on the merit and worth of a completed program. It aims to contribute to evidence-based decision making about future interventions.
Conclusion
Formative, process and summative evaluations serve different but valuable purposes in the program evaluation cycle. Formative evaluation helps improve program quality during development, process evaluation assesses implementation quality during rollout, and summative evaluation examines overall outcomes, impact and value at the end. Regular use of multiple evaluation types helps stakeholders thoroughly understand strengths, limitations and effects at each program stage to strengthen quality, maximize benefits and effectively plan for the future. Comprehensive and meaningful program evaluation requires applying scientific principles across all three types.
Andrew Stroud