|
CHAPTER FIVE: EVALUATION Evaluation is an important component of any health or education program. Through evaluation, agencies can identify which programs are effective in achieving their goals and objectives and which programs are most cost effective for a specific goal or behavior. Evaluation research has its roots in the social, behavioral, and statistical sciences and relies on their principles and methodologies of research to determine which components of a program may be considered successful, which may be considered unsuccessful, and which simply need improvement. The term “evaluation” means a lot of different things to different people. Turner, Miller, and Moses (1989) simply stated
The evaluation process can range from one that is simple, short in duration, and can be completed in-house, to one that is complex, time consuming, and requires the help of an outside consultant or consultants. Whatever the process, however, all program evaluations have one common goal: to provide information that can be used to improve the quality and effectiveness of the program or of similar programs in the future. The purpose of evaluation can vary and may be different for different stakeholders. For example, funding agencies or organizations may use evaluation to determine if continued investment in the program is justified, whereas program managers may use evaluation to determine how the program can be improved or to gather data that can be used to support proposals for continuation or expansion of the program. Evaluation data can be used by agencies to:
Evaluation data can also help clients when agencies use the data to provide services that are shown to be effective, and to identify ways to improve service delivery to better meet the needs of clients. Steps of Evaluation As explained above, evaluation is a process. This process can be broken down into five steps as follows:
Step 1: Identifying the research or evaluation question Before developing tools for evaluation and before collecting data, you should identify the question or questions that you hope to answer. What information are you trying to discover through the evaluation process? You should be as specific as possible in formulating your evaluation question or questions. Your “research question” will determine your overall approach, or evaluation model, as well as the specific data you collect. For example, suppose a clinic administrator wants to gather data that will illustrate to potential funders the effectiveness of contraceptive education strategies used at that particular clinic (i.e., the research question is “How effective are contraceptive education strategies at this clinic?”) The administrator must determine what defines “effectiveness of birth control education” in order to answer this question. This administrator may decide that one way to assess effectiveness is to determine whether or not adolescents have a greater knowledge of the different birth control options after their visit than before their visit; another administrator, in answering the same questions may assess whether or not patients can demonstrate correct use of the method; a third administrator may decide that a decrease in pregnancy rates will demonstrate the effectiveness of birth control education. As another example, suppose a consultant is hired to help formulate a new adolescent outreach program that a clinic is planning to undertake. This consultant may conduct an evaluation to answer the research question, “How can we increase the number of adolescent clients served at our clinic?” This evaluation would likely be very different in design from the previous one because they are seeking different types of information. Obviously, each of these clinics is assessing something different. It is important in conducting evaluation to specify what you are measuring, and what you can learn from that information. In order to identify a “question” for your evaluation, you should consider the following:
Step 2: Developing the evaluation plan No single approach to evaluation will be appropriate for all situations. Once you have determined the research question you will need to select the approach that will best help you achieve your goals. Specifically, you should answer the following questions:
Step 3: Conducting the evaluation Your evaluation plan will specify how the evaluation will be conducted including the evaluation model and evaluation tools to be used, who is responsible for what tasks, and when each task is to be completed. You may, however, need to revise this plan if you find that the type of results being produced is not helpful for answering your research question. You will need to allow enough time to conduct the evaluation in order to get sufficient data so that your evaluation results are relevant. The evaluation should be conducted in accordance with the plan developed in the previous step. Step 4: Analyzing the data Once you have collected the data, you need to analyze it in order to make sense of it. Depending on the type of evaluation you conducted, the tools you have used, and the reason for conducting the evaluation, you may want to conduct a thorough statistical analysis of the data, or you might just want to summarize the information you have gathered. The more complex the data and the analysis, the greater the need for a professional evaluator to gather and analyze the data. The analysis of the data should relate to the research question. Step 5: Reporting the results The final step involves reporting the results and relating these results to services currently in place and/or future services. How you report your evaluation results, who you report them to, and what you use those results for should be discussed before you conduct the evaluation. The report should indicate how you will use the data you have collected. Types of Evaluation There are several types of evaluation, and which one you use will be determined by the purpose of the evaluation and the program’s stage of development at the time the evaluation is done. Formative evaluation is an early evaluation that provides feedback to the program developers and implementers to inform project design. This is done in an effort to maximize success and minimize failures before implementing a program on a larger scale. This may answer the question, “What is the possibility that this strategy or program may work?” Process Evaluation answers questions about what core activities have occurred during the implementation process to achieve project goals or objectives. Different measurements should be taken at several phases of the project to determine what has taken place, what has not taken place, and how that may affect the short-term and long-term goals of the project. Changes to the implementation plan may need to be made pending the result of the process evaluation. This may answer the questions, “What was done” and “To whom, and how?” Summative Evaluation
aims “to assess the overall effectiveness of a program and the Impact Evaluation/Outcomes Evaluation answers questions about the long-term consequences, impacts, and results attributable to a project. This may answer the questions, “What changes were observed?” and “Has our project been successful?” Evaluation Tools Different evaluation tools collect and provide different types of data. Surveys–use interviews, observation, and questionnaires. These can be used to gather many different types of information. Experimental and Quasi-Experimental Designs–collect statistical data using randomized sampling (experimental) or alternative sampling methods (quasi-experimental). Focus Groups–meet with specific groups to discuss issues related to the evaluation. Cost Analysis–considers cost versus benefit. Using Evaluation Data Educators, clinicians,
program managers, and administrators
In general, the evaluation
process can provide information in the following three key
Accountability
involves making overall judgments about the effectiveness of your
Development involves
facilitating improvements to your program. In the category of
Knowledge involves generating information about the outcomes of your program and any related needs that remain unmet. In the category of knowledge, evaluation helps you
Sample Evaluation Plan–Evaluating a Pregnancy Prevention Counseling Program Objective: To measure the effectiveness of a pregnancy prevention counseling program. Program Description: Clinic X has implemented a brief pregnancy prevention interview with every client entering the clinic. A health educator conducts a five-minute interview while clients wait for their appointment. During this interview, the health educator begins or updates a client profile, which logs sexual activity, contraceptive practices, and attitudes on pregnancy. This program has been implemented and in practice for six months. Step 1: Identifying the Research/Evaluation Question Decide what to evaluate. To evaluate “effectiveness” you need to decide what will constitute “effectiveness. “ Some possibilities are:
How you define “effectiveness” will frame how you evaluate your program, and what aspects you evaluate. Step 2: Developing the Evaluation Plan You need to determine how to get the data you want to collect. How you completed step 1 will guide you through Step 2. For example, if you decided to evaluate any of the following: patient satisfaction with the program; increased number of patients who use their method of birth control correctly; patients having increased knowledge about pregnancy and pregnancy prevention; and change in attitude towards using birth control correctly and consistently, you would need to collect information from patients. You may want to collect information both before and after they attend your program, to see if there is a change in knowledge, attitudes, or skills. You could develop surveys specific to what you want to evaluate. For example, if you want to see if patients know more about pregnancy prevention methods after being in your program, you might want to identify key pieces of information you want patients to know, develop a knowledge test (e.g., true/false questions, multiple choice questions, etc) that is specific to that information, and give patients this test before and after attending your program. Compare results to see if patients know more after your program than they knew before. The same method, with different surveys, can be used for attitudes and skills. Having patients demonstrate how to use birth control methods is an additional way of evaluating skills they have learned. You can also use focus groups (in addition to surveys or instead of surveys) to gather information from patients about satisfaction and attitudes. Another way of surveying attitudes may be asking patients if they have recommended your services to others, or if they would recommend your services to others, and why or why not? You might conduct an outcomes/impact evaluation. First, develop a survey assessment form that covers essential information. Ask patients to complete this form each time they attend the clinic. The survey will assess changes in the client’s knowledge, attitudes, and behavior over time. To assess the long-term effects of the intervention an outcomes evaluation can be performed. After the first year of the program short interviews can be held with clients to explore their feelings and perceptions of the program. Ask specific questions to determine if the program made an impact on their behaviors. To help identify ways to enhance the effectiveness of the program, ask patients for suggestions for improvement. To check decreased numbers of pregnancies and increased numbers of patients attending the program you will probably need to collect data from patient medical records. You can count how many patients you have in the current year, and compare that with past years. You could identify how many of your contraceptive patients eventually become pregnant in a given year. You may want to compare current pregnancy rates with rates in the previous year. Or, you may want to compare numbers of patients with repeat pregnancies. You may want to compare this data with overall data in your community or nationally to see if there is a change in rates among your patients that is different from the change in rates in the same period of time in the community. To identify which patients have the greatest success (i.e., least number of unplanned pregnancies) in your program you may want to use of combination of information gathered directly from patients with surveys and focus groups, and information from patient medical charts. To evaluate the program
activities such as the method of providing information and counseling,
you may want to conduct a process evaluation. First, write a description
of what is covered during the program (e.g., contraceptive methods counseling)
and how it is covered (e.g., counselor describes and shows patients how
to use each method. Patient demonstrates use back to counselor. Or, patient
watches videotape, then meets with counselor to see demonstrations of
each method. ) The description should be as detailed as possible. The
educator or counselor could have a checklist of each topic discussed in
the interview. For each counseling visit, a note could be made in the
client’s medical chart to indicate that the session took place and
could also indicate the specific activities that the educator/counselor
covered. These data could be collected over a six-month Step 3: Conducting the Evaluation After you have decided what to evaluate and how to collect your data, the next step is to conduct the actual evaluation. In order to conduct your evaluation you need to:
You will conduct the evaluation according to the plan you developed in Step 2. Step 4: Analyzing the Data Your evaluation plan will determine how you will analyze the data. For example, if you want to determine change in patient knowledge, you need a comparison between what they knew at a baseline (what patients knew before your program) and what they knew after completing your program. If you are interested in patient satisfaction, a summary of information collected on surveys and from focus groups may suffice. If you want to look at which patients have the greatest success (i.e., least number of unplanned pregnancies) in your program, you may want to do a statistical analysis to see which populations in combination with which aspects of your program have the most success. Step 5: Reporting the Results Once you have analyzed your data, you need to report the data. The report should include:
You may also want to include in the report your future intentions for evaluation. What else would you like to evaluate in the future, and why?
|