30
|
Developing an Effective Evaluation Plan
2 3 4 6
1
5
Step 5: Planning for Conclusions
Justifying conclusions includes analyzing the information you collect, interpreting, and
drawing conclusions from your data. This step is needed to turn the data collected into
meaningful, useful, and accessible information. This is often when programs incorrectly
assume they no longer need the ESW integrally involved in decision making and instead
look to the “experts” to complete the analyses and interpretation. However, engaging the
ESW in this step is critical to ensuring the meaningfulness, credibility, and acceptance of
evaluation findings and conclusions. Actively meeting with stakeholders and discussing
preliminary findings helps to guide the interpretation phase. In fact, stakeholders often have
novel insights or perspectives to guide interpretation that evaluation staff may not have,
leading to more thoughtful conclusions.
Planning for analysis and interpretation is directly tied to the timetable begun in Step 4.
Errors or omissions in planning this step can create serious delays in the final evaluation
report and may result in missed opportunities if the report has been timed to correspond
with significant events. Often, groups fail to appreciate the resources, time, and expertise
required to clean and analyze data. This applies to both qualitative and quantitative data.
Some programs focus their efforts on collecting data, but never fully appreciate the
time it takes to work with the data to prepare for analysis, interpretation, feedback, and
conclusions. These programs are suffering from “D.R.I.P.”, that is, programs that are “Data
Rich but Information Poor.” Survey data remains “in boxes” or interviews are never fully
explored for theme identification.
After planning for the analysis of the data, you have to prepare to examine the results
to determine what the data actually say about your program. These results should be
interpreted with the goals of your program in mind, the social/political context of the
program, and the needs of the stakeholders.
Moreover, it is critical that your plans include time for interpretation and review by
stakeholders to increase transparency and validity of your process and conclusions.
The emphasis here is on justifying conclusions, not just analyzing data. This is a step
that deserves due diligence in the planning process. The propriety standard plays a role
in guiding the evaluator’s decisions on how to analyze and interpret data to assure that
all stakeholder values are respected in the process of drawing conclusions (Program
Evaluation Standards, 1994). That is to say, who needs to be involved in the evaluation for
it to be ethical. This may include one or more stakeholder interpretation meetings to review
interim data and further refine conclusions. A note of caution, as a stakeholder driven
process, there is often pressure to reach beyond the evidence when drawing conclusions.