Supporting the Evaluation Policy:
Improving the Quality of Evaluation Design
August 22, 2011 - 1:45 p.m.
||Elizabeth Roen, USAID Office of Learning, Evaluation, and Research
Christine Beggs, USAID Knowledge Services Center
Roger Rasnake, JBS International, Inc.
Marcia Odell, JBS International, Inc.
||Ron Raphael, USAID Office of Education
This session addressed the ways in which evaluation designs can be improved in light of USAID’s Evaluation Policy. Ron Raphael of the Office of Education and Elizabeth Roen of the Office of Learning, Evaluation and Research opened the session by noting that the new Evaluation Policy aims to address the decline in quality evaluations in recent years by placing new emphasis on the potential of performance and impact evaluations to increase accountability and learning, and demonstrate the outcomes of both large-scale and smaller innovative projects. The presenters contrasted the qualities of performance evaluations (90% of evaluations conducted) and impact evaluations (10% of evaluations conducted). They identified methods for improving evaluation quality, including ensuring sufficient resources (time, budget and human); limiting evaluation questions; using methods appropriate to answering those questions; allowing sufficient time for planning; refining the scope of work; providing deliverables that clearly outline methodology, limitations, sampling strategies, and instruments; and maintaining the independence of evaluators in order to limit bias.
Roger Rasnake and Marcia Odell of JBS International expanded upon this theme by providing recommendations for strengthening evaluation Statements of Work. These include careful definition of research questions, selection of appropriate evaluation methodologies, provision of adequate time to recruit and field an evaluation team, development of a sufficient timeframe for completion of fieldwork, and formulation of an appropriate budget. The presenters reviewed best practices in previously conducted evaluations.
Raphael addressed budgeting concerns in developing an evaluation Statement of Work, and Christine Beggs of USAID’s Knowledge Services Center provided examples of reasonable expectations for different types of evaluations. They shared a number of forthcoming evaluation-related resources that will be available online shortly.
Key take away points focused on the fact that USAID’s Evaluation Policy calls for a return to rigorous, high-quality evaluation. Guidance on how to improve the quality of the evaluation of education activities encompasses wide-ranging activities, from ensuring that sufficient resources are allocated for the evaluation, to maintaining evaluator independence. Points to remember in developing the evaluation Statement of Work include such core points as choosing realistic research questions, selecting appropriate evaluation methods, and allocating sufficient time and funding for the work to be undertaken. Resources currently being developed to assist education staff with evaluation activities will be available soon.