Combined Performance and Meta-Evaluation for the U.S. Department of State

Overview

ME&A conducted a combined dual design evaluation examining evaluation efforts since 2012 of the 26 Department of State (DoS) bureaus that use Foreign Assistance (FA) funds. The meta-evaluation, taking place from October 2017 through April 2018, assessed the quality of completed evaluations through an analysis of their strengths and weaknesses based on evaluation report content. The performance evaluation assessed the extent to which evaluation findings and recommendations are used, whether bureaus are meeting Evaluation Policy requirements and using available evaluation tools, and where the U.S. Office of Foreign Assistance Resources (F) should direct its assistance to bureaus moving forward.

Activities

For the meta-evaluation, the evaluation team (ET) analyzed a sample of 72 evaluation reports. Key conclusions were as follows:

• The quality of evaluation reports has improved since the Evaluation Policy was issued in 2012, with a nearly equal number of reviewed reports rated as fair (34) or good (30) and relatively few reports rated as poor (8). However, significant room remains to improve report quality.
• Areas in which evaluation reports were relatively strong included discussing the evaluation methodology, drawing on all data collection methods, and readability and accessibility to lay audiences.
• Areas in which evaluation reports were relatively weak included distinguishing findings from conclusions, providing actionable recommendations, and discussing gender and social effects.

For the performance evaluation, the ET administered an online survey to DoS evaluation staff answered by 66 persons. The survey was followed by 28 key informant interviews and two focus group discussions. Key conclusions were as follows:

• Bureaus are complying with the requirement to conduct at least one evaluation per year. Perceptions of DoS staff about compliance with other requirements of the Policy vary widely.
• The Evaluation Policy has had a significant impact on how bureaus approach evaluation, including an increase in staff evaluation capacity, number of evaluations done, focus on program learning, and availability of evaluation tools and resources. The diversity of evaluation resources and capacities across bureaus makes devising an Evaluation Policy that fits all bureaus a challenge.
• F’s assistance via direct consultations and TA has been effective, and the demand for the DoS evaluation courses is high, reflecting their perceived value among Department evaluation staff.
• Challenges in meeting the Evaluation Policy’s requirements include a lack of technical expertise, funding constraints, programs that do not lend themselves to evaluation, bureaucratic obstacles, and a lack of senior bureau leadership buy-in.
• F can continue to help bureaus implement the Evaluation Policy through promoting buy-in of bureau leadership, linking evaluation results to decision-making, and promoting cross-bureau collaboration.

Funding

ME&A is prime contractor to the U.S. Department of State on two Performance Management and Evaluation Services Indefinite Delivery Indefinite Quantity (IDIQ) contracts to provide monitoring and evaluation services to regional, functional, and management bureaus at headquarters and overseas. The ME&A contracts cover two Functional Areas: Civilian Security, Rights and Stability, and Management Platform.