Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY21?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• In compliance with the Foundations for Evidence-Based Policymaking Act, the Administrator of USAID designated the Agency’s Evaluation Officer (AEO) through an internal Executive Message that was shared with the Agency on June 4, 2019. 
• USAID’s AEO is a senior staff member who works directly with the LER Director, who, in conjunction with the Office of Learning, Evaluation, and Research (LER) in the Bureau for Policy, Planning, and Learning (PPL) helps the Agency build a body of evidence from which to learn and adapt programs. The LER Director has the authority, staff, and budget to ensure agency evaluation requirements are met, including that all projects are evaluated at some level, and that decision-making is informed by evaluation and evidence. The LER Director oversaw approximately 40 staff and an estimated $6.5 million budget in FY20.
• USAID’s Bureau for Policy, Planning, and Learning (PPL) aligns policy, resources, and evidence-based programming. It elevates evaluation as a source of evidence, through the Office of Learning, Evaluation, and Research (LER), by focusing on the Agency’s ability and capability to generate, manage, and use evidence. The office performs a leadership role in the implementation of Title 1 of the Evidence Act, including the creation and development of the Agency Learning Agenda, the Annual Evaluation Plan, and assessment of how USAID staff manage and use evidence in implementing policies and strategies. In 2021, the office is undertaking a process of revising the Agency Learning Agenda to incorporate priorities of the Biden-Harris Administration.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• The Chief Data Officer (CDO) serves as the USAID Chief Data Officer. The CDO reports to the Chief Information Officer in the Bureau for Management. In compliance with the Foundations for Evidence-Based Policymaking Act, the Administrator of USAID re-affirmed the designation of the Chief Data Officer through an internal Executive Message that was shared with the Agency on June 4, 2019. The CDO manages the USAID Data Services team which focuses exclusively on improving the usage of data and information to ensure the Agency’s development outcomes are supported and enhanced by evidence. The CDO’s team includes four direct hire data science and IT professionals along with a budget for contract professionals who provide a comprehensive portfolio of data services in support of the Agency’s mission. The CDO oversaw approximately 80 staff and an estimated $18.7 million budget in 2021. The CDO is a senior career civil servant, and the USAID Data Services team is regularly called upon to generate products and services to support the Agency’s highest priorities. USAID also invests in other complementary positions including the Chief Innovation Officer, Chief Geographer, Chief Economist, Chief Scientist, and other key roles that enhance the use of evidence across the agency. 

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, performance improvement officer, and other related officials in order to support, improve, and evaluate the agency’s major programs?

• The Agency uses several governance structures and processes currently and will be updating these in accordance with OMB guidance related to the Foundations for Evidence-Based Policymaking Act. Some notable current examples include:
  
  • Data Board: In September 2019, USAID established a Data Administration and Technical Advisory (DATA) Board, as mandated by the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) and subsequent guidance from the Office of Management and Budget (OMB) in Memoranda M-19-18 and M-19-23. The DATA Board acts as USAID’s data governance body. It serves as a central venue for seeking input from Agency stakeholders regarding data-related priorities and best practices to support Agency objectives. The DATA Board informs data-related policy, procedures and standards for the Agency. The DATA Board supports the work of the Agency Evaluation Officer by directing data services to facilitate evaluations. In addition to the Agency Evaluation Officer, Chief Data Officer and Statistical Official, its membership includes the Performance Improvement Officer, the Chief Financial Officer, the Chief Technology officer, the Senior Agency Official for Privacy and the USAID Geographer as well as broad representation from across the Agency including overseas Missions. The USAID Chief Data Officer, Agency Evaluation Officer, and Statistical Official confer regularly to coordinate policy and activities.
  • Management Operations Council: USAID also uses a Management Operations Council (MOC) as the platform for Agency leadership to assess progress toward achieving the strategic objectives in USAID’s Strategic Plan and cross-agency priority goals and additional management issues. Established in 2014, the MOC provides Agency-wide leadership for initiatives and investments to reform USAID business systems and operations worldwide. The MOC also provides a platform for senior leaders to learn about and discuss improving organizational performance, efficiency, and effectiveness. The Assistant Administrator for the Bureau for Management and the Agency’s Chief Operating Officer co-chair the MOC. Membership includes, among others, all the Agency’s Chief Executive Officers (e.g., Senior Procurement Executive, Chief Human Capital Officer, Chief Financial Officer, Chief Information Officer, Performance Improvement Officer and Project Management Improvement Officer). Depending on the agenda, it also includes the Chief Data Officer, Agency Evaluation Officer, and the Agency Senior Statistical Official.
  • Weekly/Monthly Meetings between the Chief Data Officer, Chief Evaluation Officer, and Statistical Official: USAID established a standing meeting among the CDO’s team and leadership from the Office of Learning, Evaluation and Research which manages Agency requirements on performance monitoring, evaluation and organizational learning. As this meeting pre-dated the first Chief Data Officer council and Chief Evaluation Officer council meetings, it was critical for information sharing and addressing priorities. The CDOs team also maintains an internal dashboard which is shared with the Evaluation Officer and Statistical Official to help track progress against milestones on an ongoing basis.
  • Privacy Council Meetings: USAID holds monthly Privacy Council meetings to address necessary actions and raise any privacy and confidentiality concerns. Representation includes the Senior Agency Official for Privacy, the Agency Statistical Official, and the Chief Privacy Officer, among others.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY21?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• The agency-wide USAID Evaluation Policy, published in January 2011, updated in October 2016, and in April 2021, incorporates changes that better integrate with USAID’s Program Cycle Policy and ensure compliance with the Foreign Aid Transparency and Accountability Act (FATAA), and the Foundations for Evidence-Based Policymaking Act of 2018. The 2021 changes to the evaluation policy updated evaluation requirements to simplify implementation and increase the breadth of evaluation coverage, dissemination, and utilization.
• It also establishes new requirements that will allow for the majority of program funds to be subjected to external evaluations. The requirements include the following (1) at least one evaluation per intermediate result (IR) defined in the operating unit’s strategy; (2) at least one evaluation per activity (contracts, orders, grants, and cooperative agreements) with a budget expected to be $20 million or more; and (3) an impact evaluation for any new, untested approach, anticipated to be expanded in scale and scope. The main way these requirements are communicated is through the USAID Automated Directives System (ADS) 201.
• The Evaluation Policy requires consultation with in-country partners and beneficiaries as essential, and that evaluation reports could include sufficient local contextual information. To make the conduct and practice of evaluations more inclusive and relevant to the country context, the policy requires that evaluations will be consistent with institutional aims of local ownership through respectful engagement with all partners, including local beneficiaries and stakeholders, while leveraging and building local capacity for program evaluation. As a result, the policy expects that evaluation specialists from partner countries who have appropriate expertise will lead and/or be included in evaluation teams. In addition, USAID focuses its priorities within its sectoral programming on supporting partner government and civil society capacity to undertake evaluations and use the results generated. Data from the USAID Evaluation Registry indicated that annually, about two-thirds of evaluations, were conducted by teams that included one or more local experts. Also, while local experts may be included in the team composition, it is still a rarity to have a local expert as the evaluation team lead for conducting USAID evaluations.
  

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• USAID has produced an agency-wide Annual Evaluation Plan for FY22. This plan also fulfills the Evidence Act requirement that all Federal Agencies should develop an Annual Evaluation Plan, which describes the significant evaluation activities the agency plans to conduct in the fiscal year following the year in which it is submitted. The plan contains 35 significant evaluations that each address a question from the Agency-wide Learning Agenda; performance evaluations of activities with budgets of $40 million or more; impact evaluations; and ex-post evaluations.
• USAID has an agency-wide evaluation registry that collects information on all evaluations planned to commence within the next three years (as well as tracking ongoing and completed evaluations). Currently, this information is used internally and is not published. To meet the Evidence Act requirement, in March 2021, USAID published its Annual Evaluation Plan for FY22 on the Development Experience Clearinghouse. A draft agency-wide evaluation plan for FY23 will also be submitted in the Agency’s draft Annual Performance Plan/Annual Performance Report submitted to OMB in September 2021. 
• In addition, USAID’s Office of Learning, Evaluation, and Research works with bureaus to develop internal annual Bureau Monitoring, Evaluation and Learning Plans that review evaluation quality, and evidence building and use within each bureau, and identify challenges and priorities for the year ahead.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• USAID’s agency-wide learning agenda was first established in 2018, prior to the passing of the Evidence Act. The initial set of questions, titled the Self-Reliance Learning Agenda, were developed through a strongly consultative process with internal and external stakeholders and represented the Agency’s priority learning needs related to the Journey to Self-Reliance. Throughout implementation of the learning agenda, USAID has continued to engage external stakeholders through learning events, invitations to share evidence, and by making learning agenda products and resources publicly available on USAID.gov. 
  
• As priorities shift, it is essential that the Agency Learning Agenda adapts to continue to meet the learning needs of the Agency. The Agency Learning Agenda is undergoing a revision process to incorporate new Agency priorities and align with the FY22-26 Joint Strategic Plan. A number of policy areas have been identified for inclusion are COVID-19, climate, and Diversity, Equity, and Inclusion (DEI). Although USAID is still determining where to focus learning efforts, the Agency Learning Agenda is committed to furthering generation and use of evidence to inform agency policies, programs, and operations related to DEI and other critical areas. 
• Stakeholder consultations with internal and external stakeholders are central to the revision process. Consultations aim to capture a small, prioritized set of Agency learning needs related to Agency policy priorities, and to identify opportunities for collaboration with key stakeholders on this learning. The Agency Learning Agenda team is consulting Mission staff from across all of the regions in which USAID operates and Washington Operating Units to capture a diversity of internal voices. Consultations with external stakeholders include a selection of congressional committees, interagency partners (e.g. Department of State), other donors, think tanks, nongovernmental researchers, and development-focused convening organizations. Revisions to the Agency Learning Agenda will incorporate feedback gathered through these stakeholder consultations, inputs from the Joint Strategic Planning process with the Department of State, and a stocktaking of learning agenda implementation to-date to result in a prioritized set of questions that will focus Agency learning on top policy priorities.

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 3115, subchapter II (c)(3)(9))

• USAID recognizes that sound development programming relies on strong evidence that enables policymakers and program planners to make decisions, improve practice, and achieve development outcomes. As one of the deliverables of the Evidence Act, USAID submitted an interim Capacity Assessment to OMB in September 2020. This report provided an initial overview of coverage, quality, methods, effectiveness, and independence of statistics, evaluation, research, and analysis functions and activities within USAID. The report demonstrates that evaluations conducted by operating units cover the range of program areas of USAID foreign assistance investment. Economic growth, health, and democracy, human rights, and governance, accounted for more than three-quarters of evaluations completed by the Agency in FY19.
• In addition, USAID has commissioned a Capacity Assessment in response to the Evidence Act requirements. The assessment is using a four-phased approach: assessment design, implementation and analysis, reports, and communication/ dissemination. USAID is currently in Phase 3, which involves developing a Maturity Model to assess the Agency’s capacity to generate, manage, and use evidence.
• USAID staff also review evaluation quality on an ongoing basis and review the internal Bureau Monitoring, Evaluation and Learning Plans referenced in 2.2 above. Most recently, USAID completed a review of the quality of its impact evaluations. The review assessed the quality of all 133 USAID-funded IE reports published between FY12-19. In addition, there are several studies that have looked at parts of this question over the previous several years. These include GAO reports, such as Agencies Can Improve the Quality and Dissemination of Program Evaluations; From Evidence to Learning: Recommendations to Improve Foreign Assistance Evaluations; reviews by independent organizations like the Center for Global Development’s Evaluating Evaluations: Assessing the Quality of Aid Agency Evaluations in Global Health – Working Paper 461; and studies commissioned by USAID such as the Meta-Evaluation of Quality and Coverage of USAID Evaluations 2009-2012. These studies generally show that USAID’s evaluation quality is improving over time with room for continued improvement.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• USAID uses rigorous evaluation methods, including randomized control trials (i.e. assignment studies) and quasi-experimental methods for research and evaluation purposes. For example, in FY20, USAID’s Development Innovation Ventures (DIV), funded 10 impact evaluations, nine of which used randomized control trials.
• DIV makes significant investments using randomized controlled trials and quasi-experimental evaluations to provide evidence of impact for pilot approaches to be considered for scaled funding. USAID is also experimenting with cash benchmarking –using household grants to benchmark traditional programming. USAID has undertaken five randomized control trials (RCT) of household grants or “cash transfer” programs, three of which compare more traditional programs against household grants.

Did the agency invest at least 1% of program funds in evaluations in FY21?

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY21 budget.

• USAID invested at least $192.2 million in FY20 and prior year money on a combination of evaluations completed in FY20, evaluations that are ongoing during FY20, evaluation technical assistance, and evaluation capacity-building, representing 1.02% of the agency’s $18.8 billion FY20 budget.
• Source for FY21 Agency budget: FY 2021 Congressional Budget Justification (p. 2). Bilateral Economic Assistance total ($24,500,700,000) minus State’s Global Health Programs ($5,720,000,000) is $18,780,700,000.

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• In FY20, USAID operating units invested approximately $57.8 million in FY20 and prior year money on 152 evaluations that were completed in that fiscal year. Another 180 evaluations were ongoing in FY20 (many spanning more than one year in duration) with total ongoing evaluation budgets estimated at $127.8 million. LER’s budget for evaluation technical assistance and evaluation capacity-building in FY20 was $6.6 million, coming to a total of $192.2 million. This represents 1.02% of the Agency’s $18.8 billion FY19 budget. This total does not include evaluation capacity building done by other Agency offices or field Missions or other research, studies, analysis or other data collection that is often used for evaluation, such as USAID’s investment in the Demographic Health Survey or some of the assessments done by third-parties across USAID’s innovation portfolio. It also does not include funding by agency sub-components for evaluation technical assistance.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• While specific data on this is limited, USAID estimates that investment in contracts or grants that provide support to build local organizational or governmental capacity in data collection, analysis, and use could be as high as $250 million. 
• For example, USAID’s Data for Impact (D4I) activity helps low- and middle-income countries–primarily in sub-Saharan Africa–to increase their capacity to use available data and generate new data to build evidence for improving health programs, health policies, and for decision-making. D4I’s goal is to help low-resource countries gather and use information to strengthen their health policies and programs and improve the health of their citizens.
• In another example, the MEASURE Evaluation project, funded by USAID, has a mandate to strengthen health information systems (HIS) in low-resource settings. The Project enables countries to improve lives by strengthening their capacity to generate and use high-quality health information to make evidence-informed, strategic decisions at local, subregional, and national levels.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY21?

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• USAID partners with the U.S. Department of State to jointly develop and implement clear strategic goals, strategic objectives, and performance goals, which are articulated in the FY18-22 U.S. Department of State – USAID Joint Strategic Plan (JSP). USAID and the Department of State have commenced development of the FY22-26 JSP to incorporate new strategic themes. As part of the planning process issues of racial equity, diversity, and inclusion are being considered under both management-oriented and programmatic goals. The FY22-26 JSP will include a section on evidence-building and USAID and Department of States’ respective learning agendas will be included in the annex. 
• The Agency measures progress towards its own strategic goals, strategic objectives, and performance goals using data from across the Agency, including from annual Performance Plan and Reports (PPRs) completed by operating units, and uses that information to report on performance externally through the Annual Performance Plan/Annual Performance Report (APP/APR) and the Agency Financial Report.
• To aggregate and track performance in key sectors, USAID works with the U.S. Department of State to develop and manage over 100 standard foreign assistance indicators that have common definitions and defined collection methods. Once finalized, USAID publishes illustrative indicator data on a publicly available website known as Dollars to Results. Finally, USAID reports on Agency Priority Goal (APG) and Cross Agency Priority (CAP) goal progress on www.performance.gov.

4.2 Did the agency use data/evidence to improve outcomes and return on investment?

• Most of USAID’s innovation or co-created programs and those done in partnerships reflect a data-driven “pay for results” model, where milestones are agreed by all parties, and payments are made when milestones are achieved. This means that, for some programs, if a milestone is unmet, funds may be re-applied to an innovation or intervention that is achieving results. This rapid and iterative performance model means that USAID more quickly understands what is not working and can move resources away from it and toward what is working.
• Approaches such as prizes, Grand Challenges, and ventures can also be constructed to be “pay for results only” where interventions such as “Development Impact Bonds“ are used to create approaches where USAID only pays for outcomes and not inputs or attempts only. The Agency believes this model will pave the way for much of USAID’s work to be aligned with a “pay for results” approach. USAID is also piloting the use of the impact per dollar of cash transfers as a minimum standard of cost-effectiveness for applicable program designs. Most innovations funded at USAID have a clear “cost per impact” ratio. 
• Additionally, USAID Missions develop Country Development Cooperation Strategies (CDCSs) with clear goals and objectives and a Performance Management Plan (PMP) that identifies expected results, performance indicators to measure those results, plans for data collection and analysis, and regular review of performance measures to use data and evidence to adapt programs for improved outcomes. USAID also promotes data-informed operations performance management to ensure that the Agency achieves its development objectives and aligns resources with priorities. USAID uses its Management Operations Council to conduct an annual Strategic Review of progress toward achieving the strategic objectives in the Agency’s strategic plan.
• To improve linkages and break down silos, USAID continues to develop and pilot the Development Information Solution (DIS)–an enterprise-wide management information system that will enable USAID to collect, manage, and visualize performance data across units, along with budget and procurement information, to more efficiently manage and execute programming. USAID is currently in the process of world-wide deployment of the performance management module with almost half of its operating units now live in the system. 

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• USAID’s Program Cycle policy (ADS 201.3.2.18) requires that Missions conduct at least one portfolio review per year that focuses on progress toward strategy-level results. Missions must also conduct a CDCS mid-course stocktaking at least once during the course of implementing their Country Development Cooperation Strategy, which typically spans five years.
• USAID developed an approach to explicitly ensure adaptation through learning called Collaborating, Learning, and Adapting (CLA). It is incorporated into USAID’s Program Cycle guidance (ADS 201.3.5.19) where it states: “Strategic collaboration, continuous learning, and adaptive management link together all components of the Program Cycle.” Through CLA, USAID ensures its programming is coordinated with others, grounded in a strong evidence base, and iteratively adapted to remain relative throughout implementation. 
• The CDOs team maintains an internal dashboard which is shared with the Evaluation Officer and Statistical Official to help track progress against milestones on an ongoing basis. This helps ensure that data needs are being met and achieving intended results.  
• In addition to this focus through its programming, USAID has two senior bodies which oversee Enterprise Risk Management, and meet regularly to improve the accountability and effectiveness of USAID programs and operations through holistic risk management. USAID tracks progress toward strategic goals and annual performance goals during data-driven reviews at Management Operations Council meetings. Also, through input from the Management Operations Council, an annual Agency-wide customer service survey, and other analysis, USAID regularly identifies opportunities for operational improvements at all levels of the Agency as part of its operational learning agenda as well as the agency-wide learning agenda. The initial set of learning questions under the Agency learning agenda included four questions that focused on operational aspects of the agency’s work which influence everything from internal policy, design and procurement processes, program measurement, and staff training. As the Agency Learning Agenda is being revised, the focus on including key operational questions to support continuous improvement remains.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY21?

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• USAID’s data related investments and efforts are guided by its Information Technology Strategic Plan. This includes support for the Agency’s Development Data Policy, which provides a framework for systematically collecting Agency-funded data, structuring the data to ensure usability, and making the data public while ensuring rigorous protections for privacy and security. In addition, this policy sets requirements for how USAID data is documented, submitted, and updated. Guidance for USAID’s Open Data Policy may be seen in the User Guide, FAQs, and Help Videos. 
• In 2020 USAID revised the Development Data Policy to require development activities to create and submit data management plans before collecting or acquiring data. The Development Data Library (DDL) is the Agency’s repository of USAID-funded, machine readable data, created or collected by the Agency and its implementing partners. The DDL, as a repository of structured and quantitative data, complements the DEC which publishes qualitative reports and information. The Agency’s data governance body, the DATA Board, is guided by annual data roadmaps that include concrete milestones, metrics, and objectives for Agency data programs. USAID also participates and leads in global compilations of data across the industry including the Global Innovation Exchange and in response to COVID-19. USAID also has a variety of stakeholder engagement tools available on USAID’s Development Data Library, including: Open Data Community Questions and video tutorials on using DDL.
• People-level indicators for development data are normally disaggregated by sex (male, female), sometimes by age and occasionally by other demographic markers. Development data rarely includes transgender, gender non-conforming, or non-binary disaggregation. In many countries it may be politically complicated or potentially unsafe to collect this data or data that asks about racial or ethnic identity. However, data can often be disaggregated by geographic location, region, or state, which can be mapped with other demographic data to build a picture of geographic disparities. Country expertise can then be applied to analyze racial and ethnic equity dimensions, as described in ADS 205.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• Launched in November 2018 as part of the Development Information Solution (DIS), USAID’s public-facing Development Data Library (DDL) provides a comprehensive inventory of data assets available to the Agency. DDL has posted the Data Inventory as a json file since 2015. Following the passage of the Foundations for Evidence-Based Policymaking Act, and in preparation for specific guidance expected in the upcoming release of Phase 2 guidance for the Act, USAID will make any necessary changes to its Comprehensive Data Inventory and continue reporting with quarterly updates as required. The DDL’s data catalog is also harvested via JavaScript on an ongoing basis for further distribution on the federal Data.gov website. Currently 456 USAID data assets are available to the public via USAID’s DDL, a 17% increase over last year.

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c))

• The USAID Data Services team–located in USAID’s Management Bureau’s Office of the Chief Information Officer (M/CIO)–manages a comprehensive portfolio of data services in support of the Agency’s mission. This includes enhancing the internal and external availability and ease-of use of USAID data and information via technology platforms such as the AidScape platform broadening global awareness of USAID’s data and information services, and bolstering the Agency’s capacity to use data and information via training and the provision of demand-driven analytical services. 
• The Data Services Team also manages and develops the Agency’s digital repositories, including the Development Data Library (DDL), the Agency’s central data repository. USAID and external users can search for and access datasets from completed evaluations and program monitoring by country and sector.
• USAID staff also have access to an internal database of over 100 standard foreign assistance program performance indicators and associated baseline, target, and actual data reported globally each year. This database and reporting process, known as the Performance Plan and Report (PPR) promotes evidence building and informs internal learning and decisions related to policy, strategy, budgets, and programs. 
• The United States is a signatory to the International Aid Transparency Initiative (IATI). The standard links an activity’s financial data to its evaluations. Partner country governments as well as other initiatives and websites can pull these data into their respective systems. This helps officials oversee the coordination and management of incoming foreign aid, and serves as an effective tool in standardizing and centralizing information about foreign aid flows within a country. This data can be ingested to reduce and streamline USAID’s own reporting efforts, freeing up resources for other endeavors. Further, by streamlining reporting to these partner country systems and other websites, USAID is promoting efficiency in data collection, improving the quality of data, reducing the time needed to publish updated information, as well as providing timely information to inform analysis, future decisions, and policy-making. USAID continues to improve and add to its published IATI data, and is looking into ways to utilize these data as best practice–including using it to populate partner country systems, fulfill transparency reporting as part of the US commitment to the Grand Bargain, and make decisions internally, including based on what other development actors are doing by using the Development Cooperation Landscape tool. In FY20, USAID began reporting additional data to IATI in alignment with IATI’s COVID-19 reporting guidance in order to share financial and descriptive information about USAID’s COVID-19 activities.
• USAID continues to pursue better communicating data insights. USAID’s Geocenter uses programmatic and demographic data linked with geospatial data to inform decision-making, emphasizing mapping to identify gaps in service provision and inform resource provision and decision-making (for example, to compare gender-based violence (GBV) “hotspots” and access to relevant support services; and to identify geographies and communities disparately impacted by natural disasters).

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)

• USAID’s Privacy Program and privacy policy (ADS 508) direct policies and practices for protecting personally identifiable information and data, while several policy references (ADS303maz and ADS302mbj) provide guidance for protecting information to ensure the health and safety of implementing partners. USAID’s Development Data Policy (ADS Chapter 579) details a data publication process that provides governance for data access and data release in ways that ensure protections for personal and confidential information. As a reference to the Development Data Policy, ADS579maa explains USAID’s foreign assistance data publications and the protection of any sensitive information prior to release. USAID applies extensive statistical disclosure control on all public data before publication or inclusion in the DDL.

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• While specific data on this is limited, USAID does invest in contracts or grants that provide support to build local organizational or governmental capacity in data collection, analysis, and use. In addition, to date, 526 USAID data assets are available to the public via USAID’s DDL, a 44% increase over last year. These assets include microdata related to USAID’s initiatives that provide partner countries and development partners with insight into emerging trends and opportunities for expanding peace and democracy, reducing food insecurity, and strengthening the capacity to deliver quality educational opportunities for children and youth around the globe. Grantees are encouraged to use the data on the DDL, which provides an extensive User Guide to aid in accessing, using, securing and protecting data. The Data Services team conducts communication and outreach to expand the awareness of websites with development data, how to access it, and how to contact the team for support. In addition, the Data Services team has developed a series of videos to show users how to access the data available. The [email protected] mail account responds to requests for assistance and guidance on a range of data services from both within the Agency and from implementing partners and the public.
• Starting in 2020 Data Services’ Data Literacy Training series added equitable and accessible themes woven throughout the series, including: definitions of equitable data and accessible data; guiding principles for collecting data in ways that are equitable and inclusive; and guiding questions to determine whether shared/visualized data is equitable and accessible. These learning opportunities are designed for both internal and external audiences and will be available on public-facing web pages in late 2021 or early 2022.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY21?

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• USAID has developed a draft agency-level evidence framework to clarify evidence standards for different decisions, including those related to funding. The draft was shared for voluntary comment, and is being further updated. Once finalized, it will be published as a suggested help guide for USAID staff and continue to be refined and updated as needed.  
• USAID’s evidence standards are embedded within its policies and include requirements for the use of evidence in strategic planning, project design, activity design, program monitoring, and evaluation. USAID has a Scientific Research Policy that sets out quality standards for research across the Agency. USAID’s Program Cycle Policy requires the use of evidence and data to assess the development context, challenges, potential solutions, and opportunities in all of USAID’s country strategies. Specific programs, such as the Development Innovation Ventures (DIV) use evaluation criteria based on evidence of causal impact, leading to scaling which innovations work for greater cost effectiveness. As USAID’s flagship open innovation program, DIV helps to find, test, and scale innovative solutions to any global development challenge from anyone, anywhere. By backing proven innovations, driven by rigorous evidence and ongoing monitoring, USAID’s DIV program has proven to impact millions of lives at a fraction of the usual cost. Based on recent research announced in October 2020, the early portfolio of DIV grants (covering 2010-2012) has produced $17 in social benefit per every dollar spent by USAID. This research was led by Dr. Michael Kremer, a Nobel Prize-winning economist who is part of DIV’s team. Further, GAO found in their December 2019 report Evidence-Based Policymaking: EVIDENCE-BASED POLICYMAKING Selected Agencies Coordinate Activities, but Could Enhance Collaboration that USAID reflects leading practices for collaborating when building and assessing evidence. DIV’s evaluation criteria is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale.
  1. Evidence of Impact: DIV supports the piloting and rigorous testing of innovations and helps those innovations that have successfully demonstrated impact to transition to scale. DIV looks for different indicators of impact depending on the stage of financing that the applicant is seeking and on whether the innovation has a public or commercial pathway to scale.
     1. Public Pathway to Scale: To support later-stage innovations to transition to scale with public financing, including by government or philanthropic funding, DIV requires applicants to show rigorous evidence of causal impact on improving the lives of people living in poverty. DIV does not require applicants for earlier-stage awards for piloting and testing of innovations to show rigorous evidence of causal impact and will issue awards for earlier-stage innovations to help innovations get to the point at which they have this rigorous evidence of impact. Measurement of impact can encompass either final outcomes or impacts that can be taken as worthwhile objectives per se (e.g., reducing infant mortality). Measurement of impact can also encompass improvements to intermediate outcomes on the causal chain for solutions that have been previously demonstrated to have a causal impact on final outcomes (e.g., increasing vaccination rates, which have an existing evidence base of causal impact on improved health).
     2. Commercial Pathway to Scale: To support later-stage innovations to transition to scale commercially, DIV requires applicants to show evidence of long-run market viability. DIV typically accepts customer willingness to pay the full costs of the innovation as such evidence. DIV does not require applicants for earlier-stage awards for piloting and testing to show such evidence and will issue awards for earlier-stage innovations to get to the point where they have this evidence. For example, an applicant may use a Stage 1 award to gather evidence on consumer reaction to and willingness to pay for the product. As an applicant progresses to higher stages of financing, DIV expects the applicant to demonstrate an increasingly compelling case for long-run market viability. In addition, DIV requires applicants at all stages to show stage-appropriate data on usage, or in some situations on proxies for social outcomes that track how the innovation is improving the lives of people living in poverty.
  2. Cost-Effectiveness. DIV seeks innovations that deliver more development impact per dollar than existing alternative solutions. Cost-effectiveness is a function of both cost and impact and does not mean that an innovation is the cheapest solution. Instead, an innovation can increase its cost-effectiveness either by reducing its cost or by increasing its impact. DIV recognizes that costs of an innovation at scale may be different than costs generated through earlier testing phases, so applicants should provide data around current and projected costs.
     
  3. Pathways to Scale. DIV’s ultimate goal is to support impactful development solutions that will scale to improve the lives of at least a million people in a financially sustainable manner. Accordingly, innovations must have a potential pathway to scale through commercial scaling, public sector scaling, or scaling through a hybrid of the two. DIV recognizes that innovations can take a variety of pathways to scale but expects that they will be financially sustainable and ultimately grow without continued DIV support. For more information on the terms “pathways to scale” and “financial sustainability,” see the glossary at the end of this APS.

6.2 Did the agency have a common evidence framework for funding decisions?

• USAID has a draft agency-level evidence framework to clarify evidence definitions, principles and approaches for different decisions, including those related to funding. The framework has been posted for review and comment by external stakeholders. In addition, there are specific types of programs at the sub-agency level that do use evidence framework or standards to make funding decisions. For example, the Development Innovation Ventures (DIV) uses a tiered funding approach to find, test, and scale evidence-based innovations. DIV’s grants include: Stage 1 for piloting (up to $200,000); Stage 2 for testing (up to $1,500,000); Stage 3 for scaling (up to $15,000,000); and “evidence grants” (up to $1,500,000) for research to determine causal impact of certain interventions. In particular for Stage 2 grants, DIV requires evidence of impact that must be causal and rigorous–the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period. There must be significant demonstrated demand for the innovation.
• DIV’s evaluation criteria for its funding is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale. DIV’s expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators. Most DIV grants are fixed amount awards, a unique type of federal grant instrument that is tailor-made for pay-for-results approaches. Fixed amount awards are structured by paying for milestones achieved, which emphasizes performance (not just compliance) and reduces some administrative burden for all parties (see 2 CFR 200.201(b)).
  

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• USAID does have an Agency-wide repository for development information (including evaluation reports and other studies) which is available to the public at the Development Experience Clearinghouse. In addition, USAID uses the International Initiative for Impact Evaluations (3ie) database of impact evaluations relevant to development topics (including over 4,500 entries to date), knowledge gap maps, and systematic reviews that pull the most rigorous evidence and data from across international development donors. 3ie also houses a collection of institutional policies and reports that examine findings from its database of impact evaluations on overarching policy questions to help policymakers and development practitioners improve development impact through better evidence. 
• USAID’s Agency Programs and Functions policy designates technical bureaus responsible for being the repository for latest information in the sectors they oversee; prioritizing evidence needs and taking actions to build evidence; and disseminating that evidence throughout the agency for those sectors. Several USAID bureaus and sectors have created user friendly tools to disseminate information on evidence-based solutions. These include, but are not limited to:
  • • CLIMATELINKS: A global knowledge portal for climate change and development practitioners
    • EDUCATION LINKS: Shares innovations and lessons learned on implementation of the USAID Education Policy
    • Natural Resources Management and Development Portal
    • URBANLINKS: USAID’s sharing platform for resources on sustainable urban development
• USAID also led a data-harmony initiative across the industry with other countries called the Global Innovation Exchange which surfaces, validates, and shares a repository of over 16,000 development relevant solutions across all actors, players and locations. 
• Finally, USAID recently applied Natural Language Processing Text Analysis to analyze unstructured data from the previous ten years of Evaluation Reports published by USAID and identify countries which used specific language and terminology related to racial and ethnic equity. This review included 1,208 evaluation reports and 2,525 final contractor/grantee reports that were available on USAID’s public Development Experience Clearinghouse (DEC) and converted to machine readable format. To develop an algorithm to find the most relevant information, the team consulted with experts from across the agency working on inclusive development and DEIA issues to develop a lexicon of terms that together with other factors were tested and found to identify relevant documents.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• USAID’s approach to Collaborating, Learning, and Adapting (CLA) helps ensure that evidence from evaluation of USAID programming is shared with and used by staff, partners, and stakeholders in the field. USAID requires a dissemination plan and post-evaluation action plan for each evaluation, and USAID field staff are encouraged to co-create evaluation action plans with key stakeholders based on evaluation evidence. USAID collects examples through the CLA Case Competition, held annually, which recognizes implementers, stakeholders, and USAID staff for their work generating and sharing technical evidence and learning from monitoring and evaluation. It is another way that the Agency encourages evidence-based practices among its stakeholders.
• USAID also periodically holds large learning events with partners and others in the development community around evidence including, but not limited to, Evaluation Summits, engagement around the Agency Learning Agenda, and Moving the Needle. These gatherings are designed to build interest in USAID’s evidence, build capacity around applying that evidence and learning, and elicit evidence and learning contributions.
• USAID created and led the “Million Lives Collaborative” coalition, with more than 30 partners, which has identified more than 100 social entrepreneurs who have at least a million customers in order to share the learning that this successful cohort has had and better inform how USAID funding can assist more social entrepreneurs to grow successfully and rapidly. This unique learning platform brings donors, funders, governments, and the entrepreneurial community to the table together to learn and iterate on our approaches.
• USAID recently published the Evaluations at USAID Dashboard, which provide evidence of evaluation use by Missions, and opportunities for peer learning.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY21?

7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?

• The U.S. Global Development Lab became the Innovation, Technology, and Research Hub (ITR) under the new Bureau for Development, Democracy, and Innovation (DDI). DDI became operational within USAID on November 16, 2020. DDI is USAID’s central resource for providing world-class technical assistance to its Missions. DDI’s cross-cutting, global perspective will enable the Agency to anticipate and respond to evolving trends and issues, catalyze innovation and broaden our partnership base. DDI comprises four Centers and five Hubs. The Centers serve as USAID’s leaders for technical assistance in democracy and governance, economics and market development, education, energy, environment, human rights, and infrastructure. The Hubs provide Missions with expert guidance and training on cross-cutting development priorities, including gender equality, innovation, technology and research, private-sector engagement, and partnerships with diverse organizations, such as local and faith-based groups. 
• Specifically, the ITR Hub serves as a central point for promoting and building capacity for innovation throughout development and national security strategies across USAID, the U.S. Government, and the international community. This includes helping to centrally coordinate the Agency’s innovation-related work with entrepreneurs, implementing partners, universities, donors, and others to test and scale innovative solutions and approaches to development problems around the world.  In addition to finding and supporting innovative solutions, the ITR Hub also works with other USAID Bureaus and Independent Offices to promote a culture of innovation across the Agency to enable it to be a more innovative organization itself. For example, this includes building internal capacity, skills, and outside-the-box thinking to structure and provide our funding in more creative and effective ways (e.g., using fixed amount awards as a grant instrument to pay for outcomes, not just inputs).
• For innovations specific to a particular sector, Agency leadership has supported technical staff in surfacing groundbreaking ideas, such as how the Bureau for Global Health’s Center for Innovation and Impact (CII) used open innovation approaches to issue the Saving Lives at Birth Grand Challenge and identify promising, life-saving maternal and newborn health innovations. 

7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?

• In FY20, USAID released its first Digital Strategy, moving to a “Digital by Default” position and USAID’s innovative approaches have helped get more than 40 million people in the developing world digital access. USAID’s New Partnerships Initiative (NPI) will allow USAID to work with a more diverse range of partners, strengthen existing partner relationships, and provide more entry points for organizations to work with the Agency. The principles behind NPI are outlined in the Agency’s first-ever Acquisition and Assistance (A&A) Strategy.
• USAID and its partners have launched 41 innovative programming approaches including prizes, ventures, challenges, and Grand Challenges for Development since 2011. Across the Grand Challenges portfolio, partners have jointly committed over $535 million ($155 million from USAID) in grants and technical assistance for over 528 innovators in 107 countries. To date, more than $614 million in follow-on funding has been catalyzed from external sources, a key measure of success.
• USAID’s investment in state-of-the-art geo and information intelligence centers mean that any program has the ability to leverage geospatial analysis and critical data sets to drive innovative solutions based on evidence and data. With over 20 programs experimenting with Artificial Intelligence and machine learning, and USAID’s strong work on digital finance and connectivity, the Agency is using technology to drive our programs farther and faster. USAID has also completed more than 1,500 Global Development Alliances, leveraging private sector in-kind or financial investments.
• In addition, the Center for Innovation and Impact (CII)–the Bureau for Global Health’s dedicated innovation office–takes a business-minded approach to fast-tracking the development, introduction, and scale-up of health innovations that address the world’s most important health challenges, and assessing and adopting cutting-edge approaches (such as using unmanned aerial vehicles and artificial intelligence).
• Feed the Future Partnering for Innovation partners with agribusinesses to help them commercialize and scale new agricultural innovations to help improve the livelihoods of smallholder farmers, increasing their productivity and incomes. To date the program has worked with 59 partners in 20 different countries, investing more than $43 million in new technologies and services, and leveraging nearly $100 million in private sector investment. The program has helped commercialize over 118 innovations, which resulted in an estimated $99 million in sales. It has its own Innovation site that partners can easily see and connect with promising innovations and research.
• Finally, USAID was honored when the co-founder and Scientific Director of USAID’s Development Innovations Venture (DIV) program, Dr. Michael Kremer received the 2019 Nobel prize for economics, along with Dr. Esther Duflo and Dr. Abhijit Banerjee. Some of his work that led to this honor was connected to USAID’s DIV program. DIV values rigorous testing methods such as impact evaluations or robust market tests to measure the impact of USAID innovations. Evidence of clear and measurable outcomes helps demonstrate what is working and what is not. Solutions that demonstrate rigorous evidence of impact can then be scaled to other contexts. Through the DIV program, Dr. Kremer helps USAID use evidence-based approaches to take small risks, identify what works, and scale those approaches to provide greater impact. Since 2010, the DIV program has made 225 grants to find, test, and scale evidence-based innovations directly affecting more than 55 million lives across 47 countries. Based on Dr. Kremer’s and others research announced in October 2020, the early portfolio of DIV grants (covering 2010-12) has produced $17 in social benefit per every dollar spent by USAID.

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• Monitoring, Evaluation, Research and Learning Innovations Program (MERLIN) is a USAID endeavor designed to support the Bureau for Policy, Planning and Learning (PPL) and is now a part of the Learning, Evaluation and Research (LER) office within PPL. Formerly, the MERLIN program was part of the U.S. Global Development Lab which became the Innovation, Technology, and Research (ITR) Hub within the new DDI Bureau. The MERLIN program works to innovate on traditional approaches to monitoring, evaluation, research and learning. While innovative in themselves, these approaches can also be better suited to evaluating an innovation effort. Two examples of MERLIN activities include Developmental Evaluation, which aims to provide ongoing feedback to managers on implementation through an embedded evaluator, and Rapid Feedback, which allows implementers to test various methods to reach certain targeted results (more quickly than through traditional midterm or final evaluations). Both of these approaches allow adaptive management during implementation to improve program impacts.
• Many of the agency’s programs such as Grand Challenges and Development Innovation Ventures (DIV) have been reviewed by formal audit and other performance and impact interventions.  DIV uses a tiered funding approach to find, test, and scale evidence-based innovations. DIV’s grants include: Stage 1 for piloting (up to $200,000); Stage 2 for testing (up to $1,500,000); Stage 3 for scaling (up to $15,000,000); and “evidence grants” (up to $1,500,000) for research to determine causal impact of certain interventions. In particular for Stage 2 grants, DIV requires evidence of impact that must be causal and rigorous – the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period. There must be significant demonstrated demand for the innovation.
• DIV’s evaluation criteria for its funding is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale. DIV’s expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators. Most DIV grants are fixed amount awards, a unique type of federal grant instrument that is tailor-made for pay-for-results approaches. Fixed amount awards are structured by paying for milestones achieved, which emphasizes performance (not just compliance) and reduces some administrative burden for all parties (see 2 CFR 200.201(b)).
• DIV supports innovative solutions across all countries and development sectors in which USAID operates, including education, agriculture, water, energy, and economic development. Since 2010, DIV has provided more than $149 million for 225 grants in 47 countries, reaching more than 55 million beneficiaries. 

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY21?

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY20, the five largest competitive grant programs are:

1. International Disaster Assistance ($4.40 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303); 
2. Migration and Refugee Assistance ($3.43 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303); 
3. Development Assistance ($3.4 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303);
4. Global Health (USAID) ($3.16 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303); 
5. Economic Support Fund ($3.05 billion ADS 303).

• See the U.S. Foreign Assistance Reference Guide for more information on each of these accounts. More information can also be found in the FY2021 Congressional Budget Justification (page 2 and 3, column 4). USAID generally does not limit eligibility when awarding grants and cooperative agreements; eligibility may be restricted for an individual notice of funding opportunity in accordance with the procedures in ADS 303.

8.2 Did the agency use evidence of effectiveness to allocate funds in the five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• USAID is committed to using evidence of effectiveness in all of its competitive contracts, cooperative agreements, and grants, which comprise the majority of the Agency’s work. USAID’s Program Cycle Policy ensures evidence from monitoring, evaluation and other sources informs funding decisions at all levels, including during strategic planning, project and activity design, procurement and implementation.
• USAID’s Senior Obligation Alignment Review (SOAR) helps to ensure the Agency is using evidence to design and approve funding for innovative approaches to provide long-term sustainable outcomes and provides oversight on the use of grant or contract mechanisms and proposed results. 
• USAID includes past performance to comprise 30% of the non-cost evaluation criteria for contracts. As part of determining grant awards, USAID’s policy requires an applicant to provide a list of all its cost-reimbursement contracts, grants, or cooperative agreements involving similar or related programs during the past three years. The grant Selection Committee chair must validate the applicant’s past performance reference information based on existing evaluations to the maximum extent possible, and make a reasonable, good faith effort to contact all references to verify or corroborate how well an applicant performed.
• For assistance, as required by 2 CFR 200, USAID also does a risk assessment to review an organization’s ability to meet the goals and objectives outlined by the agency. Internal procedures for conducting the risk assessment are found in ADS 303.3.9, with guidance on how to look for evidence of effectiveness from potential grantees. Per the ADS, this can be done through reviewing past performance and evaluation/performance reports such as the Contractor Performance Assessment Reporting System (CPARS). 
• Even though there is no federal requirement (as there is with CPARS), USAID also assesses grantee past performance for use when making funding decisions (detailed in ADS 303, p. 66). Per USAID’s ADS 303 policy, before making an award of any grant or cooperative agreement the Agreement Officer must state in the memorandum of negotiation that the applicant has a satisfactory record of performance. When making the award, the Agreement Officer may consider withholding authority to proceed to the next phase of a grant until provided evidence of acceptable performance within a given period. 
• USAID was recognized by GAO in its recent report published on September 5, 2018, Managing for Results: Government-wide Actions Needed to Improve Agencies’ Use of Performance Information in Decision Making (GAO-18-609SP) as one of four agencies (out of 23 surveyed) with proven practices for using performance information. USAID was also the only CFO Act agency with a statistically significant increase in the Agency Use of Performance Information Index since 2007.

8.3 Did the agency use its five largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• Grantees report on the progress of activities through documentation such as Activity Monitoring, Evaluation, and Learning (MEL) Plans, periodic performance reporting, and external and internal evaluation reports (if applicable). These reports help USAID remain transparent and accountable and also help the Agency build evidence of what does and does not work in its interventions. Any internal evaluation undertaken by a grantee must also be provided to USAID for learning purposes. All datasets compiled under USAID-funded projects, activities, and evaluations are to be submitted by grantees to the USAID Development Data Library. All final evaluation reports must also be submitted to the Agency’s Development Experience Clearinghouse (DEC), unless they receive a waiver to the USAID’s public dissemination requirements. These are rare and require the concurrence of the Director of the Office of Learning, Evaluation, and Research.

8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant programs (besides its five largest grant programs)?

• USAID is actively engaged in utilizing evidence of effectiveness to allocate funds. For example, Development Innovation Ventures (DIV) uses a tiered funding approach to find, test, and scale evidence-based innovations. DIV’s grants include: Stage 1 for piloting (up to $200,000); Stage 2 for testing (up to $1,500,000); Stage 3 for scaling (up to $15,000,000); and “evidence grants” (up to $1,500,000) for research to determine causal impact of certain interventions. In particular for Stage 2 grants, DIV requires evidence of impact that must be causal and rigorous–the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period. There must also be significant demonstrated demand for the innovation.
• DIV’s evaluation criteria for its funding is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale. DIV’s expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators. Most DIV grants are fixed amount awards, a unique type of federal grant instrument that is tailor-made for pay-for-results approaches. Fixed amount awards are structured by paying for milestones achieved, which emphasizes performance (not just compliance) and reduces some administrative burden for all parties (see 2 CFR 200.201(b)). 
• DIV supports innovative solutions across all countries and development sectors in which USAID operates, including education, agriculture, water, energy, and economic development. Since 2010, DIV has provided more than $149 million for 225 grants across 76 countries, reaching more than 55 million beneficiaries. Based on recent research announced in October 2020 led by Nobel Prize-winning economist and DIV advisor, Dr. Michael Kremer, the early portfolio of DIV grants (covering 2010-12) has produced $17 in social benefits per every dollar spent by USAID.

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?  

• USAID’s Development Innovation Ventures (DIV) specifically emphasizes rigorous evidence for causal impact in its official grant solicitation (DIV Annual Program Statement (APS), page 4): “DIV supports the piloting and rigorous testing of innovations and helps those innovations that have successfully demonstrated impact to transition to scale. DIV looks for different indicators of impact depending on the stage of financing that the applicant is seeking and on whether the innovation has a public or commercial pathway to scale.” DIV’s evaluation criteria is based on its three core principles as further outlined in its APS: (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale.
• Fenix offers expandable, lease-to-own, solar home systems (SHS) financed through ultra-affordable installments over mobile money. In 2016, DIV partnered with USAID’s Scaling Off-Grid Energy team to support Fenix’s expansion from Uganda into Zambia, a nascent and largely underserved market. By the end of the DIV award, Fenix was the leading SHS company in Zambia. In 2017, Fenix was acquired by ENGIE, a multinational electric utility company, and expanded into four new countries–Benin, Côte d’Ivoire, Nigeria, and Mozambique. Fenix has delivered clean, affordable energy to 3.5 million people across six countries in Africa.
• EarthEnable is a social enterprise that has developed durable adobe floor replacements for traditional dirt floors. EarthEnable flooring minimizes exposure to bacteria and parasites–particularly for children–and is 70% less expensive than other clean floor alternatives. Early investments by DIV supported EarthEnable to test different business models and scale up operations, expanding their geographic reach and enabling them to serve lower-income households. To date, EarthEnable has replaced more than 5,000 dirt floors and served over 20,000 people in Rwanda and Uganda. 
• In 2013, DIV funded a randomized control trial to evaluate evidence for causal impact of the program, Teaching at the Right Level (TaRL), implemented by Pratham, an Indian NGO. While progress has been made to help more children attend school, millions of students are not actually learning at their grade level. In response, TaRL helps lagging students catch up by teaching to their skill level rather than to their age or grade. The approach works by dividing children (generally in Grades 3 to 5) into groups based on learning needs rather than age or grade. It dedicates time to basic skills rather than focusing solely on the curriculum. And it regularly assesses student performance, not just end-of-year examinations. In 2017, DIV further partnered with J-PAL Africa, UNICEF, USAID/Zambia, and the Zambian Ministry of General Education to scale TaRL across Zambia. To date, DIV’s support has helped catalyze more than $25 million in additional funding beyond USAID to scale the TaRL model to 12 countries across Africa.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• USAID’s Program Cycle Policy states that “[f]unding may be dedicated within a project or activity design for implementing partners to engage in an internal evaluation for institutional learning or accountability purposes.”

• USAID does not administer non-competitive grant programs (relative score for criteria #8 applied)

In FY21, did the agency shift funds away from or within any practice, policy, interventions, or program that consistently failed to achieve desired outcomes?

10.1 Did the agency have policy(ies) for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• Based on the USAID Rapid Feedback (RF) approach USAID/Cambodia mission and implementers developed a Theory of Change, conduct implementation research, and rapid experiments on Social and Behavior Change Communication (SBCC) for communities and for donors. The Cambodia Children’s Trust (CCT) had been working at the community level to discourage families from sending their children to Residential Care Institutions. The mission also supported Friends International (FI) which had been working with donors to encourage behavior change on their support to RCIs. Both implementers adopted action plans based on the results: CCT used the findings to streamline the Social Behavior Change Campaigns (SBCC) before rolling it out to more villages; and FI used the RF findings to inform use of SBCC beyond paid social media. 
• USAID shifts funds away from ineffective grantees. For example, the Securing Water for Food Grand Challenge is designed with a Technical Assistance Facility to consult and work with grantees to identify specific growth barriers, and then connect them with vetted service providers that bring expertise and capabilities to help these grantees overcome their strategic barriers. The Technical Assistance Facility provides tailored financial and acceleration support to help these grantees improve their market-driven business development, commercial growth, and scaling.
• If a grantee is unable to meet specific performance targets, such as number of customers or product sales, further funding is not granted (per the terms of the grant), and the grantee is re-categorized into the program’s group of unsuccessful alumni. The Securing Water for Food Grand Challenge used milestone-based grants to terminate 15 awards that were not meeting their annual milestones and shifted that money to both grants and technical assistance for the remaining 25 awards in the program.
• Also, USAID’s INVEST program is designed for constant feedback loops around the partner performance. Not only are under-performing partners dropped, but new partners can be added dynamically, based on demand. This greatly increases USAID’s new partner base and increases the performance standard across the board.
• USAID’s Business Ecosystem Project (BEP), implemented by Palladium Group, is designed to increase private sector investment in strengthening domestic supply chains and workforce development in North Macedonia. BEP’s initial strategy was to mobilize corporate social responsibility (CSR) funds from investors and large international corporations toward the project’s goal, but it quickly became evident that such investments would be neither strategic nor sustainable. To achieve a lasting impact on North Macedonia’s business ecosystem, BEP partnered with companies that were better positioned to recognize the link between local economic development and their own business interest. BEP learned from its local partners and adapted its private sector engagement (PSE) strategy to target small, medium, and large enterprises that were more dependent on domestic supply chains and workers. BEP no longer focuses only on foreign direct investment (FDI) companies with CSR budgets, but approaches all companies that have a real economic incentive to invest in local supply chains and workforce development. This approach was more effective and allowed BEP to co-invest in a diverse range of supply chain and workforce development initiatives, first as a proof of concept and later at scale.

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• USAID/Food for Peace’s Sustainable Action for Resilience and Food Security (Sabal) is a five-year program in Nepal, implemented by Save the Children and a consortium of partners. Sabal’s goal is to improve food security and resilience in targeted districts in Nepal by improving livelihoods, health and nutrition, disaster risk reduction, and climate change adaptation. Sabal utilized collaborating, learning and adapting (CLA) approaches such as pause and reflect, M&E for learning, and adaptive management to be able to adapt to the changing context. In 2015, there were devastating earthquakes, which necessitated geographic program expansion and then, two years later, there were budget cuts, which meant ending implementation in those expansion areas. At that time, CLA approaches were utilized to identify sustainability strategies, assess the level of self-reliance among community groups, tailor interventions based on the data, and gain consensus and buy-in among internal staff, consortium partners, and the local government. As a result, Sabal registered high-performing community groups with the government and linked these groups with local resources and leaders. At the same time, Sabal identified poor performing groups and built their capacity through targeted trainings and community capacity building.
• USAID’s Regional Health Integration to Enhance Services in Eastern Uganda (RHITES-E) Activity (2016-2021), implemented by IntraHealth International and its partners, supports the Government of Uganda’s health “surge” strategy to find new HIV positive patients and enroll them in care and treatment. The data and results from RHITES-E’s first quarter performance review showed the Activity was way behind its target. The Activity leadership and USAID decided to shift from a “business as usual” attitude to applying collaborating, learning, and adapting (CLA) approaches to draw on and analyze existing data, from a USAID dashboard, to reflect on findings with key stakeholders and fill identified needs and gaps to improve surge efforts. By the end of the fiscal year 2017, the Activity had improved its surge performance resulting in better results and outcomes and shifted in its culture to be a learning organization. Together with stakeholders, staff identified ineffective approaches such as mass HIV testing and developed and implemented new strategies to include screening of clients before testing for efficient and effective identification and linkage of new HIV positive clients into care and treatment.
• USAID’s Empleando Futuros (Employing Futures) program, an at-risk youth program, was launched in Honduras in 2016. During its first year, a pause and reflect event found a significant number of drop-outs and the need to strengthen the program’s response to better meet the needs of youth and the labor market. USAID and its implementing partner, Banyon Global, applied USAID’s Collaborating, Learning, and Adapting (CLA) Framework and tools to establish a framework for strategic pause and reflect events throughout the year, strengthen the program’s performance monitoring system, and develop an online platform for tracking program participants’ progress. These changes helped the implementer to revisit the program’s underlying assumptions and theory of change, learn continuously, and inform evidence-based decisions. Preliminary findings suggest that the program has fewer dropouts, capacity of local systems and partners has been strengthened, and private sector engagement has improved.