2021 Federal Standard of Excellence

Common Evidence Standards / What Works Designations

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY21? (Example: What Works Clearinghouses)

Score
6
Millennium Challenge Corporation
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • For each investment, MCC’s Economic Analysis (EA) division undertakes a Constraints Analysis to determine the binding constraints to economic growth in a country. To determine the individual projects in which MCC will invest in a given sector, MCC’s EA division combines root cause analysis with a cost-benefit analysis. The results of these analyses allow MCC to determine which investments will yield the greatest development impact and return on MCC’s investment. Every investment also has its own set of indicators as well as standard, agency-wide sector indicators for monitoring during the lifecycle of the investment and an evaluation plan for determining the results and impact of a given investment. MCC’s Policy for Monitoring and Evaluation details MCC’s evidence-based research and evaluation framework. Per the Policy, each completed evaluation requires a summary of findings, now called the Evaluation Brief, to summarize the key components, results, and lessons learned from the evaluation. Evidence from previous MCC programming is considered during the development of new programs. Per the Policy, “monitoring and evaluation evidence and processes should be of the highest practical quality. They should be as rigorous as practical and affordable. Evidence and practices should be impartial. The expertise and independence of evaluators and monitoring managers should result in credible evidence. Evaluation methods should be selected that best match the evaluation questions to be answered. Indicators should be limited in number to include the most crucial indicators. Both successes and failures must be reported.”
6.2 Did the agency have a common evidence framework for funding decisions?
  • MCC uses a rigorous evidence framework to make every decision along the investment chain, from country partner eligibility to sector selection to project choices. MCC uses evidence-based selection criteria, generated by independent, objective third parties, to select countries for grant awards. To be eligible for selection, World Bank-designated low- and lower-middle-income countries must first pass the MCC – a collection of 20 independent, third-party indicators that objectively measure a country’s policy performance in the areas of economic freedom, investing in people, and ruling justly. An in-depth description of the country selection procedure can be found in the annual report.
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
  • MCC is a leader in the production of evidence on the results of its international development programs. As a data-driven agency, MCC invests in evidence-generating activities, such as due diligence surveys, willingness to pay surveys, and independent evaluations. MCC has more room to lead, however, in the accessibility and usability of its evidence. Since 2013, MCC has shared the data, documentation, and analysis underlying its independent evaluations. In terms of accessibility of evaluation materials, users have noted that MCC’s central evaluation and data repository, the Evaluation Catalog, is hard to navigate. 
  • Recognizing that transparency is not enough to achieve accountability and learning, MCC has developed the MCC Evidence Platform. The Evidence Platform will offer first-of-its-kind study and data access and usability and encourage the utilization of its vast library of evidence. MCC will invite researchers–from students to experienced professionals–to use the data and documentation provided here to reproduce and build upon MCC’s evidence base to drive development effectiveness for, and beyond, MCC.
  • The MCC Evidence Platform will share:
    • Studies–Users may search by studies to find all the related data and documentation associated with each study. Study Types include: Independent Evaluations, Monitoring, Constraints Analysis, Willingness to Pay, Due Diligence, Country-led Studies, and Other Studies.
    • Documentation–Users may search by specific documentation associated with MCC-funded studies. This documentation is shared as specific Knowledge Products Types, including: Design Report, Baseline Report, Interim Analysis Report, Final Analysis Report, MCC Learning Document, Evaluation-based Cost-Benefit Analysis, and Questionnaires.
    • Data Packages–Users may search by specific data packages associated with MCC-funded studies. Data Package Type includes: Round (Baseline, Interim, Final), Public, Restricted-Access
  • The MCC Evidence Platform will encourage the use of MCC’s data, documentation, and analysis as global public goods to support mutual accountability for the agency and its country partners, and to encourage learning from measured results. 
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • As described above, the new MCC Evidence Platform was intentionally designed and launched with utilization at its core. The Platform specifically encourages users to take MCC learning and evidence and apply and reproduce it for new learning. The Platform will then aim to also share new learning based on published MCC evidence. As a part of this comprehensive approach, Evaluation Briefs continue to be a cornerstone to promoting utilization across audience groups. Enhanced utilization of MCC’s vast evidence base and learning was a key impetus behind the creation and expansion of the Evaluation Briefs and Star Reports. A push to ensure sector-level evidence use has led to renewed emphasis of the Principles into Practice series, with recent reports on the transport, education, and water & sanitation sectors.
  • MCC has also enhanced its in-country evaluation dissemination events to ensure further results and evidence building with additional products in local languages and targeted stakeholder learning dissemination strategies. 
Score
10
U.S. Department of Education
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • ED has an agency-wide framework for impact evaluations that is based on ratings of studies’ internal validity. ED evidence-building activities are designed to meet the highest standards of internal validity (typically randomized control trials) when causality must be established for policy development or program evaluation purposes. When random assignment is not feasible, rigorous quasi-experiments are conducted. The framework was developed and is maintained by IES’s What Works Clearinghouse TM (WWC). WWC standards are maintained on the WWC website. A stylized representation of the standards can be found here, along with information about how ED reports findings from research and evaluations that meet these standards.
  • Since 2002, ED–as part of its compliance with the Information Quality Act and OMB guidance–has required that all “research and evaluation information products documenting cause and effect relationships or evidence of effectiveness should meet that quality standards that will be developed as part of the What Works Clearinghouse” (see Information Quality Guidelines). 
6.2 Did the agency have a common evidence framework for funding decisions?
  • ED employs the same evidence standards in all discretionary grant competitions that use evidence to direct funds to applicants that are proposing to implement projects that have evidence of effectiveness and/or to build new evidence through evaluation. Those standards, as outlined in the Education Department General Administrative Regulations (EDGAR), build on ED’s What Works ClearinghouseTM (WWC) research design standards. 
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
  • ED’s What Works Clearinghouse TM (WWC) identifies studies that provide valid and statistically significant evidence of effectiveness of a given practice, product, program, or policy (referred to as “interventions”), and disseminates summary information and reports on the WWC website. 
  • The WWC has published more than 600 Intervention Reports, which synthesize evidence from multiple studies about the efficacy of specific products, programs, and policies. Wherever possible, Intervention Reports also identify key characteristics of the analytic sample used in the study or studies on which the Reports are based. 
  • The WWC has published 27 Practice Guides, which synthesize across products, programs, and policies to surface generalizable practices that can transform classroom practice and improve student outcomes.
  • Finally, the WWC has completed more than 11,000 single study reviews. Each is available in a searchable database.
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • ED has several technical assistance programs designed to promote the use of evidence-based practices, most notably IES’s Regional Educational Laboratory Program and the Office of Elementary and Secondary Education’s Comprehensive Center Program. Both programs use research on evidence-based practices generated by the What Works Clearinghouse and other ED-funded Research and Development Centers to inform their work. RELs also conduct applied research and offer research-focused training, coaching, and technical support on behalf of their state and local stakeholders. Their work is reflected in Strategic Plan Objectives 1.4 and 2.2.
  • Often, those practices are highlighted in WWC Practice Guides, which are based on syntheses (most recent meta-analyses) of existing research and augmented by the experience of practitioners. These guides are designed to address challenges in classrooms and schools. 
  • To ensure continuous improvement of the kind of TA work undertaken by the RELs and Comprehensive Centers, ED has invested in both independent evaluation and grant-funded research. The REL Program is currently undergoing evaluation, and design work for the next Comprehensive Center evaluation is underway. Addition, IES has awarded two grants to study and promote knowledge utilization in education, including the Center for Research Use in Education and the National Center for Research in Policy and Practice. In June of 2020, IES released a report on How States and Districts Support Evidence Use in School Improvement, which may be of value to technical assistance providers and SEA and LEA staff in improving the adoption and implementation of evidence-based practice. 
  • Finally, the ED developed revised evidence definitions and related selection criteria for competitive programs that align with ESSA to streamline and clarify provisions for grantees. These revised definitions align with ED’s suggested criteria for states’ implementation of ESSA’s four evidence levels, included in ED’s non-regulatory guidance, Using Evidence to Strengthen Education Investments. ED also developed a fact sheet to support internal and external stakeholders in understanding the revised evidence definitions. This document has been shared with internal and external stakeholders through multiple methods, including the Office of Elementary and Secondary Education ESSA technical assistance page for grantees.
Score
6
U.S. Agency for International Development
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • USAID has developed a draft agency-level evidence framework to clarify evidence standards for different decisions, including those related to funding. The draft was shared for voluntary comment, and is being further updated. Once finalized, it will be published as a suggested help guide for USAID staff and continue to be refined and updated as needed.  
  • USAID’s evidence standards are embedded within its policies and include requirements for the use of evidence in strategic planning, project design, activity design, program monitoring, and evaluation. USAID has a Scientific Research Policy that sets out quality standards for research across the Agency. USAID’s Program Cycle Policy requires the use of evidence and data to assess the development context, challenges, potential solutions, and opportunities in all of USAID’s country strategies. Specific programs, such as the Development Innovation Ventures (DIV) use evaluation criteria based on evidence of causal impact, leading to scaling which innovations work for greater cost effectiveness. As USAID’s flagship open innovation program, DIV helps to find, test, and scale innovative solutions to any global development challenge from anyone, anywhere. By backing proven innovations, driven by rigorous evidence and ongoing monitoring, USAID’s DIV program has proven to impact millions of lives at a fraction of the usual cost. Based on recent research announced in October 2020, the early portfolio of DIV grants (covering 2010-2012) has produced $17 in social benefit per every dollar spent by USAID. This research was led by Dr. Michael Kremer, a Nobel Prize-winning economist who is part of DIV’s team. Further, GAO found in their December 2019 report Evidence-Based Policymaking: EVIDENCE-BASED POLICYMAKING Selected Agencies Coordinate Activities, but Could Enhance Collaboration that USAID reflects leading practices for collaborating when building and assessing evidence. DIV’s evaluation criteria is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale.
    1. Evidence of Impact: DIV supports the piloting and rigorous testing of innovations and helps those innovations that have successfully demonstrated impact to transition to scale. DIV looks for different indicators of impact depending on the stage of financing that the applicant is seeking and on whether the innovation has a public or commercial pathway to scale.
      1. Public Pathway to Scale: To support later-stage innovations to transition to scale with public financing, including by government or philanthropic funding, DIV requires applicants to show rigorous evidence of causal impact on improving the lives of people living in poverty. DIV does not require applicants for earlier-stage awards for piloting and testing of innovations to show rigorous evidence of causal impact and will issue awards for earlier-stage innovations to help innovations get to the point at which they have this rigorous evidence of impact. Measurement of impact can encompass either final outcomes or impacts that can be taken as worthwhile objectives per se (e.g., reducing infant mortality). Measurement of impact can also encompass improvements to intermediate outcomes on the causal chain for solutions that have been previously demonstrated to have a causal impact on final outcomes (e.g., increasing vaccination rates, which have an existing evidence base of causal impact on improved health).
      2. Commercial Pathway to Scale: To support later-stage innovations to transition to scale commercially, DIV requires applicants to show evidence of long-run market viability. DIV typically accepts customer willingness to pay the full costs of the innovation as such evidence. DIV does not require applicants for earlier-stage awards for piloting and testing to show such evidence and will issue awards for earlier-stage innovations to get to the point where they have this evidence. For example, an applicant may use a Stage 1 award to gather evidence on consumer reaction to and willingness to pay for the product. As an applicant progresses to higher stages of financing, DIV expects the applicant to demonstrate an increasingly compelling case for long-run market viability. In addition, DIV requires applicants at all stages to show stage-appropriate data on usage, or in some situations on proxies for social outcomes that track how the innovation is improving the lives of people living in poverty.
    2. Cost-Effectiveness. DIV seeks innovations that deliver more development impact per dollar than existing alternative solutions. Cost-effectiveness is a function of both cost and impact and does not mean that an innovation is the cheapest solution. Instead, an innovation can increase its cost-effectiveness either by reducing its cost or by increasing its impact. DIV recognizes that costs of an innovation at scale may be different than costs generated through earlier testing phases, so applicants should provide data around current and projected costs.
    3. Pathways to Scale. DIV’s ultimate goal is to support impactful development solutions that will scale to improve the lives of at least a million people in a financially sustainable manner. Accordingly, innovations must have a potential pathway to scale through commercial scaling, public sector scaling, or scaling through a hybrid of the two. DIV recognizes that innovations can take a variety of pathways to scale but expects that they will be financially sustainable and ultimately grow without continued DIV support. For more information on the terms “pathways to scale” and “financial sustainability,” see the glossary at the end of this APS.
6.2 Did the agency have a common evidence framework for funding decisions?
  • USAID has a draft agency-level evidence framework to clarify evidence definitions, principles and approaches for different decisions, including those related to funding. The framework has been posted for review and comment by external stakeholders. In addition, there are specific types of programs at the sub-agency level that do use evidence framework or standards to make funding decisions. For example, the Development Innovation Ventures (DIV) uses a tiered funding approach to find, test, and scale evidence-based innovations. DIV’s grants include: Stage 1 for piloting (up to $200,000); Stage 2 for testing (up to $1,500,000); Stage 3 for scaling (up to $15,000,000); and “evidence grants” (up to $1,500,000) for research to determine causal impact of certain interventions. In particular for Stage 2 grants, DIV requires evidence of impact that must be causal and rigorous–the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period. There must be significant demonstrated demand for the innovation.
  • DIV’s evaluation criteria for its funding is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale. DIV’s expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators. Most DIV grants are fixed amount awards, a unique type of federal grant instrument that is tailor-made for pay-for-results approaches. Fixed amount awards are structured by paying for milestones achieved, which emphasizes performance (not just compliance) and reduces some administrative burden for all parties (see 2 CFR 200.201(b)).
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
  • USAID does have an Agency-wide repository for development information (including evaluation reports and other studies) which is available to the public at the Development Experience Clearinghouse. In addition, USAID uses the International Initiative for Impact Evaluations (3ie) database of impact evaluations relevant to development topics (including over 4,500 entries to date), knowledge gap maps, and systematic reviews that pull the most rigorous evidence and data from across international development donors. 3ie also houses a collection of institutional policies and reports that examine findings from its database of impact evaluations on overarching policy questions to help policymakers and development practitioners improve development impact through better evidence. 
  • USAID’s Agency Programs and Functions policy designates technical bureaus responsible for being the repository for latest information in the sectors they oversee; prioritizing evidence needs and taking actions to build evidence; and disseminating that evidence throughout the agency for those sectors. Several USAID bureaus and sectors have created user friendly tools to disseminate information on evidence-based solutions. These include, but are not limited to:
  • USAID also led a data-harmony initiative across the industry with other countries called the Global Innovation Exchange which surfaces, validates, and shares a repository of over 16,000 development relevant solutions across all actors, players and locations. 
  • Finally, USAID recently applied Natural Language Processing Text Analysis to analyze unstructured data from the previous ten years of Evaluation Reports published by USAID and identify countries which used specific language and terminology related to racial and ethnic equity. This review included 1,208 evaluation reports and 2,525 final contractor/grantee reports that were available on USAID’s public Development Experience Clearinghouse (DEC) and converted to machine readable format. To develop an algorithm to find the most relevant information, the team consulted with experts from across the agency working on inclusive development and DEIA issues to develop a lexicon of terms that together with other factors were tested and found to identify relevant documents.
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • USAID’s approach to Collaborating, Learning, and Adapting (CLA) helps ensure that evidence from evaluation of USAID programming is shared with and used by staff, partners, and stakeholders in the field. USAID requires a dissemination plan and post-evaluation action plan for each evaluation, and USAID field staff are encouraged to co-create evaluation action plans with key stakeholders based on evaluation evidence. USAID collects examples through the CLA Case Competition, held annually, which recognizes implementers, stakeholders, and USAID staff for their work generating and sharing technical evidence and learning from monitoring and evaluation. It is another way that the Agency encourages evidence-based practices among its stakeholders.
  • USAID also periodically holds large learning events with partners and others in the development community around evidence including, but not limited to, Evaluation Summits, engagement around the Agency Learning Agenda, and Moving the Needle. These gatherings are designed to build interest in USAID’s evidence, build capacity around applying that evidence and learning, and elicit evidence and learning contributions.
  • USAID created and led the “Million Lives Collaborative” coalition, with more than 30 partners, which has identified more than 100 social entrepreneurs who have at least a million customers in order to share the learning that this successful cohort has had and better inform how USAID funding can assist more social entrepreneurs to grow successfully and rapidly. This unique learning platform brings donors, funders, governments, and the entrepreneurial community to the table together to learn and iterate on our approaches.
  • USAID recently published the Evaluations at USAID Dashboard, which provide evidence of evaluation use by Missions, and opportunities for peer learning.
Score
8
Administration for Children and Families (HHS)
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • ACF has established a common evidence framework adapted for the human services context from the framework for education research developed by the U.S. Department of Education and the National Science Foundation. The ACF framework, which includes the six types of studies delineated in the ED/NSF framework, aims to (1) inform ACF’s investments in research and evaluation and (2) clarify for potential grantees’ and others’ expectations for different types of studies.
6.2 Did the agency have a common evidence framework for funding decisions?
  • While ACF does not have a common evidence framework across all funding decisions, certain programs do use a common evidence framework for funding decisions. For example:
  • The Family First Prevention Services Act (FFPSA) enables states to use funds for certain evidence-based services. In April 2019, ACF published the Prevention Services Clearinghouse Handbook of Standards and Procedures, which provides a detailed description of the standards used to identify and review programs and services in order to rate programs and services as promising, supported, and well-supported practices.
  • The Personal Responsibility Education Program Competitive Grants were funded to replicate effective, evidence-based program models or substantially incorporate elements of projects that have been proven to delay sexual activity, increase condom or contraceptive use for sexually active youth, and/or reduce pregnancy among youth. Through a systematic evidence review, HHS selected 44 models that grantees could use, depending on the needs and age of the target population of each funded project.
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
  • ACF sponsors several user-friendly tools that disseminate and promote evidence-based interventions. Several evidence reviews of human services interventions have disseminated and promoted evidence-based interventions by rating the quality of evaluation studies and presenting results in a user-friendly searchable format. Current evidence reviews include: 1) Home Visiting Evidence of Effectiveness (HomVEE) Home Visiting Evidence of Effectiveness, which provides an assessment of the evidence of effectiveness for early childhood home visiting models that serve families with pregnant women and children from birth to kindergarten entry; 2) The Pathways to Work Evidence Clearinghouse, a user-friendly website that reports on “projects that used a proven approach or a promising approach in moving welfare recipients into work, based on independent, rigorous evaluations of the projects” and allows users to search for interventions based upon characteristics of the clients served by the intervention; and 3) ACF’s Title IV-E Prevention Services Clearinghouse, whose easily accessible and searchable website allows users to find information about mental health and substance abuse prevention and treatment services, in-home parent skill-based programs, and kinship navigator services designated as “promising,” “supported,” and “well-supported” practices by an independent systematic review.
  • Additionally, most ACF research and evaluation projects produce and widely disseminate short briefs, tip sheets, or infographics that capture high-level findings from the studies and make information about program services, participants, and implementation more accessible to policymakers, practitioners, and other stakeholders. For example, the Pathways for Advancing Careers and Education (PACE) project released a series of nine short briefs to accompany the implementation and early impact reports that were released for each of the nine PACE evaluation sites.
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • ACF’s evaluation policy states that it is important for evaluators to disseminate research findings in ways that are accessible and useful to policymakers, practitioners, and the communities that ACF serves and that OPRE and program offices will work in partnership to inform potential applicants, program providers, administrators, policymakers, and funders through disseminating evidence from ACF-sponsored and other good quality evaluations. OPRE research contracts include a standard clause requiring contractors to develop a dissemination plan during early project planning to identify key takeaways, target audiences, and strategies for most effectively reaching the target audiences. OPRE’s dissemination strategy is also supported by a commitment to plain language; OPRE works with its research partners to ensure that evaluation findings and other evidence are clearly communicated. OPRE also has a robust dissemination function that includes the OPRE website, including a new blog, an OPRE e-newsletter, and social media presence on Facebook, Twitter, Instagram, and LinkedIn.
  • OPRE also hosts an annual “Evaluation and Monitoring 101” training for ACF staff to help agency staff better understand how to design, conduct, and use findings from program evaluation and performance monitoring, ultimately building the capacity of agency staff and program offices to use evaluation research and data analysis to improve agency operations.
  • OPRE biennially hosts two major conferences, the Research and Evaluation Conference on Self-Sufficiency (RECS) and the National Research Conference on Early Childhood (NRCEC) to share research findings with researchers and with program administrators and policymakers at all levels. OPRE also convenes the Network of Infant and Toddler Researchers (NITR) which brings together applied researchers with policymakers and technical assistance providers to encourage research-informed practice and practice-informed research; and the Child Care and Early Education Policy Research Consortium (CCEEPRC) which brings together researchers, policymakers, and practitioners to discuss what we are learning from research that can help inform policy decisions for ACF, States, Territories, localities, and grantees and to consider the next steps in early care and education (ECE) research.
  • The Children’s Bureau (CB) sponsors the recurring National Child Welfare Evaluation Summit to bring together partners from child welfare systems and the research community to strengthen the use of data and evaluation in child welfare; disseminate information about effective and promising prevention and child welfare services, programs, and policies; and promote the use of data and evaluation to support sound decision-making and improved practice in state and local child welfare systems. ACF also sponsors several:
Score
7
AmeriCorps
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • AmeriCorps uses the same standard scientific research methods and designs for all of its studies and evaluations following the model used by clearinghouses like Department of Education’s What Works Clearinghouse, the Department of Labor’s Clearinghouse for Labor Evaluation and Research, and the Department of Health and Human Services’ Home Visiting Evidence of Effectiveness project.
6.2 Did the agency have a common evidence framework for funding decisions?
  • AmeriCorps has a common evidence framework for funding decisions in the Senior Corps and AmeriCorps State and National programs. This framework, which is articulated in the AmeriCorps State and National program notice of funding, includes the following evidence levels: pre-preliminary, preliminary, moderate, and strong.
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
  • The AmeriCorps Evidence Exchange is a virtual repository of reports and resources intended to help AmeriCorps grantees and other interested stakeholders find information about evidence- and research-based national service programs. Examples of the types of resources available in the Evidence Exchange include research briefs that describe the core components of effective interventions such as those in the areas of education, economic opportunity, and health.
  • R&E also creates campaigns and derivative products to distill complex report findings and increase their utility for practitioners (for example, this brief on a study about the health benefits of Senior Corps). R&E has categorized reports according to their research design, so that users can easily search for experimental, quasi-experimental, or non-experimental studies, and those that qualify for strong, moderate, or preliminary evidence levels.
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • AmeriCorps has an agency-wide approach to promoting the use of evidence-based practices by the field and employs a variety of strategies including evidence briefs, broad-based support to national service organizations, and targeted technical assistance to grantees. First, R&E creates campaigns and derivative products to distill complex report findings and increase their utility for practitioners (for example, this brief on a study about the health benefits of Senior Corps). Second, AmeriCorps has created user-friendly research briefs that describe the core components of effective interventions in the areas of education, economic opportunity, and health. These briefs are designed to help grantees (and potential grantees) adopt evidence-based approaches. Third, R&E funds a contractor to provide AmeriCorps grantees with evaluation capacity building support; R&E staff are also available to State Commissions for their evaluation questions and make resources (e.g., research briefs summarizing effective interventions, online evaluation planning and reporting curricula) available to them and the general public. Fourth, AmeriCorps funds and participates in grantee conferences that include specific sessions on how to incorporate evidence and data into national service programs. Fifth, as part of the AmeriCorps State and National FY20 application process, AmeriCorps provided technical assistance to grantees on using evidence-based practices through webinars and calls. R&E and AmeriCorps conducted a process evaluation of granteeswith varied replication experiences to produce a series of products designed to help grantees implement evidence-based interventions (including a forthcoming article in The Foundation Review). SeniorCorps continues to encourage and support the use of evidence-based programs, as identified by the HHS’s Administration for Community Living, by its grantee organizations.
Score
9
U.S. Department of Labor
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • DOL’s Clearinghouse for Labor Evaluation and Research (CLEAR) evidence guidelines, which describe quality standards for different types of studies, are applied to all independent evaluations, including all third party evaluations of DOL programs, determined eligible for CLEAR’s evidence reviews across different topic areas. Requests for proposals also indicate these CLEAR standards should be applied to all Chief Evaluation Office (CEO) evaluations when considering which designs are the most rigorous and appropriate to answer specific research questions.
  • In addition, the DOL Evaluation Policy identifies principles and standards for evaluation planning and dissemination. Additionally, DOL collaborates with other agencies (U.S. Department of Health and Human Services (HHS), the U.S. Department of Education’s Institute of Education Sciences (IES), the National Science Foundation (NSF), and the Corporation for National and Community Service (CNCS)) to develop technological procedures to link and share reviews across clearinghouses. 
6.2 Did the agency have a common evidence framework for funding decisions?
  • DOL uses the CLEAR evidence guidelines and standards to make decisions about discretionary program grants awarded using evidence-informed or evidence-based criteria. The published guidelines and standards are used to identify evidence-based programs and practices and to review studies to assess the strength of their causal evidence or to do a structured evidence review in a particular topic area or timeframe to help inform agencies what strategies appear promising and where gaps exist.
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
  • DOL’s CLEAR is an online evidence clearinghouse. CLEAR’s goal is to make research on labor topics more accessible to practitioners, policymakers, researchers, and the public more broadly, so that it can inform their decisions about labor policies and programs. CLEAR identifies and summarizes many types of research, including descriptive statistical studies and outcome analyses, implementation studies, and causal impact studies. For causal impact studies, CLEAR assesses the strength of the design and methodology in studies that look at the effectiveness of particular policies and programs. CLEAR’s study summaries and icons, found in each topic area, can help users quickly and easily understand what studies found and how much confidence to have in the results.
  • CLEAR’s search tool allows users to find studies based on target population, including race and other demographic characteristics.   
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • DOL promotes the utilization of evidence-based practices in a variety of ways. For example, the Employment and Training Administration (ETA) maintains a user-friendly database and a community of practice, Workforce System Strategies (WSS) that highlights the use of evidence-based interventions and the Evaluation and Research Hub (EvalHub), respectively. WSS is a comprehensive database of over 1,400 profiles that summarize a wide range of findings from reports, studies, technical assistance tools and guides that support program administration and improvement. The EvalHub is a community of practice created to support evidence and evaluation-capacity building efforts within state workforce development programs. In another effort to promote evidence-based practices, ETA has supported an Applied Data Analytics program offered through the Coleridge Initiative for multiple teams from state workforce agencies. DOL agencies, like ETA, are also making a concerted effort to help states build evaluation capacity to meet the program evaluation requirements for the Reemployment Services and Eligibility Assessment (RESEA) program through tools such as RESEA program evaluation technical assistance (RESEA EvalTA). 
Score
3
U.S. Dept. of Housing & Urban Development
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • HUD’s Program Evaluation Policy defines standards that prioritize rigorous methods for research and evaluation covering impact evaluations; implementation of process evaluations; descriptive studies; outcome evaluations; and formative evaluations; and both qualitative and quantitative approaches. It also provides for timely dissemination of such evidence to stakeholders. HUD updated the Program Evaluation Policy in August 2021 to address the rigorous inclusion of qualitative evidence and considerations related to equity.
  • HUD’s Program Evaluation Policy commits HUD to using “the most rigorous methods that are appropriate to the evaluation question and feasible within budget and other constraints… Where feasible, research should employ a treatment group and a counterfactual …. For such causal questions, experimental approaches are preferred.” The Policy also establishes standards for transparency: “HUD will release methodologically valid evaluations without regard to the findings. Evaluation reports must describe the methods used, including strengths and weaknesses, and discuss the generalizability of the findings. Evaluation reports must present comprehensive results, including favorable, unfavorable, and null findings.”
  • The consequences of HUD’s commitment to rigor may be seen in HUD’s priority research projects:
    • IWISH, a cluster-randomized health intervention at 40 treatment housing developments and 80 control housing developments
    • The long-term follow-up to the Family Options Study, which randomized over 2000 homeless families to 4 treatment regimes.
    • The Rent Reform Study, which randomized assisted non-elderly non-disabled households to two different rent structures in 4 sites.
    • The First Time Homebuyer Education and Counseling Demonstration, which randomized more than 5,800 prospective first-time homebuyers from 28 large metropolitan areas.
    • The Moving to Work Flexibilities study, which randomized 31 PHAs to partial deregulation versus status quo.
    • The Family Self-Sufficiency demonstration, which randomized voucher holders at 18 sites and followed them for 7 years.
    • The Rental Assistance Demonstration evaluation, which compared 24 RAD-conversion projects with 48 matched non-converted public housing projects on financial indicators and changes in the physical stock.
    • The Voucher Mobility Demonstration, which will randomize 14,000 voucher holders in 9 sites to receive neighborhood mobility counseling (or not).
  • Further, HUD’s commitment to independence (in the Evaluation Policy Statement) has led us to provide PII to dozens of independent research teams across the country, who match our data with other sources to measure the consequences of housing assistance for tenants of assisted housing. HUD’s commitment to independence has also led us to long-term matching of our data with surveys sponsored by the National Center for Health Statistics, to monitor trends in disease conditions among the assisted population.
6.2 Did the agency have a common evidence framework for funding decisions?
  • HUD seeks to employ tiered evidence in funding decisions by embedding implementation and impact evaluations in funding requests for program initiatives, including major program demonstrations that employ random assignment methods. These include the Moving To Work Expansion demonstration, the Rental Assistance Demonstration, the Rent Reform Demonstration, the Family Self-Sufficiency Demonstration, the Housing Counseling Demonstration, and the Family Options Demonstration, described above. Such trials provide robust evidence to inform scale-up funding decisions.
  • HUD extended its standardized data collection and reporting framework, Standards for Success, to additionaldiscretionary grant programs in FY19. The framework consists of a repository of data elements that participating programs use in their grant reporting, creating common definitions and measures across programs for greater analysis and coordination of services.
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • HUD provides resources and assistance to support community partners in evidence-based practice through the HUD Exchange web portal and through technical assistance. PD&R provides the public, policymakers, and practitioners with evidence of what works primarily through HUD USER, a portal and web store for program evaluations, case studies, and policy analysis and research; the Regulatory Barriers Clearinghouse; and through initiatives such as Sustainable Construction Methods in Indian Country and the Consumer’s Guide to Energy-Efficient and Healthy Homes. This content is designed to provide current policy information, elevate effective practices, and synthesize data and other evidence in accessible formats such as Evidence Matters. Through these resources, researchers and practitioners can see the full breadth of work on a given topic (e.g., rigorous established evidence, case studies of what has worked in the field, and new innovations currently being explored) to inform their work.
  • PD&R has increased emphasis on generating interim reports during long-term impact evaluations. Such interim reports provide practitioners with early findings about implementation practice and outcomes that can inform their own program designs. A recent example is the Interim Report from HUD’s Supportive Services Demonstration, published in 2020. A second Interim Report from this study is forthcoming in 2021.
  • Community Compass technical assistance for urban, rural, and tribal partners is designed to facilitate understanding of community and housing development issues in a way that cuts across program silos. It supports them in evaluation, evidence-building, integrating knowledge management principles, and sharing practices.
Score
5
Administration for Community Living (HHS)
6.1 Did the agency have a common evidence framework for researchand evaluation purposes?
  • ACL defines evidence-based programs on its website. ACL’s National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) has two principal frameworks that guide and inform the generation of new knowledge and products. The stages of research framework is used to guide, inform, and track the creation of new knowledge which in turn becomes part of the larger disability evidence base. The stages of development framework is used to guide, inform, and track the development of new products and technologies. Both of these frameworks are codified in federal regulations and are described on NIDILRR’s Frameworks’ page on the ACL website. The stages of research framework is codified in 45 CFR 1330.4 while the stages of the development framework codified in 45 CFR 1330.5.
6.2 Did the agency have a common evidence framework for funding decisions?
  • The Older Americans Act requires the use of evidence-based programming in Title III-D-funded activities: Disease Prevention and Health Promotion Services. In response, ACL developed a definition of the term evidence-based, and created a website containing links to a range of resources for evidence-based programs. This is a common evidence framework used for Older Americans Act funded activities. For programs that are not legislatively required to use evidence-based models, through its funding process ACL requires all programs to provide clear justification and evidence (where available) that proposed projects will achieve their stated outcomes. In 2018 ACL developed a tool to help a small number of program officers assess grantee progress towards the stated goals of their grants. Using the tool program officers have instituted corrective actions or required underperforming grantees to relinquish grant funds. ACL is developing similar tools for several other grant programs with the intention of rolling out new guidance for program officers in 2023.
6.3 Did the agency have a clearinghouse(s) or a user-friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • ACL works through its resource centers to help grantees use evidence to drive improvements in outcomes for older adults and individuals with disabilities. For example, with funding from ACL, the National Center on Aging (NCOA), in collaboration with the Evidence-Based Leadership Council, led an innovative vetting process to increase the number of programs available to ACL’s aging network that meet the Title III-D evidence-based criteria. This process resulted in adding six new health promotion programs and three new programs for preventing falls. The Alzheimer’s Disease Supportive Services Program (ADSSP) funds competitive grants to expand the availability of evidence-based services that support persons with Alzheimer’s disease and related dementia (ADRD) and their family caregivers. Extensive evaluation of the National Chronic Disease Self-Management Education (CDSME) and Falls Prevention database helped generate important insights for potential new ACL applicants in preparing their applications using data-driven estimation procedures for participant and completion targets ACL also funded several grants, such as the Lifespan Respite Care Program: State Program Enhancement Grants and Disability and Rehabilitation Research Projects (DRRP) Program: Chronic Disease Management for People with Traumatic Brain Injury (TBI) which are designed, in part, to develop an evidence base for respite care and related services and contribute to the evidence base upon which people with TBI and their health care providers can employ effective chronic disease management practices respectively. In addition, NIDILRR provides the Rehabilitation Measures Database (RMD) which is a Knowledge Translation on-line resource with succinct summaries of instruments relevant to rehabilitation populations that provides evidence-based summaries that include concise descriptions of each instrument’s psychometric properties, reliability, validity, sensitivity, instructions for administering and scoring, and a representative bibliography with citations.
Score
5
Substance Abuse and Mental Health Services Administration (HHS)
6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • The SAMHSA Strategic Plan FY19-FY23 (pp. 20-23) outlines five priority areas to carry out the vision and mission of SAMHSA, including Priority 4: Improving Data Collection, Analysis, Dissemination, and Program and Policy Evaluation. This Priority includes three objectives: 1) Develop consistent data collection strategies to identify and track mental health and substance use needs across the nation; 2) Ensure that all SAMHSA programs are evaluated in a robust, timely, and high-quality manner; and 3) Promote access to and use of the nation’s substance use and mental health data and conduct program and policy evaluations and use the results to advance the adoption of evidence-based policies, programs, and practices. 
  • When possible, SAMHSA incorporates qualitative data into its framework through the feedback received by the Project Officers, through annual narrative reports submitted by grantees, key informant interviews, site visits and focus groups. SAMHSA is in regular communication with grantees and the state/community programs regarding opportunities and challenges. SAMHSA is developing a formal process in FY22 for incorporating qualitative feedback into its evaluation process.
  • Within this strategic plan, SAMHSA is committed to Behavioral Health Equity: the right to access quality health care for all populations regardless of the individual’s race, ethnicity, gender, socioeconomic status, sexual orientation, or geographical location. This includes access to prevention, treatment, and recovery services for mental and substance use disorders. Advancing health equity involves ensuring that everyone has a fair and just opportunity to be as healthy as possible. In conjunction with quality services, equity involves addressing social determinants, such as employment and housing stability, insurance status, proximity to services, culturally responsive care–all of which have an impact on behavioral health outcomes.
6.2 Did the agency have a common evidence framework for funding decisions?
  • SAMHSA has universal language about using evidence-based practices (EBPs) that is included in its Funding Opportunity Announcements (FOAs) also known as NOFO (or Notice of Funding Opportunity) also called Funding (entitled Using Evidence-Based Practices (EBPs)). This language includes acknowledgement that, “EBPs have not been developed for all populations and/or service settings” thus encouraging applicants to “provide other forms of evidence” that a proposed practice is appropriate for the intended population.
  • Specifically, the language states that applicants should:
    1. document that the EBPs chosen are appropriate for intended outcomes;
    2. explain how the practice meets SAMHSA’s goals for the grant program;
    3. describe any modifications or adaptations needed for the practice to meet the goals of the project;
    4. explain why the EBP was selected;
    5. justify the use of multiple EBPs, if applicable; and
    6. discuss training needs or plans to ensure successful implementation.
  • Lastly, the language includes resources the applicant can use to understand EBPs. Federal grants officers work in collaboration with the SAMHSA Office of Financial Resources to ensure that grantee funding announcements clearly describe the evidence standard necessary to meet funding requirements.
  • SAMHSA developed a manual, Developing a Competitive SAMHSA Grant Application, which explains information applicants will likely need for each section of the grant application. The manual has two sections devoted to evidence-based practices (p. 8, p. 26), including: 1) A description of the EBPs applicants plan to implement; 2) Specific information about any modifications applicants plan to make to the EBPs and a justification for making them; and 3) How applicants plan to monitor the implementation of the EBPs. In addition, if applicants plan to implement services or practices that are not evidence-based, they must show that these services/practices are effective.
  • For the 10% set aside in the Mental Health Block Grant, states are directed to use evidence-based programs. SAMHSA provides guidance to states based on evidence. For First-Episode Psychosis (FEP), SAMHSA’s recommendation is to develop a state FEP program based on the Coordinated Specialty Care model, as evaluated by NIMH. For example, FEP program OnTrackNY is an evaluated model that is recommended based on evidence of success (see question 9 for more information on the EBP supported for the Block Grants).
  • SAMHSA recognizes that one size does not fit all, therefore although grantees are encouraged to consider the EBP on the SAMHSA website, grantees must provide information on the EBP they play to implement. Their description should reference why each EBP is appropriate for the problem area addressed by the grant as well as the specific population(s) of focus. SAMHSA also asks for specific information about any modifications planned to make the EBPs and a justification for making these modifications as well as how the grantee will monitor the implementation of the EBPs to ensure they are implemented according to EBP guidelines. 
  • The Evidence-Based Practices Resource Center provides communities, clinicians, policy-makers and others with the information and tools to incorporate evidence-based practices into their communities or clinical settings. The Evidence-Based Practices Resource Center (EBPRC) contains a collection of scientifically-based resources for a broad range of audiences, including Treatment Improvement Protocols, toolkits, resource guides, clinical practice guidelines, and other science-based resources. The retooled EBPRC neither accepts open submissions from outside program developers nor rates individual programs. 
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • SAMHSA promotes the utilization of evidence-based practices. Within grant applications, SAMHSA encourages innovation. For example, the FY20-21 Substance use block grant application includes the following language, “There is increased interest in having a better understanding of the evidence that supports the delivery of medical and specialty care including M/SUD services. Over the past several years, SAMHSA has collaborated with CMS, HRSA, SMAs, state M/SUD authorities, legislators, and others regarding the evidence of various mental and substance misuse prevention, treatment, and recovery support services. States and other purchasers are requesting information on evidence-based practices or other procedures that result in better health outcomes for individuals and the general population. While the emphasis on evidence-based practices will continue, there is a need to develop and create new interventions and technologies and in turn, to establish the evidence. SAMHSA supports states’ use of the block grants for this purpose. The NQF and the IOM recommend that evidence play a critical role in designing health benefits for individuals enrolled in commercial insurance, Medicaid, and Medicare. To respond to these inquiries and recommendations, SAMHSA has undertaken several activities. SAMHSA’s Evidence-Based Practices Resource Center assesses the research evaluating an intervention’s impact on outcomes and provides information on available 43 resources to facilitate the effective dissemination and implementation of the program. SAMHSA’s Evidence-Based Practices Resource Center provides the information & tools needed to incorporate evidence-based practices into communities or clinical settings.
  • SAMHSA’s Evidence-Based Practices Resource Center (EBPRC) aims to provide communities, clinicians, policymakers, and others in the field with the information and tools they need to incorporate evidence-based practices into their communities or clinical settings. The Resource Center contains a collection of science-based resources, including Treatment Improvement Protocols, toolkits, resource guides, and clinical practice guidelines, for a broad range of audiences. As of June 2021, the Resource Center includes 150 items, including 15 data reports, 23 toolkits, 6 fact sheets, and 88 practice guides. 
  • The Mental Health Technology Transfer Center (MHTTC) Network engages with organizations and treatment practitioners involved in the delivery of mental health services to strengthen their capacity to deliver effective evidence-based practices to individuals, including the full continuum of services spanning mental illness prevention, treatment, and recovery support. The State Targeted Response Technical Assistance (STR-TA), known as the Opioid Response Network, was created to support efforts to address opioid use disorder prevention, treatment, and recovery, and to provide education and training at the local level in evidence-based practices.
  • The Knowledge Application Program (KAP) supports the professional development of behavioral health workers and provides information and resources on best practices. Specifically, KAP provides substance use treatment professionals with publications that contain information on best treatment practices.
Back to the Standard

Visit Results4America.org

Copyright 2024 Results for America. All rights reserved. Privacy Policy

Site by Wide Eye Creative