2021 Federal Standard of Excellence

Innovation

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY21? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements)

Score
7
Millennium Challenge Corporation
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • MCC supports the creation of multidisciplinary country teams to manage the development and implementation of each compact and threshold program. Teams meet frequently to gather evidence, discuss progress, make project design decisions, and solve problems. Prior to moving forward with a program investment, teams are encouraged to use the lessons from completed evaluations to inform their work going forward.
  • MCC recently launched its second-ever internal Millennium Efficiency Challenge (MEC) designed to tap into the extensive knowledge of MCC’s staff to identify efficiencies and innovative solutions that can shorten the compact and threshold program development timeline while maintaining MCC’s rigorous quality standards and investment criteria. This year MCC is seeking to implement proposed innovations around making the compact development process more efficient, among other challenges.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • MCC’s approach to development assistance hinges on its innovative and extensive use of evidence to inform investment decisions, guide program implementation strategies, and assess and learn from its investment experiences. As such, MCC’s Office of Strategic Partnerships offers an Annual Program Statement (APS) opportunity that allows MCC divisions and country teams to tap the most innovative solutions to new development issues. In FY21, the Monitoring and Evaluation division, using MCC’s APS and traditional evaluation firms, continues to pilot partnerships with academics and in-country think tanks to leverage innovative, lower cost data technologies across sectors and regions. These include:
    • using satellite imagery in Sri Lanka to measure visible changes in investment on land to get early indications if improved land rights are spurring investment;  
    • leveraging big data and cell phone applications in Colombo, Sri Lanka to monitor changes in traffic congestion and the use of public transport; independently measuring power outages and voltage fluctuations using cell phones in Ghana, where utility outage data is unreliable, and where outage reduction is a critical outcome targeted by the Compact;
    • using pressure loggers on piped water at the network and household levels to get independent readings on access to water in Dar es Salaam, Tanzania; and using remote sensing to measure water supply in water kiosks in Freetown, Sierra Leone.
  • These innovations in evidence generation have been ever-more critical in the past year given the inability to do many data collection activities in person. MCC has utilized local data collection and better technology to maintain evidence generation. For example, MCC partnered with the University of Colorado on their use of satellite connected sensors on water kiosks built by the Sierra Leone Threshold Program’s Water Project. MCC partnered with a consortium of the University of Colorado and SweetSense Inc. technology firm to collect high frequency monitoring data using emerging and cost-effective technologies to understand the state of water service from the water kiosks constructed by the project. The partnership provided significant flexibility to collaboratively shape how available technology can suit MCC’s monitoring needs, including in data-challenged environments. It also offered an example of how other MCC water projects can capitalize on the use of similar technology tools to collect more reliable data more frequently.
  • MCC regularly engages in implementing test projects as part of its overall compact programs. A few examples include: (1) in Morocco, an innovative pay-for-results mechanism to replicate or expand proven programs that provide integrated support; (2) a “call-for-ideas” in Benin for information regarding potential projects that would expand access to renewable off-grid electrical power; and (3) a regulatory strengthening project in Sierra Leone that includes funding for a results-based financing system.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Although MCC rigorously evaluates all program efforts, MCC takes special care to ensure that innovative or untested programs are thoroughly evaluated. In addition to producing final program evaluations, MCC is continuously monitoring and evaluating all programs throughout the program lifecycle, including innovation efforts, to determine if mid-program course-correction actions are necessary. This interim data helps MCC continuously improve its innovation efforts so that they can be most effective and impactful. Although 32% of MCC’s evaluations use random-assignment methods, all of MCC’s evaluations – both impact and performance – use rigorous methods to achieve the three-part objectives of accountability, learning, and results in the most cost-effective way possible.  
Score
6
U.S. Department of Education
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • In FY19, the Office of Elementary and Secondary Education (OESE) made strategic investments in innovative educational programs and practices and administered discretionary grant programs. In FY19, the Innovation and Improvement account received $1.035 billion. ED reorganized in 2019, consolidating the Office of Innovation and Improvement into the Office of Elementary and Secondary Education. To lead and support innovation within the reorganized OESE, ED created the Evidence-Based Policy (EBP) team. EBP teams work within OESE and with colleagues across the agency to develop and expand efforts to inform policy and improve program practices.
  • The Innovation and Engagement Team in the Office of the Chief Data Officer promotes data integration and sharing, making data accessible, understandable, and reusable; engages the public and private sectors on how to improve access to the Department’s data assets; develops solutions that provide tiered access based on public need and privacy protocols; develops consumer information portals and products that meet the needs of external consumers; and partners with OCIO and ED data stewards to identify and evaluate new technology solutions for improving collection, access, and use of data. This team led the ED work on the ESF transparency portal, highlighted above, and also manages and updates College Scorecard. The team is currently developing ED’s first Open Data Plan and a playbook for data quality at ED. 
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • A team of analysts within the Evidence-Based Practices unit (EBP) in OESE works to support the use of evidence and data in formula and discretionary grantmaking. This unit promotes evidence consistent with relevant provisions of the Elementary and Secondary Education Act of 1965 (ESEA) as amended by Every Student Succeeds Act (ESSA). EBP supports offices as they seek ways to meaningfully incorporate evidence and data routinely as part of their grantmaking agenda and to operationalize the ESSA evidence framework for strengthening the effectiveness of ESEA investments within OESE programs. This unit engages with offices to support internal capacity by convening staff for review and discussion of the most recent and relevant resources to support the use of evidence internally and by ED grantees and responding to individual office requests for resources and information related to generation and use of evidence within their programs. Offices in OESE also invite members of the EBP to share and discuss available resources with their grantees at various grantee convenings.  
  • The Education Innovation and Research (EIR) program is ED’s primary innovation program for K-12 public education. EIR grants are focused on validating and scaling evidence-based practices and encouraging innovative approaches to persistent challenges. The EIR program incorporates a tiered-evidence framework that supports larger awards for projects with the strongest evidence base as well as promising earlier-stage projects that are willing to undergo rigorous evaluation. Lessons learned from the EIR program have been shared across the agency and have informed policy approaches in other programs.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • ED’s Experimental Sites Initiative is entirely focused on assessing the effects of statutory and regulatory flexibility in one of its most critical programs: Title IV Federal Student Aid programs. FSA collects performance and other data from all participating institutions, while IES conducts rigorous evaluations–including randomized trials–of selected experiments. Recent examples include ongoing work on Federal Work Study and Loan Counseling, as well as recently published studies on short-term Pell grants. 
  • The Education Innovation and Research (EIR) program, ED’s primary innovation program for K-12 public education, incorporates a tiered-evidence framework that supports larger awards for projects with the strongest evidence base as well as promising earlier-stage projects that are willing to undergo rigorous evaluation.
Score
7
U.S. Agency for International Development
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • The U.S. Global Development Lab became the Innovation, Technology, and Research Hub (ITR) under the new Bureau for Development, Democracy, and Innovation (DDI). DDI became operational within USAID on November 16, 2020. DDI is USAID’s central resource for providing world-class technical assistance to its Missions. DDI’s cross-cutting, global perspective will enable the Agency to anticipate and respond to evolving trends and issues, catalyze innovation and broaden our partnership base. DDI comprises four Centers and five Hubs. The Centers serve as USAID’s leaders for technical assistance in democracy and governance, economics and market development, education, energy, environment, human rights, and infrastructure. The Hubs provide Missions with expert guidance and training on cross-cutting development priorities, including gender equality, innovation, technology and research, private-sector engagement, and partnerships with diverse organizations, such as local and faith-based groups. 
  • Specifically, the ITR Hub serves as a central point for promoting and building capacity for innovation throughout development and national security strategies across USAID, the U.S. Government, and the international community. This includes helping to centrally coordinate the Agency’s innovation-related work with entrepreneurs, implementing partners, universities, donors, and others to test and scale innovative solutions and approaches to development problems around the world.  In addition to finding and supporting innovative solutions, the ITR Hub also works with other USAID Bureaus and Independent Offices to promote a culture of innovation across the Agency to enable it to be a more innovative organization itself. For example, this includes building internal capacity, skills, and outside-the-box thinking to structure and provide our funding in more creative and effective ways (e.g., using fixed amount awards as a grant instrument to pay for outcomes, not just inputs).
  • For innovations specific to a particular sector, Agency leadership has supported technical staff in surfacing groundbreaking ideas, such as how the Bureau for Global Health’s Center for Innovation and Impact (CII) used open innovation approaches to issue the Saving Lives at Birth Grand Challenge and identify promising, life-saving maternal and newborn health innovations. 
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • In FY20, USAID released its first Digital Strategy, moving to a “Digital by Default” position and USAID’s innovative approaches have helped get more than 40 million people in the developing world digital access. USAID’s New Partnerships Initiative (NPI) will allow USAID to work with a more diverse range of partners, strengthen existing partner relationships, and provide more entry points for organizations to work with the Agency. The principles behind NPI are outlined in the Agency’s first-ever Acquisition and Assistance (A&A) Strategy.
  • USAID and its partners have launched 41 innovative programming approaches including prizes, ventures, challenges, and Grand Challenges for Development since 2011. Across the Grand Challenges portfolio, partners have jointly committed over $535 million ($155 million from USAID) in grants and technical assistance for over 528 innovators in 107 countries. To date, more than $614 million in follow-on funding has been catalyzed from external sources, a key measure of success.
  • USAID’s investment in state-of-the-art geo and information intelligence centers mean that any program has the ability to leverage geospatial analysis and critical data sets to drive innovative solutions based on evidence and data. With over 20 programs experimenting with Artificial Intelligence and machine learning, and USAID’s strong work on digital finance and connectivity, the Agency is using technology to drive our programs farther and faster. USAID has also completed more than 1,500 Global Development Alliances, leveraging private sector in-kind or financial investments.
  • In addition, the Center for Innovation and Impact (CII)–the Bureau for Global Health’s dedicated innovation office–takes a business-minded approach to fast-tracking the development, introduction, and scale-up of health innovations that address the world’s most important health challenges, and assessing and adopting cutting-edge approaches (such as using unmanned aerial vehicles and artificial intelligence).
  • Feed the Future Partnering for Innovation partners with agribusinesses to help them commercialize and scale new agricultural innovations to help improve the livelihoods of smallholder farmers, increasing their productivity and incomes. To date the program has worked with 59 partners in 20 different countries, investing more than $43 million in new technologies and services, and leveraging nearly $100 million in private sector investment. The program has helped commercialize over 118 innovations, which resulted in an estimated $99 million in sales. It has its own Innovation site that partners can easily see and connect with promising innovations and research.
  • Finally, USAID was honored when the co-founder and Scientific Director of USAID’s Development Innovations Venture (DIV) program, Dr. Michael Kremer received the 2019 Nobel prize for economics, along with Dr. Esther Duflo and Dr. Abhijit Banerjee. Some of his work that led to this honor was connected to USAID’s DIV program. DIV values rigorous testing methods such as impact evaluations or robust market tests to measure the impact of USAID innovations. Evidence of clear and measurable outcomes helps demonstrate what is working and what is not. Solutions that demonstrate rigorous evidence of impact can then be scaled to other contexts. Through the DIV program, Dr. Kremer helps USAID use evidence-based approaches to take small risks, identify what works, and scale those approaches to provide greater impact. Since 2010, the DIV program has made 225 grants to find, test, and scale evidence-based innovations directly affecting more than 55 million lives across 47 countries. Based on Dr. Kremer’s and others research announced in October 2020, the early portfolio of DIV grants (covering 2010-12) has produced $17 in social benefit per every dollar spent by USAID.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Monitoring, Evaluation, Research and Learning Innovations Program (MERLIN) is a USAID endeavor designed to support the Bureau for Policy, Planning and Learning (PPL) and is now a part of the Learning, Evaluation and Research (LER) office within PPL. Formerly, the MERLIN program was part of the U.S. Global Development Lab which became the Innovation, Technology, and Research (ITR) Hub within the new DDI Bureau. The MERLIN program works to innovate on traditional approaches to monitoring, evaluation, research and learning. While innovative in themselves, these approaches can also be better suited to evaluating an innovation effort. Two examples of MERLIN activities include Developmental Evaluation, which aims to provide ongoing feedback to managers on implementation through an embedded evaluator, and Rapid Feedback, which allows implementers to test various methods to reach certain targeted results (more quickly than through traditional midterm or final evaluations). Both of these approaches allow adaptive management during implementation to improve program impacts.
  • Many of the agency’s programs such as Grand Challenges and Development Innovation Ventures (DIV) have been reviewed by formal audit and other performance and impact interventions.  DIV uses a tiered funding approach to find, test, and scale evidence-based innovations. DIV’s grants include: Stage 1 for piloting (up to $200,000); Stage 2 for testing (up to $1,500,000); Stage 3 for scaling (up to $15,000,000); and “evidence grants” (up to $1,500,000) for research to determine causal impact of certain interventions. In particular for Stage 2 grants, DIV requires evidence of impact that must be causal and rigorous – the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period. There must be significant demonstrated demand for the innovation.
  • DIV’s evaluation criteria for its funding is based on its three core principles as further outlined in its annual grant solicitation (DIV Annual Program Statement): (1) Evidence of Impact; (2) Cost-Effectiveness; and (3) Pathways to Scale. DIV’s expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators. Most DIV grants are fixed amount awards, a unique type of federal grant instrument that is tailor-made for pay-for-results approaches. Fixed amount awards are structured by paying for milestones achieved, which emphasizes performance (not just compliance) and reduces some administrative burden for all parties (see 2 CFR 200.201(b)).
  • DIV supports innovative solutions across all countries and development sectors in which USAID operates, including education, agriculture, water, energy, and economic development. Since 2010, DIV has provided more than $149 million for 225 grants in 47 countries, reaching more than 55 million beneficiaries. 
Score
6
Administration for Children and Families (HHS)
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • In late 2019, ACF stood up a customer experience initiative to enhance ACF’s delivery and administration of human services. This initiative focuses on ways to improve the experiences of both grantees and ACF employees. In 2020, ACF named a Chief Experience Officer (CEO) to lead these efforts. To date, the CEO has led efforts to understand and improve upon the experiences of ACF grantees receiving funding from multiple HHS operating divisions, evaluate and address the challenges that organizations face in applying for competitive grants, and develop an internal tool for ACF teams to assess and improve upon their capability to provide excellent technical assistance to ACF grantees. In 2021, ACF launched an Innovation Incubator initiative which began with a series of three Human-Centered Design trainings offered to ACF employees to equip staff with the skills and resources to identify problems, brainstorm ideas for improvement, and pilot solutions using an empathetic, “people-first” mindset. Participating staff also have access to the ACF Innovators community, a shared platform which supports interoffice idea generation and collaboration. ACF also has an ongoing Human-centered Design for Human Services (HCD4HS) project to explore the application of Human-centered Design across its service delivery programs at the federal, state, and local levels.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • ACF’s mission to “foster health and well-being by providing federal leadership, partnership and resources for the compassionate and effective delivery of human services” is undergirded by six values: dedication, professionalism, integrity, stewardship, respect, and excellence. ACF’s emphasis on excellence, “exemplified by innovations and solutions that are anchored in available evidence, build knowledge and transcend boundaries,” drives the agency’s support for innovation across programs and practices.
  • For example, ACF’s customer experience initiative is supporting the development of innovative practices for more efficient and responsive agency operations–improving how ACF understands and meets the needs of grantees and improving their capacity for service delivery. For example, ACF, in partnership with HRSA, convened a gathering of grantees who receive Head Start grants from ACF and federally qualified Health Center grants from HRSA to create opportunities for grantees to learn from one another and share best practices. ACF also helped a grantee analyze their data across both Head Start and Health Center programs to make operational improvements to their program.
  • ACF also administers select grant programs–through innovation projects, demonstration projects, and waivers to existing program requirements–that are designed to both implement and evaluate innovative interventions, as a part of an ACF-sponsored evaluation or an individual evaluation to accompany implementation of that innovation. For example:
  • ACF projects that support innovation include:
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
Score
5
AmeriCorps
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • The Office of Research and Evaluation hired a Learning Officer in FY21. The learning officer was a program office for the Social Innovation Fund. The learning officer and other colleagues in the Office of Research and Evaluation led the development of a concept paper for a National Service Equity, Evidence, and Innovation Fund. The purpose of the proposed fund includes:
    • To apply a developmental, tiered-grantmaking process to grow DEI organization’s access and use of National Service into community solutions by either 
      • Replicating or expanding current Social Innovation Fund (SIF) moderate or strong evidence-based models augmented with National Service in underrepresented communities, or 
      • Taking current evidence-based and evidence-informed National Service models (ASN, AmeriCorps Seniors) and scaling culturally appropriate solutions that work into underrepresented communities 
      • Ensuring that evidence building is integrated in the process to guide and document progress. 
  • The concept is informed by four agency programs, their expertise in process, award management and the body of evidence they have generated, combined with the entrepreneurial business ideas of incubator and accelerator designs. The agency plans to include this idea in its FY23 budget request and will use FY22 to further refine the ideas.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • AmeriCorps’s Evidence Exchange includes a suite of scaling products on the evidence exchange to help grantees replicate evidence-based interventions.
  • AmeriCorps continued to learn from its evidence-based planning grant program which “awards evidence-based intervention planning grants to organizations that develop new national service models seeking to integrate members into innovative evidence-based interventions.” AmeriCorps continued to learn from its research grantees, who receive grant funds to engage community residents and leaders in the development of new and innovative national service projects (more information available here). In addition to national service project development, these grants foster civic engagement through community research teams and build community capacity for using research to identify and understand local issues as well as identify possible solutions. Examples of these research-to-action projects include:
    • Researchers at the University of Nevada worked with NCCC Pacific leaders to craft a series of local projects, like building sidewalks and community cleanups, emerging from her CBPR project with youth scientists working together to understand slow violence in their own communities as well as that of people experiencing homelessness in the area. The partnership between NCCC, the University of Nevada is quite strong and the local government is supportive of the work in the region. Early in 2020, housing had been donated by a local community organization and the NCCC team was assigned and scheduled to begin on April 21st but the work was placed on hold due to the COVID-19 pandemic. Recently, PI and ADP have resumed discussions about rescheduling the team’s arrival.
    • Researchers at the Virginia Polytechnic Institute, State University Tech (Virginia Tech University), and Virginia Commonwealth University have brought together community partners and stakeholders in Martinsville, VA to address the local opioid crisis. They are using an evidence-based stakeholder engagement approach (SEED) that has led to successful outcomes in Martinsville. In Year 2, they collaborated with the Minnesota AmeriCorps state commission, ServeMinnesota, to replicate this project and approach with a focus on deploying AmeriCorps volunteers to meet unmet service needs around the opioid crisis in Minneapolis. Because of the success in rural Virginia and Martinsville, this approach will be further replicated in another town in rural Virginia and another town outside Minneapolis.
    • A researcher at Mississippi State University collaborated with NCCC Southern Campus to draft a concept paper for a NCCC team when COVID-19 struck–both agreed to table ideas for FAFSA support and ACT preparation until the crisis had subsided.
    • A researcher Drew University successfully collaborated with a former senior New Jersey state government official on a concept paper for VISTAs to work with their community partner, Family Promise, to build the organization’s capacity to work with local landlord’s and people experiencing homelessness–a role identified through their grant funded research. They were awarded two VISTAs.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • As part of the evaluation of the Social Innovation Program, which was designed to identify and rigorously test innovative approaches to social service problems, AmeriCorps continues to receive evaluation reports from grantees. As of May 2020, AmeriCorps has received 129 final SIF evaluation reports, of which 31 (24%) were experimental designs and 74 (57%) were quasi-experimental designs. Further, the evidence-based planning grant program and the research grant program both seek to generate innovative national service models. The planning grants require an evaluation plan. The research grants use evidence to inform action planning and solutions.
  • The Office of Research and Evaluation initiated a third-party evaluation of the 2018 grant program cohort (all using community-based participatory action research method) to identify outcomes, including the outcomes of national service projects developed through participatory research. The evaluation is part of a larger Program Lifecycle Evaluation project and is one of the bundled evaluations initiated in FY21.
Score
5
U.S. Department of Labor
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • The Department of Labor’s Chief Innovation Officer is responsible for efforts to use innovative technologies, partnerships and practices to accelerate the Department’s mission. The Chief Innovation Officer reports to the Deputy Secretary and also serves as the Senior Advisor for Delivery for the department.
  • DOL’s Chief Data Officer and Chief Evaluation Office (CEO) Data Analytics team developed a secure data analysis platform accessible to all DOL staff, pre-loaded with common statistical packages and offering the capability to access and merge various administrative data for analysis. DOL supports staff in executing limitless web-based A/B testing and other behaviorally-informed trials, with the shared service of the advanced Granicus platform’s GovDelivery communications tool, including free technical support. This tool enhances the Department’s ability to communicate with the public, such as through targeted email campaigns, and to adjust these communications, informed by testing and data, to increase engagement on relevant topics. The CEO also has developed toolkits and detailed resources for staff to effectively design behaviorally informed tests, shared on their new Behavioral Interventions website.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • The CEO uses a variety of communication tools to share rigorous research results, lessons learned, promising practices, and other implications of its research. These include internal briefings from independent contractors and researchers, a brownbag series that features evidence-based promising practices and results shared by DOL staff, for DOL staff, and an external expert seminar series featuring new findings or innovations in relevant areas of work. CEO staff consistently use research findings in the development of new research, and DOL agencies use findings to design and guide new discretionary grant programs, to refine performance measures for grantees, and to make decisions on compliance and enforcement practices.
  • DOL is strongly committed to promoting innovation in our policies and practices. For example, the Employment and Training Administration’s (ETA) competitive funding routinely funds innovative programming, since grantees typically bundle various program services and components to best meet the needs of the people being served by them in their local contexts. A particularly good example of where this innovation is happening is in the Administration’s high priority area of apprenticeships. In FY19, ETA issued nearly $184 million in Scaling Apprenticeship Through Sector-Based Strategies grants to public-private partnerships to expand apprenticeships to healthcare, information technology and other industries. In FY20, ETA awarded nearly $100 million in Apprenticeship: Closing the Skills Gap grants. Additionally, ETA and CEO are conducting an evaluation of the American Apprenticeship Initiative, which since 2016 has provided $175 million in grants to 45 grantees across the nation.
  • In addition, CEO’s Behavioral Insights team works with a number of DOL agencies on a continuous basis to identify and assess the feasibility of conducting studies where insights from behavioral science can be used to improve the performance and outcomes of DOL programs. The Wage and Hour Division’s (WHD) Transformation Team is one such example where continuous improvement efforts are driving innovation. Their work has identified potential areas where behavioral interventions and trials may inform program improvement. CEO is also working across agencies–including WHD, ETA, Women’s Bureau, Veterans Employment and Training Service (VETS), Office of Federal Contract Compliance Programs (OFCCP), and International Labor Affairs Bureau (ILAB)–to identify and assess the feasibility of other areas where insights from behavioral science can be used to improve the performance and outcomes of DOL programs.
  • DOL has also built capacity for staff innovation through the Performance Management Center’s Continuous Process Improvement (CPI) Program, an agency-wide opportunity which trains and certifies agency staff on Lean Six Sigma (LSS) methodologies through real-time execution of DOL process improvement projects. The program includes classroom sessions that prepare participants for LSS Black Belt certification examinations, including the American Society for Quality (ASQ) as well as DOL’s own certification.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • DOL, through the annual Learning Agenda process, systematically identifies gaps in the use of evidence. Innovation is about filling known gaps via dissemination, further research, or generation of quick turnaround assessments, like those offered to the Department by CEO’s Behavioral Insights Program.
  • DOL typically couples innovation with rigorous evaluation to learn from experiments. For example, DOL is participating in the Performance Partnership Pilots (P3) for innovative service delivery for disconnected youth which includes not only waivers and blending and braiding of federal funds, but gives bonus points in application reviews for proposing “high tier” evaluations. DOL is the lead agency for the evaluation of P3. A final report is available on the CEO’s completed studies website. 
  • DOL routinely uses Job Corps’ demonstration authority to test and evaluate innovative and promising models to improve outcomes for youth. Currently, the CEO is sponsoring a rigorous impact evaluation to examine the effectiveness of one of these pilots, the Job Corps Experimental Center Cascades, with results expected in FY22.
Score
6
U.S. Dept. of Housing & Urban Development
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • HUD has an Office of Innovation led by a Deputy Assistant Secretary that facilitates both routine innovation and improvements to departmental operations, services and working conditions. Examples of recent work include assessing the feasibility of modernizing HUD headquarters’ energy and information system and creating a standardized Paperwork Reduction Act (PRA) submission package. The Office of Innovation also organized a five-day Innovative Housing Showcase on the National Mall with federal and private sector partners in June 2019 to demonstrate new housing technology and discuss innovation barriers and opportunities. The entire local HUD staff was encouraged to attend and view the innovative technologies, and another Showcase with interagency participation is planned for 2021. The Office of Innovation is developing prize competitions to stimulate innovation in housing and HUD policy and programs.
  • Additionally, the Government National Mortgage Corporation, or Ginnie Mae, has several innovation teams and operates an Innovation Laboratory to advance the application of machine learning to strengthen operations. In 2021, the Office of Innovation is working with The Opportunity Project on a 2021 sprint on Analyzing Housing and Migration Trends Post-COVID 19. The purpose of the sprint is to develop data and visualizations to help stakeholders understand how housing needs have shifted during (and as a result of) the pandemic as some people moved away from employers and some industries collapsed or shifted to remote operations. The stakeholders that could benefit from these data and tools are public housing agencies, city and regional planners, housing developers, schools, state and local governments, and other federal agencies.
  • HUD administers five types of juried Secretary’s Awards to encourage excellence in addressing housing and community development challenges: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. An Innovation in Affordable Housing Competition engages multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • HUD established the Office of Innovation in 2019 to advance innovation in several domains. The Office managed the 2019 Innovative Housing Showcase and is developing a similar Showcase for 2021 and prize competitions to stimulate innovation in housing and HUD policy and programs. HUD’s Office of Innovation also recently worked with The Opportunity Project to improve communication between HUD’s Envision Centers and their local stakeholders and is now working with the Opportunity Project to provide data and visualizations related to the housing industry impacts of the changes to the workplace and society post-COVID.
  • FY20 grants fund cooperative agreements for pre-competitive research in homebuilding innovations, with a similar program for Historically Black Colleges and Universities, for more affordable, energy efficient, resilient, and healthier housing. HUD seeks to engage the insights and creativity of HBCUs to advance problem-solving toward greater diversity and more equitable outcomes.
  • The Department promotes evidence-based innovation by using program demonstrations to experimentally test potential policy enhancements, which have included eight low-cost, behaviorally informed experiments using interagency data matching and assistance from the GSA Office of Evaluation Sciences. Other innovative research ideas from external stakeholders are supported by the Research Partnerships program. Competitive awards for Healthy Homes Technical Studies generate innovation in the evaluation and control of housing-related health and safety hazards.
  • An interagency agreement with the Census Bureau has made datasets from HUD’s randomized control trials available for linkage with census data and administrative datasets. The RCT datasets are the first intervention data added to Federal Statistical Research Data Centers (RDCs) by any federal agency, and joint support is available to help researchers gain access and learn to use the restricted data successfully for innovative research, with seven projects currently underway.
  • HUD’s Rental Assistance Demonstration, which restructures the financing of the nation’s public housing to address capital needs backlogs, has the additional innovative feature of providing tenants with a Choice Mobility option. Choice Mobility supports self-sufficiency by offering priority receipt of a Housing Choice Voucher providing freedom to move to neighborhoods with greater economic opportunities or better schools and amenities.
  • HUD’s regulation of manufactured housing production is guided by a federal advisory committee, the Manufactured Housing Consensus Committee, to provide increased ability for the industry to produce some of the nation’s most innovative, safe, and affordable housing.
  • HUD has a Robotics Process Automation initiative devoted to freeing the workforce from low-value, repetitive work through software robotics solutions. Specialized computer programs known as “bots” automate and standardize repeatable business processes without costly investments in conventional automation. Planned efforts involving payroll, accounts receivable and payable, invoice processing, inventory management, report creation, and data migration have potential to shift over 50,000 hours of employee time from low-value to high-value work.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
Score
4
Administration for Community Living (HHS)
7.1 Did the agency have staff dedicated to learning innovation efforts to improve the impact of its programs?
  • Agency leadership promotes innovation by requiring all program offices to explain, in their annual funding proposals, how the proposed use of funds will identify innovative practices. ACL also partially funds a Forum on Aging, Disability, and Independence which engages staff to foster discussions about innovation for coordinating and integrating aging and disability stakeholders. ACL also funds resource centers, such as the Engagement and Older Adults Resource Center which provides technical assistance and serves as a repository for innovations designed to increase the aging network’s ability to tailor social engagement activities to meet the needs of older adults.
7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs? 
  • In FY21 all ACL Centers were involved in funding innovative work. ACL released several funding opportunity announcements (FOA) focused on the identification and implementation of innovative approaches to improve programming. These included funding opportunity announcements for the Innovations in Nutrition Programs and Services – Community Research and the ACL/NIDILRR Small Business Innovation Research Program (SBIR) Phase I. ACL also funded challenge competitions such as the ACL Social Care Referrals Innovation Challenge. In 2021, ACL funded a small study to identify the innovations and adaptations by grantees in the face of COVID-19. In 2022, ACL will more closely examine the most promising innovations to evaluate their effectiveness.
  • ACL is a funder of The John A. Hartford Foundation 2020 Business Innovation Award,  which recognizes aging and disability community-based organizations (CBOs) for their innovative approaches to reducing health care costs and improving the well-being of older adults and people with disabilities through strategic partnership with health care entities. 
  • There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3.5 million to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • The 2020 reauthorization of the Older Americans Act requires a new Research, Demonstration, and Evaluation Center for the Aging Network and new demonstration programs to evaluate new strategies for the recruitment, retention, or advancement of direct care workers, and the soliciting, development, and implementation of strategies; and a demonstration to address negative health impacts associated with social isolation. Further, ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches towards coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e. Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.
  • NIDILRR’s research and development activities are guided by the Stages of Research Framework and the Stages of Development Framework. NIDILRR grantees conducting research projects must identify the stage of research their projects are in while grantees conducting development projects must identify the stage of development their projects are in. The stage a research project is in depends upon what is known and what is not known about a particular disability problem or topic. Research projects where relatively little is known, or the topic area is emerging, would be classified in the Exploration and discovery stage. Over time, as more becomes known, research projects progressively move into the Intervention Development phase. The next phase, known as Intervention Efficacy, means the stage of research during which a project evaluates and tests whether an intervention is feasible, practical, and has the potential to yield positive outcomes for individuals with disabilities. The final stage, known as Scale-Up Evaluation, means the stage of research during which a project analyzes whether an intervention is effective in producing improved outcomes for individuals with disabilities when implemented in a real-world setting. 
  • Similarly, the stage of development a development project is in also depends upon what is known or not known about a need that informs the design and development of a product. The proof of concept stage means the stage of development where key technical challenges are resolved. Stage activities may include recruiting study participants, verifying product requirements, implementing and testing (typically in controlled contexts) key concepts, components, or systems, and resolving technical challenges. The proof of product stage means the stage of development where a fully-integrated and working prototype, meeting critical technical requirements, is created. The proof of adoption stage means the stage of development where a product is substantially adopted by its target population and used for its intended purpose. 
Score
4
Substance Abuse and Mental Health Services Administration (HHS)
7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • CBHSQ hired senior staff within the CBHSQ Office of the Director to focus on innovation, increasing efficiency, developing standard protocols, and developing a dissemination strategy for greater utilization of SAMHSA data (with a focus on encouraging visits to SAMHDA for data use). This work is ongoing.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • Within SAMHSA grant programs, the agency encourages innovation from every state, territory and community applicant. For example, the FY20-21 Substance use block grant application includes the following language: “There is increased interest in having a better understanding of the evidence that supports the delivery of medical and specialty care including M/SUD services. Over the past several years, SAMHSA has collaborated with CMS, HRSA, SMAs, state M/SUD authorities, legislators, and others regarding the evidence of various mental and substance misuse prevention, treatment, and recovery support services. States and other purchasers are requesting information on evidence-based practices or other procedures that result in better health outcomes for individuals and the general population. While the emphasis on evidence-based practices will continue, there is a need to develop and create new interventions and technologies and in turn, to establish the evidence. SAMHSA supports states’ use of the block grants for this purpose. The NQF and the IOM recommend that evidence play a critical role in designing health benefits for individuals enrolled in commercial insurance, Medicaid, and Medicare. To respond to these inquiries and recommendations, SAMHSA has undertaken several activities. SAMHSA’s Evidence Based Practices Resource Center assesses the research evaluating an intervention’s impact on outcomes and provides information on available 43 resources to facilitate the effective dissemination and implementation of the program. SAMHSA’s Evidence-Based Practices Resource Center provides the information & tools needed to incorporate evidence-based practices into communities or clinical settings.”
  • Pursuant to the 21st Century Cures Act, SAMHSA established the National Mental Health and Substance Use Policy Laboratory (NMHSUPL), which promotes evidence-based practices and service delivery models through evaluating models that would benefit from further development and through expanding, replicating, or scaling evidence-based programs across a wider area. Specifically NMHSUPL:
    • Identifies, coordinates, and facilitates the implementation of policy changes likely to have a significant effect on mental health, mental illness (especially severe mental illnesses such as schizophrenia and schizoaffective disorders), recovery supports, and the prevention and treatment of substance use disorder services;
    • Works with CBHSQ to collect information from grantees under programs operated by the Administration in order to evaluate and disseminate information on evidence-based practices, including culturally and linguistically appropriate services, as appropriate, and service delivery models; and 
    • Carries out other activities as deemed necessary to continue to encourage innovation and disseminate evidence-based programs and practices.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Grantees report innovation and use of evidence-based practices in their reports as required by SAMHSA. Information from these reports are included in rigorous evaluations when evaluations have been planned or used for performance management when a formal evaluation has not been scheduled.   
  • In FY21, SAMHSA completed a rigorous evaluation of its innovative SPF-Rx program. Prescription drug misuse continues to be a critical public health problem in the United States (U.S.) and the SPF (or Strategic Prevention Framework) consists of five steps. SPF is a data-driven, systemic public health planning approach to substance use prevention that is theory-based and involves the implementation of evidence-based strategies. Grantees and subrecipients apply the overarching principles of cultural competence and sustainability throughout the dynamic SPF process. Between 2017-2021, 25 state and tribal SPF-Rx grantees received funding to focus on supporting high-need community efforts to implement evidence-based practices to prevent and reduce the misuse of prescription opioids. In FY21, an evaluation was completed and the findings from this evaluation informed SAMHSA of ways to better develop, assess, and manage its prescription drug misuse prevention programs. The cross site evaluation collected quantitative and qualitative data from three instruments: an annual implementation instrument (130 items collecting data on organization type, funding levels, assessment of capacity building and sustainability, strategic planning, prevention and intervention programming, and ongoing local evaluations); grantee level and community level outcomes module (two modules including data on PDMP use and prescribing patterns, opioid overdose events, etc); and grantee level interviews (qualitative data collected at baseline and at the end of the evaluation). Results of this evaluation were used to inform the next generation of SPF Rx grants.   
  • The next evaluation of this SPF Rx program has been started with an evaluation plan drafted to include an examination of secondary data on matched communities for comparison of outcome data.   
Back to the Standard

Visit Results4America.org

Copyright 2024 Results for America. All rights reserved. Privacy Policy

Site by Wide Eye Creative