2021 Federal Standard of Excellence


U.S. Dept. of Housing & Urban Development

68
Score
9
Leadership

Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY21?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)
  • The General Deputy Assistant Secretary of the Office of Policy Development & Research (PD&R) serves as the Department of Housing and Urban Development (HUD) evaluation officer. HUD’s Office of Policy Development & Research (PD&R) is led by an Assistant Secretary and the career General Deputy Assistant Secretary. PD&R comprises six offices, 165 staff including a team of field economists in HUD’s 10 regional offices, and a budget of $105 million in FY21. The Assistant Secretary and Evaluation Officer ensures that evidence informs policy development through frequent personal engagement with other principal staff, the Secretary, and external policy officials including consultation with Congress, speeches to policy audiences, sponsorship of public research briefings, and policy implications memoranda.
1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))
1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, performance improvement officer, and other related officials in order to support, improve, and evaluate the agency’s major programs?
  • PD&R continues to lead HUD’s full implementation of the Foundations for Evidence-Based Policymaking Act. All three of the key leadership positions required by the Act–the Evaluation Officer, Chief Data Officer, and Statistical Official–are held by members of the PD&R staff. PD&R staff have engaged and coordinated within the Department its evidence-building efforts, which in FY20 included developing HUD’s learning agenda and conducting the first agency-wide assessment of evidence-building capacity. In FY21, HUD will be focused on establishing an enterprise data governance model, which will include a data governance board consisting of key decision-makers from across the agency, which will include the Evaluation Officer, Chief Data Officer, Statistical Official, and Performance Improvement Officer. HUD’s enterprise data governance model will bring together evaluation, statistical, performance, and data activities and focus on growing the agency’s evidence-based practices to improve HUD’s organizational performance.
Score
10
Evaluation & Research

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY21?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • PD&R published a Program Evaluation Policy in 2016 that established core principles and practices of PD&R’s evaluation and research activities. The six core principles are rigor, relevance, transparency, independence, ethics, and technical innovation.
  • In August 2021, PD&R updated the 2016 Program Evaluation Policy to address issues that have arisen since 2016 as well as stakeholder input received via a town hall that PD&R hosted discussing its experience with sponsoring and publishing evaluations. The new HUD Program Evaluation Policy expands the jurisdiction of the statement to all of HUD and includes principles and practices intended to ensure racial equity, diversity, and inclusion in PD&R’s evaluation and research activities. The language related to equity was developed in coordination with the Department-wide Equity Assessment that HUD is undergoing in response to Executive Order #13985, Executive Order on Advancing Racial Equity and Support for Underserved Communities.
2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • HUD’s learning agendas, called the Research Roadmap, have served as agency-wide evaluation plans that list and describe research and evaluation priorities for a five-year planning period. HUD released the 2020 Roadmap Update in December 2020. Annual evaluation plans are developed based on a selection of Roadmap proposals, newly emerging research needs, and incremental funding needs for major ongoing research and are submitted to Congress in association with PD&R’s annual budget requests. Actual research activities are substantially determined by Congressional funding and guidance. Under the Evidence Act, PD&R prepares public Annual Evaluation Plans informed by the new Research Roadmap to be submitted in conjunction with the Annual Performance Plan.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • HUD’s Research Roadmap has served as the Department’s evidence-building plan and learning agenda for eight years, and a new Roadmap was developed in FY21. HUD’s participatory process (see Appendix A of Research Roadmap: 2020 Update) engages internal and external stakeholders to identify research questions and other evidence-building activities to support effective policy-making. Stakeholders include program partners in state and local governments and the private sector; researchers and academics; policy officials; and members of the general public who frequently access the HUDuser.gov portal. Outreach mechanisms for Roadmap development include email, web forums, conferences and webcasts, and targeted listening sessions.
  • The 2020 Roadmap Update served as the Department’s draft Learning Agenda under the Evidence Act. To finalize the Learning Agenda, PD&R staff will align foundational learning questions with HUD’s new strategic plan and conduct an additional round of internal stakeholder engagement in FY21 focused on identifying priority research questions across the Department. HUD is also seeking input on the 2020 Roadmap via email and web forums. PD&R staff will coordinate with the team conducting the Department-wide Equity Assessment in response to  the executive order on Advancing Racial Equity and Support for Underserved Communities to identify priority research questions and evidence gaps that emerge as part of the assessment. HUD’s Equity Assessment has prioritized stakeholder engagement as an area for immediate analysis by all program offices. The equity assessment seeks to identify and utilize the knowledge–both lived and professional–of stakeholders who have been historically underrepresented in the Federal government and underserved by, or subject to discrimination in, federal policies and programs. Findings from this assessment will further inform HUD’s long-term “equity transformation,” which aims to sustainably embed and improve equity throughout all of HUD’s work. HUD will release its long-term Action Plan to increase equity in decision-making and access to programs and benefits on January 20, 2022 pursuant to Executive Order #13985.
2.4 Did the agency publicly release all completed program evaluations?
  • PD&R’s Program Evaluation Policy requires timely publishing and dissemination of all evaluations that meet standards of methodological rigor. Completed evaluations and research reports are posted on PD&R’s website, HUDUSER.gov. Additionally, the policy includes language in research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than six months. HUD’s publicly released program evaluations typically include data and results disaggregated by race, ethnicity, and gender, where the data permit such disaggregation. For example, in 2020 HUD expanded the detail of race and ethnicity breakouts in the Worst Case Housing Needs reports to Congress to the full extent permitted by the data. Executive summaries will highlight disparate impacts if they are found to be statistically significant; otherwise, such findings may be found in the main body of the report or its appendices.
  • PD&R is in the process of reorganizing our published research and enhancing our search capabilities on HUDUSER.gov. These steps are being implemented to enhance the usability of HUD’s research resources for researchers, policymakers, and the general public.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 3115, subchapter II (c)(3)(9))
  • PD&R is HUD’s independent evaluation office, with scope spanning all the Department’s program operations. In FY20 PD&R led an effort to assess the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts, consistent with the values established in HUD’s Evaluation Policy. The Research Roadmap: 2020 Update covers much of this content, and a formal Capacity Assessment process was designed by evaluation leaders in coordination with the Chief Data Officer and performance management personnel. The draft Capacity Assessment addresses updated content requirements of OMB Circular A-11 (2020) and includes primary data collection through an exploratory key informant survey of senior managers across the Department. The identified weaknesses in evidence-building capacity will become the focus of subsequent in-depth assessments and interventions to be integrated in the Department’s next Strategic Plan.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
Score
6
Resources

Did the agency invest at least 1% of program funds in evaluations in FY21?

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY21 budget.
  • HUD invested $105 million on evaluations, evaluation technical assistance, and evaluation capacity-building, representing 0.17% of the agency’s $60.343 billion FY21 appropriation.
3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)
  • For FY21, Congress appropriated $105 million for the Office of Policy Development and Research’s Research & Technology account. FY21 funding was up $7 million from FY20, reflecting congressional support for the value of PD&R’s research, evaluations, and demonstrations. This funding includes $59 million for core research activities; $13 million for research, evaluations, and demonstrations; and $33 million for technical assistance. The total represents an FY21 investment in evaluations and evidence amounting to 0.17 percent of HUD’s $60.343 billion gross discretionary budget authority, net of salaries and expenses, for FY21. The funding for core research is used primarily for the American Housing Survey, other surveys, data acquisition, and research dissemination that support evaluation of HUD’s mission activities in domains such as affordable housing and housing finance.
  • PD&R’s FY21 appropriation of $36 million for Salaries and Expenses, up $8 million from FY20, also supports evidence in the form of PD&R’s in-house research and evaluation program; economic analyses; data linkage initiatives; management of housing surveys, contract research, and evaluation; and the work of the Chief Data Officer.
3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?
  • PD&R’s FY21 appropriation of $36 million for Salaries and Expenses, up $8 million from FY20, also supports evidence in the form of PD&R’s in-house research and evaluation program; economic analyses; data linkage initiatives; management of housing surveys, contract research, and evaluation; and the work of the Chief Data Officer.
  • In FY21, HUD is offering nearly $10 million through its Distressed Cities technical assistance program that helps local units of government in cities and places experiencing persistent poverty and population loss build capacity for processes including financial management, data collection, analysis, and tracking outcomes. In selecting technical assistance providers, HUD will consider the provider’s experience incorporating equity into technical assistance engagement activities and equity outcomes into engagement goals or deliverables (see Rating Factor 1, Prior Experience and Performance).
  • HUD operates a Section 4 Capacity Building grant program that in FY21 provides $46 million for national intermediaries, including $10 million for rural needs, to build capacity for functions including assessing needs, planning programs, and evaluation.
  • HUD’s Community Development Block Grant (CDBG) program, which provides formula grants to entitlement jurisdictions, increases local evaluation capacity. Specifically, federal regulations (24 CFR570.200) authorize CDBG recipients (including city and state governments) to use up to 20% of their CDBG allocations for administration and planning costs that may include evaluation-capacity building efforts and evaluations of their CDBG-funded interventions (as defined in 570.205 and 570.206).
  • HUD offers grants through its Research Partnerships program to support research that addresses current research priorities. The Department is especially interested in increasing participation of Minority Serving Institutions of higher education and Historically Black Colleges and Universities in this program. The FY21 NOFO also encourages research on topics related to equity in HUD’s programs and policies.
  • HUD is conducting a Department-wide Equity Assessment in response to the executive Order on Advancing Racial Equity and Support for Underserved Communities. One of the four areas prioritized as an immediate area for assessment is procurement. As part of this assessment, the team will examine HUD’s efforts to strengthen evaluation capacity for minority and women-owned businesses, community-based evaluators, or nontraditional evaluation contractors. HUD will follow the Equity Assessment with an Equity Transformation Plan that seeks to embed and improve equity across all HUD’s work.
Score
10
Performance Management / Continuous Improvement

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY21?

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?
  • HUD’s FY18–22 Strategic Plan, as amended by HUD’s FY21 Annual Performance Plan, defines strategic objectives, priority outcome goals, and program metrics supporting each objective. Progress on program metrics is tracked through the Annual Performance Plan. In FY21, HUD began to develop a new Strategic Plan that will reflect enhanced support for evidence-building through integration with the Learning Agenda and structured Capacity Assessment as provided by the Evidence Act.
  • HUD uses data and evidence extensively to improve outcomes and return on investment. The primary means are through PD&R’s investments in data collection, program demonstrations and evaluations, and research guided by a multi-year learning agenda. HUD’s extensive use of outcome-oriented performance metrics in the Annual Performance Plan; and senior staff oversight and monitoring of key outcomes and initiatives through quarterly performance management meetings that will be supported by a new CFO performance management module under development.
  • A HUD initiative to modernize technologies for using data to improve outcomes includes elements of intelligent automation and artificial intelligence, using advanced data analytics and visualization, and building electronic records management, intelligent data extraction, and electronic forms.
  • In 2019, HUD expanded the Standards for Success data collection and reporting framework for discretionary grant programs to cover Resident Opportunities and Self-Sufficiency Service Coordinator (ROSS) grants, Multifamily Housing Service Coordinator grants, and Multifamily Housing Budget-Based Service Coordinator Sites. The framework supports better outcomes by providing a more standardized performance measurement framework, better alignment with Departmental strategies, and more granular reporting to support analytics.
4.2 Did the agency use data/evidence to improve outcomes and return on investment?
  • HUD uses data and evidence extensively to improve outcomes and return on investment. The primary means are through PD&R’s investments in data collection, program demonstrations and evaluations, and research guided by a multi-year learning agenda. HUD’s extensive use of outcome-oriented performance metrics in the Annual Performance Plan; and senior staff oversight and monitoring of key outcomes and initiatives through quarterly performance management meetings that will be supported by a new CFO performance management module under development.
  • A HUD initiative to modernize technologies for using data to improve outcomes includes elements of intelligent automation and artificial intelligence, using advanced data analytics and visualization, and building electronic records management, intelligent data extraction, and electronic forms.
  • In 2019, HUD expanded the Standards for Success data collection and reporting framework for discretionary grant programs to cover Resident Opportunities and Self-Sufficiency Service Coordinator (ROSS) grants, Multifamily Housing Service Coordinator grants, and Multifamily Housing Budget-Based Service Coordinator Sites. The framework supports better outcomes by providing a more standardized performance measurement framework, better alignment with Departmental strategies, and more granular reporting to support analytics.
4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)
  • HUD’s senior staff support continuous improvement and oversight and monitoring of key outcomes and initiatives through quarterly performance management meetings. These processes are supported by ongoing, significant investments in evidence-building as documented in the Annual Performance Plan and the iterative process of developing the Research Roadmap learning agenda, as well as development of a new performance management module by the Chief Financial Officer. Monitoring and analysis based on administrative data have a symbiotic and complementary relationship with structured evaluation and program demonstrations.
  • HUD’s Office of Policy Development and Research also hosts ongoing Knowledge Collaboratives designed to support continuous learning and improve performance. Examples include a Data Knowledge Collaborative, an RCT Knowledge Collaborative, and a Knowledge Collaborative on Equity in Evaluation, as well as a new inter-office user group that shares information and tools for using statistical software effectively. For example, a recent meeting of the Knowledge Collaborative on random assignment experiments considered the topics of research preregistration and multiple hypothesis testing. The agenda for the meeting included discussions of the concepts of preregistration and multiple hypothesis testing and of the steps that HUD PD&R could take to encourage (or require pre-registration) of its research studies and guidance it could develop for contracted researchers regarding the multiple comparisons problem. The Knowledge Collaborative on Equity in Evaluation also recently worked on revising HUD’s Evaluation Policy to incorporate considerations of equity throughout.
Score
6
Data

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY21?

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)
  • In FY21, HUD will develop a strategic data plan, which will include an open data policy. Currently, HUD’s open data program includes existing assets including administrative datasets on data.hud.gov, spatially enabled data on the eGIS portal, PD&R datasets for researchers and practitioners, a robust partnership with the Census Bureau, U.S. Postal Service vacancy data, and health data linkages with the National Center for Health Statistics. HUD’s public datasets are designed to allow analysis by race/ethnicity, gender, and other equity-related characteristics to the extent possible given the nature of the data and privacy constraints.
5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)
  • HUD has extensive data sharing processes including public sharing, interagency sharing, and internal sharing, with each mode requiring specific controls and documentation. In FY21, HUD will review its existing data inventory and update it accordingly to produce a comprehensive data inventory. HUD will also revisit its data inventory schedule to ensure the agency is performing the activities necessary to develop and maintain a comprehensive data inventory.
5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c))
  • HUD has extensively promoted data access and data linkage, including the following approaches:
    • An updated list of open data assets; numerous PD&R-produced datasets for researchers and practitioners, including tenant public use microdata samples, Picture of Subsidized Households, FMR and Income limits, Comprehensive Housing Affordability Strategy (CHAS) special tabulations of the American Community Survey; and an eGIS portal providing geo-identified open data to support public analysis of housing and community development issues using GIS tools. The eGIS portal is a comprehensive data source, covering the majority of HUD’s programs and initiatives. New mappable data added to the eGIS portal in FY21 include locations of Federally Qualified Health Centers to support COVID-19 vaccination efforts related to public and assisted housing programs.
    • Data linkage agreements with the National Center for Health Statistics and the Census Bureau to enhance major national survey datasets by identifying HUD-assisted households, with updates continuing in FY21; making available major program demonstration datasets in secure environments; and producing special open-access tabulations of census data for HUD’s partners. The agreement with the Census Bureau includes three-way data matching between HUD tenant data, American Housing Survey data, and American Community Survey data.
    • HUD has created a repository of properties, units and tenants that merge data across the various HUD rental assistance programs for use in research, evaluation and reporting. This allows for standardization and greater access to socio-demographic characteristics of HUD’s clients.
    • Engagement in cooperative agreements with research organizations, including both funded Research Partnerships and unfunded Data License Agreements, to support innovative research that leverages HUD’s data assets and informs HUD’s policies and programs. Data licensing protocols ensure that confidential information is protected.
5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)
  • HUD’s Statistical Official supports the Evidence Officer on issues related to protection of confidential data and statistical efficiency. HUD’s Evaluation Policy specifies that HUD protects client privacy by adhering to the Rule of Eleven to prevent disclosure from tabulations with small cell sizes. PD&R’s data licensing protocols ensure that researchers protect confidential information when using HUD’s administrative data or program demonstration datasets.
  • The Statistical Official collaborates with statistical agencies to create data linkages and develop data products that are machine-readable and include robust privacy protections. HUD has an interagency agreement with the Census Bureau to conduct the American Housing Survey and collaborates with Census staff to examine disclosure issues for AHS public use files and the potential for “synthetic” public datasets to support researchers in estimating summary statistics with no possibility of reidentifying survey respondents. Another interagency agreement allows the Census Bureau to link data from HUD’s randomized control trials with other administrative data collected under the privacy protections of its Title 13 authority. These RCT datasets are the first intervention data added to Federal Statistical Research Data Centers (RDCs) by any federal agency. Strict RDC protocols and review of all output ensure that confidential information is protected, and the open data and joint support for researchers are currently facilitating seven innovative research projects at minimal cost to HUD.
5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?
  • HUD has an updated list of open data assets, an open data program, numerous PD&R datasets for researchers and practitioners, and an eGIS portal providing geo-identified data to support public analysis of housing and community development issues related to multiple programs and policy domains using GIS tools. For example, HUD supports local governments in assessing and planning for housing needs by providing summary data files about HUD-supported public and assisted housing and about local housing needs. These accessible data assets have privacy protections. Researchers needing detailed microdata can obtain access through data licensing agreements.
  • HUDExchange offers numerous resources and training opportunities to help program partners use data assets more effectively. Additional technical assistance is offered through the  program, a $91 million technical assistance program to equip HUD’s customers with the knowledge, skills, tools, capacity, and systems to implement HUD programs and policies successfully and provide effective oversight of federal funding.
  • In FY21, HUD will produce a study of the feasibility of HUD producing a national database of evictions that would be available to all levels of government and the general public to track and assess evictions, including eviction trends by race, gender, disability status, ethnicity, and age. If the study determines that such a database is feasible and if Congress funds its development, such a database will be an important tool for analyzing equitable treatment of renters.
Score
3
Common Evidence Standards / What Works Designations

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY21?

6.1 Did the agency have a common evidence framework for research and evaluation purposes?
  • HUD’s Program Evaluation Policy defines standards that prioritize rigorous methods for research and evaluation covering impact evaluations; implementation of process evaluations; descriptive studies; outcome evaluations; and formative evaluations; and both qualitative and quantitative approaches. It also provides for timely dissemination of such evidence to stakeholders. HUD updated the Program Evaluation Policy in August 2021 to address the rigorous inclusion of qualitative evidence and considerations related to equity.
  • HUD’s Program Evaluation Policy commits HUD to using “the most rigorous methods that are appropriate to the evaluation question and feasible within budget and other constraints… Where feasible, research should employ a treatment group and a counterfactual …. For such causal questions, experimental approaches are preferred.” The Policy also establishes standards for transparency: “HUD will release methodologically valid evaluations without regard to the findings. Evaluation reports must describe the methods used, including strengths and weaknesses, and discuss the generalizability of the findings. Evaluation reports must present comprehensive results, including favorable, unfavorable, and null findings.”
  • The consequences of HUD’s commitment to rigor may be seen in HUD’s priority research projects:
    • IWISH, a cluster-randomized health intervention at 40 treatment housing developments and 80 control housing developments
    • The long-term follow-up to the Family Options Study, which randomized over 2000 homeless families to 4 treatment regimes.
    • The Rent Reform Study, which randomized assisted non-elderly non-disabled households to two different rent structures in 4 sites.
    • The First Time Homebuyer Education and Counseling Demonstration, which randomized more than 5,800 prospective first-time homebuyers from 28 large metropolitan areas.
    • The Moving to Work Flexibilities study, which randomized 31 PHAs to partial deregulation versus status quo.
    • The Family Self-Sufficiency demonstration, which randomized voucher holders at 18 sites and followed them for 7 years.
    • The Rental Assistance Demonstration evaluation, which compared 24 RAD-conversion projects with 48 matched non-converted public housing projects on financial indicators and changes in the physical stock.
    • The Voucher Mobility Demonstration, which will randomize 14,000 voucher holders in 9 sites to receive neighborhood mobility counseling (or not).
  • Further, HUD’s commitment to independence (in the Evaluation Policy Statement) has led us to provide PII to dozens of independent research teams across the country, who match our data with other sources to measure the consequences of housing assistance for tenants of assisted housing. HUD’s commitment to independence has also led us to long-term matching of our data with surveys sponsored by the National Center for Health Statistics, to monitor trends in disease conditions among the assisted population.
6.2 Did the agency have a common evidence framework for funding decisions?
  • HUD seeks to employ tiered evidence in funding decisions by embedding implementation and impact evaluations in funding requests for program initiatives, including major program demonstrations that employ random assignment methods. These include the Moving To Work Expansion demonstration, the Rental Assistance Demonstration, the Rent Reform Demonstration, the Family Self-Sufficiency Demonstration, the Housing Counseling Demonstration, and the Family Options Demonstration, described above. Such trials provide robust evidence to inform scale-up funding decisions.
  • HUD extended its standardized data collection and reporting framework, Standards for Success, to additionaldiscretionary grant programs in FY19. The framework consists of a repository of data elements that participating programs use in their grant reporting, creating common definitions and measures across programs for greater analysis and coordination of services.
6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?
6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?
  • HUD provides resources and assistance to support community partners in evidence-based practice through the HUD Exchange web portal and through technical assistance. PD&R provides the public, policymakers, and practitioners with evidence of what works primarily through HUD USER, a portal and web store for program evaluations, case studies, and policy analysis and research; the Regulatory Barriers Clearinghouse; and through initiatives such as Sustainable Construction Methods in Indian Country and the Consumer’s Guide to Energy-Efficient and Healthy Homes. This content is designed to provide current policy information, elevate effective practices, and synthesize data and other evidence in accessible formats such as Evidence Matters. Through these resources, researchers and practitioners can see the full breadth of work on a given topic (e.g., rigorous established evidence, case studies of what has worked in the field, and new innovations currently being explored) to inform their work.
  • PD&R has increased emphasis on generating interim reports during long-term impact evaluations. Such interim reports provide practitioners with early findings about implementation practice and outcomes that can inform their own program designs. A recent example is the Interim Report from HUD’s Supportive Services Demonstration, published in 2020. A second Interim Report from this study is forthcoming in 2021.
  • Community Compass technical assistance for urban, rural, and tribal partners is designed to facilitate understanding of community and housing development issues in a way that cuts across program silos. It supports them in evaluation, evidence-building, integrating knowledge management principles, and sharing practices.
Score
6
Innovation

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY21?

7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?
  • HUD has an Office of Innovation led by a Deputy Assistant Secretary that facilitates both routine innovation and improvements to departmental operations, services and working conditions. Examples of recent work include assessing the feasibility of modernizing HUD headquarters’ energy and information system and creating a standardized Paperwork Reduction Act (PRA) submission package. The Office of Innovation also organized a five-day Innovative Housing Showcase on the National Mall with federal and private sector partners in June 2019 to demonstrate new housing technology and discuss innovation barriers and opportunities. The entire local HUD staff was encouraged to attend and view the innovative technologies, and another Showcase with interagency participation is planned for 2021. The Office of Innovation is developing prize competitions to stimulate innovation in housing and HUD policy and programs.
  • Additionally, the Government National Mortgage Corporation, or Ginnie Mae, has several innovation teams and operates an Innovation Laboratory to advance the application of machine learning to strengthen operations. In 2021, the Office of Innovation is working with The Opportunity Project on a 2021 sprint on Analyzing Housing and Migration Trends Post-COVID 19. The purpose of the sprint is to develop data and visualizations to help stakeholders understand how housing needs have shifted during (and as a result of) the pandemic as some people moved away from employers and some industries collapsed or shifted to remote operations. The stakeholders that could benefit from these data and tools are public housing agencies, city and regional planners, housing developers, schools, state and local governments, and other federal agencies.
  • HUD administers five types of juried Secretary’s Awards to encourage excellence in addressing housing and community development challenges: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. An Innovation in Affordable Housing Competition engages multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • HUD established the Office of Innovation in 2019 to advance innovation in several domains. The Office managed the 2019 Innovative Housing Showcase and is developing a similar Showcase for 2021 and prize competitions to stimulate innovation in housing and HUD policy and programs. HUD’s Office of Innovation also recently worked with The Opportunity Project to improve communication between HUD’s Envision Centers and their local stakeholders and is now working with the Opportunity Project to provide data and visualizations related to the housing industry impacts of the changes to the workplace and society post-COVID.
  • FY20 grants fund cooperative agreements for pre-competitive research in homebuilding innovations, with a similar program for Historically Black Colleges and Universities, for more affordable, energy efficient, resilient, and healthier housing. HUD seeks to engage the insights and creativity of HBCUs to advance problem-solving toward greater diversity and more equitable outcomes.
  • The Department promotes evidence-based innovation by using program demonstrations to experimentally test potential policy enhancements, which have included eight low-cost, behaviorally informed experiments using interagency data matching and assistance from the GSA Office of Evaluation Sciences. Other innovative research ideas from external stakeholders are supported by the Research Partnerships program. Competitive awards for Healthy Homes Technical Studies generate innovation in the evaluation and control of housing-related health and safety hazards.
  • An interagency agreement with the Census Bureau has made datasets from HUD’s randomized control trials available for linkage with census data and administrative datasets. The RCT datasets are the first intervention data added to Federal Statistical Research Data Centers (RDCs) by any federal agency, and joint support is available to help researchers gain access and learn to use the restricted data successfully for innovative research, with seven projects currently underway.
  • HUD’s Rental Assistance Demonstration, which restructures the financing of the nation’s public housing to address capital needs backlogs, has the additional innovative feature of providing tenants with a Choice Mobility option. Choice Mobility supports self-sufficiency by offering priority receipt of a Housing Choice Voucher providing freedom to move to neighborhoods with greater economic opportunities or better schools and amenities.
  • HUD’s regulation of manufactured housing production is guided by a federal advisory committee, the Manufactured Housing Consensus Committee, to provide increased ability for the industry to produce some of the nation’s most innovative, safe, and affordable housing.
  • HUD has a Robotics Process Automation initiative devoted to freeing the workforce from low-value, repetitive work through software robotics solutions. Specialized computer programs known as “bots” automate and standardize repeatable business processes without costly investments in conventional automation. Planned efforts involving payroll, accounts receivable and payable, invoice processing, inventory management, report creation, and data migration have potential to shift over 50,000 hours of employee time from low-value to high-value work.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
Score
9
Use of Evidence in Competitive Grant Programs

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY21?

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?
8.2 Did the agency use evidence of effectiveness to allocate funds in the five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)
  • The Continuum of Care program (CoC) provides homelessness assistance awards on the basis of system performance measures focused on outcomes and evidence of effectiveness. This includes up to 56 points (out of 200) for past “performance related to reducing homelessness” and four points for “reallocat[ing] lower performing projects to create new higher performing projects that are based on performance review of existing projects.” Additionally, a precondition for Continuum of Care applicants to be awarded FY19 expansion bonus funding was that they rank homeless assistance projects on the basis of how they improve system performance (p. 34).
  • The Lead Hazard Reduction Grant Program is designed to maximize the number of children under the age of six protected from lead poisoning by strategically targeting lead reduction efforts to neighborhoods where children are at greatest risk. The FY21 grants require grantees to use evidence-based lead hazard control methods, meet cost-savings, effectiveness, and grant compliance benchmarks, and gather pre- and post-treatment data to support and validate their investments. The application assigns 40 points (out of 102) based on grantees’ organizational capacity and relevant experience. Past research showing large returns on investment supported HUD’s decision to request a 24 percent increase in program funding for FY21, and HUD is funding studies using an implementation science framework to continue improving efficiency and efficacy of lead interventions.
  • The Resident Opportunity & Self-Sufficiency Service Coordinator (ROSS-SC) grant program is designed to assist residents of Public and Indian Housing make progress towards economic and housing self-sufficiency by removing the educational, professional and health barriers they face. For grantees applying for renewal funding, the application assigns up to 25 points (out of 45) for past performance, including the number of residents served and the grantee’s effectiveness in spending down past funds. The application also assigns 20 points for soundness of approach, which includes the past performance of any subcontractors and the grantee’s plans to track residents’ progress. New applicants are also assessed on their relevant experience, capacity, and soundness of approach.
  • The Indian Housing competitive grant program was established to address issues of overcrowded and physically inadequate housing identified by a PD&R needs assessment completed in 2017, Housing Needs of American Indians and Alaska Natives in Tribal Areas. The FY21 grant application assigned 20 points (out of 102) based on grantees’ organizational capacity and relevant experience. The grant application also assigns points for data supporting identified needs and past efforts to address identified needs.
8.3 Did the agency use its five largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)
  • As a condition of grant award, all HUD competitive grantees are required to cooperate (p. 5) in any HUD-sponsored research or evaluation studies.
  • The Continuum of Care program is supported by the National Homeless Data Analysis Project, which provides communities with resources to improve data collection and consistent reporting about individuals experiencing homelessness to support national Annual Homeless Assessment Reports.
  • HUD Lead Paint grantees are required to integrate evidence into their work by conducting clearance testing of all housing units treated. Technical studies provide evidence to improve lead hazard detection, evaluation, and control technologies, as well as implementation, and rigorous evaluation has demonstrated the large return on investment related to children’s health from controlling lead hazards.
8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant programs (besides its five largest grant programs)?
  • HUD’s Housing Counseling Grant Program ($37 million in FY20, as well as $12.5 million in supplemental grants) provides counseling services to tenants and homeowners. One of the program’s main objectives is to “Distribute federal financial support to housing counseling agencies based on past performance.” As such, the program allocates seven points (out of 100) for past performance based on the “the positive impacts that an Applicant’s housing counseling services had on clients.” HUD scores this item based on its own performance records.
  • HUD continues to extend the Standards for Success reporting framework to additional competitive grant programs, establishing a performance outcomes framework that will both drive performance and determine future funding recipients by providing strategically aligned performance metrics that are standardized and sufficiently granular to provide information on relative effectiveness.
8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?  
  • The Continuum of Care (CoC) program is HUD’s largest program targeted to adults and children experiencing homelessness. HUD awards COC funding to over 6,500 projects through a national competition. Policy priorities for the CoC program have focused on preventing and ending homelessness through access to permanent housing, including ending homelessness for veterans, families, youth, and people experiencing chronic homelessness. Over more than a decade, increased CoC effectiveness has been supported by Homeless Management Information Systems and evidence-based funding of increased permanent supportive housing. Between 2011 and 2020, the estimated number  of people experiencing homelessness in families with children declined by 27%. After a steady decline for the first half of the last decade, the number of people experiencing chronic homelessness increased by 42% from 2016-2020 and is back up to its highest level since 2008. At the same time however, the number of veterans experiencing homelessness declined by 43%. Following federal criteria, 78 communities and three states have effectively ended veteran homelessness.
  • HUD has taken a proactive role to address the racial disparities in rates of homeless by publishing resources and providing technical; assistance to grantees. For example, in 2019 HUD created the CoC Racial Equity Access Tool to help communities understand who is accessing their homeless service system and what outcomes those families and individuals are realizing. In 2020, HUD published a guide to Increasing Equity in the Homeless Response System Through Expanding Procurement, which provides communities with recommendations for allocating CARES Act funds to address racial and ethnic disparities in the homeless response system. More recently, responding to the finding 2019 Annual Homeless Assessment Report (AHAR) that African Americans have remained considerably overrepresented among the homeless population compared to the U.S. population, HUD published a rich set of racial equity resources, data toolkits, and research reports related to identifying disparities and implementing responses to address the overrepresentation of people of color in the homeless system.
8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?
  • HUD operates a centralized evaluation program under the guidance of the evaluation officer. As a condition of grant award, all HUD competitive grantees are required to cooperate in any HUD-sponsored research or evaluation studies and to provide program monitoring data. A number of program statutes do not authorize formal evaluation as an eligible activity for use of program funds. HUD also provides technical assistance to strengthen grantees’ capacity for evaluation and performance management capacity.
  • The Continuum of Care FY19 homelessness assistance program NOFA offers one point for applicants who propose to use requested funds to improve their ability to evaluate the outcome of projects funded by the CoC Program and the Emergency Solutions Grant program (p. 39). There was no FY 2020 CoC Program Competition, HUD renewed all awards in recognition of the fact that communities have been and will continue to be consumed with COVID-19 response and have limited capacity to participate in the traditional CoC competition.
  • HUD intends to incorporate and disseminate best practices regarding racial equity identified in the ongoing equity assessment to external stakeholders as part of the agency’s long-term equity transformation. HUD has already begun this process by publishing racial equity resources, data toolkits, and research reports related to identifying disparities and implementing responses to address the overrepresentation of people of color in the homeless system. One of these resources is a CoC Racial Equity Analysis Tool, which helps CoCs identify racial disparities in their system by presenting data on poverty rates by race and ethnicity, age, and veteran status at the CoC level of geography.
Score
4
Use of Evidence in Non-Competitive Grant Programs

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY21?

9.1 What were the agency’s five largest non-competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?
9.2 Did the agency use evidence of effectiveness to allocate funds in largest five non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)
  • Although the funding formulas are prescribed in statute, evaluation-based interventions are central to each program. HUD uses evidence from a 2015 Administrative Fee study of the costs that high-performing PHAs incur in administering a HCV program to revise its approach to providing administrative fees that incentivize PHAs to improve outcomes in leasing and housing mobility. HUD has also used the results of its Landlord Taskforce to provide guidance to PHAs on working effectively with landlords and to propose policy and fee changes to ensure strong landlord participation in the new Emergency Housing Voucher program funded through the American Rescue Plan. In allocating $5 billion in Emergency Housing Voucher funding to PHAs, HUD developed an allocation formula that considered (among other factors) evidence of PHA capacity to implement the program effectively and quickly.
  • HUD’s funding of public housing is being radically shifted through the evidence-based Rental Assistance Demonstration (RAD), which enables accessing private capital to address the $26 billion backlog of capital needs funding. Based on demonstrated success of RAD, for FY20 HUD proposed to transfer $95 million from the Operating Fund and Capital Fund to the Tenant-Based Rental Assistance fund to support RAD conversions. For FY21 HUD is proposing to remove the cap on the number of public housing developments to be converted to Section 8 contracts. HUD is beginning to evaluate RAD’s impacts on children. HUD is also conducting a Rent Reform demonstration and a Moving To Work (MTW) demonstration to test efficiencies of changing rent rules and effects on tenant outcomes.
9.3 Did the agency use its five largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)
  • Evidence-building is central to HUD’s funding approach through the use of prospective program demonstrations. These include the Public Housing Operating Fund’s Rental Assistance Demonstration (RAD), the Public Housing Capital Grants’ Rent Reform demonstration, and the Housing Choice Voucher program’s Moving To Work (MTW) demonstration grants. As Congress moved to expand MTW flexibilities to additional public housing authorities (PHAs), HUD sought authority to randomly assign cohorts of PHAs to provide the ability to rigorously test specific program innovations.
  • Program funds are provided to operate demonstrations through the HCV account, Tenant-Based Rental Assistance. These include the Tribal HUD-VA Supportive Housing (Tribal HUD-VASH) demonstration of providing permanent supportive housing to Native American veterans and the FSS-Family Unification Program demonstration that tests the effect of providing vouchers to at-risk young adults who are aging out of foster care.
9.4 Did the agency use evidence of effectiveness to allocate funds in any other non-competitive grant programs (besides its five largest grant programs)?
  • HUD-Veterans Affairs Supportive Housing (HUD-VASH) vouchers are allocated in part on the administrative performance of housing agencies as measured by their past utilization of HUD-VASH vouchers in HUD’s Voucher Management System (Notice PIH-2019-15 (HA)). The performance information helps ensure that eligible recipients are actually able to lease units with the vouchers that HUD funds. The HUD-VASH Exit Study documented that 87,864 VASH vouchers were in circulation in April 2017, contributing substantially to the 47-percent decline in the number of homeless Veterans since 2010.
9.5 What are the agency’s strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?
  • To address a severe backlog of capital needs funding for the nation’s public housing stock, the Rental Assistance Demonstration was authorized in 2011 to convert the properties to project-based Section 8 contracts to attract an infusion of private capital. The 2019 final report on the RAD evaluation showed that conversions successfully raised $12.6 billion of funding, an average of $121,747 per unit to improve physical quality and stabilize project finances. Based on the program’s successes, the limit on the number of public housing conversions was increased to 455,000 units in 2018, nearly half of the stock, and HUD has been proposing to eliminate the cap. Additionally, HUD extended the conversion opportunity to legacy multifamily programs through RAD 2.
9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?
  • Communities receiving HUD block grant funding through Community Development Block Grants, HOME block grants, and other programs are required to consult local stakeholders, conduct housing needs assessments, and develop needs-driven Consolidated Plans to guide their activities. They then provide Consolidated Annual Performance and Evaluation Reports (CAPERs) to document progress toward their Consolidated Plan goals in a way that supports continued community involvement in evaluating program efforts.
  • HUD’s Community Development Block Grant program, which provides formula grants to entitlement jurisdictions, increases local evaluation capacity. Specifically, federal regulations (Section 24 CFR 570.200) authorize CDBG recipients (including city and state governments) to use up to 20% of their CDBG allocations for administration and planning costs that may include evaluation-capacity building efforts and evaluations of their CDBG-funded interventions (as defined in 570.205 and 570.206).
Score
5
Repurpose for Results

In FY21, did the agency shift funds away from or within any practice, policy, interventions, or program that consistently failed to achieve desired outcomes?

10.1 Did the agency have policy(ies) for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?
  • The Evaluation of the Housing First model of rehousing chronically homeless individuals with serious mental illness supported a policy shift toward first achieving housing stability to provide a platform for social services. Based on such evidence, HUD continues to encourage the use of more cost-effective rapid rehousing approaches combined with increased permanent supportive housing that is integrated with mainstream services provided by HHS, VA, and others. Additionally, a precondition for Continuum of Care applicants to be awarded FY19 expansion bonus funding was that they rank homeless assistance projects on the basis of how they improve system performance.
  • HUD created the Rental Assistance Demonstration (RAD) to give public housing agencies a new tool to preserve and improve their public housing properties and address the backlog of deferred maintenance. RAD is an example of HUD allowing local agencies to shift funds from public housing to Section 8 programs to make cost effective use of those funds and preserve the affordable housing stock. Since the program’s creation in 2012, public housing authorities have converted 1,390 projects covering 179,651 housing units under the RAD program. PD&R has funded a series of evaluations of RAD that have informed its implementation.
  • CDBG-DR (Disaster Recovery) is a large and growing program funded by emergency appropriations outside of HUD’s regular budgeting process. In FY18, HUD started promoting mitigation activities for disaster-prone communities, allocating $16 billion of the $28 billion in emergency disaster recovery funds for disaster mitigation in previously disaster-stricken communities. This policy shift was informed by evidence that vulnerability of communities to disasters is increasing even as frequency and severity of severe weather events might also be increasing, such that the National Institute of Building Sciences estimated that society saves $4 in future losses for every $1 spent on mitigation. HUD also drew on the evidence of mitigation pilots through the Hurricane Sandy Rebuild by Design competition and the National Disaster Resilience Competition. By investing in mitigation activities, rather than paying to rebuild existing infrastructure in its previous form, HUD shifted funds in order to help break the cycle of publicly-funded rebuilding and repeated loss.
  • HUD grant programs typically provide for recapture of funds that are not committed in a timely fashion, or that remain unexpended after the limits. Effective management by grantees can be especially crucial for timely completion of complex housing development projects, such as with the Capital Fund for public housing and Housing Trust Fund for states. Such funds are reallocated to more effective grantees. For the new $5 billion Emergency Housing Vouchers program, HUD will assess the performance of the public housing agencies (PHAs) implementing the program one year after each PHA receives its funding increment and may revoke or reallocate unissued vouchers from PHAs with substandard leasing performance.
  • Preference points used by competitive programs favor grantees that provide evidence of successful outcomes and strategies. The Continuum of Care program awards points that shift funds toward grant applications that have demonstrated better outcomes, that rank and fund better-performing projects, and that take over programs from small and struggling recipients. As noted in the notice of funding: “To encourage CoC mergers and mitigate the potential adverse scoring implications that may occur when a high performing CoC merges with one or more lower performing CoC(s), HUD will award up to 25 bonus points to CoCs that completed a merger…”
10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?
  • Through the Community Compass technical assistance program, HUD offers numerous prepared training opportunities as well as in-depth program assistance for grantees or program recipients needing intensive, tailored assistance or long-term capacity-building support to remediate challenges and achieve their potential as HUD program partners.
  • HUD has proposed to use Public Housing operating funds set aside for receivership of troubled housing authorities more proactively to address the needs of high-risk PHAs before they go into receivership, including through competitive grants for PHAs that are troubled, substandard, at-risk, or insolvent to help preserve affordable housing for the future. The Real Estate Assessment Center collects extensive data on physical condition, finances, and management to determine PHA status, and field staff have expertise to identify risk factors and useful corrective actions.
Back to the Standard

Visit Results4America.org