Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY21?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The Director of the Office of Performance and Evaluation (OPE) serves as the Administration for Community Living (ACL) evaluation officer. OPE, which oversees the agency’s performance and evaluation work, has six full time staff positions and three full-time onsite contractors. In FY21 it had a budget of approximately $13.6 million. The Director of OPE has the education, skill, and experience to meet the Evaluation Officer requirements listed in the Evidence Act and routinely gauges the coverage, quality, methods, consistency, effectiveness, independence, and balance of the portfolio of evaluations, policy research, and ongoing evaluation activities of the agency and assesses agency capacity to support the development and use of evaluation. The Director is also the designated ACL Performance Officer.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• The Director of the Office of Performance and Evaluation (OPE) serves as the Administration of Community Living’s Chief Data Officer. OPE, which oversees the agency’s performance and evaluation work, has six full time staff positions and three full-time onsite contractors. In FY21 it had a budget of approximately $13.6 million. The Director of OPE leads ACL’s Data Governance Body, including facilitating collaborative activities among the numerous actors with responsibilities and needs for data within the agency and has demonstrated training and experience in data management, governance, collection, analysis, protection, use, and dissemination and fulfills the aspects of this role which are relevant to ACL. These include coordinating with ACL’s CIO and Chief Privacy Officer on use, protection, dissemination, and generation of data to ensure that the data needs of the agency are met; ensuring that agency data conform with data management best practices; engaging agency employees, the public, and contractors in using public data assets; and encouraging collaborative approaches on improving data use. The Director of OPE serves as the agency liaison to other federal entities through, for example, serving as the ACL representative to the HHS data council, and serving on the Federal Interagency Council on Evaluation Policy as well as the HHS Data Governance Board.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, performance improvement officer, and other related officials in order to support Evidence Act implementation and improve the agency’s major programs?

• The Director of ACL’s Office of Performance and Evaluation serves the functions of evaluation officer, chief data officer, and performance officer. In order to coordinate activities relevant to these positions, the OPE Director and staff coordinate the support, improvement, and evaluation of agency programs through implementation of an agency performance strategy, learning agenda, , and the National Institute for Disability, Independent Living, and Rehabilitation Research. The structure requires semi-annual meetings with ACL leadership and management staff and annual consultation with all program managers. In FY19 ACL instituted a council to improve ACL’s data governance and quality, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data that ACL collects through its evaluations, grant reporting, and other administrative data collections. Taken together, this robust governance structure ensures cohesive collection and use of evidence across ACL regarding program performance, evaluation, and improvement and to ensure that data are gathered, processed, and curated so as to produce evidence that program staff and agency leadership use for program and operational improvement. As an operating division without a statistical unit, ACL does not have a statistical officer.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY21?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• ACL’s public evaluation policy confirms ACL’s commitment to conducting evaluations and using evidence from evaluations to inform policy and practice. ACL seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. The policy addresses each of these principles. The policy was updated in 2021 to better reflect OMB guidance provided in M-20-12 and to more explicitly affirm ACL’s commitment to equity in evaluation.

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• ACL’s agency-wide evaluation plan was submitted to the Department of Health and Human Services (HHS) in support of HHS’ requirement to submit an annual evaluation plan to OMB in conjunction with its Agency Performance Plan. ACL’s annual evaluation plan includes the evaluation activities the agency plans related to the learning agenda and any other “significant” evaluation, such as those required by statute. The plan describes the systematic collection and analysis of information about the characteristics and outcomes of programs, projects, and processes as a basis for judgments, to improve effectiveness, and/or inform decision-makers about current and future activities.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• Based on the learning agenda approach that ACL adopted in 2018, ACL published a learning agenda in FY20. In developing the plan, ACL engaged stakeholders through meetings with program staff and grantees as required under OMB M-19-23. Most meetings with stakeholder groups, such as through conference sessions, were put on hold for 2020 due to COVID-19 travel restrictions. In 2021, ACL did communicate with stakeholder groups to contribute to ACL’s learning activities. These included working with members of the RAISE Family Caregiving Advisory Council and a range of stakeholders to inform changes to the 2021 data collection under the National Survey of Older Americans Act Participants. In 2021, ACL also released a request for information (RFI) directed to small businesses to solicit research approaches related to ACL’s current research priorities.

2.4 Did the agency publicly release all completed program evaluations?

• ACL releases all evaluation reports as well as interim information such as issue briefs, webinar recordings, and factsheets based on data from its evaluation and evidence building activities. 

2.5 Did the agency conduct an Evidence Capacity Assessment that addressed the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))

• Staff from the Office of Performance and Evaluation (OPE) play an active role in HHS’s capacity assessment efforts serving on the Capacity Assessment and learning agenda Subcommittees of the HHS Evidence and Evaluation Council. ACL’s self-assessment results were provided to HHS to support HHS’ ability to submit the required information to OMB. ACL’s self-assessment results  provided information about planning and implementing evaluation activities, disseminating best practices and findings, and incorporating employee views and feedback; and carrying out capacity-building activities in order to use evaluation research and analysis approaches and data in the day-to-day operations. Based on this information, in 2021 ACL focused on developing educational materials for ACL staff and data improvement tools for ACL grantees. In 2021 the ACL Data Council published a guide to evaluating system change initiatives,  and additional documents to promote responsible data usage: Data Quality 201:Data Visualization and Data Quality 202: Data Quality Standards. While designed initially for ACL staff, they are available on the ACL website and have been promoted through several industry conferences. 

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• Starting in 2020 and continuing into 2021, ACL is funding contracts to design the most rigorous evaluations appropriate to measure the return on investment of Aging Network, the extent to which ACL services address social determinants of health, and the value of volunteers to ACL programs.  ACL typically funds evaluation design contracts, such as those for the Older Americans Act Title VI Tribal Grants Program evaluation and the Long Term Care Ombudsman Evaluation, that are used to determine the most rigorous evaluation approach that is feasible given the structure of a particular program. While the Ombudsman program is full coverage programs, where comparison groups are not possible, ACL most frequently uses propensity score matching to identify comparison group members. This was the case for the Older Americans Act Nutrition Services Program and National Family Caregivers Support Program evaluations and the Wellness Prospective Evaluation Final Report conducted by CMS in partnership with ACL.  
• ACL’s  National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funds the largest percentage of ACL’s RCTs at 151 out of 659 (23%) of research projects employing a randomized clinical trial (RCT)To ensure research quality, NIDILRR adheres to strict peer reviewer evaluation criteria that are used in the grant award process. In addition, ACL’s evaluation policy states that “In assessing the effects of programs or services, ACL evaluations will use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, high-quality quasi-experiments offer an alternative.” ACL is in the process of implementing a method for rating each proposed evaluation against OMB’s Program Evaluation Standards and Practices as defined in OMB M-20-12.

Did the agency invest at least 1% of program funds in evaluations in FY21?

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY21 budget.

• ACL invested $18.7 million on evaluations, evaluation technical assistance, and evaluation capacity-building, representing .81% of the agency’s $2.3 billion FY21 enacted budget.

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• ACL’s Office of Performance and Evaluation (OPE) budget for evaluation was $13.6 million in FY21 and there were no significant changes to the evaluation budget since the previous year. The bulk of OPE’s evaluation funds are based on a set-aside required in Title II, section 206, of the Older Americans Act, “From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed 1⁄2 of 1 percent of such amount, for purposes of conducting evaluations under this section, either directly or through grants or contracts.” In addition, in 2017 ACL’s Office of Performance and Evaluation established a mechanism that allows ACL programs not covered by the OAA set-aside to transfer funds to OPE to be able to support evaluations of their programs. In 2017, 2018, 2019, 2020, and 2021, OPE added approximately $1.0 million, $1.7 million, $3.2 million, $1.2 million, and $2.3 million from these programs to its evaluation budget respectively.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• ACL primarily provides information resources to grantees to build their evaluation and evidence building capacity. Staff record trainings on evaluation topics, including an overview of performance measurement. ACL also has several resources and TA centers that focus on evidence building including one contract dedicated to improving performance data provided by Older Americans Act Title III, VI, and VII grantees that offers live and prerecorded webinars and a range of manuals and TA supports. ACL also published toolkits for strategic planning, data quality, performance measures, logic model development, and more. ACL provides technical assistance to grantees related to using evidence-based programs and building evidence. For example, the National Resource Center on Nutrition and Aging (NRC) provides different programs and approaches that deliver nutrition-related home- and community-based services (HCBS) administered through grants to the 56 states and territories. Access to Respite Care and Help (ARCH) provides training and technical assistance to the Lifespan Respite Network with a focus on performance measurement, sustainability, best practices, and research. The National Alzheimer’s and Dementia Resource Center supports grantees as they implement evidence-based interventions and innovative practices designed to empower and assist caregivers of persons with Alzheimer’s disease and related disorders.
• In an effort to better support Older Americans Act Title VI Native American Programs grantees with their data needs, OPE created several tools for grantee use. One tool is a template for collecting program satisfaction survey data from both nutrition and caregiver clients about the services they are receiving. This template is customizable, has a user guide and an excel spreadsheet for easy data tabulation and visualization as it creates charts from the entered data. OPE also worked to create an infographic for Title VI grantees to use by plugging in their own data (into an excel template with the help of a user guide) from a variety of sources they have ready access to so that they might create a two page visual document to share with stakeholders to showcase the services they are providing to their communities. OPE also created a Title VI Data Tracking Workbook, built with grantee input to help grantees track their data on a daily basis. This workbook provides monthly and quarterly statistics, as well as produces an annual roll up of data. Based on grantee feedback it has some customization built in and has a budget feature to help grantees manage their income and expenditures–data that ACL does not currently collect for this program. OPE has held webinars and trainings and question sessions on all of these tools in an effort to make them more user friendly for grantees and to encourage wide adoption of these tools for grantee use.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY21?

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• As part of the U.S. Department of Health and Human Services Annual Performance Plan and Report, ACL reports on the following two HHS Agency Priority Goals: (1) Increase the success rate of the Protection and Advocacy Program’s individual or systemic advocacy, thereby advancing individuals with developmental disabilities’ right to receive appropriate community based services, resulting in community integration and independence, and have other rights enforced, retained, restored and/or expanded; and (2) Improve dementia capability of long-term support systems to create dementia-friendly, livable communities (Lead Agency ACL). ACL’s outcomes measures are available, by program, in its annual Congressional Budget Justification, and include measures of program efficiency. ACL submits annual reports to congress from its Administration on Disability, the Administration on Aging, and National Institute for Disability, Independent Living, and Rehabilitation Research. ACL contributes other department-wide reports to congress such as the HHS Report to Congress on Minority Health. 

4.2 Did the agency use data/evidence to improve outcomes and return on investment?

• ACL employs a moderate approach for analyzing evidence to find ways to improve return on investment that addresses multiple parts of the agency. In FY20, as part of its ongoing effort to ensure that agency funds are used effectively, ACL funded a 3-year contract, focused on ACL’s Administration in Aging, to identify approaches to measure how and to what extent parts of the Aging Network leverage Older Americans Act funds to increase their available resources as well as how the Aging Network uses resources to measure and improve the quality of services available/provided. NIDILRR conducts research as part of their new employment research agenda to continue development of return-on-investment models that can be used by Vocational Rehabilitation agencies to optimize the services they provide. In addition, in January 2021 ACL announced a new phase for the  Innovative Technology Solutions for Social Care Referrals challenge competition. This is in addition to those launched in 2020 (Innovative Solutions to Address the direct Support Professional Crisis, Mental Health Challenge, Disability Employment Challenge). The goal of all the prize competitions is to encourage effective and efficient methods for meeting ACL’s mission and improving services to its target populations. ACL also recently published the results of a study measuring the economic value of volunteerism for Older Americans Act programs.

4.3 Did the agency have continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• As part of ACL’s performance strategy and learning agenda approach OPE staff present performance data to ACL leadership several times a year. In addition, ACL leadership reviews performance data as part of the budget justification process that informs program funding decisions. OPE staff conduct annual meetings with ACL staff to report performance measure data and results to discuss methods for incorporating performance and evaluation findings into funding and operational decision-making. As part of annual evaluation planning efforts, staff from ACL’s Office of Performance and Evaluation consult with ACL center directors to identify evaluation priorities and review proposed evaluation approaches to ensure that the evaluation questions identified will provide information that will be useful for program improvement. Two projects started in late 2020 with the goal of improving agency performance are a study of how the services provided by ACL grantees influence the social determinants of health (SDOH) and an evaluation of how ACL supports grantee use of evidence-based programs that are required under Title IIID of the Older Americans Act. In 2021 ACL began using the National Standards for Culturally and Linguistically Appropriate Services (CLAS) in Health and Health Care to inform its evaluation framework. Specifically, ACL funded this project to explore the extent to which ACL grantees employ CLAS Standards in their service delivery processes, particularly their responsiveness to cultural practices, language and communication needs, LGBTQ+ needs, and health literacy.  ACL also funded a study to examine the use and financial value of volunteers to its programs.  In addition to a final report  ACL also developed an effective practice guide to help grantees use volunteers effectively.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY21?

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• As an operating division of a CFO Act Agency, the U.S. Department of Health and Human Services, ACL is not required to have its own strategic data plan and utilizes HHS’s data strategy. In 2016, ACL implemented a Public Access Plan as a mechanism for compliance with the White House Office of Science and Technology Policy’s public access policy. The plan focused on making published results of ACL/National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funded research more readily accessible to the public; making scientific data collected through ACL/NIDILRR-funded research more readily accessible to the public; and increasing the use of research results and scientific data to further advance scientific endeavors and other tangible applications. In 2019, ACL created a council to improve ACL’s data governance, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data that ACL collects through its evaluation, grant reporting, and administrative performance measures. In 2020, its first year, the ACL Data Council produced an annotated bibliography to provide essential background information about the topic,  a Primer to detail best practices in data governance specifically as they apply to ACL, a Data Quality 101 infographic to guide decision-making processes related to data quality. ACL also has an internal tracking sheet to measure ACL response to the Federal Data Strategy.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• ACL provides comprehensive public access to its programmatic data through its Aging, Independence, and Disability Program Data Portal (AGID). ACL also had two data inventories available to the public on the NARIC website. REHABDATA, a database of rehabilitation and disability literature and the Online Program Directory contains NIDILRR’s previously-funded, currently-funded, and newly-funded grants. ACL/NIDILRR has a public access plan that was first published in February 2016. Its purpose is to make available to the public peer-reviewed publications and scientific data arising from research funded in whole or part by ACL through the NIDILRR, to the extent feasible and permitted by law and available resources. The requirements outlined in this plan are being applied prospectively and not retrospectively. ACL is also creating an internal evidence inventory that staff will be able to use to search for relevant program performance and evaluation data by agency priority question.

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c)) 

• ACL’s Office of Performance and Evaluation has access to all of ACL’s performance and evaluation data and is able to link those data and advise programs about their availability and usability. In March 2019, ACL completed the ACL Data Restructuring (DR) Project to assess the data hosted on AGID, and to develop and test a potential restructuring of the data in order to make it useful and usable for stakeholders. In 2019, ACL awarded a follow-on contract to further integrate its datasets along the lines of conceptual linkages, and to better align the measures within ACL’s data collections across the agency. ACL funded several grants to promote data linkage including the Grants to Enhance State Adult Protective Services awarded in FY19 to increase intra- and inter-state sharing of information on APS cases and the 2020 Empowering Communities to Reduce Falls and Falls Risk to develop robust partnerships, develop a result-based, comprehensive strategy for reducing falls and fall risks among older adults and adults with disabilities living in your community and directs grantees to consider CDC opportunities to broaden and improve the linkage between primary care providers and evidence-based community falls prevention programs supported by ACL.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)

• As an operating division of the U.S. Department of Health and Human Services, ACL follows all departmental guidance regarding data privacy and security. This includes project-specific reviews by ACL’s Office of Information Resource Management (OIRM), which monitors all of ACL’s data collection activities to ensure the safety and security of ACL’s data assets. In FY19, ACL awarded a contract to stand up a “Data Council” to enhance the quality, security, and statistical usability of the data ACL collects through its evaluation, grant reporting, and administrative data collections, and to develop effective data governance standards. NIDILRR’s Model Systems’ data centers have extensive standard operating procedures that are designed to secure data and protect personal and confidential information. Below are a few illustrative examples from the Model Systems’ Data Centers:
  • The Burn Model System National Data and Statistical Center has a Burn Model Systems’ procedures’ page that lists all of the Standard Operating Procedures that grantees contributing to this database must follow.
  • The Traumatic Brain Injury Model Systems’ National Data and Statistical Center has a Standard Operating Procedures’ page that describes the procedures that all grantees contributing to this database must follow
  • The Spinal Cord Injury Model Systems’ National Data and Statistical Center has a page on Using the National Spinal Cord Injury Model Systems’ Database. Descriptions of what constitutes “de-identified data” can be found on this page. 
• In addition to the Model Systems’ data centers referenced above NIDILRR developed  Part 2: Preparing Data and Documentation; this page and video is part of the larger training course that NIDILRR grantees must complete entitled NIDILRR Data Archiving and Sharing Training. Additional guidance is available on the ICPSR web page entitled Resources for National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) Grantees.
• In addition, each funding opportunity announcement states that “a data and safety monitoring board (DSMB) is required for all multi-site clinical trials involving interventions” (see for example the FOA for Disability and Rehabilitation Research Projects (DRRP): Assistive Technology to Promote Independence and Community Living (Development) HHS-2019-ACL-NIDILRR-DPGE-0355).

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• ACL data sets are made publicly available through its AGID system. ACL staff provide technical assistance through presentations and ACL’s technical assistance resource centers to grantees, including state, tribal, and local governments. The resource centers providing technical assistance include: the National Resource Center on Nutrition and Aging (NRC), the Alzheimer’s Disease Supportive Services Program (ADSSP) and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service. This technical assistance includes annual workshops and presentations at the Title VI National Training and Technical Assistance Conference; training available through the ACL funded National Ombudsman Resource Center; and the Disability and Rehabilitation Research Program (DRRP), which funds capacity building for minority research entities. In addition, NIDILRR has a number of resources to help the public access its data responsibly: The National Spinal Cord Injury Statistical Center, for example, has a pdf document entitled Using the National Spinal Cord Injury Model Systems Database. This same center also has an online Data Request Form that requestors need to complete before gaining access to data. The National Data and Statistical Center for the Traumatic Brain Injury Model Systems has a web page entitled How to obtain a dataset from the TBIMS. The Burn Model Systems’ National Data and Statistical Center has a page with instructions on how to access Burn Model System data. 

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY21?

6.1 Did the agency have a common evidence framework for researchand evaluation purposes?

• ACL defines evidence-based programs on its website. ACL’s National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) has two principal frameworks that guide and inform the generation of new knowledge and products. The stages of research framework is used to guide, inform, and track the creation of new knowledge which in turn becomes part of the larger disability evidence base. The stages of development framework is used to guide, inform, and track the development of new products and technologies. Both of these frameworks are codified in federal regulations and are described on NIDILRR’s Frameworks’ page on the ACL website. The stages of research framework is codified in 45 CFR 1330.4 while the stages of the development framework codified in 45 CFR 1330.5.

6.2 Did the agency have a common evidence framework for funding decisions?

• The Older Americans Act requires the use of evidence-based programming in Title III-D-funded activities: Disease Prevention and Health Promotion Services. In response, ACL developed a definition of the term evidence-based, and created a website containing links to a range of resources for evidence-based programs. This is a common evidence framework used for Older Americans Act funded activities. For programs that are not legislatively required to use evidence-based models, through its funding process ACL requires all programs to provide clear justification and evidence (where available) that proposed projects will achieve their stated outcomes. In 2018 ACL developed a tool to help a small number of program officers assess grantee progress towards the stated goals of their grants. Using the tool program officers have instituted corrective actions or required underperforming grantees to relinquish grant funds. ACL is developing similar tools for several other grant programs with the intention of rolling out new guidance for program officers in 2023.

6.3 Did the agency have a clearinghouse(s) or a user-friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• ACL does not have a common evidence repository that applies across the entire agency. It publishes intervention summaries of aging and disability evidence-based programs and practices. ACL funds the Evidence-Based Program Review Council to identify new community programs that meet the criteria established by the Administration for Community Living/Administration on Aging (ACL/AoA) for evidence-based programs funded through the Older Americans Act (OAA) Title III-D. To support the use of evidence-based and evidence-informed programming, service providers can find out about evidence-based programs that serve people with dementia and their caregivers by consulting a white paper drafted with funds from ACL–Translating Innovation to Impact: Evidence-based interventions to support people with Alzheimer’s disease and their caregivers at home and in their communities. The Model Systems Knowledge Translation Center (MSKTC) has worked with NIDILRR’s Model Systems grantees to develop and publish a variety of evidence-based factsheets about living with spinal cord injury, traumatic brain injury, or burn injury. ACL’s Living Well demonstration program requires grantees to use evidence-based and innovative strategies to (1) improve access to and quality of community services, (2) reduce and mitigate abuse and neglect, and (3) support empowerment, self-determination, and self-advocacy.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• ACL works through its resource centers to help grantees use evidence to drive improvements in outcomes for older adults and individuals with disabilities. For example, with funding from ACL, the National Center on Aging (NCOA), in collaboration with the Evidence-Based Leadership Council, led an innovative vetting process to increase the number of programs available to ACL’s aging network that meet the Title III-D evidence-based criteria. This process resulted in adding six new health promotion programs and three new programs for preventing falls. The Alzheimer’s Disease Supportive Services Program (ADSSP) funds competitive grants to expand the availability of evidence-based services that support persons with Alzheimer’s disease and related dementia (ADRD) and their family caregivers. Extensive evaluation of the National Chronic Disease Self-Management Education (CDSME) and Falls Prevention database helped generate important insights for potential new ACL applicants in preparing their applications using data-driven estimation procedures for participant and completion targets ACL also funded several grants, such as the Lifespan Respite Care Program: State Program Enhancement Grants and Disability and Rehabilitation Research Projects (DRRP) Program: Chronic Disease Management for People with Traumatic Brain Injury (TBI) which are designed, in part, to develop an evidence base for respite care and related services and contribute to the evidence base upon which people with TBI and their health care providers can employ effective chronic disease management practices respectively. In addition, NIDILRR provides the Rehabilitation Measures Database (RMD) which is a Knowledge Translation on-line resource with succinct summaries of instruments relevant to rehabilitation populations that provides evidence-based summaries that include concise descriptions of each instrument’s psychometric properties, reliability, validity, sensitivity, instructions for administering and scoring, and a representative bibliography with citations.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY21?

7.1 Did the agency have staff dedicated to learning innovation efforts to improve the impact of its programs?

• Agency leadership promotes innovation by requiring all program offices to explain, in their annual funding proposals, how the proposed use of funds will identify innovative practices. ACL also partially funds a Forum on Aging, Disability, and Independence which engages staff to foster discussions about innovation for coordinating and integrating aging and disability stakeholders. ACL also funds resource centers, such as the Engagement and Older Adults Resource Center which provides technical assistance and serves as a repository for innovations designed to increase the aging network’s ability to tailor social engagement activities to meet the needs of older adults.
  

7.2 Did the agency have initiatives to promote innovation to improve the impact of its programs? 

• In FY21 all ACL Centers were involved in funding innovative work. ACL released several funding opportunity announcements (FOA) focused on the identification and implementation of innovative approaches to improve programming. These included funding opportunity announcements for the Innovations in Nutrition Programs and Services – Community Research and the ACL/NIDILRR Small Business Innovation Research Program (SBIR) Phase I. ACL also funded challenge competitions such as the ACL Social Care Referrals Innovation Challenge. In 2021, ACL funded a small study to identify the innovations and adaptations by grantees in the face of COVID-19. In 2022, ACL will more closely examine the most promising innovations to evaluate their effectiveness.
• ACL is a funder of The John A. Hartford Foundation 2020 Business Innovation Award,  which recognizes aging and disability community-based organizations (CBOs) for their innovative approaches to reducing health care costs and improving the well-being of older adults and people with disabilities through strategic partnership with health care entities. 
• There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3.5 million to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.
  

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• The 2020 reauthorization of the Older Americans Act requires a new Research, Demonstration, and Evaluation Center for the Aging Network and new demonstration programs to evaluate new strategies for the recruitment, retention, or advancement of direct care workers, and the soliciting, development, and implementation of strategies; and a demonstration to address negative health impacts associated with social isolation. Further, ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches towards coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e. Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.
• NIDILRR’s research and development activities are guided by the Stages of Research Framework and the Stages of Development Framework. NIDILRR grantees conducting research projects must identify the stage of research their projects are in while grantees conducting development projects must identify the stage of development their projects are in. The stage a research project is in depends upon what is known and what is not known about a particular disability problem or topic. Research projects where relatively little is known, or the topic area is emerging, would be classified in the Exploration and discovery stage. Over time, as more becomes known, research projects progressively move into the Intervention Development phase. The next phase, known as Intervention Efficacy, means the stage of research during which a project evaluates and tests whether an intervention is feasible, practical, and has the potential to yield positive outcomes for individuals with disabilities. The final stage, known as Scale-Up Evaluation, means the stage of research during which a project analyzes whether an intervention is effective in producing improved outcomes for individuals with disabilities when implemented in a real-world setting. 
• Similarly, the stage of development a development project is in also depends upon what is known or not known about a need that informs the design and development of a product. The proof of concept stage means the stage of development where key technical challenges are resolved. Stage activities may include recruiting study participants, verifying product requirements, implementing and testing (typically in controlled contexts) key concepts, components, or systems, and resolving technical challenges. The proof of product stage means the stage of development where a fully-integrated and working prototype, meeting critical technical requirements, is created. The proof of adoption stage means the stage of development where a product is substantially adopted by its target population and used for its intended purpose. 

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY21?

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY21, the five largest competitive grant programs are: 
  1. Centers for Independent Living ($116.2 million; eligible applicants: Nonprofits; Public and State controlled institutions of higher education)
     1. One of their largest competitive grants for was the Centers for Independent Living Training and Technical Assistance Grant
  2. National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) ($113.0 million; eligible applicants: State, local, and tribal governments and nonprofits, public and State controlled institutions of higher education) 
  3. State Health Insurance Assistance Program ($52.1 million; eligible applicants: Unrestricted)
     1. One of the relevant NOFAs is for 2020 State Health Insurance Assistance Program (SHIP) Base Grant 
  4. Medicare Improvements for Patients and Providers Act Programs (MIPPA) ($50 million; Eligible applicants are: Nonprofits; City or township governments; Public and State controlled institutions of higher education; Native American tribal; Public housing authorities/Indian housing authorities; Private institutions of higher education; Native American tribal organizations; Special district governments; County governments; State governments; and Independent school districts). 
     1. A relevant NOFA is for the National Center for Benefits Outreach & Enrollment 
  5. University Centers for Excellence in Developmental Disabilities Education, Research and Service ($42.1 million; eligible applicants: entities in each State designated as UCEDDs to carry out the four core functions of interdisciplinary pre-service preparation and continuing education, community services, research, and information dissemination)

8.2 Did the agency use evidence of effectiveness to allocate funds in the five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?) 

• Independent Living (IL) NOFAs describe evaluation criteria including plans for technical assistance to enhance grant effectiveness and the provision of information developed about best practices (full announcement, p. 21). To continue receiving CIL program funding, eligible centers must provide evidence that they have previously had an impact on the goals and objectives for this funding. 
• Based on a strict interpretation of the phrase “evidence of prior effectiveness to make grant awards,” NIDILRR currently does not use evidence of prior effectiveness to make grant awards. Instead, ACL makes these grant awards by largely relying on the expert evaluative judgments of ACL peer reviewers. Making grant awards by using peer review is a standard, and widely-accepted, evidenced-based practice. For example, see page 7 and page 19 of the full DPCP full announcement.
• SHIP NOFAs describe evaluation criteria including plans to improve alignment of policies, processes, and procedures to program goals and increased accountability to program expectations at all levels (full announcement, p. 25).
• MIPPA funds are awarded to State grantees and to the National Center for Benefits Outreach and Enrollment. To continue funding without restrictions, State grantees are required to submit state plans that ACL staff review for the specific strategies that grantees will employ to enhance efforts through statewide and local coalition building. The National Center applicants must describe the rationale for using the particular intervention, including factors such as evidence of intervention effectiveness. In 2019, the Center was awarded additional funding based on prior performance–specifically, assisting over 7.6 million individuals to identify over $29.6 billion in potential annual benefits. 
  
• University Centers for Excellence in Developmental Disabilities Education, Research & Service (UCEDDs) are a nationwide network of independent but interlinked centers, representing an expansive national resource for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families. Applications are also reviewed based on their description of current or previous evidence of relevant experience (p. 30).

8.3 Did the agency use its five largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• Independence Living/Centers for Independent living grants are required to show that they are working to “improve performance, outcomes, operations, and governance of CILs.” (Full Announcement, p. 2).  These include reports on “issues, goals, outcome measures, performance and compliance” (p. 6). 
• NIDILRR, and its grantees, are in the disability and rehabilitation evidence-building business. NIDILRR grantees generate new knowledge, on particular disability topics or develop new disability products which eventually becomes part of a larger evidence base. To generate this new knowledge, NIDILRR grantees must conduct a series of research and development activities that produce important outputs. These research and development activities are guided by the following two frameworks: The NIDILRR Stages of Research Framework, and the NIDILRR Stages of Development Framework. The NIDILRR Stages of Research Framework is published in 45 CFR 1330.4 while the Stages of Development Framework is published in 45 CFR 1330.5.
  
• SHIP grantees are required to build and disseminate evidence of what works through documenting and promoting “knowledge, successes, and lessons learned within the SHIP network. This includes sharing ideas, products, and materials with other SHIP grantees, ACL, and the SHIP Technical Assistance Center” (Full Announcement, p. 5).  They are required to report on specified performance measures, but also encouraged to provide additional evidence and data, such as data related to the cost changes as a result of enrollment in Medicare Part D and Medicare Advantage plans (PDP/MA-PD) (p. 7).
• MIPPA Grant funds support the identification and dissemination of (i.e., practices built upon evidence of effectiveness) improving benefits outreach and enrollment. 
• A central purpose of UCEDD grants is the building and dissemination of evidence of what works. UCEDDs are a nationwide network of independent but interlinked centers, representing an expansive national resource for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families.  Additionally, the UCEDD Annual Report requires grantees to submit information on progress made in the previous year towards achieving the projected goals (Full Announcement, p. 35). Grantees are also specifically asked to describe how innovative designs and methods are “based on evidence and can be replicated” (p. 28).

8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant programs (besides its five largest grant programs)?

• ACL requires that evidence of effectiveness is used in all award decisions. Grant officers attend training regarding ways to include information about evidence-building into funding opportunity announcements. This includes information about text that can be included in funding announcements: 1) describing requirements for developing measurable outcomes; 2) explaining how the inclusion of evidence and evidence building plans can be used to score grant applications; and 3) instructing grant reviewers regarding rating applicants’ presentation of evidence and evidence building plans. The training was recorded and is available to all staff. 
• ACL’s Alzheimer’s Disease Programs Initiative (ADPI) translates and implements evidence-based supportive services for persons with ADRD and their caregivers at the community level. Award criteria include the extent to which applicants “describe partnerships, collaborations and innovative activities that will be implemented in support of goal/objective achievement, including the dementia specific evidence-based/evidence informed intervention(s) to be implemented in the project” (Full Announcement, p. 24).
• The review criteria for the Lifespan Respite Care Program: State Program Enhancement Grants includes the applicant’s description of “how the proposed project will build upon the accomplishments made in previous Lifespan Respite Care Program grants” (Full Announcement, p. 23).
• The award for the National Paralysis Research Center requires successful applicants to provide evidence that individuals with paralysis and other disabilities will be actively and meaningfully engaged, and demonstrate experience and expertise in carrying out the kinds of activities required (Full Announcement, pp. 4-5). 
• As selection criteria for the National Technical Assistance Center on Kinship and Grandfamilies, points were awarded for demonstrating that they were based on “the most recent, relevant, and available information and knowledge (p. 20) and for demonstrating that staff, consultants, and partners possess the appropriate experience and expertise (Full Announcement, p. 22).

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not? 

• Prior to the development of visual scene displays by the NIDILRR-funded Augmentative and Alternative Communication Rehabilitation Engineering Research Center (AAC-RERC), the only Augmentative and Alternative Communication (AAC) option was traditional grid displays with isolated symbols presented in rows and columns. It was difficult for many adults with acquired conditions resulting in significant language and cognitive limitations to use these traditional grid displays. Visual Scene Displays (VSDs) offer an easier alternative to traditional grid displays. They go beyond standard pictures and symbols organized in rows and columns by providing information on the situation or context. Put more simply, VSDs are photos or pictures that people can use to communicate messages to others. These photos depict familiar scenes, objects or people–and users can touch “hot spots” on the photo to speak messages that relate to the pictured scene or object. For example, a person with aphasia might touch a hotspot on a picture of a sibling and say this is my sister. This additional information on the situation and context makes it easier for persons with complex communication needs to express their wants and needs and therefore enhances their ability to interact and participate with others in the community. Research from the AAC RERC and external researchers demonstrates the effectiveness of VSDs with adults with severe chronic aphasia, primary progressive aphasia, dementia, etc. As a result of the continued efforts of the AAC-RERC and their partners, this VSD technology has been successfully transferred to all of the major AAC manufacturers and app developers.
• NIDILRR-funded grant activities regularly produce publications that use evidence to build knowledge and promote diversity and inclusion. This included recommendations for reducing barriers to access to healthcare that face coverings pose particularly to the deaf and hard of hearing. They also produced a mixed methods study identifying barriers to access to healthcare that individuals with disabilities face, and also identifying aspects of the Affordable Care Act which have improved enforcement of laws prohibiting discrimination on the basis of disability.  
• ACL’s Alzheimer’s Disease Supportive Services Program (ADSSP) encourages the translation of dementia-specific interventions for use in communities. Examples include: the Savvy Caregiver (evidence-based) psychoeducational intervention focused on training family caregivers about the basic knowledge, skills, and attitudes needed to handle the challenges of caring for a family member with Alzheimer’s disease and to be an effective caregiver; Cuidando con Respeto (evidence-informed), Spanish version of the original Savvy Caregiver Program; and Savvy Caregiver Express (evidence-informed), a condensed version of the original Savvy Caregiver Program. ACL’s requirement for inclusion of dementia specific evidence-based interventions is demonstrated in the 2018 funding opportunity announcement entitled Alzheimer’s Disease Programs to States and Communities. 
  

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• Funding opportunity announcements and grant reviews stress the need for strong performance measurement and evaluation. ACL’s technical assistance centers–the National Resource Center on Nutrition and Aging (NRC), the Alzheimer’s Disease Supportive Services Program (ADSSP) and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service–promote the use and generation of evidence with ACL grantees. Grantees manuals also include information about the importance of and requirements for evaluation (see the Administration on Aging: Title VI Resource Manual). Staff of ACL’s Office of Performance and Evaluation make presentations regarding the importance of evidence with regional staff who are in frequent contact with State grantees and at grantee conferences (see ACL Track: The ACL Older Americans Act (OAA) Performance System–Crossing the Finish Line and ACL/CMS Track: Raising the Bar in Medicaid HCBS & Community Inclusion–Showcasing Transformation presented at the 2019 home- and community-based services (HCBS) conference; ACL Track: Assuring the Health & Welfare of Medicaid HCBS Beneficiaries: Federal Findings, Investments, & Promising Practices in Systems Change and ACL Track: Innovative Housing & Health & Human Services Collaborations: A Game-Changer in Supportive Housing & Community Living presented at the 2018 HCBS conference).

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY21?

9.1 What were the agency’s five largest non-competitive programs and their appropriation amounts (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY21, the five largest non-competitive grant programs are:
  1. Nutrition Services, ($951.8 million; eligible applicants: States)
  2. Home and Community Based Supportive Services ($392.6 million; eligible applicants: States);
  3. Caregiver Support Services ($188.9 million; eligible applicants: States)
  4. State Councils on Developmental Disabilities ($79 million; eligible applicants: States and nonprofits based in a State); 
  5. Developmental Disabilities Protection and Advocacy ($40.8 million; eligible applicants: State and nonprofits based in a States)

• As these are based on formula grants, the funding amount distributed to the States and tribal organizations are not determined using evidence-based application processes. Rather, the States and tribal organizations are responsible for directing the funds to evidence-based programs and organizations.

9.2 Did the agency use evidence of effectiveness to allocate funds in the largest five non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)

• Authorizing legislation for ACL’s largest non-competitive grant programs requires consideration of evidence-based programming as a requirement of funding. The Developmental Disabilities Assistance and Bill of Rights Act of 2000 allows for the withholding of funding if (1) the Council or agency has failed to comply substantially with any of the provisions required by section 124 to be included in the State plan, particularly provisions required by paragraphs (4)(A) and (5)(B)(vii) of section 124(c), or with any of the provisions required by section 125(b)(3); or (2) the Council or agency has failed to comply substantially with any regulations of the Secretary that are applicable.” As a condition of funding non-competitive grandees are required to “determine the extent to which each goal of the Council was achieved for that year” and report that information to ACL.
  
• States that receive Older Americans Act Home and Community-Based Supportive Services Title III-D funds are required to spend those funds on evidence-based programs to improve health and well-being, and reduce disease and injury. In order to receive funding, states must utilize programs that meet ACL’s definition of evidence-based or are defined as evidence-based by another HHS operating division. Under the Older American Act, caregiver support programs are required to track and report on their use of evidence-based caregiver support services.

9.3 Did the agency use its five largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• ACL’s Nutrition Services provides grants for innovations in nutrition programs and services. These research projects must have the potential for broad implementation and demonstrate potential to improve the quality, effectiveness, and outcomes of nutrition service programs by documenting and proving the effectiveness of these interventions and innovations. They must also target services to underserved older adults with greatest social and economic need, and individuals at risk for institutional placement, to permit such individuals to remain in home and community-based settings. Consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using its 1% Nutrition authority to fund $3.5 million for nutrition innovations and to test ways to modernize how meals are provided to a changing senior population. One promising demonstration currently being carried out by the Georgia State University Research Foundation (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission) which has drawn widespread attention is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed. 
  
• Under Home and Community-Based Services, FY12 Congressional appropriations included an evidence-based requirement for the first time. OAA Title III-D funding may be used only for programs and activities demonstrated to be evidence-based. The 
• National Council on Aging maintains a tool to search for evidence-based programs that are approved for funding through OAA Title III-D.
  
• ACL’s Caregiver Support Services builds evidence in a number of areas. These include a national survey of caregivers of older adult clients, gathering and reporting best practices regarding grandparents raising grandchildren, adapting and scaling evidence-based programs for children and older adults with disabilities through the RESILIENCE Rehabilitation Research and Training Center, and other similar efforts.
  
• State Councils on Developmental Disabilities design five-year state plans that address new ways of improving service delivery. To implement the state plans, Councils work with different groups in many ways, including funding projects to show new ways that people with disabilities can work, play, and learn, and seeking information from the public and from state and national sources.
  
• State Protection & Advocacy Systems encompass multiple avenues of protection and advocacy including specialization in individuals with developmental disabilities, assistive technology, voting accessibility, individuals with traumatic brain injury, and technical assistance. The Developmental Disabilities Assistance and Bill of Rights Act of 2000 requires Administration on Intellectual and Developmental Disabilities (AIDD) grantees to report annually on progress achieved through advocacy, capacity building, and systemic change activities.

9.4 Did the agency use evidence of effectiveness to allocate funds in any other non-competitive grant programs (besides its five largest grant programs)?

• The 2020 reauthorization of the Older Americans Act requires that Assistive technology programs are “aligned with evidence-based practice;” that person-centered, trauma informed programs “incorporate evidence-based practices based on knowledge about the role of trauma in trauma victims’ lives;” and that a newly authorized Research, Demonstration, and Evaluation Center for the Aging Network increases “the repository of information on evidence based programs and interventions available to the aging network, which information shall be applicable to existing programs and interventions, and help in the development of new evidence-based programs and interventions.”
  

9.5 What are the agency’s 1-2 strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Since 2017, ACL has awarded Innovations in Nutrition grants to 11 organizations to develop and expand evidence-based approaches to enhance the quality and effectiveness of nutrition programming. ACL is currently overseeing five grantees for innovative projects that will enhance the quality, effectiveness, and outcomes of nutrition services programs provided by the national aging services network. The grants total $1,197,205 for this year with a two-year project period. Through this grant program, ACL aims to identify innovative and promising practices that can be scaled across the country and to increase the use of evidence-informed practices within nutrition programs. 

9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• All funding opportunity announcements published by ACL include language about generating and reporting evidence about their progress towards the specific goals set for the funds. Grantee manuals include information about the importance of and requirements for evaluation. The National Ombudsman Resource Center, funded by ACL, provides self-evaluation materials for Long-Term Care Ombudsman Programs (LTCOP) funded under Title VII of the Older Americans Act.

In FY21, did the agency shift funds away from or within any practice, policy, interventions, or program that consistently failed to achieve desired outcomes?

10.1 Did the agency have a policy for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• Because much of ACL’s funding is based on non-competitive formula grants that cannot be reallocated to other programs or grantees, there is not an ACL-wide policy for this purpose. For several programs, such as most under the Older American Act, “entities such as states, U.S. territories, and tribal organizations are allotted funding based on a population-based formula factor (e.g., aged 55 and over, aged 60 and over, or aged 70 and over). Some statutory requirements for program funding allocations include a “hold harmless” provision, which guarantees that state or other entities’ allotment will remain at a certain fiscal year level or amount, provided sufficient funding in a given year. ACL is working with GSA’s Office of Evaluation Sciences (OES) to test methods for improving outcomes for its congregate meals programs. Under the Older Americans Act, congregate meal sites are required to accept donations from meal recipients. But, there has been a concern regarding how to balance the collection of funds that can be used towards meal service and making meal recipients that cannot afford to donate uncomfortable, thus suppressing attendance. This study, expected to be completed in FY 2022, will offer concrete evidence to improve program operations.

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• While much of ACL’s funding is based on noncompetitive formula grants, and therefore cannot be reallocated to other programs, evaluation staff work closely with program staff to identify ways to translate evaluation findings into technical assistance and other types of program support. For example, based on early results from an evaluation of the Tribal Grant program, ACL has developed new program support materials to improve the delivery of Tribal Caregiver programs. 
  
• ACL typically proactively provides technical assistance in order to help programs to be successful, rather than redirecting funding. For example, the State Health Insurance Assistance Program (SHIP) is funding a national Technical Assistance center for this purpose.