1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The Director of the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Behavioral Health Statistics and Quality (CBHSQ) Office of Evaluation serves as the agency’s evaluation lead with key evaluation staff housed in this division. The Office of Evaluation (OE) is responsible for providing centralized planning and management of program evaluation across SAMHSA in partnership with program originating Centers. SAMHSA evaluations are funded from program funds that are used for service grants, technical assistance, and for evaluation activities. Evaluations have also been funded from recycled funds from grants or other contract activities, as described in the FY21 Congressional Justification. OE has focused on hiring to continue to build internal expertise to boost coordination of the Foundations for Evidence-Based Policymaking Evidence Act implementation with the Program Centers and Policy Lab.  

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• The Director of CBHSQ will serve as the Chief Data Officer (pending hiring). In Fiscal Year 2021, the Deputy Director of CBHSQ served as the Chief Data Officer with the Director of the Office of Evaluation within CBHSQ serving as liaison for the HHS Data Governance Group. CBHSQ is the government’s lead agency for behavioral health data and research. In FY21, CBHSQ worked to increase technical expertise and fill staff and leadership vacancies to increase SAMHSA’s capacity to build and use evidence-informed policies and programs.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, performance improvement officer, and other related officials in order to support, improve, and evaluate the agency’s major programs?

• In FY21, the CBHSQ Deputy Director met weekly with the evaluation officer, the lead statistician (Director of the Division of Surveillance and Data Collection) and the Director of the Office of Program Analysis and Coordination (OPAC) to ensure coordination in support of the Evidence Act Implementation, data dissemination and data quality. Since 2020, SAMHSA has been building its capacity to coordinate the activities for Evidence Act implementation by filling key vacancies and onboarding subject matter experts. CBHSQ and the National Mental Health and Substance Use Policy Lab, an office within SAMHSA that is required to identify and disseminate evidence-based practices as well as to collaborate with CBHSQ, has worked closely on activities for the Data Governance Board during FY21. SAMHSA’s OE began standardizing evaluation activities and protocols across SAMHSA during FY21 by soliciting input from its internal stakeholders and will begin implementation of the feedback in FY22. This will include the development and implementation of a SAMHSA Evidence Act Governance Group.  In addition, each Center in SAMHSA has formed a Diversity, Equity and Inclusion (DEI) workgroup that meets regularly to use data to advance SAMHSA’s efforts towards health equity and inclusion.

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• Under the Evidence Act, federal agencies are expected to expand their capacity for engaging in program evaluation by not only designating evaluation officers and developing learning agendas but also producing annual evaluation plans and enabling a workforce to conduct internal evaluations. In FY2020, SAMHSA developed a Standard Operating Procedure for Program Evaluations: Program Evaluation SOP and completed a capacity assessment and evaluation plan as part of an HHS-wide initiative.
• SAMHSA’s internal Evaluation Policy and Procedures (P and P), which functions as SAMHSA’s agency-wide evaluation policy, is currently being updated. The P and P documentation is being updated in coordination with the Office of Behavioral Health Equity (OBHE), as OBHE supports efforts to reduce disparities in mental and/or substance use disorders across populations. OBHE is organized around key strategies:
  1. The data strategy utilizes federal and community data to identify, monitor, and respond to behavioral health disparities.
  2. The policy strategy promotes policy initiatives that strengthen the impact of SAMHSA programs in advancing behavioral health equity.
  3. The quality practice and workforce development strategy expands the behavioral health workforce capacity to improve outreach, engagement, and quality of care for minority and underserved populations.
  4. The communication strategy increases awareness and access to information about behavioral health disparities and strategies to promote behavioral health equity.
• OBHE seeks to impact SAMHSA policy and initiatives by:
  • Creating a more strategic focus on racial, ethnic, and LGBT+ populations in SAMHSA investments
  • Using a data-informed quality improvement approach to address racial and ethnic disparities in SAMHSA programs
  • Ensuring that SAMHSA policy, funding initiatives, and collaborations include emphasis on decreasing disparities
  • Implementing innovative, cost-effective training strategies to a diverse workforce
  • Promoting behavioral health equity at a national level
  • Serving as a trusted broker of behavioral health disparity and equity information
  • Providing consultations and presentations on issues related to behavioral health equity

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• As part of the Evidence Act, agencies within HHS submitted a plan that lists and describes the specific evaluation activities the agency plans to undertake in the fiscal year following the year in which the evaluation plan is submitted (referred to as the HHS Evaluation Plan). The HHS Evaluation Plan and Evidence-Building Plan is organized based on priority areas drawn from HHS’ Departmental Priorities, the proposed Strategic Plan goals, and proposed Agency Priority Goals. Currently, SAMHSA’s evaluation plan is aligned with the Evidence Act. For FY22, the SAMHSA’s research priority is: “How will SAMHSA collect, analyze, and disseminate data to inform policies, programs, and practices?” and has outlined four relevant objectives of the research.  
• SAMHSA, through its Office of Behavioral Health Equity, focuses on racial equity, diversity, and inclusion. As part of this work, each grantee is required to submit a DIS or Disparity Impact Statement, which requires grantee focus on access to, use of, and outcomes from SAMHSA-funded services as it applies to underserved communities.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• SAMHSA submitted a learning agenda that is currently under review with HHS and OMB and not publicly available. In the learning agenda, SAMHSA highlights key evaluation studies that reflect the administration’s priorities. In the learning agenda, SAMHSA highlighted the following evaluation activities:
  
  • SAMHSA’s Report to Congress on Garrett Lee Smith (GLS) Youth Suicide Prevention and Early Intervention Program.
  • Summative Program Evaluations (e.g., Strategic Prevention for Prescription Drugs or SPF-Rx). This program is designed to prevent prescription drug misuse among youth aged 12 to 17 and adults aged 18 and older. The program is developed to respond to a critical priority area in SAMHSA’s FY2019-FY2023 Strategic Planning Priority 1: Combating the Opioid Crisis through Expansion of Prevention, Treatment and Recovery Support Services).
  • Performance Measurement of SAMHSA’s discretionary grants (40-50 Program Profiles).
  • Internal Formative Program Evaluations (e.g., Projects for Assistance in Transition from Homelessness or PATH). The PATH evaluation report includes information on funding, staffing, numbers served/contacted and enrolled, client demographics, service provision and service referrals made and attainment. Data are submitted by the PATH providers via the SAMHSA PATH Data Exchange (PDX), though parts are to be provided through local Homeless Management Information Systems (HMIS). The PATH grantees’ State PATH Contacts (SPCs) approve the data submitted by their providers.
  • Evidence Reviews (e.g., Evidence-Based Behavioral Practice (EBBP), which is a project at SAMHSA that creates training resources to help bridge the gap between behavioral health research and practice) (e.g. MOUD implementation in CJ settings).

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 3115, subchapter II (c)(3)(9))

• As part of the HHS Evidence and Evaluation Council, all agencies conducted an internal capacity assessment. This assessment was included in the HHS report. In addition, SAMHSA shares resources for evidence-based and culturally relevant interventions – see Strategies and Lessons Learned (2011-2020).

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• SAMHSA does not apply one strategy for all evaluations but employs a variety of models including performance monitoring, formative, process, and summative evaluations using primarily quantitative data and also mixed methods when appropriate and available. SAMHSA strives for a balance between the need for collecting data and the desire to minimize grantee data collection burden. For example, in FY21, an evaluation of SAMHSA’s Naloxone Education and Distribution Program, used a mixed methods approach examining qualitative data from key informant interviews and focus groups coupled with SAMHSA’s discretionary grant data collected through the SAMHSA Performance Accountability and Reporting System (SPARS). Another example is a final report for SAMHSA’s Strategic Prevention Framework–Prescription Drug Misuse program (SPF-Rx) that included several sources of primary and secondary quantitative data (from SAMHSA, CDC, etc.) mixed with interviews all in response to three primary evaluation questions.
• SAMHSA is in the process of updating its Program Evaluation SOP. In addition, SAMHSA has developed a draft evaluation plan that includes a dissemination strategy for each of its current evaluation projects recognizing that one size does not fit all. The plan is still under review. 
• SAMHSA is partnering with the National Institute on Drug Abuse (NIDA) to support the HEALing Communities Study (HCS), which is a research initiative that intends to enhance the evidence base for opioid treatment options. Launched in 2019, HCS aims to test the integration of prevention, overdose treatment, and medication-based treatment in select communities hard hit by the opioid crisis. This comprehensive treatment model will be tested in a coordinated array of settings, including primary care, emergency departments, and other community settings. Findings will establish best practices for integrating prevention and treatment strategies that can be replicated by communities nationwide. 
• SAMHSA has also supported the National Study on Mental Health, which intends to provide national estimates of mental health and substance use disorders among U.S adults ages 18 to 65. For the first time, the NSMH will include adults living in households across the U.S. as well as prisons, jails, state psychiatric hospitals and homeless shelters Data will be available in 2023.

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY21 budget.

• SAMHSA invested $106 million on evaluations, evaluation technical assistance, and evaluation capacity-building, representing 1.8 % of the agency’s $5.9 billion FY21 budget. Information concerning the efforts to boost the share of spending on minority and women-owned businesses, community-based evaluators, or nontraditional evaluation contractors was not readily available.
  

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• SAMHSA’s FY21 evaluation budget is $135 million and it comes from the Public Health Service (PHS) Evaluations funds issued through annual appropriations. It is the same amount as FY 2020. 
• Within the PHS Act Sec. 1920 and 1921, SAMHSA is authorized to obligate 5% of the amounts appropriated for data collection and program evaluation under Sec. 1920 and for technical assistance, national database, data collection, and program evaluations under Sec. 1921. The FY21 consolidated appropriation limited the amount to $21 million for funding associated with Sec. 1920 and not more than 2% of SOR funding (2% of $1.5B or $30M) for funding associated Sec. 1921.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• SAMHSA’s Evidence-Based Practices Resource Center aims to provide communities, clinicians, policymakers, and others in the field with the information and tools they need to incorporate evidence-based practices into their communities or clinical settings. The Center lists nine technical assistance projects, two of which appear to provide financial or other resources to help city, county, and state governments or other grantees build evaluation capacity.
  • The Bringing Recovery Supports to Scale Technical Assistance Center Strategy (BRSS TACS) advances recovery supports and services for people with mental or substance use disorders and their families. The BRSS TACS website indicates it has provided training and technical assistance for building the capacity of peer-run, recovery community, and family organizations through evaluation, among six other topics. 
  • The National Training and Technical Assistance Center for Child, Youth & Family Mental Health (NTTAC) provides states, tribes, and communities with training and technical assistance on children’s behavioral health, with a focus on systems of care. NTTAC’s Training and Technical Assistance activities for clinical best practices, wraparound services, and workforce development focus on evaluation, fidelity assessment, and quality assurance, among nine other topics. 
• All SAMHSA grantees (competitive and non-competitive) may designate set-aside funds for data collection, data report into SAMHSA’s Performance Accountability and Reporting System (SPARS) and individual evaluation activities. SPARS provides access to online data entry, reporting, technical assistance, and training resources to support grantees in reporting timely and accurate data to SAMHSA. There are multiple resources available to state and community grantees including a resource library with general and Center specific information on data collection, performance monitoring, disparities impact statements, and providing trauma informed care (to name just a few). A link to these resources is not available as access to these resources require that grantees enter the password protected website. 

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• The SAMHSA Strategic Plan FY2019-FY2023 outlines five priority areas with goals and measurable objectives to carry out the vision and mission of SAMHSA. For each priority area, an overarching goal and series of measurable objectives are described followed by examples of key performance and outcome measures SAMHSA will use to track progress. 
• In addition, SAMHSA collects Disparity Impact Statements (DIS) from grantees to ensure SAMHSA programs are inclusive of underserved racial and ethnic minority populations in their services, infrastructure, prevention, and training grants. These populations have been underrepresented in SAMHSA grants. 
• The DIS is based on the framework of Access, Use and Outcomes: 
  • Access: Who are the subpopulations being served by the program? 
  • Use: What types of services does each subpopulation get? 
  • Outcomes: Given the specified outcomes of the program, how do these vary by subpopulations? 
• The DIS is a Secretarial Priority from the Department of Health & Human Services’ Action Plan to Reduce Racial and Ethnic Health Disparities. The objective is to “Assess and heighten the impact of all HHS policies, programs, processes, and resource decisions to reduce health disparities. HHS leadership will assure that: … (c) Program grantees, as applicable, will be required to submit health disparity impact statements as part of their grant applications.” The Secretarial Priority focused on underserved racial and ethnic minority populations (e.g., Black/African American; Hispanic/Latino; Asian American, Native Hawaiian and Pacific Islander; and American Indian/Alaska Native). SAMHSA’s Office of Behavioral Health Equity also includes LGBTQ+ populations as an underserved, disparity-vulnerable group.  
• The standard Government and Performance Results Act (GPRA) data collected by the grantee are used to inform the access, use and outcomes questions. No new data are collected for the DIS. Disaggregating the data by subpopulations will help target gaps in who is included in the grant, what the differences in services provided across subpopulations are, and how outcomes differ across subpopulations.

4.2 Did the agency use data/evidence to improve outcomes and return on investment?

• The Office of Evaluation in partnership with SAMHSA Program Centers, oversees the identification of a set of performance indicators to monitor SAMHSA programs in collaboration with program staff and the development of periodic program profiles for use in agency planning, program change, and reporting to departmental and external organizations. SAMHSA’s Performance Accountability and Reporting System (SPARS) serves as the mechanism for the collection of performance data from agency grantees. SAMHSA Program Centers staff examine data entered into SPARS on a regular and real-time basis to manage grant programs and improve outcomes. The data in SPARS is available in .csv file, via report or through data visualization (bar charts, etc). 
• In FY21, SAMHSA staff and grantees were able to view demographic data to compare clients by race, ethnicity, gender, age, etc. over time. On an annual basis, SAMHSA produces SPARS informed program and topical profiles to examine a program’s performance. These profiles, to be shared with grantees for FY21, include disaggregate outcomes by race and other demographics as well as changes in behavior associated with their time in the grant program. 

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• Since April 2020, CBHSQ’s Office of Evaluation has offered weekly technical assistance and training on data analysis, performance management and evaluation. These one-hour sessions offer opportunities for SAMHSA Program Center staff and CBHSQ to share challenges and opportunities faced by grantees and strategize solutions. These sessions also offer an opportunity for cross-center collaboration and process improvement as project officers share and learn from officers managing programs in other centers. These cross-center meetings allow CBHSQ to learn about challenges in the field, technological challenges using SPARS, and opportunities to make the system more user-friendly. The Project officers often share grantee questions and concerns for discussion and joint problem solving. SAMHSA collects these questions to include in FAQ documents.
• Since April 2020, every Wednesday, the Office of Evaluation offers a Center specific webinar covering selected grant programs and data visualizations providing a targeted approach to capacity building. These are designed for performance management of discretionary grants but Center and agency leadership are invited and have attended. These are internal to SAMHSA and not open to political leaders. For example, on the second Wednesday of each month, the Office of Evaluation focuses on the Center for Substance Abuse Treatment’s grant programs to offer a deep dive into performance while also discussing efforts to increase data quality. The third Wednesday includes a focus on the Center for Substance Abuse Prevention and the fourth Wednesday focuses on programs funded through the Center for Mental Health Services.
• SAMHSA has been modernizing the SPARS system to include data visualization and more useful performance management reports. The annual program profiles offer another opportunity for SAMHSA staff to work collaboratively to better understand the challenges facing grantees and allow for modifications and the development of technical assistance for continuous quality improvement. These two-page resource documents provide a snapshot of descriptive data on client-level demographics and activities completed in the previous year.

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• The SAMHSA Strategic Plan FY19-23 outlines five priority areas to carry out the vision and mission of SAMHSA, including Priority 4: Improving Data Collection, Analysis, Dissemination, and Program and Policy Evaluation. This Priority includes three objectives: 1) Develop consistent data collection strategies to identify and track mental health and substance use needs across the nation; 2) Ensure that all SAMHSA programs are evaluated in a robust, timely, and high-quality manner; and 3) Promote access to and use of the nation’s substance use and mental health data and conduct program and policy evaluations and use the results to advance the adoption of evidence-based policies, programs, and practices. 
• CBHSQ recently updated their data transfer agreement for a uniform and protected sharing of data. SAMHSA has used a DUA for data sharing for several years .
• SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) is the lead Federal government agency for behavioral health data and research. As an OMB-recognized Federal Statistical Unit, CBHSQ adheres to all laws, regulations, and guidelines related to best practices of data dissemination and data stewardship, such Statistical Policy Directive Number Four on data dissemination, the Confidential Information Protection and Statistical Efficiency Act, and the 2018 Evidence Act. SAMHSA also adheres to strict scientific guidelines, as set forth in Statistical Policy Directive Number One and our parent agency’s (HHS) Scientific Integrity Principles. Additionally, SAMHSA explicitly states the agency’s public commitment to scientific integrity.  
• CBHSQ uses a data transfer agreement for a uniform and protected sharing of data. This was updated in FY21. Additionally, as the main center within SAMHSA that collects, stewards, and disseminates data, CBHSQ is in the process of developing a short-term and long-term Strategic Data Plan. SAMHSA has in the past had an agency-wide “Strategic Plan and Data Strategy” that matches important agency priorities to data collected by SAMHSA. SAMHSA is also working with HHS on a department-wide data strategy including a data maturity model and policies for data governance and sharing.
• Within CBHSQ, SAMHSA partners with the National Center for Health Statistics to offer access to individuals for restricted use data for research and evaluation purposes. This is a carefully controlled process designed to ensure data, and the individuals that provide the data, are protected. 
• The Office of Behavioral Health Equity within SAMHSA coordinates efforts to reduce disparities in mental and/or substance use disorders across populations. OBHE is organized around key strategies: data strategy, policy strategy, quality practice and workforce development strategy and communication strategy.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• SAMHSA’s Report and Dissemination site identifies seven data collections: the National Survey on Drug Use and Health (NSDUH): Treatment Episode Data Set (TEDS); National Survey of Substance Abuse Treatment Services (N-SSATS); the National Mental Health Services Survey (N-MHSS); Drug Abuse Warning Network (DAWN); Mental Health Client-Level Data (MH-CLD); and the Uniform Reporting System (URS). SAMHSA has also made numerous data collection and survey datasets publicly available at the Substance Abuse and Mental Health Data Archive (SAMHDA), which include online analytic capabilities and downloadable datasets.  

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c))

• The Center for Behavioral Health Statistics and Quality (CBHSQ) oversees data collection initiatives and provides publicly available datasets so that some data can be shared with researchers and other stakeholders while preserving client confidentiality and privacy. 
• In FY21, SAMHSA’s CBHSQ built internal technical capacity for data collections and began the process of modernizing them. For example, the N-SSATS and N-MHSS have been combined into the National Substance Use and Mental Health Services Survey (NSUMHSS) in an effort to decrease burden and duplication of responses. In addition, CBHSQ, in partnership with the Center for Mental Health Services is now SAMHSA’s Substance Abuse and Mental Health Data Archive (SAMHDA) which contains substance use disorder and mental illness research data available from CBHSQ’s seven data collections for restricted and public use. SAMHDA promotes the access and use of SAMHSA’s substance abuse and mental health data by providing public-use data files and documentation for download and online analysis tools to support a better understanding of this critical area of public health.
• SAMHSA’s Substance Abuse and Mental Health Data Archive (SAMHDA) contains substance use disorder and mental illness research data available from CBHSQ’s seven data collections for restricted and public use. SAMHDA promotes the access and use of SAMHSA’s substance abuse and mental health data by providing public-use data files and documentation for download and online analysis tools to support a better understanding of this critical area of public health. In addition, SAMHSA partners with the National Center for Health Statistics to make restricted use data available through the Research Data Center (RDC). The National Center for Health Statistics (NCHS) operates the Research Data Center (RDC) to allow researchers access to restricted-use data. For access to the restricted-use data, researchers must submit a research proposal outlining the need for restricted-use data. In FY21, many of the procedures for the application process moved in-house from NCHS and a CBHSQ RDC website was created.
• Also, SAMHSA implements the Disparity Impact Statement (DIS). DIS is a Secretarial Priority from the Department of Health & Human Services’ Action Plan to Reduce Racial and Ethnic Health Disparities (2011). The objective is to “Assess and heighten the impact of all HHS policies, programs, processes, and resource decisions to reduce health disparities. HHS leadership will assure that: … (c) Program grantees, as applicable, will be required to submit health disparity impact statements as part of their grant applications.” The Secretarial Priority focused on underserved racial and ethnic minority populations, e.g., Black/African American; Hispanic/Latino; Asian American, Native Hawaiian and Pacific Islander; and American Indian/Alaska Native. SAMHSA’s Office of Behavioral Health Equity also includes LGBT populations as underserved, disparity-vulnerable groups.
• Through the SAMHSA Performance Accountability and Reporting System (SPARS), grantees and SAMHSA program staff monitor the performance of grantees and, when performance is below targets, provide technical assistance and support. This allows SAMHSA to support communities during the grant process. SAMHSA staff meet with grantees regularly to discuss progress and to examine data entered into SPARS ensuring a timely submission of data.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)

• The National Survey on Drug Use and Health (NSDUH) annual, national survey has developed a statistical disclosure control technique called MASSC to protect confidentiality of the data. MASSC stands for Micro-Agglomeration, Substitution, Subsampling, and Calibration. It was a disclosure limitation methodology specifically developed for NSDUH to meet the requirements of CIPSEA. SAMHSA recognizes the inherent trade-off between disclosure risk and information loss. The goal of MASSC is to control the disclosure risks while minimizing the impact of the disclosure control measures on the quality of the data in a comprehensive and integrated manner. MASSC has been successfully used to create NSDUH public use files (PUFs) since 1999.
• In addition to having a Confidentiality Officer within CBHSQ who ensures staff complete training and sign a confidentiality statement, SAMHSA offers a certificate of confidentiality (CC) that protects grantees from legal requests for names or other information that would personally identify participants in the evaluation of a grant, project, or contract. CBHSQ trains all staff in good data stewardship, whether the data is covered by CIPSEA or the Privacy Act (5 U.S.C. 552a) and the Public Health Service Act (42 U.S.C.290aa(n)).
• For the CBHSQ national data sets, SAMHSA uses multiple means to protect data and ensure the protection of personally identifiable information including encryption, multifactor identification (MASSC) and limiting access to data. 
• SAMHSA’s Performance and Accountability and Reporting System (SPARS) hosts the data entry, technical assistance request, and training system for grantees to report performance data to SAMHSA. SPARS serves as the data repository for the Administration’s three centers, Center for Substance Abuse and Prevention (CSAP), Center for Mental Health Services (CMHS), and Center for Substance Abuse Treatment (CSAT). In order to safeguard confidentiality and privacy, the current data transfer agreement limits the use of grantee data to internal reports so that data collected by SAMHSA grantees will not be available to share with researchers or stakeholders beyond SAMHSA, and publications based on grantee data will not be permitted.
  

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• SAMHSA provides both public access and restricted use access to its datasets in a variety of ways. Specific examples are highlighted below.
  • The Center of Excellence for Protected Health Information (CoE for PHI) is a SAMHSA funded technical assistance project designed to develop and increase access to simple, clear, and actionable educational resources, training, and technical assistance for consumers and their families, state agencies, and communities to promote patient care while protecting confidentiality. 
  • CBHSQ’s various data collection’s data are available (1) as pre-published estimates, (2) via online systems, and (3) as microdata files.” A description of CBHSQ’s products can be found under the Substance Abuse and Mental Health Data Archive page (SAMHDA). 
  • SAMHSA partners with the National Center for Health Statistics to make restricted use data available through the Research Data Center (RDC). The National Center for Health Statistics (NCHS) operates the Research Data Center (RDC) to allow researchers access to restricted-use data. For access to the restricted-use data, researchers must submit a research proposal outlining the need for restricted-use data. The proposal provides a framework for CBHSQ to identify potential disclosure risks and how the data will be used.

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• The SAMHSA Strategic Plan FY19-FY23 (pp. 20-23) outlines five priority areas to carry out the vision and mission of SAMHSA, including Priority 4: Improving Data Collection, Analysis, Dissemination, and Program and Policy Evaluation. This Priority includes three objectives: 1) Develop consistent data collection strategies to identify and track mental health and substance use needs across the nation; 2) Ensure that all SAMHSA programs are evaluated in a robust, timely, and high-quality manner; and 3) Promote access to and use of the nation’s substance use and mental health data and conduct program and policy evaluations and use the results to advance the adoption of evidence-based policies, programs, and practices. 
• When possible, SAMHSA incorporates qualitative data into its framework through the feedback received by the Project Officers, through annual narrative reports submitted by grantees, key informant interviews, site visits and focus groups. SAMHSA is in regular communication with grantees and the state/community programs regarding opportunities and challenges. SAMHSA is developing a formal process in FY22 for incorporating qualitative feedback into its evaluation process.
• Within this strategic plan, SAMHSA is committed to Behavioral Health Equity: the right to access quality health care for all populations regardless of the individual’s race, ethnicity, gender, socioeconomic status, sexual orientation, or geographical location. This includes access to prevention, treatment, and recovery services for mental and substance use disorders. Advancing health equity involves ensuring that everyone has a fair and just opportunity to be as healthy as possible. In conjunction with quality services, equity involves addressing social determinants, such as employment and housing stability, insurance status, proximity to services, culturally responsive care–all of which have an impact on behavioral health outcomes.

6.2 Did the agency have a common evidence framework for funding decisions?

• SAMHSA has universal language about using evidence-based practices (EBPs) that is included in its Funding Opportunity Announcements (FOAs) also known as NOFO (or Notice of Funding Opportunity) also called Funding (entitled Using Evidence-Based Practices (EBPs)). This language includes acknowledgement that, “EBPs have not been developed for all populations and/or service settings” thus encouraging applicants to “provide other forms of evidence” that a proposed practice is appropriate for the intended population.
• Specifically, the language states that applicants should:
  1. document that the EBPs chosen are appropriate for intended outcomes;
  2. explain how the practice meets SAMHSA’s goals for the grant program;
  3. describe any modifications or adaptations needed for the practice to meet the goals of the project;
  4. explain why the EBP was selected;
  5. justify the use of multiple EBPs, if applicable; and
  6. discuss training needs or plans to ensure successful implementation.
• Lastly, the language includes resources the applicant can use to understand EBPs. Federal grants officers work in collaboration with the SAMHSA Office of Financial Resources to ensure that grantee funding announcements clearly describe the evidence standard necessary to meet funding requirements.
• SAMHSA developed a manual, Developing a Competitive SAMHSA Grant Application, which explains information applicants will likely need for each section of the grant application. The manual has two sections devoted to evidence-based practices (p. 8, p. 26), including: 1) A description of the EBPs applicants plan to implement; 2) Specific information about any modifications applicants plan to make to the EBPs and a justification for making them; and 3) How applicants plan to monitor the implementation of the EBPs. In addition, if applicants plan to implement services or practices that are not evidence-based, they must show that these services/practices are effective.
• For the 10% set aside in the Mental Health Block Grant, states are directed to use evidence-based programs. SAMHSA provides guidance to states based on evidence. For First-Episode Psychosis (FEP), SAMHSA’s recommendation is to develop a state FEP program based on the Coordinated Specialty Care model, as evaluated by NIMH. For example, FEP program OnTrackNY is an evaluated model that is recommended based on evidence of success (see question 9 for more information on the EBP supported for the Block Grants).
• SAMHSA recognizes that one size does not fit all, therefore although grantees are encouraged to consider the EBP on the SAMHSA website, grantees must provide information on the EBP they play to implement. Their description should reference why each EBP is appropriate for the problem area addressed by the grant as well as the specific population(s) of focus. SAMHSA also asks for specific information about any modifications planned to make the EBPs and a justification for making these modifications as well as how the grantee will monitor the implementation of the EBPs to ensure they are implemented according to EBP guidelines. 
• The Evidence-Based Practices Resource Center provides communities, clinicians, policy-makers and others with the information and tools to incorporate evidence-based practices into their communities or clinical settings. The Evidence-Based Practices Resource Center (EBPRC) contains a collection of scientifically-based resources for a broad range of audiences, including Treatment Improvement Protocols, toolkits, resource guides, clinical practice guidelines, and other science-based resources. The retooled EBPRC neither accepts open submissions from outside program developers nor rates individual programs. 

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• The Evidence-Based Practices Resource Center provides communities, clinicians, policymakers, and others with the information and tools to incorporate evidence-based practices into their communities or clinical settings. In FY21, five additional EBP guides were added to the website sharing recommendations, evidence ratings, and reviews of interventions that offer insight into available evidence regarding the following priority issues: Telehealth for Treatment of Serious Mental Illness and Substance Use Disorders, Substance Use Disorders Recovery with a Focus on Employment and Education, Treatment Considerations for Youth and Young Adults with Serious Emotional Disturbances and Serious Mental Illnesses and Co-occurring Substance Use, Use of Medication-Assisted Treatment in Emergency Departments and Treatment for Suicidal Ideation, Self-harm, and Suicide Attempts Among Youth and Prevention and Treatment of HIV Among People Living with Substance Use and/or Mental Disorders.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• SAMHSA promotes the utilization of evidence-based practices. Within grant applications, SAMHSA encourages innovation. For example, the FY20-21 Substance use block grant application includes the following language, “There is increased interest in having a better understanding of the evidence that supports the delivery of medical and specialty care including M/SUD services. Over the past several years, SAMHSA has collaborated with CMS, HRSA, SMAs, state M/SUD authorities, legislators, and others regarding the evidence of various mental and substance misuse prevention, treatment, and recovery support services. States and other purchasers are requesting information on evidence-based practices or other procedures that result in better health outcomes for individuals and the general population. While the emphasis on evidence-based practices will continue, there is a need to develop and create new interventions and technologies and in turn, to establish the evidence. SAMHSA supports states’ use of the block grants for this purpose. The NQF and the IOM recommend that evidence play a critical role in designing health benefits for individuals enrolled in commercial insurance, Medicaid, and Medicare. To respond to these inquiries and recommendations, SAMHSA has undertaken several activities. SAMHSA’s Evidence-Based Practices Resource Center assesses the research evaluating an intervention’s impact on outcomes and provides information on available 43 resources to facilitate the effective dissemination and implementation of the program. SAMHSA’s Evidence-Based Practices Resource Center provides the information & tools needed to incorporate evidence-based practices into communities or clinical settings.
• SAMHSA’s Evidence-Based Practices Resource Center (EBPRC) aims to provide communities, clinicians, policymakers, and others in the field with the information and tools they need to incorporate evidence-based practices into their communities or clinical settings. The Resource Center contains a collection of science-based resources, including Treatment Improvement Protocols, toolkits, resource guides, and clinical practice guidelines, for a broad range of audiences. As of June 2021, the Resource Center includes 150 items, including 15 data reports, 23 toolkits, 6 fact sheets, and 88 practice guides. 
• The Mental Health Technology Transfer Center (MHTTC) Network engages with organizations and treatment practitioners involved in the delivery of mental health services to strengthen their capacity to deliver effective evidence-based practices to individuals, including the full continuum of services spanning mental illness prevention, treatment, and recovery support. The State Targeted Response Technical Assistance (STR-TA), known as the Opioid Response Network, was created to support efforts to address opioid use disorder prevention, treatment, and recovery, and to provide education and training at the local level in evidence-based practices.
• The Knowledge Application Program (KAP) supports the professional development of behavioral health workers and provides information and resources on best practices. Specifically, KAP provides substance use treatment professionals with publications that contain information on best treatment practices.

7.1 Did the agency have staff dedicated to leading its innovation efforts to improve the impact of its programs?

• CBHSQ hired senior staff within the CBHSQ Office of the Director to focus on innovation, increasing efficiency, developing standard protocols, and developing a dissemination strategy for greater utilization of SAMHSA data (with a focus on encouraging visits to SAMHDA for data use). This work is ongoing.

7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?

• Within SAMHSA grant programs, the agency encourages innovation from every state, territory and community applicant. For example, the FY20-21 Substance use block grant application includes the following language:“There is increased interest in having a better understanding of the evidence that supports the delivery of medical and specialty care including M/SUD services. Over the past several years, SAMHSA has collaborated with CMS, HRSA, SMAs, state M/SUD authorities, legislators, and others regarding the evidence of various mental and substance misuse prevention, treatment, and recovery support services. States and other purchasers are requesting information on evidence-based practices or other procedures that result in better health outcomes for individuals and the general population. While the emphasis on evidence-based practices will continue, there is a need to develop and create new interventions and technologies and in turn, to establish the evidence. SAMHSA supports states’ use of the block grants for this purpose. The NQF and the IOM recommend that evidence play a critical role in designing health benefits for individuals enrolled in commercial insurance, Medicaid, and Medicare. To respond to these inquiries and recommendations, SAMHSA has undertaken several activities. SAMHSA’s Evidence Based Practices Resource Center assesses the research evaluating an intervention’s impact on outcomes and provides information on available 43 resources to facilitate the effective dissemination and implementation of the program. SAMHSA’s Evidence-Based Practices Resource Center provides the information & tools needed to incorporate evidence-based practices into communities or clinical settings.”
• Pursuant to the 21st Century Cures Act, SAMHSA established the National Mental Health and Substance Use Policy Laboratory (NMHSUPL), which promotes evidence-based practices and service delivery models through evaluating models that would benefit from further development and through expanding, replicating, or scaling evidence-based programs across a wider area. Specifically NMHSUPL:
  • Identifies, coordinates, and facilitates the implementation of policy changes likely to have a significant effect on mental health, mental illness (especially severe mental illnesses such as schizophrenia and schizoaffective disorders), recovery supports, and the prevention and treatment of substance use disorder services;
  • Works with CBHSQ to collect information from grantees under programs operated by the Administration in order to evaluate and disseminate information on evidence-based practices, including culturally and linguistically appropriate services, as appropriate, and service delivery models; and 
  • Carries out other activities as deemed necessary to continue to encourage innovation and disseminate evidence-based programs and practices.

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• Grantees report innovation and use of evidence-based practices in their reports as required by SAMHSA. Information from these reports are included in rigorous evaluations when evaluations have been planned or used for performance management when a formal evaluation has not been scheduled.   
• In FY21, SAMHSA completed a rigorous evaluation of its innovative SPF-Rx program. Prescription drug misuse continues to be a critical public health problem in the United States (U.S.) and the SPF (or Strategic Prevention Framework) consists of five steps. SPF is a data-driven, systemic public health planning approach to substance use prevention that is theory-based and involves the implementation of evidence-based strategies. Grantees and subrecipients apply the overarching principles of cultural competence and sustainability throughout the dynamic SPF process. Between 2017-2021, 25 state and tribal SPF-Rx grantees received funding to focus on supporting high-need community efforts to implement evidence-based practices to prevent and reduce the misuse of prescription opioids. In FY21, an evaluation was completed and the findings from this evaluation informed SAMHSA of ways to better develop, assess, and manage its prescription drug misuse prevention programs. The cross site evaluation collected quantitative and qualitative data from three instruments: an annual implementation instrument (130 items collecting data on organization type, funding levels, assessment of capacity building and sustainability, strategic planning, prevention and intervention programming, and ongoing local evaluations); grantee level and community level outcomes module (two modules including data on PDMP use and prescribing patterns, opioid overdose events, etc); and grantee level interviews (qualitative data collected at baseline and at the end of the evaluation). Results of this evaluation were used to inform the next generation of SPF Rx grants.   
• The next evaluation of this SPF Rx program has been started with an evaluation plan drafted to include an examination of secondary data on matched communities for comparison of outcome data.   

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY21, the five largest competitive grant programs are:
  1. State Opioid Response (SOR) Grant ($1.5 billion; States, Existing USDA Cooperative Extension Grantees, and U.S. Territories, Tribes and tribal organizations are eligible to apply to set-aside funds only)
  2. Children’s Mental Health Services ($125 million; States, Tribes, Communities, Territories)
  3. Strategic Prevention Framework (SPF) ($109 million; States, Tribes, and Territories)
  4. Targeted Capacity Expansion-Special Projects ($102 million; Domestic Public and Private Non-Profit Entities, States, Opioid Medication-Assisted SPF Rx Treatment Service Providers, Outpatient Substance Abuse Providers, Community Mental Health Centers, Federally Qualified Health Centers)
  5. Certified Community Behavioral Health Clinic (CCBHC) ($250 million; Certified Community Behavioral Health Clinics, Community-Based Behavioral Health Clinics)

8.2 Did the agency use evidence of effectiveness to allocate funds in the five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• As with all SAMHSA grants, the five largest competitive grants programs require applicants to include evidence-based practices and activities that are backed by science to their proposals. The allocation of funds is based on an application that includes a request for evidence of effective work in the area of reducing substance use and mental health disorders. 
• State Opioid Response (SOR) Grant requires grantees to describe their evidence-based service and practice. The grantee must describe how the EBP meets the population(s) needs and the outcomes to be achieved. The grantee must also indicate how their practice might be modified and reasons for such modifications. 
• Children Mental Health Services requires grantees to describe the evidence-based and culturally competent mental health services to children with SED.
• Strategic Prevention Framework requires grantees to report on the number and percent of evidence-based programs, policies, and or practices that are implemented and describe the types of evidence-based interventions implemented at the community level. Additionally, grantees must coordinate with the Evidence-Based Practices Workgroups. 
• Targeted Capacity Expansion–Special Projects requires applicants to describe their proposed evidence-based service/practice. The grantee must describe how the EBP meets the population(s) needs and the outcomes to be achieved. The grantee must also indicate how their practice might be modified and reasons for such modifications.
  
• Certified Community Behavioral Health Clinic Expansion Grants requires applicants to describe their proposed evidence-based service/practice. The grantee must describe how the EBP meets the population(s) needs and the outcomes to be achieved. The grantee must also indicate how their practice might be modified and reasons for such modifications.

8.3 Did the agency use its five largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• Evaluating some of the largest competitive grant programs is in SAMHSA’s evaluation plan. These evaluations will inform and enable SAMHSA to build evidence. One mechanism for this is through the grantmaking process. In some grants, SAMHSA includes additional terms and conditions that state, depending on the funding opportunity and grant application, a grantee may be asked to participate in a cross-site evaluation.
• All grant programs at SAMHSA are required to submit data on race, ethnicity, gender and sexual orientation (among other demographic data). In addition, SAMHSA’s surveys collect national data in these areas allowing SAMHSA’s Office of Behavioral Health Equity, to utilize federal and community data to identify, monitor and respond to behavioral health disparities. 

8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant programs (besides its five largest grant programs)?

• All grant programs at SAMHSA are required to submit data on race, ethnicity, gender and sexual orientation (among other demographic data). In addition, SAMHSA’s surveys collect national data in these areas allowing SAMHSA’s Office of Behavioral Health Equity, to utilize federal and community data to identify, monitor and respond to behavioral health disparities. 

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Competitive grant programs are required to consider evidence-based practices in their application and are referred to the SAMHSA Evidence-based Practices Resources Center for tools they need to achieve better outcomes based on what works. An additional example might be found in SAMHSA’s trauma and justice  portfolio, which provided a comprehensive public health approach to addressing trauma and establishing a trauma-informed approach in health, behavioral health, human services, and related systems. https://www.samhsa.gov/criminal-juvenile-justice
• The intent of this initiative was to reduce both the observable and less visible harmful effects of trauma and violence on children and youth, adults, families, and communities. As part of this initiative, the SPARS team presented the video series, A Trauma-Informed Approach to Data Collection, with commentary from subject matter experts and clientele from the People Encouraging People (PEP) program in Baltimore, MD. This series advised grantees and GPOs about using a trauma-informed approach to collecting client-level data.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• Grantees are encouraged to allocate grants funds for data collection, data analysis, and program evaluation. Some grantees hire external evaluators using grant funds to assist them in the evaluation process. For example, one funding announcement states, “Provide specific information about how you will collect the required data for this program and how the data will be utilized to manage, monitor and enhance the program.” In addition, up to 20% of the total grant award for the budget period may be used for data collection, performance measurement, and performance assessment expenses. 

9.1 What were the agency’s five largest non-competitive programs and their appropriation amounts (and were city, county, and/or state governments are eligible to receive funds from these programs)?

• In FY21, the five largest non-competitive grant programs are:
  1. Substance Abuse Prevention and Treatment Block Grant Program ($1.8 billion; States, Territories, Freely Associated States, District of Columbia, Red Lake Band of Chippewa Indians of Minnesota)
  2. Community Mental Health Services Block Grant ($758 million; States, Territories, Freely Associated States, and District of Columbia)
  3. Projects for Assistance in Transition from Homelessness (PATH) ($65 million; States and Territories) 
  4. The Protection and Advocacy for Individuals with Mental Illness (PAIMI) ( $36 million; States and Territories)

9.2 Did the agency use evidence of effectiveness to allocate funds in largest five non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)

• The allocation of these state grants are based on formularies that include data provided by national data sets (including SAMHSA’s National Survey on Drug Use and Health), population estimates, as well as estimates of substance use and mental health disorders. 
• The Substance Abuse Prevention and Treatment Block Grant requires recipients to describe and report on the evidence-based prevention programs to implement. Through the SABG, states should “identify, implement, and evaluate evidence-based programs, practices, and policies that have the ability to reduce substance use and improve health and well-being in all communities.” Funds can also be used for program implementation fidelity training. It also asks recipients about the Evidence-Based Workgroup that helps identify evidence-based strategies and programs for implementation. 
• The Community Mental Health Services Block Grant requires recipients to describe in their plans use of not less than 10% of the MHBG funds to carry evidence-based programs that address the needs of individuals with early serious mental illness, including psychotic disorders, regardless of the age of the individual at onset. Recipients are required to describe any existing and implemented evidence-based practices, how the state promotes evidence-based practices and details on data collection and program implementation strategies. 

9.3 Did the agency use its five largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• Information on how to use funds for data collection and evaluation is covered in the Block Grant application. Grantees are encouraged to allocate grants funds for data collection, data analysis and program evaluation. Some grantees hire external evaluators using grant funds to assist them in the evaluation process. In FY21, SAMHSA updated their application manual to include a section on developing goals and measurable objectives (see p. 38). Specifically, the document states, “To be able to effectively evaluate your project, it is critical that you develop realistic goals and measurable objectives. This chapter will provide information on developing goals and measurable objectives. It will also provide examples of well-written goals and measurable objectives.” 
• Grantees in non-competitive grant programs are required to submit quantitative data to SAMHSA using reporting systems associated with their grant. For example, State Mental Health Agencies receive noncompetitive grants and compile and report annual data collected from SAMHSA’s Community Mental Health Block Grant. More information on the URS or Uniform Reporting System can be found online. In this way, noncompetitive grant programs not only allow the sharing of data for research and evaluation but this allows grantees to explore data from other state grantees.
• In the FY20-21 Block Grant Application, SAMHSA asks states to base their administrative operations and service delivery on principles of Continuous Quality Improvement/Total Quality Management (CQI/TQM). These CQI processes should identify and track critical outcomes and 72 performance measures, based on valid and reliable data, consistent with the NBHQF, which will describe the health and functioning of the mental health and addiction systems. The CQI processes should continuously measure the effectiveness of services and supports and ensure that they continue to reflect this evidence of effectiveness. The state’s CQI process should also track programmatic improvements using stakeholder input, including the general population and individuals in treatment and recovery and their families. In addition, the CQI plan should include a description of the process for responding to emergencies, critical incidents, complaints, and grievances.

9.4 Did the agency use evidence of effectiveness to allocate funds in any other non-competitive grant programs (besides its five largest grant programs)?

• The majority of SAMHSA grants are competitively awarded. SAMHSA only has four non-competitive grants which are included above.
• As stated in the Block Grant application (p. 45): “States may implement models that have demonstrated efficacy, including the range of services and principles identified by National Institute of Mental Health (NIMH) via its Recovery After an Initial Schizophrenia Episode (RAISE) initiative. Utilizing these principles, regardless of the amount of investment, and by leveraging funds through inclusion of services reimbursed by Medicaid or private insurance, states should move their system to address the needs of individuals with a first episode of psychosis (FEP). RAISE was a set of NIMH sponsored studies beginning in 2008, focusing on the early identification and provision of evidence-based treatments to persons experiencing FEP. The NIMH RAISE studies, as well as similar early intervention programs tested worldwide, consist of multiple evidence-based treatment components used in tandem as part of a Coordinated Specialty Care (CSC) model, and have been shown to improve symptoms, reduce relapse, and lead to better outcomes. State shall expend not less than 10% of the MHBG amount the State receives for carrying out this section for each fiscal year to support evidence-based programs that address the needs of individuals with early serious mental illness, including psychotic disorders, regardless of the age of the individual at onset. In lieu of expending 10% of the amount the State receives under 45 this section for a fiscal year as required a state may elect to expend not less than 20% of such amount by the end of such succeeding fiscal year.”

9.5 What are the agency’s strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• SOR Grantees have implemented evidence-based practices focused on safe prescribing, naloxone and medication for opioids use disorder to help support and build knowledge around the use of these EBPs.  
• Currently, State Opioid Response grantees are using funds to examine Fentanyl Test Strips. This recent change under the Biden-Harris Administration will help to build knowledge on the utility of these evidence-based practices.

9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• Information on how to use funds for data collection and evaluation is covered in the Block Grant application. Grantees are encouraged to allocate grants funds for data collection, data analysis and program evaluation. Some grantees hire external evaluators using grant funds to assist them in the evaluation process. 

10.1 Did the agency have policy(ies) for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• As a matter of policy, SAMHSA uses the term “restricted status” to describe grant recipients that are financially unstable, have inadequate financial management systems, or are poor programmatic performers. Grants placed on restricted status require additional monitoring and have additional award conditions that must be met before funds can be drawn. SAMHSA adheres to HHS Grants Policy Statement, including the policy on suspension or termination, which states: “If a recipient has failed to materially comply with the terms and conditions of award, the OPDIV [Grant-Awarding Operating Division] may suspend the grant, pending corrective action, or may terminate the grant for cause” (p. II-89).

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• The SAMHSA Performance Accountability and Reporting System allows SAMHSA staff to regularly monitor discretionary grant status as well as meet with grant program directors. If a grantee is falling behind or not meeting proposed targets, SAMHSA staff access the data in real time to provide the support or technical assistance needed to make sure the grantee does not fail. Given the important mission of SAMHSA, to reduce the impact of substance use and mental illness on America’s communities, it is critical that struggling communities are identified early with the goal of continuous quality improvement and support.