Abstract
School-based mental health services (SBMH) may increase students’ access to care, which could yield benefits for mental health status and human capital-related outcomes. We use a difference-in-differences design with 19 years of survey and administrative data to estimate the impacts of SBMH on a range of K–12 student outcomes. SBMH increases average outpatient mental health service use and reduces self-reported suicide attempts. There is weaker evidence that SBMH reduces suspensions and juvenile justice involvement and no evidence that SBMH affects average attendance, standardized test scores, or self-reported substance use.
I. Introduction
The prevalence and implications of mental health problems among children and adolescents are well established. Mental disorders are relatively common for children and adolescents (Kessler et al. 2012; Merikangas et al. 2010; Substance Abuse and Mental Health Services Administration 2021), and key indicators such as suicides have been deteriorating for more than a decade (Curtin 2020). Beyond the actual symptoms of mental health problems, mental health can affect human capital formation (Currie 2020). Mental health problems in childhood and adolescence lead to worse subsequent academic outcomes and greater use of public assistance in early adulthood (Currie et al. 2010), and depression in adolescence leads to increased criminal behavior in early adulthood (Anderson, Cesur, and Tekin 2015; Cornaglia, Crivellaro, and McNally 2015). Childhood psychological problems are also associated with long-term consequences, including worse labor market and marital outcomes into middle-aged adulthood (Goodman, Joyce, and Smith 2011).
While correlations between mental health problems in childhood and adolescence and health, human capital, and economic-related outcomes are well established, less is known about how mental health service interventions affect these outcomes. A range of psychotherapy and pharmaceutical services exist for mental disorders that commonly occur in childhood and adolescence, including depression, anxiety, and attention disorders. However, evidence suggests that mental health services are underused among children and adolescents, despite recent trends of increased service use. Between 2010 and 2012, only 44 percent of youths aged 6–17 who had relatively severe mental health impairments were estimated to have received any outpatient services in the past year (Olfson, Druss, and Marcus 2015). Even among children and adolescents who had a current mental health diagnosis, only half received specialty mental health services in 2016 (Whitney and Peterson 2019). Moreover, among adolescents who do receive treatment, services often fall short of established care guidelines (Cuddy and Currie 2020). As such, policies and interventions that facilitate diagnosis and improve treatment of mental health disorders might improve a range of youth outcomes. As noted by Currie (2020), there is a paucity of economic research on what intervention strategies are most effective for addressing mental health problems among adolescents.
Several papers examine the causal effects of policy shocks and interventions on specific childhood mental health disorders. Busch, Golberstein, and Meara (2014) examine the effect of the FDA’s black box warnings for pediatric antidepressant use, which caused an unanticipated downward shock to the use of depression treatment for adolescents, and find that it led to poorer academic outcomes and greater levels of substance use, with stronger effects for girls than for boys. Currie, Stabile, and Jones (2014) examine changes in ADHD drug use driven by the expansion of prescription drug insurance in Quebec and find no evidence of improvements in academic outcomes. Chorniy and Kitashima (2016) leverage variation in provider treatment patterns and find that ADHD drug treatment reduces risky behavior–related outcomes among youth in South Carolina Medicaid. Dalsgaard, Nielsen, and Simonsen (2014) use a similar source of variation and find that ADHD drug treatment reduces both hospital use and interactions with police among Danish youth. Other research suggests an important role of youth mental health treatment on social and economic outcomes, such as incarceration (Cuellar and Dave 2016; Deza, Lu, and Maclean 2021; Jácome 2022), along with experimental evidence that cognitive behavioral therapy among youths leads to improved school engagement and reduced criminal justice involvement (Heller et al. 2017).
In two papers that have important similarities to our study, Reback examines the effect of school counselors, whose roles can include mental health support and services, on student outcomes. One paper (Reback 2010a) leverages variation in state-level school counselor policy as an instrument for school counseling use and finds that schools with more counseling use have better test scores and reports of problem behaviors but no difference in interest or confidence in specific social or academic topics in the third grade. The other paper (Reback 2010b) uses a regression-discontinuity design based on a threshold that influenced the number of elementary school counselors in Alabama schools and finds that greater subsidies for counselors led to reductions in behavioral problems but no effect on standardized test scores.
We add to the evidence on how mental health interventions for children and adolescents affect a range of outcomes related to health and human capital. We examine a specific intervention that places mental health clinicians in school settings. We construct a novel policy data set of the implementation of this intervention across many K–12 public schools in Minnesota’s largest county. Hennepin County includes the city of Minneapolis and 44 surrounding suburbs, encompasses 17 school districts, and is socioeconomically, racially, and ethnically diverse. The staggered implementation of SBMH across many schools provides an opportunity to assess SBMH’s effects on mental health services and on other human capital-related outcomes that might be affected by improvements in mental health services use. We merge the policy data with linked administrative data that include student characteristics, educational-related outcomes, Medicaid claims data, juvenile justice involvement, and child welfare involvement. We also link the policy data set with seven waves of a large-scale triennial survey from 2001 to 2019 that include suicidality and substance use outcomes.1 We apply the Borusyak, Xavier, and Speiss (2023) heterogeneity-robust, difference-in-differences estimator to leverage the staggered implementation of SBMH across schools.
II. The School-Based Mental Health Model
School-based mental health services (SBMH) embody a model of service delivery that aims to enhance the diagnosis and treatment of mental health services among youth by placing mental health clinicians directly in school settings. Even in the absence of the SBMH model, schools generally offer some mental health services and supports, typically from school social workers, school counselors, or school psychologists, though these resources are frequently limited. Nevertheless, schools are an important source of mental health services. For instance, data from the 2012–2015 National Survey of Drug Use and Health show that 57 percent of all adolescents who reported receiving any mental health services in the past year indicated that they received some sort of services from a school setting (Ali et al. 2019).
The specific SBMH model that we examine is also known as the “expanded” SBMH model and has four key features. First, licensed mental health clinicians (master’s or doctoral-trained social workers, psychologists, or mental health counselors) work directly inside of schools, complementing existing school counselors and social workers to expand capacity for mental health services. School counselors and school social workers are a main source of referral to SBMH services. Second, SBMH clinicians deliver a variety of services that include diagnosing and assessing mental health problems and delivering treatments in the form of individual, group, and family psychotherapy. Third, SBMH clinicians coordinate with and refer to other professionals as needed. Notably, SBMH clinicians generally do not have authority to prescribe prescription drugs and would refer a client to a physician to prescribe drugs. Fourth, SBMH clinicians train the school’s teachers in identifying mental health problems among students, and teachers subsequently are a major source of referring students to SBMH services.
The specific intervention we study has two other important features. First, the clinicians are not employed by schools, but by community mental health services agencies. Second, if a student has insurance, either private or Medicaid, then the SBMH provider’s agency would bill the insurer for the services. Students who were uninsured were not charged for services, and those services were paid for with state grant funds, and any cost-sharing was also paid for with state grant funds.
SBMH could improve mental health services for children through several mechanisms. Proponents argue that SBMH reduces the time, hassle, and disruption for children and parents by delivering services where children already spend their time. Providing SBMH with no cost-sharing could remove cost-related barriers and also reduce the significant search costs of finding an available clinician. Students might feel less stigma in receiving care in a school setting than at a specialty clinic (Committee on School Health 2004). Delivering services in schools could help to identify and treat symptoms quickly, as teachers, school social workers, and counselors are well positioned to recognize potential problems and can easily refer students to SBMH (Stephan et al. 2007). SBMH could enhance equity in mental health care as well, to the extent that it overcomes existing barriers for underserved populations (for example, low-income, uninsured, and racial/ethnic minorities) (Alegría et al. 2015).
While the SBMH model is a promising approach to improving child mental health and related outcomes, existing causal evidence on the model is limited. A large literature describes a variety of specific SBMH models and school settings (Farahmand et al. 2011; Reddy et al. 2009; Rones and Hoagwood 2000), but this literature has important limitations. Many studies either lack a control group or directly compare SBMH users with other students without accounting for selection (Armbruster and Lichtman 1999; Ballard, Sander, and Klimes-Dougan 2014). Many other studies, including some experimental designs, examine very small samples of students or schools (Reddy et al. 2009) or either examine a very narrow version of school-based intervention or target a very narrow type of mental health problem (Farahmand et al. 2011).
Despite the limited evidence base for the causal effects of SBMH, policy interest in this model is intense in the United States. Some states are appropriating direct funding for SBMH services, some states are creating subsidies to increase the SBMH workforce, and other states are changing their Medicaid policies to allow for more Medicaid financing of SBMH (Anderson 2021; Hill 2021; Kennedy 2022). At the federal level, recent House and Senate bills have been proposed to enhance funding for SBMH (Napolitano 2021; Smith 2021). Every single witness in Senate Finance Committee hearings on child mental health in February 2022 testified to the importance of SBMH, and the Surgeon General called for greater investment in SBMH in a 2021 advisory (U.S. Surgeon General 2021).
III. Methods
A. Analytic Approach
We leverage the staggered adoption of SBMH services across Hennepin County public schools to assess the effects of SBMH on the study outcomes in a difference-in-differences framework. The first SBMH programs started in five Minneapolis schools in the 2005–2006 academic year, with funding from a federal Safe Schools/Healthy Student grant awarded to Minneapolis Public Schools. In 2008, the state of Minnesota offered grant funding to support SBMH, and since then SBMH has expanded more rapidly. Figure 1 describes the expansion of SBMH across schools in Hennepin County.
Trend in SBMH Implementation across Hennepin County Schools
Notes: Data on SBMH collected by the authors.
Our basic design compares outcome changes following the introduction of SBMH into a school to outcome changes in school–years without SBMH. The credibility of the research design hinges on the assumption that the timing of particular schools’ SBMH adoption is independent of changes in unobservable influences on outcomes, as outcomes in implementing schools would have changed similarly to nonimplementing schools over the same time periods but for SBMH. A key issue is thus the way that SBMH was rolled out across the county. We have had conversations with state and county officials and reviewed documents to understand that process. SBMH grants were awarded to community mental health agencies, which then partnered with school districts in their service areas to decide which specific schools would receive SBMH services. The state had no discretion over which specific schools received services. We understand that at least some school districts allocated SBMH services on the basis of schools’ perceived to have a “high need” for mental health resources, but our conversations have not revealed any sense that this was based on trends in need for services. We examine predictors of SBMH adoption below. To further address differences between schools that did and did not implement SBMH, we also create a complementary analytic sample of schools that are matched based on observable school characteristics that include percentage of minority students, percentage of students eligible for free or reduced-price lunch, percentage of limited English proficiency students, and percentage of students enrolled in special education programs. We use the nearest-neighbor matching method to identify the schools for a matched analytic sample, similar to methods used in Cabral et al. (2022).2
Our analytic approach uses difference-in-differences models that compare changes in outcomes in schools after implementing SBMH to those changes in schools that never implement SBMH. A recent econometric literature identifies problems with applying conventional two-way fixed-effects models to difference-in-differences models in settings with staggered adoption and heterogeneous treatment effects across units (Goodman-Bacon 2021). These concerns are salient for us since it is quite plausible that the effects of SBMH vary over time.
We use a recently developed imputation estimator (Borusyak, Jaravel, and Speiss 2023) that is robust to those issues to estimate our difference-in-differences models. Treatment occurs at the school–year level, but we analyze student–year data. We seek to identify and estimate the average treatment-on-treated effect (ATT) across academic years on any outcome for student i in academic year t in school s (Ysti). We assume the treatment effect for a given student–year is the difference in the student’s potential outcomes, τti = Yti(1) – Yti(0), where each potential outcome’s argument indicates whether the student is treated with SBMH (Dst). Averaging τti across treated student–years will yield the estimand τ = E[τti|Dst = 1]. We impose parallel trends by assuming the never-treated potential outcome would be
where the never-treated potential outcome is produced by observable control variables varying at the individual–school–year level (Xist), individual-school fixed effects (δs), academic-year fixed effects (θt), and the interaction of level of school (elementary, middle, and high school: l(s)) by academic year fixed effects (γl(s)t), and an idiosyncratic unobservable with conditional mean zero (E[ɛist|Xst,1s,1t, 1l(s)t] = 0). As such, the model relies on variation in SBMH implementation across schools that are within the same level. We impose no-anticipation by assuming outcomes observed among never-treated individuals equal their never-treated potential outcomes; thus, there is no effect of a possible future treatment. Under these assumptions, the never-treated observations and the pretreatment ever-treated observations identify (α, δ, θ, γ), implying a model of E[Y(0)|Xst, 1s, 1t, 1l(s)t] for all observations. Because the observed outcome equals treated outcome Ysti = Ysti(1) for all treated observations, this identifies each τti as the difference between observed Ysti and imputed E[Y(0)|Xst, 1s, 1t, 1l(s)t]. Then, the ATT is estimated by the average τti among treated observations. We also estimate dynamic effects that allow heterogeneous ATT effects by years elapsed from initial treatment and placebo tests for “effects” in years prior to SBMH treatment. The individual–school–year-level (Xist) covariates differ depending on the data used for analysis and are described below. In all models, we estimate standard errors that are robust to heteroskedasticity and correlation within school. The interpretation of the results of these models is at the school level, measuring how average outcomes change in a school after SBMH is adopted. We address student-level dynamics with respect to these school-level estimates below.
B. Data and Measures
We use three main data sources. First, data on the intervention come from a novel SBMH adoption data set. Second, outcome and covariate data come from either (i) public agencies’ administrative records, which are linked together at the student level and deidentified, or (ii) from the Minnesota Student Survey (MSS), a state survey of students conducted every three years. The Minnesota-Linking Information for Kids project (Minn-LInK) linked the administrative data.3 The Online Appendix includes more details about data and measures.
1. Policy data: School-based mental health implementation
We created a novel data set of the implementation of SBMH services for K–12 public schools in Hennepin County for 2001–2019. We worked with each of the 17 community mental health agencies that delivered SBMH to identify which schools they worked in and when they started working in each school. Where there was ambiguity about when an agency started working in a school, we worked with school health staff to verify the start date. We were unable to verify SBMH start dates for 16 schools that had implemented SBMH with one agency, and those 16 schools were dropped from the analysis. The sample of schools is limited to “standard” schools, which excludes online-only schools, alternative learning centers, and schools affiliated with a mental health or substance-use treatment program. The key intervention variable varies at the school–year level and is coded as one if a school implemented SBMH by the middle of the fall semester of an academic year, and zero otherwise. Figure 1 displays the trend in the share of schools that implemented SBMH.
We use public data at the school–year level on staffing of school social workers, counselors, and psychologists from the Minnesota Department of Education to measure whether the adoption of SBMH and the attendant expertise brought in from external agencies crowds out internal district staff with relevant expertise. We add up full-time-equivalent (FTE) staffing in these three licensed occupations and divide by student enrollment to get a measure of relevant internal FTE staff per 100 students enrolled in each school, each year. These data are available only for academic years ending in 2007 and forward.
2. Administrative data: Minn-LInK
The Minn-LInK administrative data cover children ages 5–18 in Hennepin County from 2001 to 2018. Over that period, we observe the 477,991 unique students who attended any of 263 schools using student–school–year enrollment data from the Minnesota Automated Reporting Student System (MARSS). MARSS follows each unique student across any Hennepin County schools and districts they attended during the study period. We merged the administrative data on which school each child attended in each year with our own data set on SBMH implementation, relying on the school that each student attended at the start of the school year. These data represent the broadest of the administrative data samples, and all other samples are subsets of it. Summary statistics of student demographics and outcomes from this administrative data sample are in Online Appendix Table 1. Student–year covariates in the administrative data sample include gender, age, race/ethnicity, and free or reduced-price lunch eligibility.
Outcomes related to academic performance are derived from Minnesota Department of Education administrative data. The first academic outcome is average daily attendance, defined as the proportion of school days attended per academic year, and is available for all students in all years. The second academic outcome is standardized test scores. We examine the Minnesota Comprehensive Assessment reading, math, and science scores from standardized tests that are administered annually. Test scores are available for 2006–2018 for math and reading and 2011–2018 for science. We pool information on standardized scores across any tested subjects within child–year to measure achievement. The third academic outcome is in-school disciplinary action of suspension in the academic year, which was available for 2009–2017.
A second administrative data outcome domain is juvenile justice involvement and comes from administrative data in the State Court Administrator’s Office. We measure whether each child had any juvenile justice involvement that initiated within an academic year. This outcome is available for all students in the years 2010–2017.
A third administrative data outcome domain is mental health services use. These outcomes are derived from administrative Medicaid data made available by Hennepin County, which include month- and child-level enrollment data along with complete administrative claims for inpatient, outpatient, residential, and pharmaceutical services. We created school–year-level units of analysis for each child that appeared in the Medicaid data, and required at least six out of nine school–year months of Medicaid enrollment for inclusion in the analysis sample. The Medicaid data are available for the 2002–2003 through 2017–2018 academic years, and 19 percent of the sample are enrolled in Medicaid in those years.
We examine three outcomes related to outpatient mental health services and prescription drug use. We assess whether there was any psychotropic prescription drug use in the academic year, and also look separately by drug classes: (i) antidepressants, (ii) antipsychotics, (iii) anxiolytics (for treating anxiety disorders), (iv) mood stabilizers, and (v) stimulants (that is, ADHD drugs). We assess whether there was any outpatient mental health therapy received in the academic year. Our preferred approach to identify outpatient mental health therapy uses CPT/HCPCS procedure codes for services that do not have an institutional (that is, hospital, emergency department, or residential facility) place-of-service code. These procedures include psychotherapy and psychosocial services (Finnerty et al. 2016; Hoagwood et al. 2016), and the specific codes are in the Online Appendix.4 We also identify the subset of outpatient mental health services with a place-of-service code for “school.”5 Additionally, we create a measure of a new mental health diagnosis within a school year, defined as having any new mental health diagnosis compared to the previous year (for those students observed in the previous year).
We also examine the use of hospital-based mental health services, including inpatient and emergency-department use (defined by place-of-service codes for general hospital, psychiatric/substance use disorder facility, or emergency department) for a primary mental health diagnosis (listed in the Online Appendix). We also examine the subset of those hospital-based services that are suicide-related. To the extent that SBMH improves access to outpatient mental health services, we would predict a reduction of intensive services that are delivered in inpatient or emergency settings. However, it is also possible that increased identification of mental health problems could lead to an increase in hospital-based services, including for suicidality.
For context, we also examine how adoption of SBMH affects children’s probability of being enrolled in Medicaid. To the extent that SBMH providers have an incentive to enroll eligible children in Medicaid to receive payment for services, it is possible that exposure to SBMH will increase Medicaid enrollment. We code Medicaid enrollment in a school year as being enrolled in Medicaid at least six out of nine months.
3. Survey data: Minnesota Student Survey
We also used student-level data from the 2001–2019 Minnesota Department of Education’s Minnesota Student Survey (MSS). The MSS is a cross-sectional survey conducted every three years in January, with seven waves during our study period. More than 81 percent of school districts in the state opted to participate in each MSS wave in 2001–2019. The MSS only includes specific grades in each wave: Grades 6, 9, and 12 before 2010 and Grades 5, 8, 9, and 11 afterwards. All students in participating districts and eligible grades are eligible for the MSS, but they or their parents can opt out of participating. Nevertheless, the MSS includes the majority of Minnesota students in the eligible grades. Districts, rather than students within districts, drive almost all the variance in MSS participation. The MSS is an unbalanced panel of schools, due to school openings and closings over the study period and due to school district decisions to participate in the MSS in given years. Within Hennepin County, the MSS included an average of 31,000 students per wave between 2001 and 2019, from an average of 120 schools per wave. With the permission of all 17 school districts in Hennepin County, we obtained confidential school identifiers for each survey respondent to link to our SBMH adoption data set. Unlike the Minn-LInK data, we are unable to follow individual students over time in the MSS. Summary statistics of student demographics and outcomes for the MSS sample are in Online Appendix Table 2. Student–year covariates in the MSS include gender, age, and race/ethnicity.
The MSS data include five relevant outcomes that were collected in all seven waves from 2001 to 2019. Two measures assessed suicidal behavior for students in sixth grade and higher. One measured suicidal thoughts, as students were asked, “Have you ever thought about killing yourself?” We code this outcome as one if the student indicated “Yes, during the last year,” and zero otherwise. Among respondents, 13.8 percent reported past-year suicidal thoughts, which is identical to the most comparable national measure for high school students indicating having seriously considered attempting suicide in the past 12 months in the national 2009 Youth Risk Behavior Survey (YRBS) (Eaton et al. 2010). The other measure is the more severe outcome of suicide attempt, using the MSS question, “Have you ever tried to kill yourself?” We code this as one if the student responded “Yes, during the last year,” and zero otherwise; 3.7 percent of MSS respondents reported a suicide attempt in the past year. In comparison, 6.3 percent of high school students in the national 2009 YRBS reported a past-year suicide attempt (Eaton et al. 2010).
We also examine any self-reported past-30-day substance use, which we view as a second-order outcome because substance use might be affected by mental health status but was not the main target of the SBMH intervention. The any-substance-use outcome considers self-reports of any past 30-day alcohol use, any marijuana or hashish use (both were collected from respondents in Grade 6 and higher), and any cigarette use (collected from respondents in Grade 5 and higher). The sample means for these three types of substance use are 17.7 percent, 10.4 percent, and 7.7 percent, which are comparable to national estimates for 12–17-year-olds from the National Survey of Drug Use and Health (Substance Abuse and Mental Health Services Administration 2021).
Two measures were not available in every wave. We do not focus on these outcomes but include them, as they were pre-registered. First is a mental health status measure from an index of four items from the General Well-Being Schedule (Dupuy 1977) that were collected in the 2001–2010 MSS waves from students in sixth grade or higher. Each question refers to the past 30 days and includes five response categories, with one indicating the lowest level of well-being and five indicating the best level of well-being.6 Summing the response values forms an index ranging from 4 to 20. Second is a binary measure of mental health services use in the past year from a question in the 2007–2019 MSS waves only for students in eighth grade or higher: “Have you ever been treated for a mental health, emotional, or behavioral problem?” with possible answers of “Never,” “During the past year,” or “More than a year ago.”
In our preregistration, we hypothesized that after a school implements SBMH: (i) outpatient and prescription drug mental health services use will increase; (ii) new mental health diagnoses will increase; (iii) mental health status will improve; (iv) hospital-based mental health care, including for suicidality, will decrease; (v) academic outcomes (attendance and academic achievement) will improve; (vi) juvenile justice outcomes will improve; (vii) substance use will fall, and (viii) these effects will be strongest for students with the greatest needs for mental health services, including those involved in the child welfare system, and for racial/ethnic minorities, who face relatively greater barriers to accessing mental health care. Because we present results for many outcomes, we also calculate sharpened False Discovery Rate (FDR) q-values (Benjamini, Krieger, and Yekutieli 2006) for the 24 ATT results that we present in the main body of the paper. This adjustment does not have a large effect on our statistical inference, since as we note below, only one ATT estimate with a p < 0.10 has a q-value > 0.10.
C. Subgroup Analyses
We investigate whether the effects of adopting SBMH are stronger in certain subpopulations where theory and prior evidence would predict different magnitudes of effects. We estimated stratified models across seven dimensions:
Grade level, as mental health risks differ across age groups—we stratify the sample into elementary school (K–5), middle school (Grades 6–8), and high school (Grades 9–12).
Gender, as there are gender differences in youth mental health problems, along with evidence that the effects of mental health services on academic outcomes vary by gender.
Race and ethnicity, as racial/ethnic minority children may face greater barriers to mental health care because of various forms of racism or discrimination, and to assess SBMH’s effects on equity in mental health services.
Household economic status, proxied by whether the student qualifies for free or reduced-price lunch, based on data from the Minnesota Department of Education.
Whether a child ever had Medicaid coverage during the study period, which both reflects low family income and the difficulty in accessing specialty mental health services for children on Medicaid.
Child welfare system involvement, as children involved in the child welfare system are at higher risk of mental health problems—we identify children who ever had a child protective services investigation or any out-of-home placement using data from the Department of Human Services.
Risk of mental health problems, using a predictive modeling approach to identify high- and low-risk children.
The predictive modeling approach uses external data to identify children with higher risks of mental health problems in the Minn-LInK data. We used the 2001–2017 National Health Interview Survey, which included two measures of mental health problems among youth aged 4–17 and had variables for age, sex, race/ethnicity, and Medicaid enrollment that were measured as in the Minn-LInK data. We constructed an outcome for presence of mental health problems from two measures in the NHIS that had a sample average of 7.4 percent and then estimated a LASSO model of that outcome on a random “training” sample. The LASSO model yielded meaningful variation in predicted mental health problems, with the 10th, 25th, 75th, and 90th percentiles being 3.0 percent, 4.6 percent, 8.6 percent, and 14 percent, respectively. We assigned the predictive values from the LASSO model to the Minn-LInK sample, and we partition the sample into high (>75th percentile), medium (25th–75th percentile), and low (<25th percentile) risk of mental health problems. Further details on the predictive modeling are in the Online Appendix.
Subgroup analyses differ for outcomes derived from Minn-LInK data and outcomes derived from the MSS data. We implement all of the aforementioned subgroup analyses with Minn-LInK data. However, we are only able to examine subgroups for grade level, race/ethnicity, and gender in the MSS because other student-level information is unavailable. We do not implement the predicted mental health risk stratification in the MSS because the MSS has a narrower age range and does not include Medicaid status, which greatly diminishes the variation in predicted mental health problems.
IV. Results
A. Selection into Treatment and Contemporaneous Changes in School Resources
We begin by comparing the 123 schools that ever implement SBMH to the 140 that never implement SBMH, in terms of their average outcomes and other observable characteristics (Online Appendix Table 3a). We also split implementers into early and late implementers. A greater proportion of implementing schools than of nonimplementing schools were senior high schools. Senior high schools were also disproportionately early adopters, making up 26 percent of the 35 schools that adopted up to the academic year ending in 2010, but only 10 percent of the 88 schools that implemented afterward. In contrast, elementary schools were disproportionately less likely to implement, and middle schools were as likely to implement as not, constituting 10 percent of both groups. Implementing schools had a slightly higher average share of students of racial or ethnic minority groups and students qualifying for free or reduced-price lunch (FRPL) compared to nonimplementing schools, and there was little difference between early and later implementers. There was even more similarity across implementation status in the share of students with limited English proficiency (LEP) and those using special education services. While the analytic approach we use is not biased by differences in fixed observable or unobservable school characteristics, this gives some sense of the types of students who are most likely to be treated and hence included in the average treatment on treated (ATT) estimates. We also created a matched sample of implementing and nonimplementing schools, and the characteristics of this subsample are described in Online Appendix Table 3b.
To help understand the magnitude of resource changes implied by SBMH implementation, we talked with SBMH providers, reviewed public reports on the program, and drew on coauthor Sander’s deep experience with the SBMH program. The typical school participating in SBMH has 0.75 full-time-equivalent clinical staff, along with some additional offsite support by the agency. An onsite clinician FTE costs the agency approximately $110,000 annually. With overhead and support, we estimate it costs about $100,000 for the typical school’s participation for both onsite and offsite resources. The onsite SBMH staffing provides services to students at school, and the agency helps arrange for payment behind the scenes, which can include helping students enroll in Medicaid. Based on knowledge of the program and data from several agencies, we estimate that approximately 5 percent of students use services in a year if their school has SBMH.
To assess whether there were changes in school resources associated with the implementation of SBMH, we estimate difference-in-differences models of school–year-level measures of counseling, social work, and psychologist staffing using the empirical methods described above. The ATT estimate of SBMH implementation on support staff per 100 students is −0.032 (p = 0.452), which corresponds to a 6 percent reduction relative to the sample mean that seems to attenuate with time (Online Appendix Figure 1). This could partially offset the increase in staffing brought in through SBMH and would imply an offset to costs as well, though it is important to note that such offsets are not an intended feature of the SBMH model.
B. Effects on Study Outcomes
For every outcome, we report estimates of the average treatment effect on the treated (ATT) from models estimated from the full sample and the matched sample. In the interest of space, we show some of the heterogeneity analysis results and event study models in the main body of the paper, with the remainder in the Online Appendix. The event study and heterogeneity graphs that we present are estimated from the full sample. Analogous versions for the matched sample were quite similar and the text notes any exceptions.7
1. Administrative Minn-LInK data outcomes
We begin by analyzing positive academic outcomes: attendance rate and academic achievement on standardized tests. SBMH does not appear to affect school attendance rates across all students (Table 1). Students attend an average 94 percent of school days. ATT estimates from both the full and the matched samples are very close to zero, and estimates rule out even small effects on attendance. As displayed in Figure 2, in the years leading up to SBMH implementation, attendance rate trends are similar between the soon-to-implement and never-implementer schools, which lends credibility to the maintained assumption of parallel trends. With the exception of a drop in the fourth year after implementation, the estimated treatment effects show no effect (the version of this model from the matched sample shows no effects in any of the first five years after implementation).
Event Study Model of Average Daily Attendance
Notes: Data are from administrative data on all students. Results are from an event study model with individual-level covariates (age, race/ethnicity, sex, free/reduced-price lunch status), and grade level by year fixed effects. Significance of test of pre-intervention trends: p = 0.823.
Estimated Effects of School SBMH on School and Juvenile Justice Administrative Data Outcomes
For this first outcome of attendance rate, we describe the analysis of heterogeneous effects across all subgroups in Figure 3 to provide interpretative guidance for other outcomes. Figures describing ATT heterogeneity are expressed in relative terms to account for different levels of outcomes across groups. SBMH is estimated to affect attendance negatively among students who are white, who are in grades K–5, and who have low risk for mental health problems. SBMH is estimated to have a null effect on attendance rate among all other groups. However, even when statistically significant, effect sizes are very small.
Estimated Relative ATT Effect and 95 Percent Confidence Interval of SBMH on Attendance Rate, by Subgroup
Notes: Daily attendance rate ranges from zero to one. AIAN is American Indian or Alaska Native. API is Asian or Pacific Islander. FRPL is free/reduced-price lunch, and FRPL eligibility is a proxy for low family income. MH is mental health, and the risk model is based on predictions made from a model trained in a separate, nationally representative sample. CPS represents a child protective services investigation, and OHP is out-of-home (foster care) placement; these two things proxy for child welfare system involvement.
Estimates indicate that SBMH does not seem to affect students’ average standardized test scores, pooling across all tested subjects and grades. In the full sample (Table 1, Column 3), the estimated ATT is −0.0124 standard deviations with a 95 percent confidence interval of −0.0446, −0.0198. In the matched sample (Column 4), the estimated ATT is closer to zero (−0.0085 standard deviations, with a 95 percent confidence interval of −0.045, 0.020).8
The event study model shows parallel trends in standardized test scores leading up to SBMH adoption and no effects on test scores with the exception of a negative effect the third year after implementation (Figure 4). Results from the matched sample are very similar. The heterogeneity analysis (Figure 5) reveals negative effects for American Indian, Black students, boys, K–5 students, those eligible for FRPL, and those with child welfare involvement, along with positive effects for students not eligible for FRPL. As shown at the bottom of Figure 5, none of the ATTs for specific subjects are significant at p < 0.05, but the ATT for math scores is negative, in contrast to null reading and science score ATTs.
Dynamic Effects on Standardized Test z-Scores (Pooled Academic Subjects)
Notes: Test score data are available for all students who took tests in years they were eligible to do so. Results are from an event study model with individual-level covariates (age, race/ethnicity, sex, free/reduced-price lunch status), and grade level by year fixed effects. Significance of test of pre-intervention trends: p = 0.865.
Estimated Relative ATT Effect and 95 Percent Confidence Interval of SBMH on Standardized Test z-Score, by Subgroup
Notes: AIAN is American Indian or Alaska Native. API is Asian or Pacific Islander. FRPL is free/reduced-price lunch, and FRPL eligibility is a proxy for low family income. MH is mental health, and the risk model is based on predictions made from a model trained in a separate, nationally representative sample. CPS represents a child protective services investigation, and OHP is out-of-home (foster care) placement; these two things proxy for child welfare system involvement.
There is weak evidence that negative behaviors towards others, measured by out-of-school suspensions and by any juvenile justice involvement, improve after SBMH is implemented. ATT estimates are similar in the full and matched samples (Table 1, Columns 4–5) and correspond to an approximate 10 percent relative reduction in the likelihood of out-of-school suspension. However, the ATT is only statistically significant at the p < 0.10 level in the matched sample (ATT = –0.0044, 95 percent confidence inverval of −0.0095, 0.0006), and the sharpened FDR q-value is 0.183. The event study model (Figure 6) shows parallel trends prior to implementation and decreasing suspension rates subsequently. Matched sample results are similar. No subgroup had a positive ATT estimate for suspensions (Online Appendix Figure 3). Asian and Hispanic students, high school students, students with high mental health problem risk, and students without child welfare involvement all had significantly negative ATT estimates.
Event Study Model of the Probability of Any Out-of-School Suspension
Notes: Data are from administrative data on all students. Results are from an event study model with individual-level covariates (age, race/ethnicity, sex, free/reduced-price lunch status), and grade level by year fixed effects. Significance of test of pre-intervention trends: p = 0.306.
Results for the likelihood of annual juvenile justice involvement are more sensitive to the sample than results for suspensions. The ATT estimate for the full sample is positive but close to zero and quite imprecise (Table 1, Column 5). In contrast, the ATT estimate from the matched sample is −0.001 and precise (Table 1, Column 6), corresponding to a 25 percent relative reduction in the low average likelihood of juvenile justice involvement. For the full sample, pre-intervention trends are not significant, and there is little evidence of effect in the first five years after the intervention, except for an increase in the fourth year (Online Appendix Figure 4). However, the matched sample event study shows flat and nonsignificant pre-trends, followed by several years with negative and significant effects (Online Appendix Figure 5). In the full sample, most subgroups have null ATTs for juvenile justice, except for high-schoolers, medium mental health risk students, and students without child welfare involvement (Online Appendix Figure 6).
Turning to mental health services use outcomes, SBMH increases use of either outpatient mental health or prescription drug services. The ATT estimates are very similar in the full and matched samples, and both are precise (Table 2, Columns 1–2). In the full sample, estimates indicate that SBMH increases the likelihood of any outpatient or prescription drug mental health services among the treated by 0.013 (confidence interval: 0.004–0.023). In the full sample, there is a weak and nonsignificant increasing pre-intervention trend (Online Appendix Figure 7). In the full sample, most subgroups had significant increases in outpatient or prescription drug mental health service use, except for students who were white, boys, high-schoolers, not eligible for FRPL, at low risk of mental health problems, or had no child welfare involvement (Online Appendix Figure 8).
Event Study Model of Any Suicide Attempts in Past 12 Months
Notes: Data are from Minnesota Student Surveys. Results are from an event study model with individual-level covariates (age, race/ethnicity, sex), and grade level by year fixed effects. Significance of test of pre-intervention trends: p = 0.173.
Estimated ATT Effects of SBMH on Mental Health Services Administrative Data Outcomes
Within types of mental health services, the observed increase in any outpatient mental health or prescription drug use is driven entirely by outpatient mental health services, specifically by psychotherapy services (Table 2, Columns 3–4). The psychotherapy use results are similar and precisely estimated in both the full and matched sample, about a 13 percent relative increase. For other outpatient services, we find no effect on nonpsychotherapy psychosocial services but a precisely estimated increase in outpatient services with a school place-of-service code. This is small in absolute value but a large increase relative to the mean (Online Appendix Table 5). SBMH did not affect the likelihood of using prescription drugs for mental health overall (Table 2, Columns 5–6). Looking within specific drug classes, there is little evidence SBMH affected prescription drug classes, with the exception of increasing antipsychotic use (Online Appendix Table 6). We find little evidence SBMH affected the probability of hospital-based mental health services, with positive but imprecise ATT estimates (Table 2, Columns 7–8; Online Appendix Figure 9 presents event study results). Within hospital-based services for mental health, however, the likelihood of suicide-related hospital or emergency department use increases significantly (Online Appendix Table 5). We also find weak evidence that SBMH increased the likelihood of a new mental health diagnosis (Table 2, Columns 9–10). In the full sample, the ATT is a precisely estimated 0.009, an 11 percent relative increase. However, the ATT estimate is smaller and not significant at p < 0.10 in the matched sample.
Aside from the effects of SBMH on mental health services use within Medicaid, we find that after schools implement SBMH, the likelihood students are enrolled in Medicaid increases in both the full and matched sample (Online Appendix Table 5). While it is possible that SBMH clinicians would have an incentive to help eligible students enroll in Medicaid, the event study graphs illustrate a clear pre-trend in this outcome, the only outcome we examine with a statistically significant pre-intervention trend (Online Appendix Figure 10).9
2. Minnesota Student Survey data outcomes
We focus here on the three outcomes available in all seven MSS waves. ATT estimates indicate that SBMH did not affect self-reports of any substance use in the past 30 days (Table 3, Columns 1–2), and there are no clear pre-implementation trends in substance use (Online Appendix Figure 11). ATT estimates for the models of past-12-month suicidal ideation are negative but imprecise (Table 3, Columns 3–4). ATT from the full sample is −0.002 (p = 0.564) and from the matched sample is −0.004 (p = 0.390), corresponding to small relative changes in suicidal ideation. The event study model of suicidal ideation does not reveal any clear pre-trend but has two years post-intervention with significant declines (Online Appendix Figure 12). Evidence is stronger that SBMH affects past-12-month self-reports of suicide attempts. ATTs for the full sample and matched sample are −0.0038 (p = 0.050) and −0.0053 (p = 0.034), respectively, corresponding to relative reductions of 15 percent in each sample. The event study models do not show a clear pre-intervention trend in suicide attempts, and the pre-intervention trends are not statistically significant (Figure 7). The only clear heterogeneous effects are that SBMH led to larger decreases for white students compared to other racial and ethnic groups and an increase for the small group of students who had multiple or missing race or ethnicity (Online Appendix Figure 13). For the two MSS outcomes where data were available only for a subset of survey waves, there is no evidence that SBMH affected mental health status as measured by the general well-being index, with small and very imprecise estimates. However, we find small but somewhat precise negative ATTs on past-12-month self-reported mental treatments (Online Appendix Table 8). This latter result is from data that were only collected from students in Grade 8 and higher, and consistent with this, effects on mental health service use in the administrative Medicaid data are not statistically significant for the high school group.
Survey Data Outcomes from MSS: Estimated ATT Effects
3. Dynamic effects of SBMH
The above results are all interpreted at the school–year level and do not account for heterogeneity in effects based on students’ prior SBMH exposure. Such dynamics could be important. For instance, a student who starts using mental health services because their school has SBMH is likely to continue using services even after moving to another school, regardless of whether that next school has SBMH. Similarly, the estimated effect of SBMH on a school is likely to vary by whether students in that school had previous exposure to SBMH, which would vary over time as adoption expanded. To explore these dynamics we focused on two grade-level groups whose prior exposure to SBMH was most relevant: middle school and high school (Grades 6–8 and 9–12, respectively). For all students, we assessed whether they were exposed to any SBMH in Grades K–5 and in Grades 6–8. We then reestimated models focusing on middle school (high school) and stratified by whether students had any K–5 (Grades 6–8) SBMH exposure. We can only do this for the Minn-LInK data, where we follow students over time. The overall pattern of results is mixed (Online Appendix Table 9). SBMH seems to increase any mental health services use and average daily attendance more among students with prior exposure than those without prior exposure. For other outcomes (suspensions, juvenile justice involvement, test scores) the prior SBMH exposure patterns are less consistent between middle school and high school. Overall, we find some evidence of dynamic complementarity in SBMH’s effects, but not robust evidence that SBMH has stronger effects among previously unexposed students.
V. Discussion
We examine an example of expanding mental health services to children and adolescents that is policy-relevant, as many U.S. localities are currently implementing or considering the SBMH model, and as federal and state policymakers support expanding funding for this model. This research sheds new light on how expanding mental health services affects human capital-related outcomes, along with evaluating this specific model.
When considering the full population in our data on K–12 public school students in Hennepin County, Minnesota, we find no evidence that SBMH improved average attendance, test scores, substance use, or suicidal ideation. We do find some evidence that SBMH reduced antisocial behaviors (suspensions and juvenile justice involvement) and stronger evidence it reduced self-reports of suicide attempts. It could be that the “first-stage” effect of SBMH on mental health services is not big enough to detect broader effects, as we estimate that only 5 percent of students typically use SBMH services in schools that offer them. Our estimated effects on any outpatient mental health or prescription drug use for children on Medicaid were small in absolute magnitude (1.3–1.4 percentage points). That absolute increase is only moderately lower than the absolute increase in stimulant use reported in Currie, Stabile, and Jones (2014). In addition, a back-of-the-envelope calculation suggests that the drops in adolescent antidepressant use reported in Busch, Golberstein, and Meara (2014) were at most 1 percent for the full population. Thus, our shock to mental health treatment is reasonably similar to other relevant research, but we are limited in our ability to identify the specific students who actually need and use services.
Our approach to identify students who are at higher or lower risk of mental health problems, and thus might be more or less likely to use and benefit from SBMH, did not yield a consistent set of results. This may reflect insufficient predictive power of our predictive modeling approach and/or a disconnect between the predictions of need for mental health services and which students were most likely to receive SBMH in schools that implemented the model.
Our results should be interpreted with several limitations in mind. We are unable to identify all the students who use SBMH services or quantify the amount of SBMH services that they used. We only attempt to identify mental health services with a place-of-service code for a school among the subset of students who were enrolled in Medicaid. SBMH clinicians delivered services to students who were not billable to insurance and thus not observable in the Medicaid data. Some SBMH agencies did not immediately bill Medicaid when they started working in schools. As such, it is hard to assess the degree to which SBMH services were truly marginal or just represented a shift in the location of inframarginal services. However, the overall effects on Medicaid service use and, albeit weak, evidence of an increase in new mental health diagnoses suggest net increases in service use, though these effects must be interpreted in the context of our estimate that SBMH led to increases in Medicaid enrollment. Although we cannot identify all students who actually used SBMH themselves, it is still useful to consider effects across the whole student population. Aside from direct service provision, SBMH clinicians also did some amount of teacher education and engagement. And to the extent that treating mental health yields spillovers for classmates (Aizer 2011), it is useful to look at the full population.
Another limitation is that this study was limited to a single urban and suburban county, albeit one that is large and reasonably diverse. The effects of SBMH likely depend on the accessibility of mental health services outside of school settings. To the extent that Hennepin County, Minnesota, has greater access to youth mental health services than other areas, our estimated effects on SBMH there might be lower than if the intervention were implemented elsewhere.
As policymakers grapple with how best to address the crisis of child and adolescent mental health, information on the costs and benefits of different interventions is critical. As discussed early in the “Results” section, we estimate that the intervention costs about $100,000 per school year.10 In the midpoint of the intervention period, schools that implemented SBMH had an average enrollment of 858 students, indicating that the per student cost of SBMH is approximately $117 per year. In the specific SBMH model that we study, these costs were shared across state grant funds and health insurers, to the extent children had insurance. Other potential SBMH benefits were not quantified here, such as the reduced time and hassle costs of obtaining services for both children and parents. Nevertheless, the evidence of reductions in self-reported suicide attempts and the more suggestive evidence of improvements in disciplinary behavior and juvenile justice involvement indicate the potential for important benefits. As schools continue to be an important place for identifying and treating mental health problems, additional research that further quantifies costs, benefits, and optimal size of school-based services will be valuable.
Acknowledgments
Sponsored by the NOMIS Foundation and the Center for Health and Wellbeing at Princeton University.
The authors are grateful for Ann Kinney’s assistance in obtaining Minnesota Student Survey data, and Michelle Englund and Kristine Piescher’s assistance in obtaining Minn-LInK data. They are also grateful for helpful comments from Randall Reback, Brad Shapiro, seminar participants at Johns Hopkins University, University of Pittsburgh, and Texas A&M, and session participants at the 2021 Conference of the American Society of Health Economists and the Causes and Consequences of Child Mental Health conference hosted by the Center for Health and Wellbeing at Princeton University. The authors gratefully acknowledge grant funding from the Robert Wood Johnson Foundation Evidence for Action program (Grant No. 7588) and the University of Minnesota Center for Urban and Regional Affairs Faculty Interactive Research Program. The authors disclose that Dr. Mark Sander has been employed by Hennepin County and Minneapolis Public Schools and has had direct involvement with the program studied here. The authors have no other disclosures. The University of Minnesota IRB approved all aspects of this study. The authors are unable to share the administrative data, nor the school identifiers in the Minnesota student survey data. The authors are willing to assist (Ezra Golberstein, egolber{at}umn.edu with sharing analysis code and providing guidance on accessing the data used here.
Footnotes
↵1. The outcomes and analysis plan were preregistered at https://osf.io/u8h5r (accessed October 11, 2023).
↵2. We are grateful to a referee for this suggestion.
↵3. More information is available at https://cascw.umn.edu/community-engagement-2/minn-link/ (accessed October 11, 2023).
↵4. We also use two other approaches to identifying outpatient mental health services. One uses the Berenson–Eggers type-of-service algorithm, with similar results. The other relies on diagnosis codes on claims. This latter approach is only viable for 2009–2018, since diagnosis information was missing for the managed care claims that are the majority of the data in earlier years. We provide more detail on this in the Online Appendix.
↵5. The Medicaid data represent a lower bound on mental health services use, as we learned in discussions with local officials that some SBMH agencies did not start billing Medicaid for services until several years after starting their work in schools.
↵6. The four questions are as follows: “During the last 30 days, have you: 1) felt you were under any stress or pressure? 2) felt sad? 3) felt so discouraged or hopeless that you wondered if anything was worthwhile? and 4) felt nervous, worried, or upset?”
↵7. Other heterogeneity-robust difference-in-differences and event study estimators have been recently developed (for example, Callaway and Sant’Anna 2021; Sun and Abraham 2021). In Online Appendix Tables 3a–3b and Figures 2a–2b, we show that the main estimates are similar to those from other estimators.
↵8. We investigated and did not find evidence that SBMH affected the number of students who took standardized tests.
↵9. We also examined whether SBMH affected the likelihood of a student having an individualized education plan (IEP). SBMH did not affect the overall likelihood of having an IEP, but reduced the likelihood of having an IEP for an emotional or behavioral disability (Online Appendix Table 7).
↵10. The SBMH model is not intended to replace existing student support staff, but in theory school districts could respond to it by reducing support staffing. Combining our admittedly imprecise estimate of the effect of SBMH on student support staffing per 100 students with the average school size yields an estimated reduction of 0.3 support staff FTE, which according to BLS data would save $30,400 per school year.
- Received December 2022.
- Accepted August 2023.
This open access article is distributed under the terms of the CC-BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) and is freely available online at: https://jhr.uwpress.org.
