Abstract
We systematically investigate the effect heterogeneity of job search programs for unemployed workers. To investigate possibly heterogeneous employment effects, we combine nonexperimental causal empirical models with Lassotype estimators. The empirical analyses are based on rich administrative data from Swiss social security records. We find considerable heterogeneities during the first six months after the start of training. Consistent with previous results in the literature, unemployed persons with fewer employment opportunities profit more from participating in these programs. Finally, we show the potential of easy-to-implement program participation rules for improving average employment effects of these active labor market programs.
I. Introduction
In this study, we employ machine learning methods for a systematic investigation of effect heterogeneity of job search programs (JSPs) in Switzerland. Program evaluation studies widely acknowledge the possibility of effect heterogeneity for different groups. Stratifying the data in mutually exclusive groups or including interactions in a regression framework are two baseline approaches to investigate effect heterogeneity (see, for example, Athey and Imbens 2017a, for a review). However, these approaches may overlook important heterogeneities because they usually do not include a systematic search based on clear, spelled-out statistical rules. Furthermore, for large-scale investigations of effect heterogeneity, standard p-values of classical (single) hypothesis tests are no longer valid because of the multiple-hypothesis testing problem (see, for example, Lan et al. 2016; List, Shaikh, and Xu 2019). For example, for 50 single hypotheses tests, the probability that at least one test falsely rejects the null hypotheses at the 5 percent significance level could be up to 92 percent.1 This could lead to so-called ex post selection and the reporting of spurious heterogeneity that, in fact, resulted from so-called false positives.
The disadvantages of ex post selection of significant effects have been widely recognized in the program-evaluation literature. For example, in randomized experiments researchers may be required to define their analysis plan for heterogeneity prior to the experiment to avoid only reporting (and searching for) significant effects (for example, Casey, Glennerster, and Miguel 2012; Olken 2015). However, these pre-analysis plans are inflexible and usually not demanded (by funding bodies or editors of economic journals) in the common case of observational studies. An alternative approach that partly alleviates the ex post selection problem is to report effect heterogeneity for all possible groups. For large-scale investigations, an approach that takes account of all possible differences might lead to very small groups and thus imprecise estimates and overfitting. Further, the large number of different results makes it difficult to report the results in an intuitive way.
A developing part of the literature proposes to use machine learning algorithms (adapted for causal analysis) to search systematically for groups with heterogeneous effects (see, for example, the reviews of Athey and Imbens 2017b; Knaus, Lechner, and Strittmatter 2021). Machine learning approaches are potentially attractive because they could provide a principled approach to heterogeneity detection that makes it less likely to leave out important heterogeneities and can reduce concerns about the multiple testing problem. In addition, they enable flexible modeling and remain computationally feasible, even when the covariate space becomes high-dimensional and possibly exceeds the sample size. As opposed to conventional estimation methods, machine learning methods can balance the trade-off between including too few or too many heterogeneity variables (the so-called bias–variance trade-off) in a systematic, data-driven way.
This is one of the first large-scale economic applications of causal machine learning methods investigating effect heterogeneity for observational studies. In contrast to the few existing studies, which mainly exploit randomized control trials (for example, Bertrand et al. 2017; Denisova-Schmidt et al. 2021; Strittmatter 2019), we pursue the more common but also more demanding selection-on-observables strategy (see, for example, Imbens and Wooldridge 2009, for standard econometric approaches and their properties). For this reason, we exemplify the application of causal machine learning methods by a well-studied training program for which the evaluation literature tackled the selection issues plausibly. See, for example, for example, Card, Kluve, and Weber 2017, for an overview and meta-analysis of evaluation studies of active labor market programs (ALMP). This enables us to focus on the econometric innovations of the causal machine learning approach.
In particular, we contribute to the causal machine learning literature in at least three ways. First, we systematically investigate effect heterogeneity of JSPs. For this purpose, we propose concrete procedures that combine machine learning and program evaluation methods. We base the algorithm searching for heterogeneity on many attributes of the unemployed persons, as well as their caseworkers. For example, we consider the employment and welfare history of unemployed persons, sociodemographic characteristics, caseworkers’ subjective employability ratings of their clients, and measures for the cooperativeness of caseworkers. The latter could uncover effect heterogeneity by different monitoring intensities, which we consider an important mechanism of JSPs (Behncke, Frölich, and Lechner 2010a). Overall, we consider 1,268 different variables, including interactions and polynomials. Second, we report the effect heterogeneity in an interpretable way. We report marginalized effects and use clustering algorithms to aggregate the results of several outcomes. Third, based on the detected heterogeneities, we document the potential of different assignment rules to improve JSPs’ effects.
Importantly, we investigate the consistency of our findings across a variety of different machine learning algorithms.2 The causal machine learning literature is still lacking large-scale sensitivity checks with regard to methodological choices in credible applications, particularly in observational studies. Obviously, the robustness of the results to possible misspecifications of the empirical model is essential for drawing coherent policy conclusions.
For our main results on effect heterogeneity, we combine inverse probability weighting (IPW) with the so-called modified covariate method (MCM) (Tian et al. 2014; Chen et al. 2017). We use exceptionally rich linked unemployed–caseworker data obtained from Swiss social security records. The selection of relevant heterogeneity is carried out with Tibshirani’s (1996) least absolute shrinkage and selection operator (LASSO). For the quantification of the effects and their inference, we follow the sample splitting approach (see recent discussion in Rinaldo et al. 2019). We use half of the sample to select variables that are relevant to predict the size of the heterogeneous treatment effect, that is, that are responsible for deviations from the average effects. We use the other half of the sample for inference on the (possibly low-dimensional) selected variables and the heterogeneous effects.
Our results suggest substantial effect heterogeneity of Swiss JSPs during the first six months after the start of participation. During this so-called “lock-in” period, we observe negative effects for most participants. However, the size of the heterogeneity is strongly related to the characteristics of the unemployed. Consistent with the previous literature, participants with disadvantaged labor market characteristics benefit more from JSPs (for example, Card, Kluve, and Weber 2017). A major reason is that they face generally lower lock-in effects, and, thus, these indirect program costs are lower. Additionally, this study appears to be the first to uncover substantial effect heterogeneity by residence status. We show that JSPs are more effective for foreigners, who have less access to informal job search networks compared to locals. For caseworker characteristics, however, there is only little heterogeneity. There is also only limited effect heterogeneity beyond six months after the start of training. Finally, the paper presents easy-to-implement assignment rules that would improve the average employment effects of the JSPs. An extensive sensitivity analysis shows that the main conclusions remain robust across a variety of different estimation methods.
In the next section, we provide information about the institutional background of the Swiss ALMP. In Section III, we document the sample selection and show basic descriptive statistics. In Section IV, we discuss the econometric approach for a principled investigation of effect heterogeneity. In Section V, we report the empirical findings and robustness checks. Section VI explains our conclusions. Additional descriptive statistics, detailed information on the selection procedures, results for additional outcome variables, as well as extensive sensitivity analyses are reported in Online Appendixes A–F.
II. Background
A. Swiss Institutions
Switzerland is a federal country with 26 cantons and three major language regions (French, German, and Italian). It is a relatively wealthy country with approximately 78,000 CHF (approximately 77,000 USD) GDP per capita and a low unemployment rate of 3 to 4 percent (State Secretary for Economic Affairs 2017; Federal Statistical Office 2016). Unemployed persons have to register at the regional employment agency closest to their home.3 The employment agency pays income maintenance. Benefits amount to 70–80 percent of the former salary depending on age, children, and past salary (see Behncke, Frölich, and Lechner 2010b). The maximum benefit entitlement period is 24 months.
The yearly expenditures for Swiss ALMPs exceed 500 million CHF (Morlok et al. 2014). Unemployed persons can participate in a variety of different ALMPs. Gerfin and Lechner (2002) classify these ALMPs as (i) training courses, (ii) employment programs, and (iii) temporary employment schemes. Training courses include job search, personality, language, computer, and vocational programs. We focus on JSP training courses in this study, which is the most common ALMP in Switzerland (more than 50 percent of the assigned ALMPs are JSPs; see Huber, Lechner, and Mellace 2017). The JSPs provide training in effective job search and application strategies (for example, training in résumé writing). Furthermore, actual applications are screened and monitored. JSPs are relatively short, with an average duration of about three weeks. Training takes place in classrooms. The employment agency covers the costs of training and travel. Participants are obliged to continue to search for jobs during the course.
In Switzerland, regional employment agencies have a large degree of autonomy, which is partly related to the country’s federal organization. Caseworkers make the decision to assign unemployed persons to a training course on the basis of information about the unemployed person (for example, employment history, subjective employability rating, etc.). Additionally, employment agency policies and federal eligibility rules are relevant for the assignment decision. The federal eligibility rules are rather vague. They imply, for example, that the training has to be necessary and adequate to improve the individual’s employment chances. Caseworkers can essentially force the unemployed into such courses by threatening to impose sanctions. Unemployed persons have the option to apply to participate in such courses, but the final decision is always made by the caseworkers.
B. Related Literature on Job Search Programs
An assignment to a JSP may affect the matching process and quality alignment between the participant and their potential new job (see, for example, Blasco and Rosholm 2011; Cottier et al. 2017). Push effects could occur if participants accept jobs with low matching quality because of actual or perceived sanctions or perceived future ALMP assignments. Push effects decrease the duration of unemployment, but may reduce employment stability. On the other hand, JSP participation could improve the visibility of suitable job vacancies and the efficiency of the application process, which may improve employment stability. Furthermore, many studies are concerned with the crowding out of nonparticipants (see, for example, Blundell et al. 2004; Crépon et al. 2013; Gautier et al. 2018).
Empirical evidence about the effectiveness of JSPs is mixed. The review studies of Card, Kluve, and Weber (2010, 2017) and Crépon and van den Berg (2016) document a weak tendency towards positive effects of JSPs, especially in the short term.4 However, for Swiss JSPs, the literature finds negative employment effects, which taper off one year after the start of participation (see Gerfin and Lechner 2002; Lalive, van Ours, and Zweimüller 2008). The ambiguous findings of the literature could suggest that JSPs might have positive effects on some groups and negative effects on other groups of participants. This potential effect heterogeneity is mostly unexplored.
One reason for the ambiguous effectiveness of JSPs might be the different relative intensities of job search training and monitoring. Van den Berg and van der Klaauw (2006) are concerned that intensive monitoring reduces informal job search, which might be a more efficient strategy than formal job search for some unemployed persons. They suggest formal job search is more effective for individuals with fewer labor market opportunities. Consistent with their arguments, Card, Kluve, and Weber (2017) document that JSPs are relatively more effective for disadvantaged participants. Vikström, Rosholm, and Svarer (2013) find slightly more positive effects of JSPs for women and younger participants. Dolton and O’Neill (2002) report negative employment effects of JSPs for men and insignificant effects for women five years after the program begins. Surprisingly, the program evaluation literature is lacking large-scale evidence about the effect heterogeneity of JSPs.
III. Data
A. General
The data we use include all individuals who are registered as unemployed at a Swiss regional employment agency in the year 2003. The data contain rich information from different unemployment insurance databases (AVAM/ASAL) and social security records (AHV). This is the standard data used for many Swiss ALMP evaluations (for example, Gerfin and Lechner 2002; Lalive, van Ours, and Zweimüller 2008; Lechner and Smith 2007). We observe (among other attributes) residence status, qualification, education, language skills, employment history, profession, job position, industry of last job, and desired occupation and industry. The administrative data are linked with regional labor market characteristics, such as the population size of municipalities and the cantonal unemployment rate. The availability of extensive caseworker information and their subjective assessment of the employability of their clients is what distinguishes our data. Swiss caseworkers employed in the period of 2003–2004 were surveyed through a written questionnaire in December 2004 (see Behncke, Frölich, and Lechner 2010a,b). The questionnaire asked about the caseworker’s aims and strategies and information about the regional employment agency.
B. Sample Definition
In total, 238,902 persons registered as being unemployed in 2003. We only consider the first unemployment registration per individual in 2003. Each registered unemployed person is assigned to a caseworker. In most cases, the same caseworker is responsible for the entire unemployment duration of their client. If this is not the case, we focus on the first caseworker to avoid concerns about (rare) endogenous caseworker changes (see Behncke, Frölich, and Lechner 2010b). We only consider unemployed persons aged 24–55 years who receive unemployment insurance benefits. We omitted unemployed persons who apply for disability insurance benefits, persons whose responsible caseworker is not clearly defined, and persons whose caseworker did not answer the questionnaire (the response rate is 84 percent). We omitted unemployed foreigners with a residence permit that is valid for less than a year. Finally, we omitted unemployed persons from five regional employment agencies that are not comparable to the other regional employment agencies. This sample is identical to the data used in Huber, Lechner, and Mellace (2017). It contains 100,120 unemployed persons.
One concern regarding the treatment definition is the timing with respect to the elapsed unemployment duration prior to participation. Caseworkers may assign unemployed persons to job training programs at essentially anytime during their unemployment spell. Such dynamic or sequential program assignment has received considerable attention in the evaluation literature (see, among others, the discussions in Abbring and van den Berg 2003, 2004; Fredriksson and Johansson 2008; Heckman and Navarro 2007; Lechner 2009; Robins 1986; Sianesi 2004). We consider a classical static evaluation model and define treatment as the first participation in a JSP during the first six months of unemployment (83 percent of JSP are assigned within the first six months of unemployment). We exclude individuals who participate in other ALMPs within the first six months of unemployment from the sample, such that our control group represents nonparticipants of all programs (8,787 other ALMP participants are dropped). Potentially, this approach could lead to a higher share of individuals with better labor market characteristics among the control group than among the training participants because individuals in the control group may have possibly found another job prior to their potential treatment times. This would negatively bias the results. To overcome this concern, we assign conditional random (pseudo) participation starts to each individual in the control group. Thereby, we recover the distribution of the elapsed unemployment duration at the time of training participation from the treatment group (similarly to, for example, Lechner 1999; Lechner and Smith 2007). To ensure comparability of the treatment definitions of the participants and nonparticipants, we only consider individuals who are unemployed at their (pseudo) treatment dates.5 This makes the groups of participants and nonparticipants comparable with respect to the duration of unemployment and ensures that the treated and control groups are eligible for program participation at their respective assigned start dates.
The final sample contains 85,198 unemployed persons (Table A.1 in Online Appendix A provides the details of the sample selections steps). From this sample, 12,998 unemployed persons participate in a JSP, and 72,200 are members of the control group. These 85,198 unemployed persons are assigned to 1,282 different caseworkers.
C. Descriptive Statistics
Table 1 reports the means and standard deviations by JSP participation for some selected variables. During the first six months after training begins, JSP participants are employed for fewer months than nonparticipants. The standardized difference is above 20.6 During the first 12 and 31 months after training begins, JSP participants also have a shorter employment duration than nonparticipants, but the standardized differences decline. During Months 25–31 after training begins, the difference in the employment duration is minor.
Descriptive Statistics of Some Important Variables by JSP Participation Status
Furthermore, Table 1 documents descriptive statistics of the characteristics of the unemployed persons, the characteristics of their caseworkers, and local labor market conditions. We report descriptive statistics for additional control variables in Table B.1 of Online Appendix B. The JSP participants have spent more months employed and received a higher income than nonparticipants in the last two years prior to the program’s start. We document minimal difference between the caseworkers of participants and nonparticipants.7 The JSP participants are more often registered at German-speaking regional employment agencies and live in cantons with better economic conditions (in terms of local GDP and unemployment rate) than nonparticipants. Accordingly, JSP participants have on average better employment opportunities than nonparticipants.
IV. Econometric Approach
A. Parameters of Interest
We describe the parameters of interest using the potential outcome framework of Rubin (1974). Following the conventional notation, we indicate random variables by capital letters and the realizations of these random variables by lowercase letters. The binary treatment dummy Di, indicates JSP participation. Let 
denote the potential outcome (for example, employment) when individual i (i = 1,…,N) participates in a JSP (Di = 1). Conversely, 
denotes the potential outcome when individual i is not participating in a JSP (Di = 0). Obviously, each individual can either participate in a JSP or not, but both participation states cannot occur simultaneously. This implies only one potential outcome is observable. The observed outcome equals
The causal effect of D on Y for individual i is
However, we cannot identify the parameter γi, without assumptions that are implausible in many applications (for example, effect homogeneity). Nevertheless, group averages of γi may be identifiable under plausible assumptions. For example, the identification of the average treatment effect (ATE), γ = E[γi], the average treatment effect on the treated (ATET), θ = E[γi | Di = 1], and the average treatment effect on the nontreated (ATENT), ρ = E[γi | Di=0], are standard econometric problems (see, for example, Imbens and Wooldridge 2009). Furthermore, conditional average treatment effects (CATEs) can potentially uncover effect heterogeneity based on exogenous pretreatment variables Zi chosen by the researcher based on the policy interest,
Knowledge about CATEs could help, for example, to improve the assignment mechanism to JSPs.8
B. Econometric Background and Intuition
Machine learning methods are powerful tools for out-of-sample predictions of observable variables. However, the fundamental problem of causal analyses is the inability to observe individual causal effects because at least one potential outcome is unobservable. Recently, several methods have been proposed that apply machine learning methods in ways that overcome this fundamental problem (see, for example, the reviews by Belloni, Chernozhukov, and Hansen 2014; Horowitz 2015; Varian 2014).
Concerning effect heterogeneity, Imai and Ratkovic (2013) suggest a LASSO-type algorithm, while Athey and Imbens (2016) propose a regression tree method. Foster, Taylor, and Ruberg (2011); Wager and Athey (2018); Athey, Tibshirani, and Wager (2019); and Lechner (2018) use random forest algorithms to estimate effect heterogeneities. These algorithms are flexible and are effective at capturing multidimensional and nonlinear interactions among covariates. Imai and Strauss (2011), Green and Kern (2012), and Taddy et al. (2018) propose alternative Bayesian machine learning methods to estimate effect heterogeneity. Grimmer, Messing, and Westwood (2017) do not attempt to use the best method available. Instead, they suggest combining many different machine learning tools to estimate the conditional treatment responses. Athey and Wager (2021), Qian and Murphy (2011), Xu et al. (2015), and Zhao et al. (2012) focus on the estimation of individualized treatment rules, which primarily focus on decision rules instead of effect heterogeneity.9
All of these studies consider heterogeneity in randomized experiments. In many fields of economics, randomized experiments are expensive and minimally socially acceptable. Therefore, we consider a selection-on-observables identification strategy (for example, Imbens and Wooldridge 2009). A promising approach to estimate group-specific causal effects in nonexperimental contexts is the modified covariate method (MCM).10
To gain some intuition about the MCM, we consider the stylized case where participation in a program is randomly assigned to 50 percent of the unemployed persons. Accordingly, in this introductory example there is no need to adjust for selection into training participation. Throughout the analyses the first element in Zi is a constant term (Zi0 = 1), and the remaining elements of Zi contain additional p ≥ 1 pretreatment variables that are potentially related to the effect heterogeneity in which the researcher is interested. A standard approach to estimate effect heterogeneity is to use the interaction model,
(1)
The first term on the right side of Equation 1 provides a linear approximation of the conditional expectation of the potential outcome under nonparticipation, 
. We call this the main effects in the following. The second term on the right-hand side of Equation 1 provides a linear approximation of the CATE:
Accordingly, δ(z) describes how the treatment effect changes in response to a marginal change in z. Vansteelandt et al. (2008) point at possible sensitivities of the empirical model in Equation 1 when the main effects are misspecified. Tian et al. (2014) propose to transform the treatment dummy Ti = 2Di – 1 and rearrange the interaction model in Equation 1 to:
(2)
The treatment indicator shifts from Di ∈ {0,1} to Ti/2 ∈ {−0.5, 0.5}. The modification does not alter the coefficient vector δ. However, this transformation alters the main effects. In Equation 2, E[Yi | Zi = z] = zβt is the linear approximation of the conditional expectation of the observed outcome. Notice that Cov(Zij, TiZik) = Cov(Zij, Zik)E[Ti] = 0 for j, k ∈ {1,…, p}. The first equality holds under random assignment of training participation, and the second equality holds because E[Ti] = 0.11 Accordingly, the righthand terms of Equation 2 are independent of each other, and we can estimate the coefficients βt and δ in two separate regressions. For example, we can estimate CATEs with the model
which is the baseline model of the MCM. The MCM is suitable when only the interaction effects and not the main effects are of interest. Parsimony and robustness to misspecification of the main effects are two advantages of the MCM compared to the specification in Equation 1. We can adopt the basic idea of the MCM to nonexperimental identification strategies (see Chen et al. 2017). Furthermore, we can combine the MCM with different machine learning methods to select the variables for heterogeneity. Procedure 1 summarizes our (main) estimation algorithm of the adapted MCM approach, which we describe in detail below.
| Step 1 | Estimate the propensity score and calculate the IPW weights. |
| Step 2 | a. Randomly split the sample into training and estimation sample s. b. Select the relevant heterogeneity variables in the training sample using the LASSO approach with or without efficiency augmentation (explained in Online Appendix F.2). c. Estimate the coefficients δs: d. Set the coefficients of deselected variables to zero. e. Estimate the coefficients of the selected variables in the estimation sample. f. Calculate |
| Step 3 | Repeat Step 2 S times. Calculate the AITEs |
| Step 4 | Bootstrap the variance of  and  . (For computational feasibility, we do not reestimate Step 2b in the bootstrap replications.) |
Estimation algorithm of the adapted MCM.
C. Identification
In addition to the pretreatment variables included in the vector Zi (which are potentially related to effect heterogeneity), we consider the possibility of confounding variables, which are included in the vector Xi. Confounders are pretreatment variables that jointly affect the probability to participate in a JSP and the employment outcome. The vector Zi, may be larger, smaller and partially or fully overlapping with Xi depending on the question under investigation.
Assumption 1 (Conditional independence):
for all values of x and z in the support of X and Z.Assumption 2 (Common support): 0 < P(Di = 1| Xi = x, Zi = z) = p(Xi, Zi) < 1 for all values of x and z in the support (of interest) of X and Z.
Assumption 3 (Exogeneity of controls):
and .Assumption 4 (Stable Unit Treatment Value Assumption, SUTVA):
.
Assumption 1 states that the potential outcomes are independent of program participation conditional on the confounding pretreatment variables. The plausibility of this assumption is justified by the availability of a detailed set of confounding variables containing characteristics of the unemployed and the caseworkers. The studies of Biewen et al. (2014) and Lechner and Wunsch (2013) discuss the selection of confounders in ALMP evaluations based on rich administrative data. Within the employment agency, caseworkers have high autonomy to decide about assignment of JSPs. Our data contain the same objective measures about labor market history, education, and sociodemographics of the unemployed, as well as local labor market characteristics that are observable to the caseworkers when choosing who participates in JSPs. We observe caseworkers’ subjective ratings of the employability of their clients. Furthermore, we observe detailed information about the caseworkers’ characteristics and counseling styles. These are potential confounders because caseworker characteristics might affect the probability of JSP participation and labor market outcomes simultaneously.
According to Assumption 2, the conditional probability to participate in a JSP is bounded away from zero and one. The common support assumption has to hold when conditioning jointly on X and Z. We enforce common support by trimming observations below the 0.5 quantile of participants and above the 99.5 quantile of nonparticipants.12 This procedure shows good finite sample performance in the study of Lechner and Strittmatter (2019). Assumption 3 requires exogeneity of confounding and heterogeneity variables. To account for this assumption, we use only control variables that are determined prior to the start of JSP participation. Assumption 4 excludes spillover effects between participants and nonparticipants. In our application, this assumption appears to be at least approximately valid, as the group of unemployed individuals participating in this program is not very large.
(Identification): Under Assumptions 1–4 (and regularity conditions ensuring the existence of appropriate moments) the following equality holds:
Thus, γ(z) are identified from observable data on 
. For completeness, the proof of Theorem 1 is in Online Appendix C (see also, for example, Rosenbaum and Rubin 1983).
D. Search for Effect Heterogeneity
Chen et al. (2017) outline how we can combine MCM with inverse probability weighting (IPW), a standard approach to balance covariates in observational studies (see, for example, Hirano, Imbens, and Ridder 2003; Horvitz and Thompson 1952). We can estimate the parameter vector δ using weighted ordinary least squares (WOLS), that is, by minimizing the objective function
(3)
with the IPW weights
which we calculate using the estimated propensity score 
(we discuss the consistency of the estimator in Equation 3 in Online Appendix C). Several studies emphasize the instability of IPW when the propensity score is close to zero or one (for example, Kahn and Tamer 2010).
In Equation 3, we multiply the IPW weights by Ti, such that the weights are positive. The denominator of the IPW weights is a small sample adjustment to ensure that the weights for treated and controls sum to one (see, for example, Busso, DiNardo, and McCrary 2014). In our baseline model, we adapt the propensity score specification of Huber, Lechner, and Mellace (2017), which we report in Table D.1 of Online Appendix D. This means that in the main specifications we rely on the propensity score specification 
estimated with a probit model. However, in the robustness section we show that our results do not change when we consider the more flexible propensity score 
estimated with LASSO.
The variables included in Zi, which are potentially related to effect heterogeneity, consist of individual and caseworker characteristics, their second-order interactions, up to fourth-order polynomials, and logarithms of nonbinary variables. Additionally, we consider dummy variables for the 103 employment agencies, as well as 29 category dummies for previous industry and 29 category dummies describing the previous job. In total, this leads to 1,268 heterogeneity variables that we consider in the analyses.13
In our main specifications, we employ LASSO estimators. The weighted LASSO estimator of the MCM minimizes the objective function,
(4)
where we add a penalty term for the sum of the absolute values of the coefficients of the p variables appearing in Z. Importantly, we do not penalize the constant 
. The penalizing parameter λ specifies the amount of penalization. If λ = 0, then Equation 4 is equivalent to the WOLS model in Equation 3. However, when λ > 0 some coefficients are shrunken towards zero. For sufficiently large values of λ, some (or all) coefficients are exactly zero. Therefore, the LASSO serves as a model selector, omitting variables with little predictive power from the model.14 A challenge is the optimization of the penalty term, such that only the relevant predictors of the effect heterogeneity remain in the model. Too-low penalties lead to overfitting; too-high penalties lead to models that miss important variables (that is, we have a bias–variance trade-off).
We apply tenfold cross-validation to find the penalty term λ with the best out-of-sample performance in terms of mean-squared-error (MSE) (for example, Bühlmann and van de Geer 2011). This means we partition the sample into ten subsamples, estimate the model in nine subsamples, and evaluate the MSE of the LASSO predictions in the tenth subsample (hold-out sample). Then we repeat this procedure with alternating hold-out samples and eventually select the λ with the lowest mean MSE. The LASSO coefficients are biased when λ > 0 (regularization bias, see, for example, Zou 2006). For this reason, we use the so-called post-LASSO estimates to calculate the MSE. The post-LASSO is a standard ordinary least squares (or in our case WOLS) estimator, which includes all variables with nonzero coefficients in the corresponding LASSO model (see, for example, Belloni and Chernozhukov 2013). We choose the LASSO model with the penalty parameter λ that minimizes the post-LASSO MSE.15
There is no need to specify the main effects in the MCM approach. Nevertheless, Tian et al. (2014) and Chen et al. (2017) show that accounting for the main effects can improve the finite sample performance of the MCM because they can absorb variation in the outcome, which is unrelated to the effect heterogeneity. In Online Appendix F.2, we document two ways to implement an efficiency augmenting procedure.
Note that in case Z contains variables not included among the confounders X, there is some concern that including Z in the estimation of the propensity score might increase the predictive power of the propensity score (with the possibility of resulting common support issues) without removing additional selection bias. Therefore, our main specification is based on p(x) only. We also estimate specifications allowing Z to enter the propensity score as well, p(x,z) (see Online Appendix F.5). However, besides decreasing the precision of the estimates, the main results are not altered.
E. Estimation of CATEs
To avoid the situation in which the LASSO approach models idiosyncratic within-sample effects, we randomly partition the sample into two equally sized parts. We assume independence between the two samples. We use the first sample to select the relevant effect heterogeneity variables (training sample). We use the second sample for the estimation of a WOLS model including all selected heterogeneity variables (estimation sample). This is called the “honest” inference procedure (see the discussion about the general properties, for example, in Fithian, Sun, and Taylor 2017).
The CATE for individual i is estimated as 
. All coefficients of variables not selected in the training sample are set to zero. The coefficients of the selected variables are estimated in the estimation sample and extrapolated to the full sample. The medical and biometric literature calls 
individualized treatment effects (ITE) (for example, Chen et al. 2017). The estimates of 
vary with respect to the random sample split. To reduce the dependency of the results on a particular split, we run the analyses S = 30 times with different random splits. The choice of replicating the entire estimation procedure 30 times is a trade-off between computational feasibility and avoiding that our results depend on a specific sample split. We calculate the individualized CATEs, 
, for each split, where the post-LASSO coefficients, 
, are from the random sample split s. We use these parameters to calculate the aggregated individualized CATEs (AITEs), 
, which still vary on the individual level. This procedure is in the spirit of bootstrap aggregation (“bagging”) in the machine learning literature (see, for example, Breiman 1996). It reduces model dependency and smooths the estimated CATEs, but the estimation model of 
is more difficult to interpret than the model of 
because it is assembled out of 30 different models. To understand which factors influence the AITEs, we report averages by different groups,
where the binary variable Gi indicates whether individual i belongs to the group (Gi = 1) or not (Gi = 0). These groups could, for example, be all JSP participants, all nonparticipants, or unemployed persons with specific characteristics.
F. Variance Estimation
It appears natural to estimate the variance with a bootstrap approach over the whole estimation algorithm, including the variable selection step. However, this is computationally infeasible for a reasonable number of bootstrap replications. Thus, we use a computationally feasible bootstrap approach in which we fix the selected heterogeneity variables in each sample split.
First, we draw a random bootstrap sample b (with replacement) clustered on the caseworker level. Second, we compare the observations in the original estimation sample of split s with the observations in the bootstrap sample and use the overlapping observations to form the bootstrap estimation sample of split s. Third, based on these samples, we reestimate the CATEs for each sample split using the heterogeneity variables selected in the original training sample of the respective sample split. We repeat these three steps 1,000 times. This procedure takes into account the dependencies that stem from overlapping observations across sample splits.16
For each sample split s and bootstrap replication b we obtain the bootstrapped CATEs, 
. The bootstrapped AITEs are 
. We estimate the standard error for the AITEs with
and the standard errors of AITEs by groups with
where 
is the estimate of 
in the respective bootstrap replication b.
V. Results
A. Propensity Score Model
Table D.1 in Online Appendix D reports the average marginal effects of the estimated propensity score model. The propensity score estimates serve as inputs into the IPW algorithm. The results confirm the impression from the descriptive statistics in Table 1, namely that the participation probability is generally increasing with previous labor market success. Unemployed persons with good labor market opportunities have a greater probability to participate in a JSP. Such a selection of training participants is called “cream-skimming” (for example, Bell and Orr 2002). The effect of training is not necessarily higher for participants with good labor market opportunities because these participants would have good labor market opportunities even in the absence of training (see, for example, the discussion in Berger, Black, and Smith 2000).
When performing IPW, it is a best practice to check for potential issues of (i) insufficient support in the propensity scores across treatment states that may result in incomparable matches, as well as large IPW weights of some nontreated observations with specific propensity scores, and (ii) imbalances in covariates after IPW reweighting (due to inappropriate propensity score specifications). We document the distribution of the baseline propensity score in Figure D.1 of Online Appendix D. Furthermore, we document the balancing of the control variables after IPW reweighting in Table D.2 of Online Appendix D. We find only small imbalances between JSP participants and nonparticipants. The standardized differences are always below three.
B. Average Effects
Figure 1 shows the estimated average potential outcomes and average program effects on employment for each of the first 31 months after the program’s start. We observe substantial negative lock-in effects. The employment probability in the first three months is about 15 percentage points lower for JSP participants than nonparticipants. However, differences in the two groups’ employment probabilities disappear after 16 months. In Months 22–24 after a program’s start, we find small positive effects, but this seems to be only of short duration. Overall, the long-term effects are insignificant and close to zero. The negative lock-in effects are consistent with the findings of the previous Swiss JSP evaluations (for example, Gerfin and Lechner 2002; Lalive, van Ours, and Zweimüller 2008). Moreover, the effectiveness of JSPs is also negative in other countries (see for example, Dolton and O’Neill 2002; Wunsch and Lechner 2008). It is possible that participants reduce the intensity of informal job search during participation in a JSP, which could explain negative employment effects.
ATE, ATET, and Potential Outcome Levels by Months since the Start of JSP Participation
Notes: We estimate the ATE and ATET separately for each of the first 31 months after the start of JSP participation. Circles/triangles indicate significant effects at the 5 percent level. We obtain standard errors from a clustered bootstrap at the caseworker level with 4,999 replications.
Searching for effect heterogeneity in each month after a program’s start is computationally expensive and hard to summarize intuitively, at least if it varies over time. Therefore, we estimate the effects of JSP participation on cumulated months employed during the first 6, 12, and 31 months after a program begins, as well as during Months 25–31. Table 2 shows the respective average effects that mirror the findings in Figure 1. The lower employment probabilities after program participation translate into an average decline of 0.8 employment months (≈−24 days) during the first six months after the start of participation. This decreases to –1.1 months (≈ –33 days) during the first 12 and 31 months. We find no significant employment effects during Months 25–31 after the start of participation.
ATE, ATET, and ATENT by Duration since the Start of JSP Participation
C. Effect Heterogeneity
Table 3 reports the estimated heterogeneity coefficients, 
, obtained from one of the considered random partitions into training and estimation samples.17 The coefficients are the marginal effects of the respective variables on the treatment effect of JSP, as opposed to the marginal effects of the respective variables on the outcome level in standard linear regression models.
Post-LASSO Coefficients for the Interaction Effects of Some Outcome Variables
The first column of Table 3 reports the estimated coefficients for the outcome cumulated employment during the first six months after training participation begins. In this specification, the post-LASSO estimation selects 17 out of 1,268 potential variables. In the estimation sample, five of these variables are significant; for example, the treatment effect increments by 0.3 months (about nine days) for unskilled workers with previous earnings below 25,000 CHF a year (see Row 3). When all other selected variables equal zero, the predicted effect of JSP participation for unskilled workers with previous earnings below 25,000 CHF a year would be –0.89 + 0.3 = –0.59 months employment (≈−17 days). However, we must be cautious when interpreting the coefficients because they are selected to maximize prediction power and might differ from the structural (causal) model (see, for example, discussion in Mullainathan and Spiess 2017).
The second column of Table 3 shows the coefficients for the 13 selected heterogeneity variables for the outcome cumulated employment during the first 12 months after training participation begins. The selected heterogeneity variables are partially overlapping between the two outcomes (compare Columns 1 and 2). In Table E.1 in Online Appendix E, we report the selected heterogeneity parameters for the outcome cumulated months employed in the first 31 months after training participation begins. We omit the results for the outcome cumulated months employed between months 25–31 after training participation begins because we do not detect any effect heterogeneity in the considered sample split.
To improve precision and check the sensitivity of our results, we investigate the post-LASSO models for different random sample splits. Table 4 documents the average number of selected variables in the different random sample splits. For each random partition, we obtain different post-LASSO models. This is unsurprising because many of the variables we consider are highly correlated (for example, different measures of the employment history). Therefore, the same CATE can be obtained from different post-LASSO models, each considering different variables or different functions of variables. Table 5 reports the average number of times a variable was selected (no matter whether in levels or some interaction/transformation). Past income was on average selected eight times by the LASSO. Other variables, like the age of the unemployed, are almost never selected (see Table E.2 in Online Appendix E). Table F.1 in Online Appendix F documents the average correlation between CATEs for different sample splits. The correlations are positive and relatively large. Accordingly, the CATEs are consistent across the sample splits we consider. The selected models are not identical, but each model essentially predicts similar CATEs. Mullainathan and Spiess (2017) show a similar result in their famous barcode graph (their Figure 2). The LASSO is a prediction tool and does not reveal the structural model.
Distribution of AITEs for Months Employed during First Six Months after the Start of Participation
Notes: Kernel smoothed distribution of AITEs. Gaussian kernel with bandwidth 0.02, chosen by Silverman’s rule-of-thumb. We apply one-step efficiency augmentation. We partition the data randomly into selection and estimation samples. We choose the penalty term based on post-LASSO RMSE, which we optimize with tenfold cross-validation. The dashed vertical line shows the ATE.
Number of Selected Variables over 30 Sample Splits
Average Number of Times a Specific Variable Is Selected over 30 Sample Splits
One approach to get an overview of the heterogeneities we detect is to plot the distribution of the predicted effects. Therefore, Figure 2 reports the distribution of the AITEs of JSPs on cumulated months employed during the first six months after participation begins. The figure documents substantial variation in the AITEs. For most groups of unemployed persons, the AITE of JSP participation is between −0.8 and −1 months of employment (approximately a decline of between 24 and 30 days). However, the AITEs are less negative or even positive for a nonnegligible fraction of the unemployed persons. This points at potential ways to improve assignment to a JSP.
Table 6 reports summary statistics for the AITEs. For all outcomes, the means of the AITEs are close to the (semiparametrically) estimated average treatment effects (ATEs) (compare Table 2). For all outcomes, the median is slightly lower than the mean. This suggests a right-skewed distribution (similar to Figure 2). We find substantial heterogeneity for the outcomes cumulated months employed during the first 12 months and the first 31 months after the start of JSP participation. During the first 12 months, the JSP effect ranges from minus two to plus one employment months. During the first 31 months, the JSP effect ranges from minus four to plus four employment months. However, for the outcome cumulated months employed between month 25 and 31 after the start of JSP participation we find little heterogeneity.
Descriptive Statistics of AITEs
It is not strictly necessary to use machine learning to estimate effect heterogeneity. It is also possible to combine the MCM approach with ordinary least squares (OLS). However, then the relevant heterogeneity variables have to be selected manually in a non-data-driven, usually unsystematic, fashion. For reasons of comparison, we replicate our results using two different OLS estimations. First, we consider an OLS estimation allowing for gender, age, and citizenship. We selected these variables because previous studies of JSPs applied separate subgroup analyses along these three variables (for example, Blasco and Rosholm 2011; Dolton and O’Neill 2002; Gerfin and Lechner 2002; Vikström, Rosholm, and Svarer 2013). We go beyond these studies and include all interactions of the three variables to increase the flexibility. However, these variables account for only little heterogeneity (see Figure 3). Second, we consider a fully flexible linear model (including all 1,268 covariates). The distribution of the predicted effect heterogeneity is much more dispersed than the results obtained with the LASSO approach (see Figure 4). The fully flexible OLS estimation overfits as the range of the predicted CATEs going from −7.1 to 7.3 indicates, which is impossible for this bounded outcome (the largest/smallest possible effect is ±6 months).18 The LASSO approach is preferable to OLS because it does not underfit like the first OLS estimator, and it does not overfit like the second model. The LASSO addresses this bias–variance trade-off as part of its optimization problem. Obviously, an OLS estimation could replicate the LASSO results after the relevant covariates dimensions are known. However, these covariate dimensions are not clear a priori.
Distribution of Employment Effects of Low-Dimensional OLS Model
Notes: Subgroup analyses by gender, age, and citizenship (for example, Blasco and Rosholm 2011; Dolton and O’Neill 2002; Gerfin and Lechner 2002; Vikström, Rosholm, and Svare 2013), as well as their interactions.
Distribution of Employment Effects of Fully Flexible OLS Model
Notes: Comparison of baseline MCM with one-step efficiency augmentation and post-LASSO and the fully flexible OLS model with 1,268 variables.
D. Description of Effect Heterogeneity
In this section, we average AITEs by characteristics of unemployed persons and their caseworkers. For each characteristic, we partition the sample into two mutually exclusive groups (high g = 1 and low g = 0), by using a binary characteristic itself or by discretizing at the median of nonbinary characteristics. The parameters 
and 
average the AITEs over all unemployed in the respective group.
Figure 5 reports effect heterogeneity of JSP participation on cumulated months employed during the first six months after the start of participation by low and high values of the characteristics of unemployed persons. The groups in the top of Figure 5 show the largest effect heterogeneities. For groups at the bottom of Figure 5 we find only little effect heterogeneity. We estimate the largest degree of effect heterogeneity for unskilled workers. The average effect of JSP for the unskilled unemployed is 0.26 months (about eight days) longer than for unemployed persons in other skill categories (see Table E.2 in Online Appendix E).
AITEs on Cumulated Employment during the First Six Months after Start of JSP Participation by Characteristics of Unemployed Persons
Notes: AITEs by low and high values of the respective characteristic of unemployed persons. A low value is zero when the variable is binary or below the median when the variable is nonbinary. A high value is one when the variable is binary or not below the median when the variable is nonbinary. The AITEs are based on 30 random sample splits. For each partition, we choose the penalty term based on post-LASSO RMSE, which we optimize with tenfold cross-validation. We apply one-step efficiency augmentation. We report the 95 percent confidence interval based on the bootstrap procedure described in Section IV.F. Statistical significance: *p<0.10, **p< 0.05, ***p< 0.01. The differences and respective standard errors are reported in Table E.2 in Online Appendix E. We report results for additional outcomes in Figures E.1, E.3, E.5, E.7, and E.9 in Online Appendix E.
Conversely, Figure 5 documents that individuals with some degree of education suffer on average more from JSP than individuals with no degree. In general, we observe that the negative lock-in effect is much less pronounced for unemployed persons with lesser qualifications. This suggests that cream-skimming reduces the effectiveness of JSP participation. These findings are consistent with the evaluation literature (for example, Card, Kluve, and Weber 2017; van den Berg and van der Klaauw 2006). Furthermore, the lockin effects are less negative for foreigners. One potential explanation is that foreigners have a relatively small network for an informal job search. Therefore, the formal job search strategy might be relatively more successful for them. This suggests more foreigners should be assigned to JSPs. We find only a little heterogeneity by gender and age, which is in line with the findings of Vikström, Rosholm, and Svarer (2013) for JSPs in Denmark. In our application, the effect heterogeneity by gender is not statistically significant (see standard errors in Table E.2 in Online Appendix E).
Figure 6 reports effect heterogeneity of JSP participation on cumulated months employed during the first six months after the start of participation by low and high values of caseworker characteristics. The interpretation of Figure 6 corresponds to the interpretation of Figure 5. Although we find some statistically significant differences, they are much less pronounced than for the characteristics of unemployed persons. Most effect heterogeneity is observed by caseworkers’ own unemployment experience, but the difference is only 0.07 months (about two days). However, the difference is statistically significant (see Table E.3 in Online Appendix E). Interestingly, the cooperativeness of caseworkers has no statistically significant influence on the effectiveness of JSP participation. We would have expected this characteristic to be a good predictor for effect heterogeneity because it might approximate different monitoring intensities of the caseworker.
AITEs on Cumulated Employment during the First Six Months after Start of JSP Participation by Caseworker Characteristics
Notes: AITEs by low and high values of the respective caseworker characteristic. A low value is zero when the variable is binary or below the median when the variable is nonbinary. A high value is one when the variable is binary or not below the median when the variable is nonbinary. The AITEs are based on 30 random sample splits. For each partition, we choose the penalty term based on post-LASSO RMSE, which we optimize with tenfold cross-validation. We apply one-step efficiency augmentation. We report the 95 percent confidence interval based on the bootstrap procedure described in Section IV.F. Statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. The differences and respective standard errors are reported in Table E.3 in Online Appendix E. We report results for additional outcomes in Figures E.2, E.4, E.6, E.8, and E.10 in Online Appendix E. CW is the abbreviation for caseworker.
E. The Effects of Different Assignment Rules on Program Effectiveness
Next, we investigate the characteristics of unemployed persons with positive AITEs (Table 7). The number of individuals with positive AITEs amounts to 674, which corresponds to 0.9 percent of the unemployed persons in the sample. The first row of Table 7 reports the share of unemployed persons assigned to a JSP by the sign of the AITEs. Only 7 percent of unemployed persons with positive AITEs participate in a JSP, whereas 16 percent of the unemployed persons with negative AITEs participate in a JSP. These statistics point to the potential to improve the actual selection of JSP participants. Figure 7 provides further evidence in this direction. It documents the relationship between the AITEs and the propensity score. An effective selection of JSPs would show a positive relationship, indicating that those who benefit more have a higher JSP participation probability. However, Figure 7 shows, over a wide range of participation probabilities (that is, for unemployed persons with more than 10 percent participation probability), a negative relationship between the AITEs and the propensity score. This implies that those unemployed persons who suffer more from JSPs have a higher participation probability.
Correlation of the Propensity Score and the AITEs
Notes: Kernel smoothed regression of propensity score and AITEs. Local constant kernel regression used with Gaussian kernel and bandwidth 0.02 computed by the plug-in bandwidth selector (Fan and Gijbles 1996). The dashed horizontal line shows the ATE. The gray bars show the histogram of the propensity score.
Characteristics of Unemployed by the Sign of AITE
Additionally, Table 7 reports characteristics of unemployed persons with positive and negative AITEs. The difference gives explicit advice on how assignment rules for JSPs could be improved. For example, unemployed persons with a lower past income and lesser past employment experience tend to have positive effects from participation. Participants with lower degrees of education and foreigners seem to have a higher probability to profit from a JSP.
Strikingly, those unemployed persons who receive a low employability rating by their caseworker are more likely to experience positive effects from a JSP than unemployed persons with a medium or high rating. These results are further evidence that creamskimming does not improve JSP effectiveness.
Furthermore, we document the effectiveness of hypothetical statistical assignment rules in Table 8. Statistical assignment rules have already received considerable attention in the context of ALMPs (see, for example, Bell and Orr 2002; Caliendo, Hujer, and Thomsen 2008; Frölich 2008; Dehejia 2005; O’Leary, Decker, and Wandner 2002, among many others). However, we are not aware of any application using machine learning methods to investigate assignment rules for ALMP that systematically consider a high-dimensional covariate space.
Average CATE of Hypothetical Participants under Different Assignment Rules
For the proposed assignment rules, we keep the number of 12,712 (hypothetical) JSP participants constant.19 Therefore, the proposed assignment rules improve the average employment effects of JSPs compared to the existing assignment mechanism. However, we account for possible capacity limits only at the national level and not regional training centers in particular. We consider five hypothetical assignment rules: (i) random allocation (called “random” in the following), (ii) assignment of unemployed persons with the highest AITEs (called “best case” in the following), (iii) assignment of unemployed persons with the lowest AITEs (called “worst case” in the following), (iv) all unemployed persons with at least one unemployment spell in the previous two years and unskilled plus a random selection of the remaining unemployed persons with at least one unemployment spell in the previous two years and no degree (called “previous unemployment” in the following), and (v) all unemployed with a low employability rating by their caseworkers plus a random sample with a medium employability rating (called “employability rating” in the following). The random addition of participants in assignment rules (iv) and (v) enables us to maintain the number of 12,712 participants. The “previous unemployment” rule (iv) is inspired by the variables that show the highest treatment effects in Table 3 and Figure 5. The “employability rating” rule (v) assigns unemployed persons to a JSP for whom the caseworkers give a low employability rating, as opposed to cream-skimming, which assigns more unemployed persons with high employability ratings.
Table 8 reports the average CATE under the different assignment rules. The average CATE represents the hypothetical ATET under this treatment assignment. The “worst case” and “best case” assignment rules are the lower and upper bounds of the ATET (for a fixed number of 12,712 participants). The difference between the lower and upper bounds are about 0.65 employment months (≈ 20 days). The ATET under random assignment is –0.78 months (≈ –24 days) employment during the first six months after the start of participation. This is the benchmark assignment rule. Any imposed assignment rule should be better than random assignment. However, the observed ATET is –0.82 months (≈ –25 days) employment during the first six months after the start of the program. It appears that the current assignment mechanism is not better than a random assignment rule. In the context of Swiss ALMPs, Lechner and Smith (2007) also find that the allocation by caseworkers performs no better than random assignment. Furthermore, this is consistent with the findings of Bell and Orr (2002) and Frölich (2008), who reject the idea that caseworkers allocate training programs efficiently in the United States and Sweden. Applying the optimal assignment “best case” rule would reduce the negative employment effects by 60 percent { = [(0.82 – 0.33)/0.82] · 100%}.
For the proposed assignment rule “previous employment” the predicted ATET is – 0.51 months (≈−15 days) employment during the first six months after the start of participation. On average, each participant has nine days more employment under this assignment rule than under random assignment. The negative employment effect of the current assignment mechanism would be reduced by 38 percent { = [(0.82 – 0.51)/0.82] · 100%}. For the proposed “employability rating” assignment rule the predicted ATET is –0.61 months (≈−18 days) employment during the first six months after the start of participation. On average, each participant has six days more employment under this assignment rule than under random assignment. The negative employment effect of the current assignment mechanism would be reduced by 21 percent { = [(0.82 – 0.61)/0.82] · 100%}. These results are consistent with the argument that assignments based on expected treatment effects rather than on predicted outcomes can be more successful (Ascarza 2016). However, the average effects remain negative, and the program does not seem useful in improving the employment opportunities of unemployed persons in general. Nevertheless, the easy-to-implement assignment rules document the potential to improve the current allocation mechanism. For example, reducing the number of participants and assigning only the 674 unemployed persons with expected positive effects would increase the average employment by 0.16 months.
F. Using Clustering of the Estimated Effects to Improve the Interpretability of the Results
Until now, we have focused our discussions only on the cumulated employment in the six months after program start. The reason is that the heterogeneity of the longer cumulated time windows (12 and 31 months) seems to be very similar at first glance. One approach to find patterns that are not observable from the previous descriptions of the estimated heterogeneities is to apply k-means clustering. This is an unsupervised machine learning method that searches for clusters in the data (see, for example, Hastie, Tibshirani, and Friedman 2009). We apply k-means clustering to the predicted treatment effects of all four time periods, without using the covariates. To ensure interpretability we consider only three clusters.20 The first four rows of Table 9 provide the mean predicted effects in the three clusters. The first two rows show that all three clusters have on average substantial negative effects within the first months after program start. However, the third cluster turns positive over the time period of 31 months. While Clusters 1 and 2 show very similar negative effects after 12 and 31 months, the average effects of cluster three turn from –0.5 to +2. Thus, the 1,037 unemployed persons in Cluster 3 benefit on average from the JSP in the longer run. The remaining rows of Table 9 show that the third cluster can be described by specific covariate values, although these covariates were not directly used to construct them. Ninety-one percent of the unemployed in Cluster 3 are skilled without any degree, compared to only 1 percent and 6 percent in Clusters 1 and 2, respectively. The fraction of foreigners is also considerably higher in Cluster 3 than in the other two clusters.
k-Means Clustering of AITEs
G. Sensitivity Checks
We perform several sensitivity analyses to investigate the robustness of our results with respect to the choice of the empirical method and the selection of tuning parameters. We replicate our estimates using different forms of efficiency augmentation (see Online Appendix F.2). As an alternative variable selector, we consider the adaptive LASSO (Zou 2006, see Online Appendix F.3). Furthermore, we replicate the results with the modified outcome method (MOM) (Signorovitch 2007; Zhang et al. 2012; see Online Appendix F.1) instead of the MCM. Moreover, we employ radius-matching with bias adjustment (Lechner, Miquel, and Wunsch 2011) to balance the observable covariates between the treatment and control group instead of the IPW weights. This method shows good finite sample performance (Huber, Lechner, and Wunsch 2013). Furthermore, we compare the robustness of the main results with two different sets of additional confounders (see Online Appendix F.5 for a description of how we select the additional confounders). Finally, we compare our results with the causal forest approach (Wager and Athey 2018; see Online Appendix F.4).
Table 10 reports the Pearson correlation between the AITEs for different empirical procedures.21 No matter which specification we use, the correlation between the AITEs is always positive and mostly above 0.5. The causal forest AITEs are less strongly correlated, but they still show a decently strong positive association. Accordingly, our main findings are not overly sensitive to the choice of empirical methods or selection of tuning parameters. We report additional sensitivity checks in Online Appendix F.6. The estimation results are widely consistent across a variety of different methodological choices and estimation procedures.
Correlation between AITEs Obtained from Different Empirical Procedures
Knaus, Lechner, and Strittmatter (2021) provide a more rigorous comparison of the different machine learning approaches using an empirical Monte Carlo study. The MCM with efficiency augmentation is among the four approaches with the best finite sample performance in their simulations.
VI. Conclusion
We investigate recently developed machine learning methods to uncover systematically treatment effect heterogeneity. We apply these methods to estimate the heterogeneous effects of Swiss Job Search Programs (JSPs) on different employment outcomes by allowing for a high-dimensional set of variables potentially related to effect heterogeneity.
The employment effects of JSPs are negative during the first six months after the start of participation and taper off afterwards. Parallel to this finding, we discover substantial effect heterogeneity during the first six months after the start of program participation, but not afterwards. While an appropriate assignment rule could substantially decrease the negative lock-in effects, the negative effects of Swiss JSPs disappear only when we substantially reduce the number of participants. This is surprising given the positive effects of JSPs in several other countries (see, for example, Card, Kluve, and Weber 2010). In particular, we find that unemployed persons with low employment opportunities and foreigners experience less negative effects. The data used contain the caseworkers’ subjective employability rating of their clients. Using this measure alone for program assignment, that is, if caseworkers assign mainly unemployed persons with a low employability rating, then negative lock-in effects are already reduced by approximately 22 percent. The results remain consistent across a range of alternative estimators and different implementation choices, showing the robustness of the findings.
In practice, it is important to know whether the results of these causal machine learning estimators differ from those of conventional estimators. Of course, the (post-) LASSO results could be replicated with a conventional approach (for example, subgroup analysis using OLS) if heterogeneity margins were known. However, too little model flexibility would not reveal much effect heterogeneity, and too much model flexibility would lead to overfitting. The machine learning approaches allow addressing explicitly this bias–variance trade-off. The previous literature that uses the conventional approach considered only effect heterogeneity with regard to few characteristics in a one-by-one fashion, which tends to reveal only little effect heterogeneity.
There are still many open questions that are, however, beyond the scope of this work. On the substantive side, for example, it is not clear that the largely negative results will generalize to other economic environments and other versions of JSPs implemented in other times and other countries. On the methodological side, it is acknowledged that despite the extensive robustness checks, these methods are still very new, and there could be practical problems not yet uncovered. Furthermore, we investigate the heterogeneous employment effects only of a particular program for different unemployed persons. Finally, this study abstracts from the questions about an optimal program for a particular unemployed person, which is also relevant because of the usually rich program structure of ALMPs. Such a modified goal raises several additional statistical issues that may be addressed in future research.
Footnotes
The authors presented a previous version of this paper at Humboldt University Berlin, Cal Poly in San Luis Obispo, University of Maastricht, the Workshop on Unemployment and Labour Market Policies in Barcelona, the IZA Summer School in Labor Economics in Ammersee, the Institute for Employment Research (IAB) in Nuremberg, and the Computational Social Science Workshop in Konstanz. They thank participants, in particular Hugo Bodory, Bruno Crépon, Carlos Flores, Bernd Fitzenberger, Chris Hansen, Jeff Smith, Martin Spindler, as well as two anonymous referees for helpful comments and suggestions. The usual disclaimer applies. Financial support from the Swiss National Science Foundation (SNSF) is gratefully acknowledged. The study is part of the project “Causal Analysis with Big Data,” which has grant number SNSF 407540_166999 and is included in the Swiss National Research Programme “Big Data” (NRP 75). The data set is available as a restricted use file via the data sharing platform FORSbase. More information and the possibility to request the data for research purposes can be found here:https://forsbase.unil.ch/project/study-public-overview/17035/1/
↵1. Assuming independent test statistics as an extreme case (1 – 0.9550 = 0.92).
↵2. However, it is not the goal of the study to compare all available algorithms that have been proposed in the literature (see Knaus, Lechner, and Strittmatter 2021 for an overview). For example, we do not consider R-learning (Nie and Wager 2021), X-Learning (Künzel, Sekhon, Bickel, and Yu 2019), generalized random forest (Athey, Tibshirani, and Wager 2019), or modified causal forests (Lechner 2018).
↵3. At the beginning of the unemployment spell, newly registered unemployed persons are often sent to a one-day workshop providing information about the unemployment law, obligations and rights, job search requirements, etc.
↵4. Meyer (1995) reports negative effects on unemployment benefit payments and positive earnings effects of JSPs in the United States. Graversen and van Ours (2008) and Rosholm (2008) report positive effects of JSPs on the unemployment exit rate in Denmark. Wunsch and Lechner (2008) find that JSPs have negative effects during the first two years after a program begins, which fade out afterwards in Germany. They also show that training sequences are responsible for long-lasting negative lock-in effects.
↵5. In particular, we distinguish between two elapsed unemployment durations when the JSP participation begins. Either the elapsed unemployment duration is between one and three months or between four and six months. First, we use the treated sample to estimate the conditional probability that the JSP participation begins between four and six months after the start of unemployment. We condition on all control variables. Second, we extrapolate the estimated conditional probabilities to the control group. Third, we generate a variable ui, which has a uniform distribution between zero and one. Fourth, we assign the pseudo treatment start time “four and six months” to the control group when ui is smaller than the conditional probability that the JSP participation begins between four and six months after the start of unemployment. Control group members are more likely to end up in pseudo treatment time four to six months when the conditional probability is high. Fifth, we drop control group members with a pseudo treatment start time between four and six months in the case that they already found new employment before Month 3 begins.
↵6. The standardized difference of a variable X between Samples A and B is defined as
where denotes the mean of Sample A and denotes the mean of Sample B. Rosenbaum and Rubin (1983) consider a standardized difference of more than 20 as being “large.”↵7. In Table B.1 of Online Appendix B we also show caseworker characteristics interacted with the language of the regional employment agency. For some interacted variables, we find strong differences between participants and nonparticipants.
↵8. Additional parameters are CATEs for JSP participants θ(z) = E[γi| Di = 1, Zi = z] and CATEs for nonparticipants ρ(z) = E[γi | Di = 0, Zi = z]. The parameters γ(z), θ(z), and ρ(z) can differ from each other when confounders and heterogeneity variables are not equivalent (which is the case in our application). However, we are interested in the heterogeneities for a randomly chosen unemployed person with specific characteristics because this mirrors the decision problem of the caseworker when assigning a policy to the unemployed based on the unemployed’s observed characteristics. Thus, we focus on γ(z).
↵9. Closely related is the study of Ciarleglio et al. (2015), who proposea method to select the optimal treatment conditional on observed individual characteristics. Zhao et al. (2015) investigate the optimal dynamic order of sequential treatments.
↵10. Furthermore, Zhang et al. (2012) develop alternative nonexperimental approaches for a principled effect heterogeneity search, which is an adaptation of the Modified Outcome Method (MOM) (Signorovitch 2007). We describe the MOM in Online Appendix F.1. For one of the robustness checks, we replicate our results using the MOM. Furthermore, the tree and forest methods of Athey and Imbens (2016) and Wager and Athey (2018) are applicable in nonexperimental settings. All robustness checks are provided in Section V.G and in Online Appendix F. The main findings are not altered.
↵11. In contrast, Cov(Zij, DiZik) = Cov(Zij, Zik)E[Di] = Cov(Zij, Zik)/2, which can be different from zero.
↵12. In total, we trim 6,700 observations (582 participants, 6,188 nonparticipants). As an additional robustness check, we trim observations below the 2.5 quantile of participants and above the 97.5 quantile of nonparticipants. The main results do not change significantly (see Figure F.2 in Online Appendix F).
↵13. We exclude binary variables where less than 1 percent of (non)participants show values of zero or one. Furthermore, we keep only one variable of variable combinations that show correlations of larger magnitude than ±0.99 to speed up computation.
↵14. The larger the values of λ the fewer variables remain in the model. By gradually increasing the penalty term one can obtain a path from a full model to a model that only contains the parameter
.↵15. In robustness checks, we base the selection of the penalty parameter on the LASSO MSE. The main results are not altered.
↵16. We are not aware of any previous study that provides theoretical guarantees for our bootstrap algorithm. The theoretical properties are beyond the scope of this study.
↵17. We omit the coefficients of the main effects because they are only used for efficiency augmentation and are irrelevant for the interpretation.
↵18. Approximately 9 percent of the predicted potential outcomes are out of bounds based on the fully flexible OLS estimation.
↵19. We consider only participants on the common support. Therefore, the number of participants considered here is lower than previous numbers.
↵20. Increasing the cluster size to five does not alter the finding discussed below; see Table E.6 in Online Appendix E.
↵21. Table F.7 in Online Appendix F additionally reports Spearman’s rank correlation. Both metrics lead to the same conclusions.
- Received July 2018.
- Accepted January 2020.
