Abstract
We study the effect of U.S. Workforce Investment Act (WIA) training in two states using matched employer–employee data. This allows us to estimate the impact of training on firm characteristics and to assess the value of firm characteristics measured prior to training as conditioning variables. We find moderate positive impacts of training on employment and earnings for adults, but not for dislocated workers. We find limited evidence of positive effects on firm characteristics for adults in one state, but clear evidence of effects on industry of employment for most groups. Firm characteristics add little value as conditioning variables.
I. Introduction
Experimental evaluations of large‐scale active labor market programs have great value, but occur mostly in the United States and, even there, only rarely. In other times, in other places, and for many other programs, both policymakers and scholars must continue to rely on nonexperimental evaluations.1 As such, improving our stock of nonexperimental evidence based on current methods and the best available data, as well as continuing to advance the methodological frontier, have great value. In that spirit, this work provides new substantive findings on the training provided under the Workforce Investment Act (WIA), until recently the largest federal employment and training program in the United States.2 We also provide important new evidence on several open methodological questions in the literature.3
Our first substantive contribution consists of estimating the earnings and employment impacts of receiving WIA training (and possibly other services) rather than just WIA nontraining services using administrative data from two states. Building on the rich set of relevant conditioning variables in our data, particularly the pre‐program labor market outcomes shown by the literature to matter in this context, most of our estimates rely on a “selection on observed variables” identification strategy. Moreover, because we focus on the impact of service type conditional on WIA participation, rather than the impact of WIA versus no WIA that receives most of the attention in the literature, we do not have to worry about selection into WIA, just selection into training conditional on selection into WIA. Our data, and the WIA institutional setup, suggest that this constitutes a less challenging identification problem.
More broadly, we read the literature that uses experiments as benchmarks to study the performance of alternative nonexperimental sources of identification as indicating that high‐quality nonexperimental evaluations, where quality refers to both data and methods, can successfully replicate the broad conclusions obtained with experimental methods. See in particular the evidence on the importance of what goes into the conditioning set in Heckman et al. (1998) and the comparison of experimental and high‐quality (mostly European) nonexperimental evaluations in the meta‐analyses of Card, Kluve, and Weber (2010, 2018). The U.S.‐focused meta‐analysis of Greenberg, Michalopoulos, and Robins (2006) reaches a similar conclusion.
Our preferred estimates show positive, substantively relevant, and statistically significant impacts of WIA training on earnings and employment over the 12 calendar quarters following WIA registration for workers served under the adult funding stream. In contrast, for workers served under the dislocated worker funding stream, we find persistently negative impacts in one state and initially negative and later marginally positive impacts in the other. These findings parallel those in earlier nonexperimental work by Heinrich et al. (2013).
Our second substantive contribution centers on our analysis of the determinants of training receipt conditional on WIA registration. We find standard patterns for age and schooling, with younger workers more likely to receive training, along with those in the middle of the educational distribution. These patterns appear in both our univariate and multivariate analyses and suggest that WIA allocates training in reasonable ways along these dimensions. In contrast, we find large unconditional differences by race/ethnicity but only very small conditional differences. Here, our multivariate analysis sheds important light on the source of univariate disparities.
For our first methodological contribution, we estimate impacts on the types of firms at which WIA participants obtain jobs after WIA. We can do this because, for the first time in the literature, we link administrative data from the WIA programs in our two states to the rich Longitudinal Employer‐Household Dynamics (LEHD) data maintained by the U.S. Census Bureau. Motivating this analysis is the well‐known fact that, controlling for worker characteristics, employer characteristics contribute importantly to worker earnings and other outcomes; see, for example, Abowd and Kramarz (2002); Andersson, Holzer, and Lane (2005); and Holzer et al. (2011). If WIA and other training programs can improve the quality of the firms to which workers “match,” then we have identified an important mechanism through which job training programs might work, over and above their effects on the worker’s stock of human capital.
Empirically, our data provide limited evidence of effects of WIA training on the propensity of adult workers to obtain jobs at firms with desirable characteristics. We find clear evidence of effects of WIA training on changing industries and modest positive effects on other firm characteristics for adult trainees in one state.
Our second methodological contribution lies in the examination of alternative sets of conditioning variables in the context of our preferred “selection on observed variables” identification strategy. The greatest novelty arises in our examination of variables related to the firm at which the worker last worked prior to WIA participation. We expected, based on the literature, that firm characteristics might proxy for otherwise unobserved worker characteristics and so have an effect on our estimates by reducing any remaining selection bias. In fact, we find to our surprise that the firm characteristics add essentially no value to the conditioning variables available from state unemployment insurance (UI) wage record data. We also examine the value of conditioning on an additional year of pre‐program earnings data relative to Heinrich et al. (2013) and the value of conditioning on a proxy for local labor markets. The former does not move the estimates, somewhat to our surprise. In contrast, local labor markets do matter.
Our third methodological contribution arises from comparing our estimates based on “selection on observed variables” with estimates based on the (conditional) bias stability assumption, which underlies difference‐in‐differences estimators. Broadly, we find little qualitative difference, which suggests either little selection into training based on time‐invariant unobserved variables or that the pre‐program outcomes in our conditioning set do an adequate job of capturing stable factors, like motivation and ability, that we do not directly observe.
II. The WIA Program
The Workforce Investment Act (WIA) was passed by Congress and signed into law by President Clinton in 1998 and became operational in 1999–2000; Besharov and Cottingham (2011) provide a detailed history. It was replaced in 2015 by the Workforce Innovation and Opportunity Act (WIOA). As noted in Barnow and Smith (2016), WIOA changes little of substance relative to WIA, with the implication that our findings should generalize to the new program.
Title I of WIA provided general funding streams for services to adults, dislocated workers, and youth. Adults comprise individuals 18 years and over meeting various broad criteria. Dislocated workers comprise adults who have recently lost a job or are about to lose a job. WIA youth programs serve in‐school and out‐of‐school youth ages 18–21. The target populations for these three streams partly overlap; in particular, almost all clients served as dislocated workers could be served as adults, while some adult clients could be served as dislocated workers.4 In fiscal year 2011, these three streams received just $2.8 billion in funds, a dramatic decline in real dollars relative to peak funding for federal job training in 1980 (Holzer 2009). Because youth receive different services from other participants, we restrict our attention and our data to individuals served under the adult and dislocated worker programs.
Participation in WIA often begins at a “one‐stop” office. These offices aim to provide workers with access to a variety of programs including WIA, Unemployment Insurance (UI), and the Employment Service (ES) in a single location. Unlike some European programs, WIA participants enroll as volunteers. As described in Blank, Heald, and Fagnoni (2011), WIA‐funded services for adults and dislocated workers fall into four categories: self‐service core services, staff‐assisted core services, intensive services, and training services. Recipients of self‐service core services typically use computers or other resources related to job search and do not formally register for the program. As a result, they do not appear in our data. Staff‐assisted core services (hereinafter just core services) consist of low‐intensity interventions, such as assisted job search or provision of labor market information. Intensive services include assessment, case management, and short courses in topics not specific to a particular occupation, such as anger management. Finally, training services include basic skills training, classroom training in occupational skills (CT‐OS), and subsidized on‐the‐job training at private firms. As shown in, for example, Table II‐12 of Social Policy Research Associates (2005), the vast majority (over 85 percent in 2003) receive CT‐OS. WIA imposes no eligibility rules on core services. Low‐income individuals and individuals on public assistance receive priority for intensive and training services, but WIA imposes no income‐based eligibility rules (Blank, Heald, and Fagnoni 2011).
Individual Training Accounts (ITA) fund most training under WIA; on‐the‐job training constitutes the key exception. ITAs represent vouchers that participants can use to purchase training at certain providers. King and Barnow (2011) document the provider pre‐approval process and its variation over time and space. In addition to restrictions on providers and expenditures, participants typically make their decision in consultation with a caseworker, who may encourage some choices and discourage (or even prohibit) others. This feature of WIA also varies widely in implementation (see, for example, Perez‐Johnson, Moore, and Santilano 2011 and Van Horn and Fichtner 2011). While caseworkers affect the services offered to WIA enrollees, they do not, unlike some of their European counterparts, have the power to sanction participants. Most of the training funded under ITAs consists of relatively short occupational training courses provided by community colleges, other nonprofit institutions, or (less often) by for‐profit providers.
Normatively, WIA participants follow a path from core services to intensive services to training, where each succeeding step only occurs when the prior step fails to yield employment. In practice, as described by D’Amico et al. (2004), for example, there exists tremendous variation across states, across local offices (called WIBs for Workforce Investment Boards) within states, and even across caseworkers within WIBs, as well as over time at all of these levels, in how registrants select into specific services, including training. Some states structure their programs to train almost everyone who registers. Others, as documented for Wisconsin in Almandsmith, Adams, and Bos (2006), rely in part on the information provided by the Worker Profiling and Reemployment Services (WPRS) system, which provides predictions of likely benefit exhaustion or of benefit receipt duration among UI claimants. Other states, including the states in our study, have a more ad hoc approach. Some claimants follow the normative sequence while others do not, perhaps because they have a referral from a training provider or because a caseworker identifies job skills as their key deficiency. Exhibit II‐1 in D’Amico et al. (2009) illustrates this type of system. Even those enrollees who follow the normative sequence typically do so relatively quickly, that is, over a few days or weeks rather than many months as in some European programs, which largely removes concerns about dynamic selection (that is, about the selection process changing over the course of worker unemployment spells).5
III. Data
A. Data
The analysis builds on WIA data from two states linked to earnings, employment, and firm data from all 50 states. Individual records from the Workforce Investment Act Standard Record Data (WIASRD) from a medium‐sized state on the Atlantic seaboard (“State A”) and a large, Midwestern state (“State B”) constitute our first major data source. The WIASRD files contain data on every individual who registered for WIA services in each state. For each registrant, the microdata contain the following: dates of WIA registration and exit, location of WIA service receipt, demographics (including age, education, race, and sex), membership in certain categories (for example, disabled or veteran), specific services received, and employment and earnings for quarters before and after service receipt drawn from state UI records.
The Longitudinal Employer–Household Dynamics (LEHD) data constitute our other primary data source. UI wage records and Quarterly Census of Employment and Wages (QCEW) data for all 50 states sit at the core of the LEHD. The latter consist of quarterly reports filed by employers for each worker in covered employment, which includes roughly 96 percent of private nonfarm wage and salary employment (Stevens 2007). As described in Abowd et al. (2009), the LEHD data also include demographic information, such as date and place of birth, sex, and a crude race/ethnicity measure obtained by matching to the SSA Numident file and other sources.
In addition to the large samples and long earnings histories, the matched employer data represent a marquee advantage of the LEHD data. They enable us to construct several important time‐varying and fixed characteristics of workers and their employers.6 For workers, the time‐varying characteristics include measures of overall experience and job tenure with each employer. For employers, we calculate industry, firm size, average payroll per worker paid per quarter, and measures of turnover. In both cases, we follow Holzer et al. (2011) in the details. We also use worker and firm “fixed effects” constructed following the methodology outlined in Abowd, Lengermann, and McKinney (2002).
We limit our WIASRD data to all program participants in each state who registered between 1999 and 2005, with a valid exit date. The Census Bureau matched the individuals in this group to the LEHD. For each individual included in the data, we retain employment and earnings in the LEHD data for 25 calendar quarters including the quarter of registration and 12 quarters before (Q–12 to Q–1) and 12 quarters after (Q + 1 to Q + 12) that quarter. We adjust the earnings values into 2020 constant dollars using the CPI‐U‐RS. Characteristics of employers, including industry, size, and turnover rates, refer to the last dominant employer prior to registration (when used as conditioning variables) and to the last dominant employer in our data (when used as outcomes). Online Appendix 1 provides more details about the LEHD and the construction of our sample, as well as variable definitions and descriptive statistics.
In all, we have more than 26,000 WIA registrants for State A, with nearly 16,000 in the adult program and nearly 11,000 among the dislocated; comparable sample sizes for State B are more than 50,000 for the state, with around 23,000 and 28,000 in the adult and dislocated streams, respectively. We perform all of our analyses separately by state and stream. The states have distinct WIA programs, as illustrated by the quite different fractions of registrants who receive training. In addition, the prior literature, for example, Heinrich et al. (2013), finds quite different impacts for registrants served in the adult and dislocated worker streams.
B. Descriptive Statistics
Tables 1–4 describe our samples of WIA registrants. Tables 1 and 2 present a subset of our conditioning variables (registration year, sex, race/ethnicity, age, and education), while Tables 3 and 4 present quarterly earnings and employment.
Descriptive Statistics for Characteristics, State A
Descriptive Statistics for Characteristics, State B
Descriptive Statistics for Earnings and Employment, State A
Descriptive Statistics for Earnings and Employment, State B
In terms of timing, roughly three‐fourths of the sample registered during the labor market downturn in 2001–2003, but the sample also includes many participants before and after that period. Overall, the patterns in our two states broadly comport with the national WIASRD summaries published by the Department of Labor for the years 2001–2005. In terms of who receives training and who does not, Tables 1 and 2 reveal several important patterns. First, in both states and for both funding streams, whites have a higher probability of receiving training than nonwhites. Second, women receive more training among adults, while men receive more among the dislocated. Third, WIA participants in the middle of the education distribution (that is, those with high school or some college) receive training more often in both funding streams. Fourth, relatively younger participants differentially sort into training.
At least some of these distinctions make sense in terms of who might make a good candidate for an investment in skills training. For instance, as in the standard human capital model, training likely makes more sense for younger workers, who have a longer time horizon over which to realize the returns from their investment. As suggested in Jacobson, LaLonde, and Sullivan (2005), workers in the middle of the education distribution likely have the basic skills to master somewhat technical occupational training, while those with less than high school may lack them. At the same time, workers with a college education likely already have sufficient skills or will learn what they need on the job, as noted in Barron, Berger, and Black (1997).
The quarterly earnings and employment data tabulated in Tables 3 and 4 and graphed in Figure 1 also have stories to tell. We focus on three patterns here. First, in both states, WIA participants served as dislocated workers have substantially higher pre‐program mean earnings and employment than those served as adults. This signals that even the relatively vague eligibility rules for the dislocated funding stream have real‐world bite. Second, particularly in State A, trainees have about the same earnings as nontrainees within each funding stream. This suggests relatively little selection into training based on time‐invariant outcome differences. Third, the mean earnings and employment of WIA participants in our states decline in the period leading up to registration. This literature calls this oft‐observed pattern the “Ashenfelter dip” (Ashenfelter 1978); see also Heckman and Smith (1999). In this population, the dip begins about four quarters prior to registration. The dip indicates strong selection into WIA based on transitory outcome shocks, but the quite modest differences in the nature and extent of the dip between trainees and nontrainees within each funding stream and state suggest little selection into training based on differences in transitory labor market shocks among registrants, especially in State A.
IV. Treatment and the Parameter of Interest
This study compares WIA participants who receive training, and perhaps also core and/or intensive services, to WIA participants who receive only core and/or intensive services.7 Our estimates, combined with data on costs, allow us to determine whether the additional resources spent on training services, on average, have labor market effects that cover their costs. This represents an important policy question; the wide divergence across states in the fraction of WIA participants receiving training documented in Heinrich et al. (2013) suggests that policymakers do not perceive a clear consensus on this question in the literature.
As shown in Tables 3 and 4, many WIA participants in both our states and both funding streams receive training, and many do not. Median WIA enrollment duration for trainees is around three quarters in State A and around four quarters in State B. In contrast, the median enrollment duration for registrants receiving only core and/or intensive services is typically about a quarter shorter.
These values provide only a crude (and upward‐biased for most registrants) guide to treatment intensity for several reasons, including data quality issues (for example, we observe large spikes in enrollment durations at particular values in one state), the potential for strategic manipulation for performance management reasons as in Courty and Marschke (2004), and because training participants may have to wait for a course to start and may also receive job search assistance following training completion.
Unlike Hollenbeck (2009) and Heinrich et al. (2013), we do not attempt to estimate the impact of participating in WIA versus not participating. Instead, we focus on the training analysis because we think it represents a simpler nonexperimental evaluation problem. First, WIA core/intensive participants represent a natural comparison group for trainees in a way that ES recipients and/or UI claimants do not for all WIA participants. Second, given the common earnings patterns for trainees and nontrainees shown in Tables 3 and 4, combined with the substantial amount of caseworker discretion and local heterogeneity in the WIA program, we think that a selection on observed variables identification strategy has greater plausibility in the training context. Finally, focusing on training allows us to avoid difficult questions of temporal alignment that arise when comparing WIA enrollees to other groups.
To represent our parameter of interest more formally, consider the standard potential outcomes framework wherein Y1 denotes the outcome with training (that is, the treated outcome) and Y0 denotes the outcome without training. We estimate the average treatment effect on the treated (ATET) given by E(Y1 – Y0 | D = 1), where D = 1 indicates receipt of WIA training. When combined with data on the average incremental cost of training, the ATET allows a cost–benefit analysis on the training for the trainees who currently receive it. We do not purport to provide evidence on the effect of training on those not currently receiving it.
V. Identification and Estimation
A. Conditional Independence
Our primary identification strategy builds on the conditional (mean) independence assumption (CIA) and assumes that, conditional on a set of observed characteristics X, the untreated outcome does not depend on treatment status. The literature also calls this “selection on observed variables.” In notation, we assume E(Y0 | X, D = 1) = E(Y0 | X, D = 0). This (very strong) assumption suffices to identify the ATET of training in our context; we offer a defense of it shortly. Following standard practice, we match on the propensity score, P(X) = Pr(D = 1 | X), which in our context corresponds to the conditional probability of training receipt given WIA enrollment. Rosenbaum and Rubin (1983) show that if the CIA holds for X, then a version based on the propensity score holds as well; that is, E(Y0 | P(X), D = 1) = E(Y0 | P(X), D = 0). We estimate the propensity score using a probit model, promising to increase its flexibility (but not too quickly!) on those magic days when our sample size increases.
For the CIA to hold in practice requires that X include all the variables that affect both training receipt and outcomes in the absence of training receipt—a tall order. To justify this assumption, we find it helpful to think about what we know from theory, the institutional context, and existing empirical evidence about the determinants of training participation and of labor market outcomes in this population. We can then list the conditioning variables that knowledge implies and argue that our data contain them and/or reasonable proxies for them.
Recall from the institutional background in Section II that receipt of WIA training embodies a two‐sided selection problem. The WIA participant has to want training, and the WIA caseworker has to want to give them training. WIA participants’ desires for training will depend on their perception of the job market for their current skill set, their age, their motivation, their ability to learn in a classroom setting (and disutility from doing so), their beliefs about the value of the training they might receive, the availability of government transfers or helpful family members to provide the basics of life during training, and so on. Many of these factors, such as motivation, ability and inclination to learn, and the market value of their skills, also affect outcomes in the absence of training. The caseworker will have beliefs about all these factors as well. In addition, the caseworker will have information about the budget set of the local WIB and about local training providers.
We observe some of these variables directly, such as age and the most recent industry of employment. For others, such as motivation and ability, we have to rely on proxies. In our case, realized pre‐program labor market outcomes serve as the proxies. Motivated workers and high ability workers should do persistently well in the labor market; if so, conditioning on earlier labor market outcomes will remove any selection bias that results from motivation and ability also affecting training receipt. The extant literature—for example, Heckman et al. (1998) and Smith and Todd (2005)—clearly shows the importance of conditioning on pre‐program labor market outcomes. The literature is less clear about whether they can serve as all, or just part, of a successful identification strategy. In addition to lagged outcomes, we can use geographic indicators to proxy for the state of the local economy, as well as differences in local training emphasis, opportunity, and quality.
The broader literature claims that we have most of what we need. In particular, the important study of Lechner and Wunsch (2013) compares alternative sets of conditioning variables in the context of rich German administrative data and identifies those that solve the selection problem in their context.8 Of the variables they identify, we lack variables related to family status, part‐time work, occupation in last job, and more detailed information on past qualifications than the simple years of schooling variable present in our data. Also worth noting are variables that do not make a difference in their study, which include firm characteristics (just as we find below), industry‐ and occupation‐specific experience, health‐related variables, openness to mobility, various caseworker‐related variables, job‐search behavior, and local labor markets. Along this same line, Caliendo, Mahlstedt, and Mitnik (2017) link survey data on various noncognitive skills to the German administrative data and find that they do not make a difference to their impact estimates. While we advocate further work along the lines of Lechner and Wunsch (2013) and Caliendo, Mahlstedt, and Mitnik (2017), and we generalize from Germany to the United States only very cautiously, these studies increase our confidence in our identification strategy.
We consider six different sets of conditioning variables X. Model 1, inspired by Dehejia and Wahba (1999, 2002), contains only sex, race, age, education, and two years of pre‐enrollment earnings. It serves as a minimalist baseline that incorporates only variables available in commonly used data sets such as the U.S. Current Population Survey. Our prior, given the performance of similar specifications in other contexts in Smith and Todd (2005) and Heckman, Ichimura, and Todd (1997, Table 6a), is that this conditioning set does not suffice for the CIA.
Model 2 represents our approximation of the specification employed in Heinrich et al. (2013), but leaving aside indicators for geographic location. Relative to Model 1, it includes calendar year and calendar quarter indicators, a more flexible age specification, disability and veteran status indicators, flexible employment and earnings histories over the eight calendar quarters prior to registration including the industry of the most recent employer, and UI benefit receipt and ES participation over the same period.
Model 3 adds indicator variables for the local one‐stop office at which the participant enrolled. These approximately represent indicators for counties. Model 4 adds several characteristics of the firm for which each worker most recently worked; we describe those characteristics in detail in Section VIII. Model 5 conditions on an extra four quarters of pre‐program earnings and employment information relative to Model 3. Finally, Model 6, our preferred specification, starts with Model 3 and adds the firm variables included in Model 4 and the additional quarters of pre‐program earnings included in Model 5 to create one grand propensity score model.
To evaluate the importance of including particular variables in the conditioning set, we compare the nonexperimental impacts they imply because we do not have an experimental benchmark. This strategy, which loosely says that if you keep adding conditioning variables, and the estimates do not change, then probably you have solved the selection problem, is not an uncommon one in the literature. See, for example, Black, Daniel, and Smith (2005, Tables 2–4).9 Heckman and Navarro‐Lozano (2004) provide a very useful formalization of this strategy that makes clear that it requires the absence of a variable or variables affecting both treatment and outcomes unrelated to the conditioning sets under consideration; see also Oster (2019).
B. Conditional Bias Stability
The conditional bias stability assumption (BSA) assumes that unconfoundness holds but only conditional on a person‐specific fixed effect. Put differently, BSA assumes that the available conditioning variables do not suffice to solve the selection problem on their own, but do suffice once the fixed effect has been removed, as we do by differencing. The propensity score version of the BSA corresponds to:
where t denotes an “after” period and tʹ denotes a “before” period. Written in this way the BSA is sometimes called the “common trends” assumption, as it implies that, conditional on X, the mean of the untreated outcome has the same time trend for the treated and untreated units. See, for example, Heckman et al. (1998), Rosenbaum (2001), or Lechner (2010) for more on the BSA.
Substantively, the motivation for the BSA relative to the CIA comes from the concern that some relatively stable unobserved characteristic, such as ability or motivation or attractiveness, may persistently affect labor market outcomes, but not get fully captured by conditioning on the available pre‐program labor market outcomes. The literature suggests that this issue may arise in some contexts but not others. For example, Smith and Todd (2005) find that difference‐in‐differences matching, motivated by the BSA, comes closer to the experimental estimates in the context of the National Supported Work (NSW) Demonstration data on men studied in LaLonde (1986) than does cross‐sectional matching motivated by the CIA. In contrast, Calónico and Smith (2017) study the NSW data on women and find little difference between the two identification strategies. They conclude that the differences for men in Smith and Todd (2005) likely result from failure of conditional independence for that group. Finally, while Heckman and Smith (1999) find selection on transitory shocks to labor market outcomes and on persistent differences in the unconditional earnings patterns in the National Job Training Partnership Act Study data, they find no strong differences between their cross‐sectional and conditional difference‐in‐differences estimates. This indicates that conditioning on pre‐program outcomes suffices in that context to capture the not directly observed time‐invariant characteristics. In the absence of clear direction from the literature, we pursue both identification strategies.10
C. Estimation
The literature offers a wide variety of estimators for use with our identification strategies. We employ inverse propensity weighting (IPW) as our primary estimator. Nearest neighbor matching plays the role of secondary estimator. Online Appendix 2 justifies our choice of estimators and provides econometric and implementation details.
VI. Results: Determinants of Training Receipt
Table 5 presents average derivatives for select covariates from propensity score Model 6 described in Section V.A; Online Appendix 3 presents estimates for the other models and for the remaining covariates in Model 6. Many of the univariate differences between trainees and nontrainees prove robust to conditioning. In particular, differences in the incidence of training by age (younger participants get more) and education (participants in the middle of the education distribution get more) generally persist as we enrich the conditioning set. The large unconditional differences we observe by race represent the partial exception to this broader pattern. In the models with the richest covariate sets, this difference falls dramatically.
Probit Models of WIA Training Receipt, Model 6
VII. Results: Impacts on Earnings and Employment
A. Full Sample
Tables 6 and 7 present estimates from Model 6, our preferred specification, obtained using our preferred IPW estimators. Figure 1 presents the corresponding estimated untreated means. In each table, we present impacts on earnings and employment for quarters Q + 1 to Q + 12. We also present two summary measures of earnings impacts: the sum over all 12 quarters and the sum over quarters 9–12. The latter measure completely avoids the “lock‐in” period when trainees are receiving training. Online Appendix 3 describes the distributions of the estimated propensity scores along with the results from conventional balance tests; our main‐effects‐only specification exhibits surprisingly good balance and occasions no common support issues.
Impacts on Earnings and Employment, Inverse Propensity Score Weighting, Model 6, State A
Impacts on Earnings and Employment, Inverse Propensity Score Weighting, Model 6, State B
Our estimates display several important patterns. First, in both states and both streams we observe lock‐in effects in the form of negative impacts in the initial quarters after WIA registration. These result from trainees reducing or eliminating job search during their training while the comparison group actively engages the job market during and after their receipt of core and/or intensive services. Second, we find for both states and streams that the quarterly impacts gradually increase until they appear to stabilize, usually around eight quarters after registration.
Third, we find substantively important and surprisingly large differences in impacts between the adult and dislocated worker funding streams. In State A, the impacts for adults stabilize around $350 per quarter, while those for dislocated workers stabilize at around –$180. Similarly, in State B, the adults stabilize at $500 or so, while the dislocated workers, who start out with large negative impacts, have positive and significant impacts of more than $350 in the last two quarters of the data. Given that essentially all the dislocated worker participants could have received services under the adult funding stream and that many of the adults could have received services under the dislocated worker stream, this difference puzzles us. Fourth, the employment impacts track the earnings impacts except for the dislocated worker stream in State A, where negative earnings impacts in later quarters coincide with positive employment impacts. Fifth, we find modest differences across states, with larger impacts in State B than in State A for both funding streams. Finally, as expected, the nearest neighbor matching estimates presented in Online Appendix 4 tell the same substantive story, but less precisely.
B. Men and Women
As discussed in LaLonde (1995) and Greenberg, Michalopoulos, and Robins (2003), for example, earlier evaluations of employment and training programs in the United States have typically found larger impacts for women than for men. Table 8 presents separate estimates for men and women in our context for the two cumulative earnings impact measures. The point estimates for the adult stream match the usual finding, with women having larger impacts than men over the full 12 quarters.
Impacts on Earnings, Inverse Propensity Score Weighting, Model 6
However, the close similarity of the estimates for men and women in the four final quarters suggests that the overall difference derives mainly from differences in opportunity cost, with men giving up higher earnings while in training. In contrast, for the dislocated worker stream, women have more negative impacts overall but, as with the adult stream, a much smaller difference in quarters 9–12. None of these differences attain conventional levels of statistical significance. Given the lack of clear evidence of meaningful differences in impacts, we pool men and women in our remaining analyses.11
C. Comparisons to the Literature
Heinrich et al. (2013) provide the closest analogue to our work. Before turning to their results, we note some differences in the analyses. First, Heinrich et al. (2013) look at different states, and our two states do not constitute a proper subset of those considered in Heinrich et al. (2013). Second, the time periods differ, with Heinrich et al. (2013) considering participants who exited between July 2003 and June 2007. Third, though both studies assume conditional independence and rely on administrative data for their conditioning variables, the exact propensity score specifications and estimators differ in the details.
Figures 5 and 8 of Heinrich et al. (2013) present estimates of the impact of training versus core/intensive services for adults and dislocated workers, respectively. They pool the data from the states in their analysis but present separate impacts for men and women. At a broad level, they find the same patterns that we do, including a substantial lock‐in effect in the initial quarters (other than male adults), impacts that increase in relative time and then stabilize (other than male adults), slightly more positive impacts for women than men, and much more positive impacts for adults than for dislocated workers.12
Leung and Pei (2020) study the labor market effects of enrollment in public community colleges by Ohio workers with a recent UI claim using state administrative data. They pursue identification and estimation strategies similar to our own. As part of their broader analysis, they provide estimates for a subpopulation of WIA enrollees who roughly correspond to our dislocated worker funding stream registrants. The impact estimates in their Figure 4 bear more than a passing resemblance to our estimates for State B in Table 7, as they start out with large negative lock‐in effects but turn positive after around 10 quarters.13
In sum, our estimates generally tell the same stories about WIA training impacts as the two other studies applying roughly the same methods to similar data in other times and places.
VIII. Results: Firm Characteristics
From a long list of possible (and interesting!) candidate firm‐related outcomes, we confine ourselves to just six to avoid concerns about multiple comparisons. We chose our six based on the prior literature and before looking at any impact estimates.14 Three of the six relate to the firm fixed effect described in Holzer et al. (2011). These fixed effects capture differences in conditional mean worker earnings across firms. They condition on worker fixed effects and some firm characteristics and thus offer a relatively clean measure of firm “quality.”
The outcomes are: (i) an indicator for having no available firm fixed effect, which implies either having no job in Q + 1 to Q + 12 or else only jobs at firms with fewer than 25 employees, (ii) an indicator for having a job at a firm with a fixed effect above the median, and (iii) the continuous firm fixed effect itself. Coding outcomes (ii) and (iii) for workers who lack a firm fixed effect because they did not work in the “post” period or because they worked but at a firm too small to have a fixed effect raises an unhappy trade‐off. On the one hand, if we code the outcomes for such workers as missing, we create a selection problem by nonrandomly dropping the corresponding observations from the analysis for that outcome. On the other hand, if we code the outcomes to zero, we complicate the interpretation. We split the difference and code (ii) to zero when the fixed effect is missing but drop observations with a missing fixed effect when analyzing (iii).
Our fourth firm‐related outcome consists of an indicator for working at a firm with at least 100 employees. A large literature (for example, Brown, Hamilton, and Medoff 1990) links firm size and compensation levels. The fifth measure consists of an indicator for working at a high‐turnover firm, defined as having a turnover rate above the median, as the literature has found that employment at high‐turnover firms is systematically associated with lower compensation (Holzer, Lane, and Vilhuber 2004). The sixth measure indicates a change in industries between the last pre‐registration dominant employer and the last post‐registration dominant employer.
Changing industries may represent a good thing, as when moving from a declining to a growing industry, or a bad thing, as when leaving behind industry‐specific human capital, or some combination of the two. For all three of these variables we code workers without post‐registration employment as zeros. Andersson, Holzer and Lane (2005) and Holzer et al. (2011) document the correlations among the firm‐related variables in the LEHD.
All six measures refer to the last dominant firm that employed the worker in our 12 quarters of post‐registration data. While WIA might arguably have the largest direct effect on the first firm with which the worker matches, we thought it more important to focus on longer‐run outcomes. By looking at the last dominant firm we may capture the result of a sequence of jobs induced by the program, while missing potentially promising job matches that end quickly.
How should we think about these outcomes? First, we can think about them as illustrating mechanisms through which WIA generates impacts on earnings. Second, we can think about them as providing information relative to the likely duration of impacts. The literature on worker–firm matches suggests that placements at larger, lower‐turnover firms bode well for the persistence of any impacts we find. Third, we can think about them as informative of the potential for general equilibrium effects, particularly displacement, to lead our partial equilibrium estimates to overstate the social benefits of WIA training. If WIA training only changes who gets the “good jobs” at the larger, lower‐turnover, higher‐fixed‐effect firms, then it may have some equity benefits, but it will not increase efficiency.15
Table 9 present impacts on the firm‐related outcomes. We note three patterns. First, for adults the point estimates lean towards a positive effect on firm quality, though only the estimates on the high fixed effect indicator and the firm size indicator achieve statistical significance at conventional levels, and then only in State B. Second, other than the firm size outcome, the estimates all suggest moves to lower‐quality firms for the displaced workers, though with small point estimates that fail to achieve statistical significance. Third, for the (ambiguous) industry switching outcome, we find positive, statistically significant, and substantively large impacts for adults in State B and dislocated workers in both states, with a smaller positive estimate that lacks statistical significance for adults in State A.
Impacts on Firm Characteristics, Inverse Propensity Score Weighting, Model 6
IX. Results: Alternative Identification Strategies
A. CIA with Alternative Conditioning Sets
This section addresses one of our primary questions of interest: does having information on the firms at which WIA participants worked prior to participation move the estimates relative to the conditioning variables available only from the union of the WIASRD and UI administrative data sets? It does so by comparing estimates obtained with propensity scores containing the sets of conditioning variables described in Section V.A.
Tables 10 and 11 presents the impact estimates associated with the different specifications. To save space, we focus solely on the aggregated impacts for Q + 1 to Q + 12 and Q + 9 to Q + 12. The estimates differ substantially between the rather sparse specification of Model 1 and the richer conditioning sets in the other models. Models 1 and 2 always yield less positive (or more negative) estimates than Models 3, 4, 5, and 6—often substantially so. Given our earlier arguments regarding the variables required for credibility of the CIA, we interpret these differences as bias and conclude that producing credible impact estimates for WIA training requires (at least) conditioning relatively flexibly on the information available from the UI administrative data.
Impacts on Earnings, Inverse Propensity Score Weighting, Alternative Conditioning Variables, State A
Impacts on Earnings, Inverse Propensity Score Weighting, Alternative Conditioning Variables, State B
The data also provide clear responses to the other substantive questions that motivated our investigation of alternative specifications. The value of investing in obtaining the firm variables from the LEHD to use as conditioning variables shows up in the differences in estimates, if any, between Model 4 and Model 3 and between Model 6 and Model 5. In every case, those differences turn out to be quite minor indeed. For example, looking at the sum of earnings over Q + 9 to Q + 12, the changes are quite small (less than $20 in two cases). The additional firm variables in the LEHD data clearly do not pass a researcher cost–benefit test in this context based on what they add in terms of the credibility of the CIA.
A comparison of Model 5 with Model 4 provides evidence on the importance of adding an additional year of quarterly pre‐program earnings (from Q–9 to Q–12) to the conditioning set we modeled on that of Heinrich et al. (2013). In our data, the additional year of quarterly earnings barely moves the estimates, with the exception of adults in State B for the Q + 1 to Q + 12 outcome, where the estimate moves by about $390. We conclude that the additional pre‐program earnings variables do not add much to the credibility of the CIA in this context.
Finally, a comparison of Model 3 with Model 2 signals the value of adding indicators for the local one‐stop at which the participant enrolled in WIA. As noted above, we interpret these primarily as indicators for local labor markets. Our states each contain multiple urban labor markets, as well as rural ones. As a result, we expected these variables to matter, though the ambivalent findings in the literature muted our prior. For example, while Heckman et al. (1998) and Friedlander and Robins (1995) find that local labor markets matter a lot, Dolton and Smith (2011) do not. In our data, even given the conditioning already present in Model 2, additional conditioning on local labor markets moves the impact estimates substantially. This finding reflects substantial geographic heterogeneity in training receipt within our states.
B. Conditional Bias Stability
Table 12 presents the estimates that assume conditional bias stability. We again report estimates for two after periods: Q + 1 to Q + 12 and Q + 9 to Q + 12. In both cases, we use the symmetric pre‐program period as the “before” period for the difference‐in‐differences estimator; that is, we use Q–1 to Q–12 as the “before” period in the first case and Q–9 to Q–12 in the second case.16
Differences‐in‐Differences Impacts on Earnings, Inverse Propensity Score Weighting, Model 6
Somewhat to our surprise, the difference‐in‐differences estimates in Table 12 differ only modestly from the cross‐sectional estimates discussed above and shown in Tables 6 and 7. The impacts are just a bit more positive among the dislocated in both states. This similarity suggests that our conditioning variables do a very fine job of capturing time‐invariant differences in outcome levels between the trainees and the nontrainees in our sample of WIA enrollees.17
X. WIA Costs and Benefits
This section performs relatively crude cost–benefit calculations for WIA training versus no training using our preferred estimates from Tables 6 and 7; using the estimates from Table 12 does not change the qualitative story. The relatively low average social costs of core and intensive services imply that our qualitative conclusions roughly generalize to WIA versus no WIA for the trainees. Before presenting the results of our calculations, we briefly discuss several issues that arise when performing them.18
First, WIA, like JTPA before it, stands out for its lack of serious data on program costs. As such, we use two quite different values for the costs, neither of them particularly satisfactory as an estimate of the difference in average social direct costs between WIA registrants who receive training and those who do not. The lower value of $3,000 draws on Heinberg et al. (2005) and Heinrich et al. (2011). Heinberg et al. (2005) present estimates of the unit costs (to WIA, and in earlier dollars) of training in three states, two of which likely face lower costs than those in our study. Heinrich et al. (2011) cites average direct cost (to WIA) values for the 12 states in their study. Their numbers correspond to all WIA participants (that is, to WIA services versus no WIA services), rather than to WIA trainees versus WIA nontrainees, which implies a modest upward bias. On the other hand, both studies omit the public subsidy implicit in the prices that WIA programs pay community colleges for training courses, which implies a large downward bias. Taken together, we view $3,000 as a lower bound on the true difference in social direct costs.
The higher value of $9,000 draws on Table 4 in Hollenbeck (2012). It reflects data from Washington state on the cost of providing career and technical education inclusive of the large state subsidies to the community colleges providing the training. We have trimmed down Hollenbeck’s numbers a bit to account for the fact that we seek the difference in costs between training and other services, rather than the difference between training and no services, and because receiving training from WIA may reduce the costs to the public of training subsidized via other programs (for example, Pell grants).19 Our calculations assume that all direct costs occur in the first quarter following enrollment.
Second, our data limit us to estimating impacts for 12 quarters following WIA enrollment, but the impacts of WIA training may last far longer than that. The literature features studies with both impacts that persist in the long term and with impacts that fade out, but too few of both to sort out confidently what factors predict persistence; Smith (2011) and Greenberg, Michalopoulos, and Robins (2004) provide further discussion. To account for the uncertainty about the persistence of the impacts, our calculations embody three scenarios: (i) zero impacts after the data run out, (ii) the impacts in Q + 12 persist for an additional eight quarters then fall to zero, and (iii) the impacts in Q + 12 continue indefinitely.
Third, we need to say something about the social discount rate and the marginal social cost of public funds (MSCPF). We follow Heckman, LaLonde, and Smith (1999) and use 0.05 and 0.10 as annual discount rates; these rates bracket the 0.07 rate called for in U.S. Office of Management and Budget (1992). The MSCPF includes the direct costs of operating the tax system (for example, the IRS) and the lost output due to the use of distortionary rather than lump sum taxes. Estimates of the MSCPF vary widely across studies; see Auerbach and Hines (2002) and Dahlby (2008) for overviews. To account for this variability, we consider values of 1.00, 1.25, and 1.50, where, for example, 1.25 means that each dollar of government funds costs society $1.25 in output.
Finally, our cost–benefit calculations omit both general equilibrium effects and effects on outcomes other than earnings. General equilibrium effects can result from displacement, wherein the trained worker takes a job that otherwise would have gone to someone else (who is likely not in our comparison group) or from changes in relative skill prices. We expect our failure to consider equilibrium effects to lead us to overstate the social benefits of WIA training. See, for example, Lise, Seitz, and Smith (2004) or Cahuc and Le Barbanchon (2010) for more on these issues. Effects on other outcomes, such as crime or the well‐being of children could in principal go either way. For example, Job Corps’ negative effects on crime represent a large portion of its benefits, as shown in Schochet, Burghardt, and McConnell (2008), while Morris and Michalopoulos (2003) find that Canada’s Self‐Sufficiency Project had mixed effects on child outcomes that depend on child age. On net, we think our failure (a direct result of the limitations imposed by our data) to take other outcomes into consideration likely leads us to understate the social benefits of WIA training. Though it would make life easier for us and for the reader, the literature provides no reason for thinking the biases from these two omissions cancel each other out.
With all the foregoing in mind, turn now to Tables 13 and 14, which present the results of our calculations for State A and State B, respectively. In each table, the first column indicates the assumed duration of the program impacts, the second the assumed MSCPF, and the third the assumed discount rate. The next two present discounted net present values per participant from the perspective of society for adults and dislocated workers assuming $3,000 in direct costs, while the final two columns repeat the exercise under the assumption of $9,000 in direct costs.
Cost–Benefit Analysis, State A
Cost–Benefit Analysis, State B
For the adult stream, the assumptions make a difference. In both states, the adult stream always passes a cost–benefit test when the impacts last forever and never passes when the benefits last only three years. When the benefits last five years, it passes only under the most “optimistic” assumptions about the discount rate and MSCPF. On the other hand (and not at all surprisingly given the impact estimates) the dislocated worker stream essentially never passes a cost–benefit test, except for State B under the optimistic assumption that program impacts never fade out and even then only for certain low values of the discount rate and MSCPF.
XI. Conclusions
Our examination of the impacts of receiving WIA training rather than solely core and/or intensive services in two anonymous states has yielded a wealth of important findings both substantive and methodological. We start by reviewing our substantive findings.
We find reasonable differences in probabilities of training as a function of age and education. Substantial unconditional differences by race largely, but not entirely, disappear with conditioning. Our preferred estimates indicate positive earnings and employment effects for the adult funding stream and mostly negative estimates for the dislocated worker stream. This difference, for which we currently lack a good explanation, parallels findings in the broader literature and represents an important (and highly policy‐relevant) topic for future research.20 Surprisingly, we do not find statistically different impacts for men and women, though our point estimates generally show larger impacts for women, in line with the literature. Our estimates imply that training provided to WIA registrants in the dislocated worker programs in our states does not pass a social cost–benefit test, while training provided to adult registrants does so only when the impacts extend beyond our data and, even then, only under certain assumptions regarding discount rates and the marginal social cost of public funds.
Our data cover WIA participants who entered the program between 1999 and 2005, inclusive, in two states. To what extent would we expect our findings to generalize to later years under WIA and WIOA and/or to other states and countries? Temporally, the institutional framework of WIA did not change much between 2005 and the advent of WIOA in 2015. It also did not change much when WIOA replaced WIA, though the former now requires local boards to “develop, convene, or implement industry or sector partnerships.”
External validity over time also hinges on changes in the participant population and on changes in program impacts with the business cycle. Published WIASRD data show some important changes in the participant population since 2005. For one thing, the sizes of the adult and dislocated enrollee populations grew quite dramatically during the Great Recession. In program years 2001–2005, the adult and dislocated worker populations nationwide average about 230,000 and 185,000, respectively, while in PY 2009 the number of adults served alone rose to more than 1.1 million. During the Great Recession, the WIA population contained relatively fewer minority, less‐educated, and/or low‐earning workers. Given our finding of no substantively important subgroup differences by sex, race, or education, changes in the mix of participants along these variables should not lead to concerns about generalizability. The literature—see Lechner and Wunsch (2009) and Heinrich and Mueser (2014) and the references therein—suggests that partial equilibrium impacts of active labor market programs increase modestly during downturns for some groups. Finally, expanding the scale of the program dramatically, as was done with WIA via the “stimulus” at the start of the Great Recession, may change the nature of any general equilibrium effects, on which point one could tell stories in either direction, and may result (in the short run) in reduced attention to matching registrants to training due to inexperienced caseworkers and a desire to serve many registrants quickly, leading to lower impacts.
Spatially, while Barnow and King (2005) and D’Amico et al. (2009) show that WIA implementation varied meaningfully among states, the broad institutional commonalities as well as the similarity of our findings to those in the Heinrich et al. (2013) paper for many other states lead us to think that our results generalize to other states in the aggregate and, with more uncertainty, to other individual states, particularly in periods outside the height of the Great Recession. We would not generalize our substantive findings to other countries, due to large differences in data and active labor market program institutions.
On the methodological side, using the linked LEHD data, we find no consistent impacts on the characteristics of firms where workers get jobs other than some positive effects for the adult stream in State B. We do find modest positive impacts for most groups on the probability of switching industries relative to the last pre‐WIA employer. We expect that sectoral‐based training programs would have larger impacts than we found for WIA training, as they explicitly aim to lead their participants to good jobs in a new industry. In our view, the measurement issues we noted above and in Online Appendix 1, combined with the potential importance of firms as mediators, imply a high priority for the development of improved measures of firm and establishment characteristics based on administrative data.
Our second—and, to us, rather surprising—methodological conclusion is that adding variables related to the last employer prior to WIA participation to the conditioning set does not budge our “selection on observed variables” estimates. Nor does adding an additional four quarters of lagged earnings move the estimates. In contrast, conditioning on indicators for local labor markets does move the estimates. In our view, these findings complement those from the analyses by Biewen et al. (2014); Caliendo, Mahlstedt, and Mitnik (2017); and Lechner and Wunsch (2013) in the rather different German institutional and data context.
Comparing our cross‐sectional estimates to the difference‐in‐differences estimates also shows little qualitative change, suggesting either that selection into training occurs mainly on something other than relatively time‐invariant differences in labor market performance and/or that our conditioning set does a good job of capturing the time‐invariant differences. While further research linking the UI administrative data to other data sets that would provide alternative conditioning variables, such as psychometric tests, information on other household members, and more detailed educational histories remains of great value, we think that, taken together, our methodological findings suggest that current practice, as embodied particularly in this paper and in Heinrich et al. (2013), likely removes much of the bias for this estimand.
Acknowledgments
The authors thank the Charles Stewart Mott Foundation and the National Science Foundation for generous financial support of this project and Will Carrington, Bernd Fitzenberger, Carolyn Heinrich, Kevin Hollenbeck, Adriana Kugler, Lucy Msall, Peter Mueser, and Ken Troske for helpful discussions. They thank seminar participants at Alabama, Clemson, Cornell, Duke, IMPAQ, McGill, McMaster, Minnesota (Carlson School), St. Louis Fed, Simon Fraser, Syracuse, Tel Aviv, Texas A&M, Virginia, Wisconsin, and the Institute for Fiscal Studies and conference participants at CESifo, IZA/SoLE, the Institute for Research on Poverty Summer Research Workshop, the New Developments in Human Capital in Jerusalem conference, the 2014 UM‐MSU‐Western Labo(u)r Day conference, the 2014 Danish CAFÉ conference, the 2015 Canadian Labour Studies Research Network meetings, the 2015 APPAM fall research conference, the 2018 Rigorous Impact Evaluation Conference in Torino, and the 11th IZA Conference on Labor Market Policy Evaluation for their helpful comments. They thank Elan Segarra for his superlative research assistance inside the Wisconsin RDC, Jason Fletcher for gifting Elan’s time, the staff of the Wisconsin RDC for their patient assistance, and Lois Miller for careful readings and help with tables, figures, and formatting. The views expressed here are solely those of the authors and not necessarily those of the U.S. Census Bureau. Each author declares that they have no relevant or material financial interests that relate to the research described in this paper. The data used in this paper cannot be published but are available through the Federal Statistical Research Data Centers (FSRDCs). For more information, contact the author (econjeff{at}ssc.wisc.edu) or your local FSRDC administrator.
Footnotes
↵1. Experiments have issues too; see for example, Heckman and Smith (1995); Heckman, LaLonde, and Smith (1999); and Rothstein and von Wachter (2017).
↵2. As noted in Baum, Holzer, and Luetmer (2020), in recent years, Pell grants have increasingly funded training in certificate programs at community colleges. Interest has also shifted towards “sector‐based” training programs, which have generated larger positive impacts on earnings than WIA training in some (but not all) experimental studies (for example, Katz et al. 2020), though the evaluated programs remain small and unscaled. Roder and Elliott (2019) provide evidence of persistent positive impacts for an early sectoral program.
↵3. Literature reviews include Lalonde (1995); Friedlander, Greenberg, and Robins (1997); Heckman, LaLonde, and Smith (1999); King (2004); Bergemann and van den Berg (2008); Holzer (2009); Card et al. (2010, 2018); Heinrich and King (2010); Decker (2011); Barnow and Smith (2016); and McCall, Smith, and Wunsch (2016).
↵4. Barnow and Smith (2016) note that the U.S. Department of Labor provided surprisingly vague eligibility rules. Caseworkers assigned individuals eligible for both the adult stream and the dislocated worker stream based on their employment histories and on the funds remaining in the two streams.
↵5. Lurking in the background is the WIA performance management system. This system provides rewards and punishments to states and WIBs primarily based on the measured outcome levels of registrants and so may encourage sites to “cream‐skim” at the registration stage. It may also lead them to focus expensive training services on those they deem unlikely to realize good labor market outcomes without them. For more on the WIA performance system, see, for example, Barnow and Smith (2004), Barnow (2011), and Heckman et al. (2011).
↵6. For workers with multiple employers in a calendar quarter, we code employer characteristics based on the employer from which the worker receives the most earnings in that quarter, which we call the “dominant” employer.
↵7. Mechanically, the treatment consists of having enrolled in, but not necessarily completed, training as recorded in the WIA administrative data.
↵8. This list includes control variable sets 0–6 and 8a in their Table 2; see also the discussion in their Section 5.3.
↵9. From the narrower perspective of whether or not it passes a researcher cost–benefit test to go to the time and trouble of linking to the LEHD for the additional conditioning variables it provides, what really matters is whether the estimates change rather than their bias.
↵10. Imbens and Wooldridge (2009) and Chabé‐Ferret (2015) provide further discussion of the choice between estimators based on the CIA (with pre‐program outcomes) and the BSA.
↵11. We also produced subgroup estimates broken down by race/ethnicity and by years of schooling. We did not observe substantively important impact differences for these subgroups.
↵12. The WIA Gold Standard Evaluation documented in Fortson et al. (2017) does not provide direct estimates of the effect of WIA with training versus WIA without training (nor direct estimates of the effect of WIA versus no WIA). Instead, it essentially estimates the effect of the marginal additional training (whether paid for by WIA or not) received by individuals randomly assigned access to either intensive services or both training and intensive services, rather than just core services. This difference in causal estimands precludes meaningful comparisons.
↵13. To calm the suspicious, we note that Ohio is not State B.
↵14. Many of the measures we omit surely merit consideration in future studies of programs aimed at disadvantaged populations.
↵15. The (vast) literature that builds on Abowd, Kramarz, and Margolis (1999) interprets the firm fixed effects as capturing a one‐dimensional measure of overall firm quality. In a world of multidimensional firms, the interpretation of the fixed effects becomes less clear as higher wages may imply less of other forms of remuneration, for example, lower fringes or less pleasant work, as in a compensating differences world.
↵16. An alternative strategy would avoid the Ashenfelter dip by using Q–9 to Q–12 as the before period in both cases; Heckman and Smith (1999) document the grave problems that result from including the period of the dip in their context. We do not adopt this alternative strategy here due to the lack of apparent selection into training based on transitory shocks shown in Figure 1.
↵17. Online Appendix 5 presents difference‐in‐differences IPW estimates obtained from a version of Model 6 that omits all of the pre‐program labor market outcomes. Doing so tends to yield less positive or more negative estimates. This result points to the importance of these variables in capturing transitory changes in labor market outcomes that affect participation.
↵18. See, for example, Bloom et al. (1997); Heckman, LaLonde, and Smith (1999, their Section 10.1); Schochet, Burghardt, and McConnell (2006); and Hollenbeck (2012) for more detailed discussions.
↵19. The WIA Gold Standard Evaluation did not collect administrative data on training costs, though Mastri and McCutcheon (2015) provide detailed information on the costs of specific core and intensive services. Table VIII.1 of Fortson et al. (2017) estimates average total direct social costs of training in the “full WIA” arm at $3,223 = $1,521 from taxpayers + $1,702 from the participants themselves based on survey reports of training costs from enrollees. Their Figure V.2 indicates that half of the enrollees in the full WIA arm of the experiment received any training; rescaling yields an estimated social direct of $6,446 for training, which lies toward the upper end of our range for the difference in direct costs between WIA with training and WIA without training in our two states.
↵20. We see four (not mutually exclusive) candidate explanations. First, caseworkers may sort enrollees into funding streams in a manner that correlates with the treatment effect of training. Second, enrollees may sort differently into training within the two funding streams in a manner correlated with the treatment effect of training. Third, the content of training may differ between trainees in the two funding streams. Fourth, the treatment effect of training may vary with trainee characteristics that themselves correlate with funding stream, such as age.
- Received August 1, 2016.
- Accepted January 1, 2022.
This open access article is distributed under the terms of the CC-BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0) and is freely available online at: https://jhr.uwpress.org.
References
In this issue
Jump to section
- Article
- Abstract
- I. Introduction
- II. The WIA Program
- III. Data
- IV. Treatment and the Parameter of Interest
- V. Identification and Estimation
- VI. Results: Determinants of Training Receipt
- VII. Results: Impacts on Earnings and Employment
- VIII. Results: Firm Characteristics
- IX. Results: Alternative Identification Strategies
- X. WIA Costs and Benefits
- XI. Conclusions
- Acknowledgments
- Footnotes
- References
- Figures & Data
- Supplemental
- Info & Metrics
- References
Related Articles
Cited By...
- No citing articles found.