Abstract
Despite progress in recent decades, a substantial fraction of children in developing countries attain little schooling, and many adults lack skills that are valued in the labor market. We evaluate the medium-term effects of a program that provided scholarships for three years to poor children upon graduation from elementary school in Cambodia, a low-income country. To do this we use a sharp regression discontinuity design. We show that scholarships have substantial effects on school attainment. By the time children would have been in grade 11 had they remained in school, two years after they stopped being eligible for scholarships, those who were offered scholarships have attained 0.6 more grades of completed schooling. Nevertheless, we find no evidence that scholarships had significant effects on test scores, employment, earnings, or the probability of getting married or having a child in adolescence.
I. Introduction
Despite progress in recent decades, a substantial fraction of children in developing countries attain little schooling, and many adults lack skills that are valued in the labor market. For this reason, policymakers and academics continue to search for programs and policies that can raise educational attainment and learning in poor countries.
A number of interventions have recently been shown to increase school enrollment and attendance in some settings. These include merit-based scholarships (Kremer, Miguel, and Thornton 2009), deworming for school-aged children (Miguel and Kremer 2004), school construction (Duflo 2001), the provision of additional teachers (Banerjee et al. 2007), vouchers for private schooling (Angrist et al. 2002; Angrist, Bettinger, and Kremer 2006), and conditional cash transfers—transfers that are made to poor households, conditional on them keeping their children enrolled in school and attending regularly. (See the review by Fiszbein and Schady 2009.) Nevertheless, the evidence base on interventions that can effectively increase school attainment in poor countries is still thin. Of particular interest are evaluations that assess whether any effects that are observed in the short run are still apparent over a somewhat longer time horizon.
In this paper, we evaluate the medium-term effects of a program that provided scholarships to poor children upon graduation from elementary school in Cambodia, a low-income country. To do this we use a sharp regression discontinuity design that exploits the fact that eligibility for scholarships was determined on the basis of a composite measure that predicts the likelihood that a child would drop out of school. We present a variety of checks to support our identification strategy.
Cambodia is one of the poorest countries in Asia. Most children enroll in primary school—the net primary school enrollment rate in Cambodia is 96 percent—but only a minority continues on to secondary school. The net secondary school enrollment rate is 35 percent, a figure that is more closely comparable to countries in sub-Saharan Africa like Mali (29 percent) and Malawi (28 percent) than to neighboring countries like Thailand (73 percent) and Vietnam (69 percent). To address these low enrollment rates, Cambodia has put in place a number of scholarship programs for children in lower secondary school (grades 7–9).
In earlier work, we have analyzed the short-term effects of scholarships on school enrollment. One scholarship program, known as the Japanese Fund for Poverty Reduction (JFPR) program, increased school enrollment by 30 percentage points in grade 7 (Filmer and Schady 2008). That scholarship program was eventually discontinued and replaced by the program we analyze in this paper, known as the CESSP Scholarship Program (CSP).1 The CSP increased school enrollment rates in grade 8 by approximately 25 percentage points (Filmer and Schady 2011). These effects are substantially larger than those that have been reported for conditional cash transfer programs in Latin America, including the much-analyzed Mexican PROGRESA program—presumably, in part, because of the much lower baseline school enrollment rates in Cambodia than in Mexico. (See Schultz 2004; Filmer and Schady 2011; Fiszbein and Schady 2009, pp. 128–29.)2
In this paper, we consider the effect of the CSP five years after children started receiving scholarships. We show that scholarships have substantial effects on school attainment. By the time children would have been in grade 11 had they continued in school, two years after they stopped being eligible for scholarships, those who were offered scholarships have attained 0.6 more grades of completed schooling. Nevertheless, we find no evidence that scholarships had significant effects on test scores, employment, earnings, or the probability of getting married or having a child in adolescence. Because we focus on medium-term effects, our paper is closely related to a recent paper that analyzes the medium-term effects of PROGRESA (Behrman, Parker, and Todd 2011), and we discuss this in more detail in our conclusions.
The rest of the paper proceeds as follows. In Section II, we describe the CSP scholarship program and the data. Section III discusses our identification strategy. Section IV presents our main results. We conclude in Section VI.
II. The Program and Data3
A. Scholarships in Cambodia
Cambodia has a tradition of demand-side incentives that seek to raise school enrollment and attendance at the secondary school level. There are several relatively small-scale programs that distribute bicycles, uniforms, and school materials to children in order to lower the travel and other direct costs associated with schooling. In addition, Cambodia has had a number of scholarship programs financed by nongovernmental organizations (NGOs), international donors, and the government.
The CSP scholarship program, which we analyze in this paper, works as follows. The government first selected 100 lower secondary schools throughout the country (from a total of approximately 800) to participate in the program. These CSP-eligible schools were chosen because they served poor areas, as indicated by a poverty map, and because there appeared to be high levels of school nonenrollment and dropout, as indicated by administrative data; schools covered by other scholarship programs were excluded. Next, each primary “feeder” school was mapped to a CSP-eligible secondary school.4
Finally, in every feeder school, all students in sixth grade, the last year of primary school, filled out an “application form” for the CSP scholarship program—regardless of whether children or their parents had expressed an interest in attending secondary school.
Application forms consisted of 26 questions that were easy for sixth graders to answer and for other students and teachers to verify. In practice, the form elicited information on household size and composition; parental education; the characteristics of the home; availability of a toilet, running water, and electricity; and ownership of a number of household durables. Forms were filled out in school on a single day. Students and parents were not told beforehand of the questions on the forms nor were they ever told the scoring formula—both decisions designed to minimize the possibility of strategic responses (for example, by a student seeking to maximize her chances of receiving a scholarship). Head teachers in each school collected the completed forms and sent them to Phnom Penh, the capital. The median age of children when they completed the application survey was 14; the tenth percentile of the distribution was 12; and the 90th percentile 16.
We worked closely with officials from the Ministry of Education in Cambodia to design a process that assigned CSP scholarships on the basis of the information on the application forms. We hired an independent firm to digitize the information on the application forms, and provided the weights used to aggregate responses on the form into a composite “dropout-risk score.”5
Separately for each CSP school, applicants were then ranked by their dropout-risk score. As requested by the Ministry, in “large” CSP schools, with total enrollment above 200, 50 students with the highest value of the score were then offered a scholarship for grades 7, 8, and 9; in “small” CSP schools, with total enrollment below 200 students, 30 students with the highest value of the score were offered scholarships. In total, just over 3,800 scholarships were offered.6
The list of students offered scholarships was posted in each CSP school, as well as in the corresponding feeder schools. The program allowed for a complaints mechanism if an applicant felt they had been wrongly denied a scholarship but there were virtually no revisions made as a result of this process. We closely monitored every step in the process whereby application forms were filled out, dropout scores calculated, and schools given the list of scholarship recipients.7
Children who were offered a scholarship were automatically eligible to receive it for grades 7, 8, and 9 unless they repeated a grade or dropped out of school, in which case the scholarship was withdrawn. Two-thirds of the scholarships were given to girls; this is because girls were more likely than boys to drop out of school in Cambodia, which was factored into the dropout-risk score. The value of the scholarship was equivalent to US $45.8 This is roughly 2 percent of the total consumption of the average recipient household in Cambodia, and is almost exactly equal to the direct cost of schooling, including fees, uniforms, supplies, and transportation but excluding the opportunity cost of going to school (Ferreira, Filmer, and Schady 2009).
B. Data
We make use of two sources of data in this paper. The first data set includes the composite dropout-risk score, as well as the individual characteristics that make up the score for all 26,537 scholarship applicants. The second data set is based on a household survey we fielded, and which collected information on 3,020 applicants and their families.
The sample for the survey was constructed as follows. First, we purposefully selected five provinces in different parts of the country where the program was operating, and where there were a reasonably large number of program schools. In total, there were 57 program schools in these five provinces: Battambang (9), Kampong Thom (14), Kratie (4), Prey Veng (25), and Takeo (5). Second, within each school, the survey sample included all children who had been offered scholarships (30 or 50 children, depending on whether a school had been designated as “small” or “large”) and 20 children who had been turned down for scholarships, beginning with the “first” child turned down for a scholarship (the child whose score was just below the cutoff for eligibility) and up to the 20th child.9
Data were collected between February and April of 2010, almost five years after children filled out the application forms.10 For applicants enrolled in school who had not repeated grades, the household survey therefore refers to school attendance in the second half of grade 11. The median age of children at the time of the household survey was 19.
The household survey collected information on a large number of child outcomes. Children were asked about the highest grade of schooling they had completed to date, whether they were enrolled in school in each academic year between 2005 (when they completed the applications) and 2010 (at the time of the survey), and, if so, in what grade. They were also asked whether they were currently working (separate questions for work for pay and work without pay), their earnings over the last pay period, and how many hours they worked in the last week.
We administered three tests during the household survey. The first is a math test, which included 20 multiple choice items. Areas covered included algebra, geometry, and several questions that required using mathematical tools to answer simulated real world situations—including reading a simple graph or interpreting a bar chart. The test included mathematical concepts that students would have been exposed to over the lower secondary school cycle.11 The second test was a vocabulary test based on picture recognition. This test asked respondents to identify the picture corresponding to a word which the enumerator read out loud. For each word the respondent was then asked to select from a choice of four pictures. While the initial words are relatively easy to identify (“shoulder,” “arrow,” “hut”) the test is structured such that items become increasingly difficult. (Later words in the test we administered included, for example, “speed,” “selecting,” “adjustable”.) There were in total 72 words that each applicant was asked to identify. The third test is a test of puzzles and shapes loosely based on the Ravens Progressive Matrices.12 These tests are not linked to the curriculum taught in Cambodia’s lower secondary schools. We normalize the scores on all three tests by subtracting the mean and dividing by the standard deviation of nonrecipients. The Cronbach alpha values for all of our tests are reasonably high—0.68 for the math test, 0.90 for the vocabulary test, and 0.65 for the test of puzzles and shapes.13
The household survey also collected data on adolescent mental health using an adaptation of the Center for Epidemiological Studies Depression scale (CESD), a widely used measure of depression (Radloff 1977).14 Subjective social status was assessed using the “MacArthur ladders.”15 Adolescents were shown a picture of a ladder with ten rungs, and were told that higher rungs correspond to higher socioeconomic status. They were asked to place themselves on the ladder in relation to everyone in their communities, and in relation to everyone in Cambodia. Finally, we asked respondents if they were married or had children.
III. Identification Strategy
The identification strategy we use in this paper is based on regression discontinuity (RD). These regressions take the following form:
(1)
where Yis is an outcome (for example, school enrollment, or the grades of completed schooling) for child i who applied to CSP school s; αs is a set of fixed effects for lower secondary schools; f(Cs) is a flexible formulation of the control function; I(Tis = 1) is an indicator variable that takes on the value of one if a child was offered a CSP scholarship; and εis is the regression error term. In this set-up, the coefficient β is a measure of the impact of the scholarship. To allow for different slopes, we interact the control function with the indicator variable for children who were eligible for scholarships. Standard errors are clustered at the level of the primary feeder school.
Unusually, in our application of RD there are 57 school-specific cutoffs, rather than a single cutoff, and the cutoff falls at different values of the dropout-risk score in different schools.16 (We return to this point in our discussion of heterogeneity below.) This raises the question of how best to pool the results from what are, in effect, 57 separate applications of RD. One way to address this challenge would be to normalize the dropout-risk score by subtracting the value of the score at the school-specific cutoff. However, this gives a score of exactly zero to at least one applicant in each school (more if there are tied scores), and mechanically results in a piling of mass at the cutoff. To avoid this, we normalize the cutoff in each school as the value of the score that is the midpoint between the scores of the last scholarship recipient and the first nonrecipient in that school.17 Also, to ensure that identification is based on comparisons of scholarship recipients and nonrecipients within schools, not across schools, we include school fixed effects.
The dropout-risk score perfectly predicts whether an applicant was offered a scholarship, with the exception of a very small number of changes that resulted from the public complaint mechanism. This is therefore a case of sharp (as opposed to fuzzy) RD. Also, because we focus on the impact of being offered a scholarship, rather than that of actually taking up a scholarship, these are Intent-to-Treat (ITT) estimates of program impact. (For convenience, in the paper we interchangeably refer to children with values of the dropout-risk score above the cutoff, all of whom were offered scholarships, as children who were eligible for scholarships, recipients, or treated children.)
As in other applications of RD, it is important to ensure that results are not driven by a particular parametrization of the control function. Following Lee and Lemieux (2010), we present results based on two ways of implementing RD. The first approach uses data on all children in the household survey. In this approach, the estimation choice revolves around the selection of the polynomial in f(Cs). We follow Lee and Lemieux (2010) and use both the Akaike information criterion (AIC) and bin regressions to select the optimal order of polynomial. As it turns out, both approaches produce very similar results, and in virtually every case, no matter which dependent variable, the optimal order of the polynomial is one (meaning that the control function is linear in the score, with slopes that can be different on each side of the cutoff). Nevertheless, to ensure that our results do not depend on the choice of polynomial, we also present results for polynomial order two (with quadratic terms interacted with the indicator variable for scholarship recipients).18
The second approach, based on local linear regressions (LLRs), restricts the sample to those who are relatively “close to” the cutoff. In this approach, the estimation choice revolves around the selection of the bandwidth around the cutoff. Here too we follow Lee and Lemieux (2010) and present results based on a rule-of-thumb (ROT) selection of bandwidth, as well as a cross-validation (or “leave-one-out”) procedure. (In the ROT approach we use a rectangular kernel and follow the estimation steps detailed in Imbens and Kalyanaraman 2012.) Based on these calculations, we estimate that the optimal bandwidth is between four and eight, depending on the outcome in question. To avoid presenting different bandwidths in different tables, we report results from LLRs with a bandwidth of six for all outcomes, but generally also report results with a bandwidth of three (half the optimal) and 12 (twice the optimal). As we show below, our results are not sensitive to the selection of polynomial or the selection of bandwidth in the LLRs.
There are two main threats to our identification strategy: attrition and manipulation of the dropout-risk score. Attrition is a potential source of concern because the firm that collected the household survey was given a list of households to visit on the basis of the roster of children who had completed the application form in 2005, but not all of these households could be located in 2010 (for example, if the information regarding the name or place of residence on the application form was inaccurate or if a household had moved to another province). In practice, almost five years after students filled out their scholarship application forms (as described above), 14.1 percent of children on the list given to the firm could not be located. We note that these values are comparable to those for other surveys in developing countries—for example, Behrman, Parker, and Todd (2011) report an attrition rate of 14 percent for their analysis of medium-term effects of PROGRESA transfers.
Figure 1 graphs the proportion attrited as a function of the normalized dropout-risk score.19 Attrition is higher among scholarship recipients, who are poorer, but there is no evidence of a discrete jump in attrition at the cutoff that determines eligibility for scholarships. We conclude that it is unlikely that attrition is a source of bias for our estimates of scholarship effects.20
Analysis of Attrition
Notes: Dashed lines are from the linear, interacted regression of attrition dummy on recipient status, recipient status interacted with normalized dropout-risk score, normalized-dropout risk score, including secondary school fixed effects. Circles proportional to number of observations.
An identifying assumption for RD is that, conditional on a flexible parametrization of the control function, there is no discrete jump at the cutoff in characteristics (including unobservables) that could be correlated with the regression error term. Although one cannot test this assumption directly, we provide two standard RD identification checks. (See Imbens and Lemieux 2008; Lee and Lemieux 2010.) The first checks for unusual heaping of mass at the cutoff. Figure 2 graphs the proportion of observations in 60 bins of the normalized dropout-risk score. The figure shows no unusual heaping of mass above or below the cutoff for scholarship eligibility.21
Density Estimates
A second standard check on RD involves regressions of predetermined characteristics on the indicator value for treatment (in our case, children who were offered scholarships) and the control function. In general, if the RD design is valid, we would expect the coefficient on the treatment indicator in these “placebo” regressions to be small and not significant.
Table 1 presents results of these regressions for a number of variables from the household survey that are arguably fixed—including the gender and age of applicants, education levels of all adults in the household, and various measures of household composition. In the polynomial order one regressions and the LLRs with a bandwidth of six, none of the coefficients are significant at the 5 percent level or higher. We conclude that the RD approach we take in this paper is valid, and is likely to provide causal estimates of the medium-term effects of scholarships.
Applicant Household Characteristics at Endline
IV. Results
A. Effects on Enrollment and Grades of Completed Schooling
Figure 3 graphs the mean grades attained by scholarship recipients and nonrecipients in 2010, almost five years after both groups completed application forms, and two years after recipients stopped being eligible for scholarships. The figure shows a clear jump in grade attainment at the cutoff, equivalent to about half of an additional grade of completed schooling. Table 2 reports the results from polynomial regressions and LLRs of grade attainment on the indicator variable for scholarship recipients and the control function. The results suggest a scholarship impact of about 0.6 grades of schooling, relative to a control group mean of 8.35 grades.
Scholarship Effects on Years of Completed Schooling
Scholarship Effects on Grades of Completed Schooling
Notes: Dashed lines are from polynomial order 1 regression of years of completed schooling on indicator variable for children with dropout-risk score above the cutoff, including secondary school fixed effects. Circles proportional to number of observations.
Table 3 and Figure 4 summarize the results from a number of robustness checks on the results in Table 2. In Figure 4 we present the coefficient and 95 percent confidence interval on the scholarship indicator for LLRs with bandwidths ranging from 1 to 30. The figure clearly shows that the results are insensitive to bandwidth choice. As expected, estimates become more precise with larger bandwidths.
Robustness Checks
Scholarship Effects on Grades of Completed Schooling, Varying Bandwidths
Notes: Coefficients and 95 percent confidence interval are from Local Linear Regression of years of completed schooling on indicator variable for children with dropout-risk score above the cutoff, including secondary school fixed effects, varying bandwidth.
Additional robustness checks based on the polynomial order 1 regressions are in Table 3. One possible concern with our estimates is reporting bias—specifically, that scholarship recipients may be more likely to overstate the grades of completed schooling they have attained than nonrecipients. In earlier work (Filmer and Schady 2011) on the short-term effect of scholarships we collected self-reported school enrollment data in a household survey conducted in late 2006, approximately 18 months after children filled out the application forms, as well as school attendance data from four unannounced school visits we conducted in 2006 and 2007. We showed that the estimated effect of scholarships on enrollment was very similar using both sources of data, suggesting that scholarship recipients were no more likely to overstate their school enrollment than nonrecipients, at least in the short run. As a robustness test on the results in Table 2 we run regressions using the 2010 household survey data but limit the sample to children who were also interviewed in 2006 (and for whom, as discussed above, we had earlier found no evidence of reporting bias). These results are in Column 1 of Table 3. Not surprisingly, given that 99 percent of the children we interviewed in 2010 were also interviewed in 2006, excluding children who were not interviewed in 2006 has no bearing on our results.
The following column in Table 3 reports the results from a specification that includes the controls in Table 1. Including these controls does not substantially alter the results. As a final specification check, we report the results from a “donut” estimator proposed by Barreca et al. (2011). Barreca et al. point out that the motivating assumption for RD is a comparison of means approaching the cutoff (rather than a comparison of means at the cutoff itself). Because any manipulation of eligibility is most likely to occur around the cutoff, Barreca et al. suggest it is useful to check whether RD results are robust to symmetrically discarding observations in the immediate vicinity of the cutoff. In Column 3 of Table 3 we report the results from regressions that exclude children whose score places them within one full point of the cutoff; this is equivalent to dropping observations in the two bins that are closest to the cutoff in Figure 2. In Column 4 we exclude those with scores within two points of the cutoff; this is equivalent to dropping observations in the four bins that are closest to the cutoff in Figure 2. These results suggest a scholarship impact of about 0.5 grades of schooling. In sum, the results in Figure 4 and Table 3 show that our main results on grade attainment are robust to a variety of specification checks.
In Table 4, we use retrospective questions in the household survey to estimate program effects on school enrollment in grades 7–11. For this purpose, we define the dependent variable in two alternative ways. In the first set of specifications, the dependent variable takes on the value of one if a child is enrolled in a given school year, no matter what grade she is enrolled in, and zero otherwise; in the second set of specifications, the dependent variable takes on the value of one if a child is enrolled in school in the appropriate grade in a given school year (for example, enrolled in grade 7 in 2005–2006, enrolled in grade 8 in 2006–2007, and so on), and zero otherwise. These specifications therefore also account for any effect the scholarship may have had on grade repetition (in addition to the effect on school enrollment).
Scholarship Effects on School Enrollment, Various Years
The first two columns report the mean for the control group. Two things are notable about these results. First, school enrollment declines monotonically and sharply with age. For example, enrollment in the age-appropriate grade declines from 91 percent in grade 7 to 18 percent in grade 11. Second, the differences between the means in the two columns are modest, suggesting that there is relatively little grade repetition in this sample.
The remaining columns in Table 4 provide our estimates of scholarship effects. We limit the discussion to the coefficients from the polynomial order one and the LLRs with a bandwidth of six, although we note that other results in the table are generally similar. In grade 7, where counterfactual enrollment rates are high, the impact of the scholarship is modest—nine to ten percentage points in our preferred specifications. In grades 8 and 9, program effects are substantial, 18 to 20 percentage points.22 Program effects in grade 10 are smaller, between seven and eight percentage points, but these represent increases of almost one-third, relative to the control group mean of 23 percent for age-appropriate school enrollment. The results in the last row show even smaller effects on school enrollment in grade 11, and these are not significant at conventional levels in our preferred specifications. In sum, and consistent with the results on grade attainment discussed earlier, the results in Table 4 show substantial program effects on school enrollment.
B. Effects on Learning Outcomes
We next turn to program effects on the tests of vocabulary, math, and puzzles and shapes. Note that these tests were applied to children in their homes, regardless of whether they were enrolled in school or not. Also, nonresponse rates on all three tests are negligible—only three children missed the math test, one child missed the test of puzzles and words, and no children missed the vocabulary test. This considerably simplifies estimation, as there is no need to correct for nonresponse or for the effect of scholarships on school enrollment, a complication with school-based tests. (See, for example, the analysis in Angrist et al. 2002; Angrist, Bettinger, and Kremer 2006 of a school voucher program in Colombia.) The reduced-form estimates we report in Table 5 below are therefore the impact of the scholarships on test scores, inclusive of any program effects on grade attainment, any income effects, any program effects on other determinants of learning, such as motivation or effort expended in school, and any Hawthorne effects. (For example, if children who were offered scholarships chose to put more effort into test-taking than those who were turned down for scholarships.)
Scholarship Effects on Test Scores
To motivate our results, we first note that children with more grades of completed schooling do better on all three tests. Among children who were turned down for scholarships, an additional grade of schooling is associated with a 0.21 standard deviation better performance on the test of math (with a standard error of 0.02), a 0.23 standard deviation better performance on the vocabulary test (with a standard error of 0.02), and a 0.13 standard deviation higher score on the test of puzzles and shapes (with a standard error of 0.02).23 As is well understood, these associations need not be causal, as they could in part reflect the selection of higher-ability children into higher grades. Nevertheless, the fact that there is an association between grades attained and test scores makes it more likely that scholarships would also increase test scores.
Figure 5 graphs the test score averaged across the three tests at different values of the dropout-risk score.24 The figure shows that, as expected, children with lower values of the score generally do better on the tests. However, there is no jump in test scores at the cutoff for scholarship eligibility. In Table 5 we report the coefficients and standard errors for scholarship recipients for regressions of different tests, as well as the average effect on all tests. There is no evidence that the scholarship program improved scores on any test. For the average across all three tests, and based on the polynomial specification with order one, we can rule out effect sizes of 0.13 standard deviations or larger with 95 percent confidence.25
Scholarship Effects on Average Test Scores
Notes: Dashed lines are from polynomial order 1 regression of test score on indicator variable for children with dropout-risk score above the cutoff, including secondary school fixed effects. Circles proportional to number of observations.
In sum, our results show that, in spite of a substantial effect on grades of completed schooling, scholarships did not improve test scores. One possible explanation for these results is that schooling in Cambodia generally has little effect on learning outcomes. An alternative, more subtle, possibility is that there are differences in the latent distribution of test scores between children who were brought into school by the scholarships and other children. This explanation is consistent with selection into schooling on the basis of expected gains (Willis and Rosen 1979; Heckman, Urzúa, and Vytlacil 2006; Carneiro and Lee 2011). In the absence of scholarships, children with higher ability could be more likely to stay in school than lower-ability children because for them the expected returns are high; higher-ability children also learn more while in school. Conversely, lower-ability children, for whom the expected returns to school are low, could be more likely to drop out of school; these children would not have learned much had they stayed in school. The scholarship program lowers the costs of education for recipients, and some of the lower-ability children who under normal circumstances would have dropped out now stay in school. However, the test performance of these children is not improved by their additional schooling. Although we believe this explanation is plausible, we do not have the data that would allow us to test it convincingly. We note, however, that in their evaluation of PROGRESA in Mexico, Behrman, Parker, and Todd (2009) also find program effects on grade attainment but not on test scores.
C. Heterogeneity
We next turn to a discussion of heterogeneity in program effects, focusing on gender, school quality, and the value of the dropout-risk score at the school-specific cutoff. These results are presented in Table 6 and Figures 6 and 7. To keep the number of results manageable, we focus on the values from polynomial regressions of order one, and LLRs with a bandwidth of six. In each case, the table reports the coefficient on the main scholarship effect, and the coefficient on an interaction between scholarship recipients and a given characteristic—for example, in the lefthand panel, the coefficient in the second row corresponds to the interaction between scholarship recipients and an indicator variable for girls. The top panel of the table reports effects on grades of completed schooling, and the bottom panel reports effects on average test scores.
Heterogeneity
Distributions of Dropout-Risk Score and School-Level Cutoff
Notes: Dark bars correspond to distribution of dropout-risk score across children, hollow bars correspond to distribution of value of cutoff of dropout-risk score across schools.
Heterogeneity by School-Level Cutoff
Notes: School level regressions are based on polynomial order 1 specification. Solid (dark) circles correspond to schools where the effect is significant; hollow (light) circles to schools where the coefficient is not significant at conventional levels.
The lefthand panel of Table 6 shows there is no evidence that the scholarship program had differential effects by gender. The main effects are very similar to those in Tables 2 and 5, and the interaction between the indicator variable for girls and scholarship recipients is not significant.
To construct a measure of school quality, we focus on differences across schools in performance on a math test, after netting out differences in socioeconomic status across schools.26 We then divide schools into those with above-average and below-average school quality, and interact the indicator variable for scholarship recipients with the indicator variable for children who attended schools of above-average quality. The results from these regressions, reported in the middle panel of Table 6, show no evidence that scholarships had larger effects on grade attainment or test scores for children who attended higher-quality schools, at least when school quality is measured in this way.
Finally, we test for heterogeneity of scholarship effects for different values of the school-specific cutoff. We begin by graphing the density of the dropout-risk score of all applicants to the 57 CSP schools in our sample (dark bars) and the value of the dropout risk score at the school-specific cutoff (hollow bars) in Figure 6. The figure shows there is considerable variation in the value of the school-specific cutoff, which ranges from 61 (corresponding to the fifth percentile of the distribution of the dropout-risk score across all applicants) to 80 (corresponding to the 77th percentile)
In the righthand column of Table 6 we report the main effect of scholarships, and the coefficient on an interaction between scholarship recipients and an indicator variable for children who attended schools where the value of the dropout-risk score at the cutoff was above average (that is, schools where children were more likely to drop out). As with the other results in Table 6, we find no evidence of heterogeneity.
To extend these results, we next run polynomial order one regressions of a given outcome (grades of completed schooling, average test scores) on the indicator variable for scholarship recipients, separately by school. This gives us 57 coefficients for each of the two outcomes we consider. In Figure 7 we then plot the coefficients of these regressions as a function of the value of the school-specific cutoff. The figure also includes a line corresponding to a nonparametric (Fan) regression.27 If scholarships had larger effects on children who were more (less) likely to drop out of school in the absence of the CSP, we would expect this line to slope upward (downward) from left to right. Figure 7 shows that this does not appear to be the case.28 In sum, we conclude from Table 6 and Figure 7 that there is no evidence of heterogeneity in the effects of scholarships on grades of completed schooling or test scores by gender, school quality, or the dropout-risk score.
D. Other Outcomes
As can be seen in Table 4, less than one-quarter of scholarship applicants were enrolled in school in 2010. In the household survey most children report they work—37 percent report they work for pay, and 40 percent report they work without pay.29 There are clear differences in employment by gender, especially in work for pay. Among girls who work for pay, 75 percent of girls work in the garment industry, and 15 percent in services and sales; among boys who work for pay, 31 percent work in services and sales, 26 percent in agriculture, 21 percent in construction and mining, and 16 percent in the garment industry.
Since scholarship recipients completed more schooling, we might expect them to have better labor market outcomes. In fact, Table 7 shows that the scholarship program did not affect the likelihood of working for pay or without pay. There is some evidence of reductions in the intensive margin in work for pay: Scholarship recipients work between one and four fewer hours per week for pay, although the coefficients in these regressions are not always significant at conventional levels. The last two rows in Table 7 shows that total monthly earnings are not significantly different among scholarship recipients and nonrecipients.
Scholarship Effects on Labor Outcomes
A number of factors could account for the absence of significant scholarship effects on earnings, in spite of the positive effects on completed grades of schooling. One possibility is that, because the additional schooling did not result in more learning (as measured by test scores), it did not raise productivity and therefore earnings. Another possibility is that the returns to the additional schooling are positive but that they are initially offset by the fact that scholarship recipients will on average have fewer years of work experience (because some stayed in school longer, and it is difficult to combine schooling and full-time work for pay); under this scenario, with a longer time horizon, we would expect the earnings of scholarship recipients to eventually surpass those of nonrecipients. A third possibility is that there is a great deal of measurement error in earnings, and that this makes it hard to identify program effects with reasonable precision.30
Finally, we analyze program effects on subjective social status (as measured by the MacArthur ladders with the village or the country as reference groups), on mental health (as measured by the CESD depression scale), and on the probability that adolescents are married or have had a child. Table 8 shows that the scholarship program did not affect subjective social status. The third row of the table suggests that scholarships may have improved mental health, although the coefficients in these regressions are not always significant at conventional levels. There is no evidence that the scholarship program led recipients to delay marriage or childbirth, although we note that the fractions married (19 percent) and who have children (8 percent) were still relatively low at the time we carried out the household survey. For these outcomes, as for earnings, a longer time horizon may be needed to meaningfully estimate the impact of the scholarship program.
Scholarship Effects on Other Outcomes
V. Conclusion
Many programs in developing countries seek to raise school enrollment and grade attainment. Drawing on randomized evaluations of conditional cash transfers, school feeding programs, programs that provide school uniforms free of charge, and merit-based scholarships, a recent review concludes that interventions that reduce the price of schooling can have substantial effects on access to education (Kremer and Holla 2009).
In this paper we show that a program that gave scholarships to poor children in Cambodia for the three years of lower secondary school had substantial effects on school enrollment and grade attainment. Our paper thus adds to the sparse literature on the medium-term (as opposed to short-term) effects of monetary incentives on schooling in developing countries.
Because we focus on medium-term effects, our paper is closely related to a recent paper that analyzes the medium-term effects of PROGRESA in Mexico (Behrman, Parker, and Todd 2011). Behrman, Parker, and Todd present a variety of estimates of program effects on completed grades of schooling, using different samples and different identification strategies. Based on their double-difference matching estimates, the authors conclude that three years of PROGRESA transfers (relative to no transfers) result in approximately 0.3 more grades of completed schooling (Figures 5 and 6 in Behrman, Parker, and Todd 2011, pp. 112–13). In comparison, we estimate effect sizes of roughly 0.6 additional grades of completed schooling. Thus, the program effects in Cambodia appear to be substantially larger than those found for PROGRESA. This is particularly noteworthy because transfers made by PROGRESA are on average an order of magnitude larger than the value of scholarships awarded by the program in Cambodia we analyze in this paper. (They represent 20 percent of the consumption of the average recipient household in Mexico, compared to two percent in Cambodia—see Fiszbein and Schady 2010, pp. 128–29.)31 Taken together with our earlier results on short-term effects (Filmer and Schady 2008, 2011), the results in this paper suggest that quite small monetary incentives can have large effects on school enrollment and grade attainment in poor countries.
In spite of this large effect on grade attainment, however, we fail to find any program effects on learning outcomes or earnings. There are a variety of reasons, both substantive and measurement-related, why this could be the case, and our data do not allow us to credibly test competing hypotheses. Also, children could benefit from programs that increase grade attainment levels even if their cognitive skills, as measured by tests, are no higher than they would have been without the additional schooling. They may acquire important noncognitive skills, including discipline, perseverance, motivation and a work ethic. (See Heckman and Rubinstein 2001; Heckman, Stixrud, and Urzúa 2006.) Despite the medium-term time frame we analyze, it may still be too early to tell whether the labor market outcomes of children who received scholarships will eventually be better than those of children who did not receive them. Schooling could also have other benefits, including better health status for scholarship recipients and (eventually) their children—even if learning, as measured by test scores, is limited. Nevertheless, the absence of scholarship effects on test scores, employment, earnings, and subjective social status all lead us to be cautious. It would be important in future research to continue to follow the beneficiaries of this and similar programs into adulthood.
Footnotes
Deon Filmer is a lead economist at the World Bank.
Norbert Schady is the principal economic advisor for the Social Sector at the Inter-American Development Bank. They thank Felipe Barrera-Osorio, Luis Benveniste, Pedro Carneiro, Stephanie Cellini, Richard Murnane, Jamele Rigolini, T. Paul Schultz, three anonymous referees and various seminar participants for very helpful comments, as well as Beng Simeth, Tsuyoshi Fukao and the members of Scholarship Team of the Royal Government of Cambodia’s Ministry of Education for valuable assistance in carrying out this work. Rodrigo Azuero and Haijing Huang provided outstanding research assistance. This work benefited from funding from the World Bank’s Research Support Budget (P094396; P095662); the Bank-Netherlands Partnership Program Trust Fund (TF055023); the Gender Action Plan Trust Fund (TF094922); and a Hewlett Foundation Trust Fund (TF096771). The findings, interpretations, and conclusions expressed in this paper are those of the authors and do not necessarily represent the views of the World Bank or the Inter-American Development Bank, their Executive Directors, or the governments they represent. The data used in this article can be obtained beginning January 2015 though December 2017 from Deon Filmer at 1818 H Street NW Washington DC 20433, email dfilmer{at}worldbank.org.
↵1. CESSP stands for Cambodia Education Sector Support Program.
↵2. PROGRESA was renamed Oportunidades in 2001. In this paper we use the name PROGRESA for simplicity.
↵3. This section draws on Filmer and Schady (2009).
↵4. A primary school was designated a feeder school if it had sent graduates to a given secondary school in recent years. In the rare cases where primary schools had sent students to more than one secondary school, the primary school was designated as a feeder to the secondary school where it had sent most students. In principle, scholarships were not portable. To receive a scholarship, a student graduating from a given primary school had to attend the designated CSP school into which her primary school fed. In practice, if a student moved to a secondary school within five 5 kilometers of a program school, she could retain her scholarship. Given the low density of secondary schools in Cambodia at the time, this was a very rare occurrence.
↵5. All of the variables in the scholarship application form had also been collected in previous nationally-representative household surveys. Using the most recently available Cambodia Socio-Economic Survey, Demographic and Health Survey, and Cambodia Child Labor Survey, we ran regressions of grade 7 enrollment, conditional on grade 5 completion, in a sample of 12–17-year-olds, on the variables in the application form. We used the coefficients from this regression to construct the weights given to individual responses on the application form. All application forms were double-entered. We verified the code used by the firm to code individual questions, and to calculate the aggregate dropout-risk score.
↵6. When there were tied scores at the eligibility cutoff all applicants with the tied score were offered a scholarship.
↵7. The final list of scholarship recipients was included in an official government proclamation (“Prakas”) for each CSP school. We cross-checked the list on the “Prakas” against a list that we produced independently based on the applicant database. There were no discrepancies (allowing for a handful of adjustments based on the public comment phase).
↵8. In practice, within every large school, the 25 students with the highest dropout-risk score were offered a scholarship of $60, and the 25 students with the next highest scores were offered a scholarship of $45; in small schools, the comparable numbers were 15 students with scholarships of $60, and 15 students with scholarships of $45. We do not focus on this distinction in this paper. Rather, we compare applicants who were offered a scholarship, regardless of the amount, with others that were not. Because the identification strategy is regression discontinuity, as discussed below, we are implicitly comparing applicants who were offered a $45 scholarship, with those who were offered no scholarship at all. Students who were offered a $60 scholarship help estimate the control function that relates outcomes (for example, grades of completed schooling or test scores) to the dropout-risk score. We have shown elsewhere that there is no evidence that the short-term effect of the CSP on enrollment is larger among students who were offered the $60 scholarship rather than the $45 scholarship (Filmer and Schady 2011).
↵9. Between October and December 2006, we fielded a “midline” survey that is the basis of our earlier work on the CSP scholarship program (Filmer and Schady 2011). The sample for this midline survey was constructed in the same way as described above for the “endline” survey we use in the current paper. During the midline, if the survey firm could not contact an applicant after several attempts, they were provided a list of “replacements.” These were selected from those with the next lower dropout-risk scores below those already on the list (for example, if either a scholarship recipient or one of the 20 “first” children denied scholarships could not be found, the 21st child denied a scholarship was given as a replacement). The endline survey included these “replacements” in the list of applicants that the firm was supposed to contact.
↵10. Some surveys were conducted as late as July 2010 because tracking these applicants took additional time. Eighty-eight percent of interviews were carried out before the end of April 2010; 98 percent before the end of June 2010.
↵11. The test questions ranged from simple addition and subtraction of two-and three-digit numbers, reading the numbers off a bar chart or a graph, interpreting the area of shapes on a grid, manipulating fractions, and completing a progressive sequence of shapes.
↵12. The Ravens has been applied in many settings in the developing world. It is considered a test of cognitive capacity or general intelligence, and it is therefore not clear whether we would expect scholarships to improve scores on this test.
↵13. Cronbach’s alpha is the average covariance across the item responses, divided by the sum of the average variance of all items plus the average covariance across all items. It is often used as a measure of the reliability of a test, although it is best interpreted here as the extent to which the various items on the test capture a single underlying concept (such as ability).
↵14. Specifically, the survey included 12 questions about mental health (such as “Over the past month have you been feeling unhappy or depressed?”), which respondents answered as “always,” “sometimes,” or “never,” which we code with values of 1, 2, or 3. The total score is then the sum of all 12 scores, normalized by subtracting the mean and dividing by the standard deviation of nonrecipients. Our results are robust to different ways of aggregating the individual responses.
↵15. For a description and bibliography of papers that use MacArthur ladders, see the MacArthur Foundation’s Network on SES and Health website: http://www.macses.ucsf.edu/Research/Psychosocial/notebook/subjective.html.
↵16. Because the number of scholarship recipients is fixed in “small” and “large” schools, as discussed above, the value of the dropout-risk score at the cutoff will on average be higher in schools that received more applications and in schools which serve poorer children.
↵17. We thank an anonymous referee for this suggestion. Pop-Eleches and Urquiola (2013) analyze the effects of school quality in Romania in an RD setting with multiple cutoffs. They normalize the score by subtracting the value of the score at the school-specific cutoff but exclude the observation whose score is exactly equal to zero to avoid the mechanical piling of mass at the cutoff. Our results are robust to this approach to normalizing the score.
↵18. We also calculated results for polynomial order zero, which is equivalent to a simple comparison of mean outcomes for children who were offered scholarships and those who were turned down. We do not report these results as they make no adjustment for differences in socioeconomic status between scholarship recipients and nonrecipients.
↵19. In this, as in other figures, the size of each circle is proportional to the number of observations at each value of the score. Lower values of the normalized dropout-risk score, on the lefthand side of the graph, correspond to children with characteristics that make them less likely to drop out of school in the absence of scholarships. The regression line is based on polynomial order one, with school fixed effects, and with the cutoff centered at the midpoint between the score of the last scholarship recipient and the first nonrecipient.
↵20. As a further robustness check, we ran separate regressions in which the indicator variable for attrited children was regressed on an indicator for a given characteristic (for example, age of the child) and the relevant interaction with treatment (for example, age interacted with treatment). We repeated this for each of the 26 variables on the application form. In the polynomial order 1 regressions, the interaction term is only significant at the 5 percent level or higher in one case (corresponding to the interaction between treatment and the availability of drinking water within the house). We take this as further confirmation that attrition is unlikely to be a source of concern for our estimates.
↵21. McCrary (2008) proposes a formal test of the log difference in the heights of the bins corresponding to observations just to the right and to the left of the cutoff. On the basis of this test, we fail to reject the null of equal mass on both sides of the cutoff. (The regression coefficient is 0.118, with a standard error of 0.075.)
↵22. In our earlier work we use a household survey carried out in 2006 and data from unannounced school visits, and report similar effects on school enrollment, between 23 and 28 percentage points (Filmer and Schady 2011).
↵23. These regressions include school fixed effects and a quadratic term in the school dropout-risk score. However, results are very similar if we exclude the fixed effects, the dropout-risk score, or both.
↵24. The average has been normalized so that it, too, has a mean of zero and a standard deviation of one.
↵25. In the 2006 household survey discussed above we also tested children in math and vocabulary. The coefficients in regressions of test scores on the control function and the indicator variable for scholarship recipients using the 2006 data are very similar to those that use the 2010 survey. In the polynomial order 1 regression, the coefficients on scholarship recipients using the 2006 data are 0.044 (with a standard error of 0.064) for vocabulary, and 0.063 (with a standard error of 0.066) for math—very close to the values in Table 5.
↵26. We use a math test that was applied to all children in CSP-eligible schools in 2006, when scholarship recipients were in grade 7. We limit the sample to children who were not offered scholarships, and regress individual test scores on the 26 characteristics that were collected on the scholarship application form. We then calculate the average value of the residuals, by school. These are, in effect, the mean test scores of children turned down for scholarships in every CSP-eligible school, purged of differences in socioeconomic status across schools.
↵27. Solid (dark) circles correspond to schools where the effect is significant; empty (light) circles to schools where the coefficient is not significant at conventional levels.
↵28. In our earlier work (Filmer and Schady 2011) we found evidence that the short-term impact of scholarships on school enrollment was somewhat larger in schools where the cutoff corresponded to children who were on average more likely to drop out.
↵29. Respondents were asked whether “they do any work for pay at all, even one hour, during the past seven days (worked on farm, private or public sector, own account or in a business belonging to themselves or someone else in their household, etc.)?” The question on work for no pay is identical, with the words “for no pay” instead of the words “for pay.” Work for no pay is not meant to include household chores, as there is a separate question that asks respondents whether “they engaged in housekeeping activities or household chores (for example, collecting firewood, collecting water, cooking, looking after other household members such as children, elderly, sick, and so on) in their household during the past seven days.”
↵30. We find no evidence that scholarships resulted in substantial shifts in employment into modern economic sectors with higher value-added. For example, among girls who work for pay, scholarship recipients are more likely than nonrecipients to work in agriculture (where the return to the additional schooling would appear to be low) and less likely to work in the garment sector (where the return to the additional schooling would appear to be higher).
↵31. PROGRESA and other conditional cash transfer programs in Latin America are meant to cover both the direct cost of schooling, and the opportunity cost of schooling. In Cambodia, by contrast, the scholarships are only meant to cover the direct cost of schooling (fees, textbooks, uniforms).
- Received April 2012.
- Accepted May 2013.