Subjective Performance Evaluation in the Public Sector

Footnotes

• Iftikhar Hussain is a lecturer (assistant professor) in economics, University of Sussex. Email: iftikhar.hussain{at}sussex.ac.uk. For helpful discussions and comments the author would like to thank Orazio Attanasio, Oriana Bandiera, Tim Besley Steve Bond, Martin Browning, Rajashri Chakrabarti, Ian Crawford, Avinash Dixit, David Figlio, Sergio Firpo, Rachel Griffith, Caroline Hoxby, Andrea Ichino, Ian Jewitt, Kevin Lang, Valentino Larcinese, Clare Leaver, Susanna Loeb, Steve Machin, Bhash Mazumder, Meg Meyer, Andy Newell, Marianne Page, Imran Rasul, Randall Reback, Jonah Rockoff, Jeff Smith, David Ulph, John Van Reenen, and Burt Weisbrod, as well as seminar participants at the Chicago Fed, Northwestern (IPR), LSE, Manchester, Nottingham, Oxford, Sussex, the NBER Summer Institute, Association for Education Finance and Policy annual conference, the Royal Economic Society annual conference, the Society of Labor Economists annual meeting, the Educational Governance and Finance Workshop, Oslo and the Second Workshop on the Economics of Education, Barcelona. The data belong to the U.K. Department for Education and researchers can apply directly to the DfE in order to gain access to the administrative data. The author is happy to provide researchers with the Stata ‘do’ files used to create the tables in the paper.

• ↵1. See Holmstrom and Milgrom (1991) for a formal statement of the multitasking model. Dixit (2002) discusses incentive issues in the public sector. For empirical examples of dysfunctional behavior in the education sector see the references below.

• ↵2. However, a system where the evaluator is allowed to exercise his or her own judgment, rather than following a formal decision rule, raises a new set of concerns. Results from the theoretical literature emphasize influence activities and favoritism (Milgrom and Roberts 1988; Prendergast and Topel 1996) that make the subjective measure “corruptible” (Dixit 2002). See Prendergast (1999) and Lazear and Oyer (2012) on the limited empirical evidence on the effectiveness of subjective evaluation. In many settings, good objective measures may not be immediately available. For example, in its analysis of active labor market programs, Heckman et al. (2011, p. 10) notes that: “the short-term measures that continue to be used in… performance standards systems are only weakly related to the true long-run impacts of the program.” Whether complementing objective performance evaluation with subjective assessment is an effective strategy in such settings remains an open question.

• ↵3. Hussain (2009) analyzes the effects of inspections on the teacher labor market. The evidence shows that when schools receive a “severe fail” inspection rating, there is a significant rise in the probability that the school principal exits the teaching labor market. The next section provides further details.

• ↵4. Note that an important question not addressed in this study is whether introducing an inspection regime raises performance of the schooling sector as a whole.

• ↵5. So that the May test outcome for these schools is not affected by the subsequent fail but neither do inspectors select them for failure on the basis of this outcome.

• ↵6. The descriptive analysis demonstrates that there is little difference in observable characteristics between schools failed in June (the control group) and schools failed in the early part of the academic year (the treatment group). This, combined with the fact that timing is determined by a mechanical rule, suggests that there are unlikely to be unobservable differences between control and treatment schools. The claim then is that when comparing early and late fail schools within a year, treatment (early inspection) is as good as randomly assigned. Potential threats to this empirical strategy are addressed below.

• ↵7. Quantile regression analysis reveals substantial gains across all quantiles of the test score distribution.

• ↵8. On the usefulness of in-class teacher evaluations, see also Kane et al. (2010) and Rockoff and Speroni (2010).

• ↵9. In Section V below I discuss a number of aspects of the inspection process that sets it apart from the teacher evaluation system described by Taylor and Tyler (2001).

• ↵10. Taylor and Tyler (2013) finds that teacher performance increases in the year of the evaluation, although the effects in subsequent years are larger.

• ↵11. Of course, the possibility that both these effects cancel each other out cannot be ruled out.

• ↵12. For evidence on the efficacy of test-based accountability systems, such as the U.S. federal No Child Left Behind Act of 2001, see the survey by Figlio and Loeb (2011). There is a large descriptive literature on the role of school inspections (see, for example, the surveys by Faubert 2009 and OECD 2013) but very little hard evidence exists on the effects of inspections. Two exceptions from the English setting are Allen and Burgess (2012) and Rosenthal (2004), discussed below.

• ↵13. Hastings and Weinstein (2008) provides evidence on the importance of information constraints for poor households when choosing among schools.

• ↵14. Note that the age-14 (Key Stage 3) tests were abolished in 2008.

• ↵15. Online Appendix A outlines the relevant theoretical background.

• ↵16. Hussain (2013) demonstrates that parents are responsive to inspection ratings when making school choice decisions.

• ↵17. In its own words, the inspectorate reports the following as the primary purpose of a school inspection: “The inspection of a school provides an independent external evaluation of its effectiveness and a diagnosis of what it should do to improve, based upon a range of evidence including that from first-hand observation. Ofsted’s school inspection reports present a written commentary on the outcomes achieved and the quality of a school’s provision (especially the quality of teaching and its impact on learning), the effectiveness of leadership and management and the school’s capacity to improve.” (Ofsted 2011, p. 4).

• ↵18. As of 2011, Ofsted tendered school inspections to three organizations, two are private sector firms, the third is not-for-profit.

• ↵19. Inspection cycles typically lasted between three and six years.

• ↵20. English primary schools cater for students between the ages of 5 and 11.

• ↵21. This short notice period has been in place since September 2005. Prior to this, schools had many weeks, sometimes many months, of notice of the exact date of the inspection. Anecdotal evidence suggests that these long notice periods resulted in disruptive preparations for the inspections.

• ↵22. These can be obtained from http://www.ofsted.gov.uk/.

• ↵23. For evidence of the negligible effects of a moderate fail on principal turnover and the contrasting large effects for severe fail schools, see Hussain (2009).

• ↵24. The government views test scores as an “anchor” in the English accountability system. See, for example, the statement by the secretary of state for education in oral evidence to the House of Commons education select committee (Uncorrected Transcript of Oral Evidence, 31 January 2012, http://www.publications.parliament.uk/pa/cm201012/cmselect/cmeduc/uc1786-i/uc178601.htm, accessed February 10, 2012).

• ↵25. Note that exogenous timing of inspections, exploited by Rosenthal (2004) and Luginbuhl et al. (2009), cannot by itself identify the effect of inspection for schools receiving a specific rating. This is a consequence of the fact that ratings are determined by past realizations of the outcome variable, test scores. For example, comparing test scores in the year after inspection for failed schools with test scores for yet-to-be-failed schools does not yield unbiased estimates. See Section V below.

• ↵26. For an evaluation of the effects on test scores it is important to note that the latest available test score information at the time of inspection is the same for both control and treated schools: that is, from the year before inspection.

• ↵27. I focus on these four years because 2005–2006 is the first year when the inspection system moved from one where schools were given many weeks notice to one where inspectors arrived in schools with a maximum of two days notice. In order to analyze the effect of a first fail (a “fresh fail”), schools that may have failed in 2004–2005 or earlier are dropped from the analysis. This results in a loss of 10 percent of schools.

• ↵28. The early inspection category is expanded to three months in order to increase the sample of treated schools.

• ↵29. Note that an inspection in the academic year 1999–2000 is recorded as “2000”; an inspection in 2000–2001 is recorded as “2001,” and so on.

• ↵30. Further analysis shows that in a regression predicting assignment status (early versus late inspection) for the set of fail schools, the year of previous inspection is statistically significant while past test scores, prior inspection rating, and the proportion of students eligible for free lunch are all statistically insignificant at conventional levels.

• ↵31. One remaining concern is that the worst schools may be closed down after a fail inspection. If this happens immediately after an inspection then any test score gains posted by the early fail schools may be due to these selection effects. In fact, three pieces of evidence suggest that this is not the case. Although such a “weeding out” or “culling” process may be important in the medium term, the evidence in Table 1 demonstrating the comparability of the early and late inspected group of schools suggests that such a process does not take place immediately, that is, in the year of inspection.

  Second, an examination of the data shows that the probability of dropping schools from the estimation sample because of a lack of test data from the May tests in the year of inspection—perhaps because the school is closed down—when test data are available in previous years, is similar for treated and control schools. In total, for the years 2005–2006 to 2008–2009, 4 per cent of schools (6 schools) from the control group and 5 percent (14 schools) from the treatment group are dropped because of lack of test score data in the year of inspection. These schools appear to be comparable to the treatment and control schools on characteristics such as student attainment in the year before inspection. For example, the proportion of students attaining the mandated attainment level for age 11 students in the year before inspection is 62 percent for the 14 treated (early inspected) schools dropped from the estimation sample; the corresponding mean is 63 per cent for the 258 treated schools included in the estimation sample.

  Finally, a comparison of results for the moderate fail schools versus the severe fail ones also sheds light on this issue. Note that a moderate fail is unlikely to lead to changes in school leadership and school closure. If test gains following a fail are observed for moderate fail schools, then selection effects arising from school closures are unlikely to be driving the results. I report on these findings below.

• ↵32. In the online Appendix Table 8, I show that pooling all four years together yields the same conclusion: Treatment and control schools appear to be balanced on observable characteristics.

• ↵33. In a heterogenous treatment effect setting where δ_i is the student-specific gain from treatment, the key assumption is that D_s is uncorrelated with both u_is and δ_i. The evidence presented above suggests that this assumption is satisfied. In this case a comparison of means for the treatment and control outcomes yields the Average Effect of Treatment on the Treated. This is the effect of a fail inspection rating for schools inspectors judge to be failing. Another parameter of policy interest—not estimated—is the Marginal Treatment Effect; that is, the test score gain for students in schools on the margin of being failed.

• ↵34. The key DID assumption, which embodies the assumption of common trends across treatment and control groups, is that conditional on the school fixed effect (μ_s) and year (post₀₆) the treatment dummy D_st is uncorrelated with the error—that is, E(u_ist | post₀₆, D_st, μ_s) = 0.

• ↵35. There is no grade retention in the English system and reduced scope for excluding students from the testing pool. Nevertheless, anecdotal evidence suggests that schools may selectively bar or exclude students and popular discourse suggests that schools target students on the margin of performance thresholds (see, for example, “Schools focusing attention on middle-ability pupils to boost results,” The Guardian, 21 September 2010).

• ↵36. It should be noted that potentially distortionary incentives may well exist prior to the fail rating. However, these incentives become even more powerful once a school is failed. Thus the tests for gaming behavior outlined here shed light on the effects of any extra incentives to game the system following a fail inspection. As noted previously, the incentives to improve test score performance following a fail inspection are indeed very strong.

• ↵37. In primary schools, national tests are administered to students in England at ages seven and 11.

• ↵38. The online appendix can be found at http://jhr.uwpress.org/.

• ↵39. Note that “2006” refers to the academic year 2005–2006 and so on for the other years.

• ↵40. As described earlier, changes to the inspection process were introduced in September 2005. Arguably, the biggest change was a move to very short (two days) notice for inspections, down from a notice period of many months. This regime has remained in place since September 2005.

• ↵41. Treatment effects, reported in the online Appendix B, estimated for schools failed in each individual month from September through to April yield a pattern that suggests a steadily declining effect as we move closer to May.

• ↵42. The following school-level controls are included in all the regressions reported in Panel A of Table 2: pre-inspection mathematics and English attainment, percent of students eligible for free lunch, and percent of students who are nonwhite. Dropping these from the regressions makes very little difference to the estimates. For example, without any controls at all, the estimated effect for mathematics is 0.10 of a standard deviation.

• ↵43. Note that the above results from data pooled over the four inspection years are in line with results from regression analysis conducted for each year separately, reported in online Appendix Tables 1 and 2.

• ↵44. Results for individual inspection years confirm the finding that the placebo treatment produces no discernible effect.

• ↵45. In order to see this, suppose that for each school i inspectors assess overall quality in year t and month m to be some weighted average of past test scores (w_t–1) and inspection evidence of school quality, which is composed of fixed (q_i) and time-varying components:

  (*)

  where is an idiosyncratic component and it is assumed that follows a covariance stationary AR(1) process. In addition, suppose that inspectors fail schools when quality falls below some threshold level, Q₀:

  (**)

  Thus, for schools failed in September, say, the temporary component of quality (observed by the inspectors, but not the analyst) is such that: . Then because of the selection rule (**) and the AR(1) nature of , it follows that by the time of the May test, unobserved quality for September failed schools has improved, that is: . The idea behind the robustness test described above is that schools which failed largely because of low w_t–1 will be likely less affected by within-year (for example, September to May) mean reversion in . Thus for these schools the estimated treatment effect of a fail rating is less likely to be driven by differences between and .

  Note that further regression results, not reported here for brevity, using all schools to explain the probability of failure by month of inspection, test scores and the interaction of these two variables, show that inspectors attach the same weight to test scores in the early months of the year and later months such as June. This supports the use of a time-invariant coefficient on in (*) above.

• ↵46. The online Appendix Table 3 highlights the relationship between prior ability—measured by the age-7 Key Stage 1 test score—and the probability of attaining the target level on the age-11 Key Stage 2 test. This shows that only around a quarter to one-third of students in the lowest prior ability quartile attain the stipulated Level 4.

• ↵47. Note that running four separate regressions by prior ability quartile subgroup leads to results virtually identical to those reported in Columns 2 and 4 of Table 4.

• ↵48. For example, Neal and Schanzenbach (2010) finds that test scores improve for students in the middle of the prior ability distribution while low-ability students experience zero or even negative effects on test scores.

• ↵49. Dustmann et al. (2010) shows that even though students from most minority groups lag behind white British students upon entry to compulsory schooling, they catch up strongly in the subsequent years. The relatively large gains for this subgroup reported in Table 5 suggest that one mechanism driving the results reported in Dustmann et al. may be the school inspection system.

• ↵50. This interpretation of the results is also supported by the subgroup analysis of Table 5, which shows that children from poorer, minority groups tend to gain relatively more from the fail inspection. Children from families where English is not the first language at home most likely have parents who are less able to interrogate teachers and hold them accountable. The results in Table 5 boost the conclusion that it is children from these sorts of families who are helped most by the fail inspection.

• ↵51. Note however that in the absence of data on allocation of resources within the classroom, I cannot definitively rule out the possibility that teachers raise effort equally across all students and a differential marginal return to this rise in effort leads to the observed heterogeneous treatment effects.

• ↵52. Note that such fadeout of initial gains is in fact common in settings even where educators are not under pressure to artificially distort measured student performance (see, for example, Currie and Thomas 1995). Thus, the fading of test score gains does not necessarily indicate distortionary response on the part of teachers. On the other hand, if some of the initial test score gains persist into the medium term then this would suggest that the initial gains from the fail treatment are “real.” Prior evidence supports the notion that pressure on schools via test-based accountability leads to test score gains for students even after they have left the sanctioned schools (Rouse et al. 2013; Chiang 2007).

• ↵53. One other notable feature of the results in Table 7 worth highlighting is the contrasting nature of mean reversion in the moderate and severe fail categories. The extent of reversion-to-mean for the control groups is depicted in the “post” row of Table 7. For the moderate fail schools, there appears to be substantial mean reversion: There is bounceback, compared to the prior year, in mathematics and English test scores for the moderate fail schools of 0.06 and 0.14 of a standard deviation, respectively. For the severe fail schools, however, there is no bounceback. One interpretation of this evidence is that inspectors are able to distinguish between poorly performing schools and the very worst performing schools, with the latter category exhibiting no improvements in the absence of treatment.

• ↵54. The precise question in the survey is: “Are there any children in the study child’s class whose behavior in class prevents other children from learning?” (check Yes or No).

• ↵55. Any schools that were also failed in one of the previous inspection cycles between 2003–2004 and 2008–2009 are dropped from this control group.

• ↵56. School controls included in the regression: percentage of students eligible for a free lunch, the school’s national test score percentile, and type of school.

• ↵57. This result is unlikely to be driven by student selection into schools. Hussain (2009) shows that enrolment declines following a fail inspection. More motivated parents are more likely to switch schools, in which case discipline should deteriorate following a fail inspection.

• ↵58. As a robustness check, using different sample selection rules, for example, 2005–2006 and 2006–2007 fail and satisfactory schools, yields qualitatively similar results, though the reduction in sample size results in larger standard errors.

• ↵59. As Taylor and Tyler (2013) shows, there is a positive effect on student test scores in response to structured classroom teacher observations even in the year of the evaluation, although the effects in subsequent years are larger. However, there are a number of aspects of the inspection process that set it apart from the teacher evaluation system described by Taylor and Tyler (2013). First, the period covered by this analysis typically corresponds to the third or fourth inspection a school has received since the 1990s; the Taylor-Tyler study evaluates the effects of the first rigorous evaluation for teachers after 8–17 years in the job. If marginal gains from evaluation are declining then later evaluations and inspections may garner negligible returns. Second, whereas inspectors typically observe each teacher for no more than an hour, in the system described by Taylor and Tyler (2013) teachers receive detailed feedback from four classroom observations over the course of a whole year. Finally, Taylor and Tyler (2013) finds that gains from evaluation increase over time, at least for mathematics. The results in Table 9 relate to effects in the year of inspection only and hence longer-term gains cannot be ruled out. Unfortunately, given the nature of the mean reversion problem with respect to test scores, within school analysis exploiting the panel dimension of the data cannot be employed to uncover longer term effects of inspections.

• ↵60. Effects of a Satisfactory rating may incorporate effects of feedback as well as potential threats or sanctions, especially if the school is close to slipping into the fail category. Taylor and Tyler (2013) finds that the biggest effects from providing feedback are for teachers who receive relatively low scores. If this also holds for the case of inspection feedback at the school level, then one may expect to find larger effects for satisfactory schools than for higher-rated schools.

• ↵61. I also estimate the effect of an inspection on test scores for all schools (unconditional on the grade the school receives) by exploiting the variation in timing of inspections across years, as in Rosenthal (2004). Comparing test scores in 2007 for all schools inspected in 2006 (the treatment group) with schools inspected in 2008 (the control group), a difference-in-differences model using student-level data reveals that there are no significant differences between early and late inspected schools (the estimate is –0.8 of one percent of a standard deviation; p-value 0.24; regressions include school fixed effects and age-7 test scores).

• ↵62. A key concern under a subjective assessment regime is that it is open to manipulation by the bureaucrats charged with oversight. Results reported in Hussain (2012) on the validity of inspection ratings suggest that inspector ratings are correlated with underlying measures of school quality—constructed using survey measures from the school’s current stock of students and parents—even after conditioning on standard observed school characteristics.

• ↵63. The gains are large when compared to other possible policy interventions, such as the effects of attending a school with higher attainment levels (Hastings et al. 2009) or enrolling in a charter school (Abdulkadiroglu et al. 2011).