Abstract
Traditional teacher evaluation systems have come under scrutiny for not identifying, supporting, and, if necessary, removing low-performing teachers from the classroom. Leveraging the experimental rollout of a pilot evaluation system in Chicago, we find that, while there was no main effect of the pilot on teacher exit, the pilot system increased exit for low-rated and nontenured teachers. Furthermore, teachers who exited were lower performing than those who stayed and those who replaced them. These findings suggest that reformed evaluation systems can induce low-performing teachers to exit schools and may also improve the overall quality of the teacher labor force.
The vast majority of people … know that [teacher] evaluation has been generally meaningless, has failed to support the development of teachers and principals, and that the system is broken. [Policymakers and practitioners] are working together to help educators strengthen their craft—and to build real career ladders that recognize and reward excellence.1
—Secretary of Education Arne Duncan
I. Introduction
Over the last decade, increased attention has been paid to improving the quality of our nation’s public school teachers. The No Child Left Behind (NCLB) Act of 2002, for example, represented the first national legislative effort to set teacher-quality benchmarks, requiring states to ensure that all of their teachers were “highly qualified.” Under NCLB, teacher quality was defined by a teacher’s credentials—receipt of a bachelor’s degree, state certification or licensure, and proof of content-area expertise. However, these characteristics have been shown to have a limited relationship with student achievement (Goldhaber and Brewer 2000; Goldhaber 2002; Darling-Hammond and Sykes 2003; Clotfelter, Ladd, and Vigdor 2007; Goldhaber 2007; Kane, Rockoff, and Staiger 2008; Harris and Sass 2011). Recent empirical evidence confirms that high-quality teaching is vital for student achievement and later labor market outcomes (Goldhaber 2002; Rockoff 2004; Rivkin, Hanushek, and Kain 2005; Aaronson, Barrow, and Sander 2007; Chetty, Friedman, and Rockoff 2014), and we know that teachers vary dramatically in their ability to improve student performance (Rivkin, Hanushek, and Kain 2005; Aaronson, Barrow, and Sander 2007). Despite the variability in teacher effectiveness, traditional evaluation systems—those that rely on cursory classroom observations of teacher practice and binary ratings of a teacher’s summative performance—have not provided a meaningful signal of teacher quality, with little to no differentiation in the ratings of teacher performance (Weisberg et al. 2009). As a result, traditional teacher evaluation systems have come under scrutiny for failing to meet two basic objectives of personnel evaluation: providing guidance to teachers to improve their classroom practice and identifying low-performing teachers for remediation or removal from the classroom.
Under the federal Race to the Top (RTTT) initiative, recent policy efforts have emphasized and provided financial incentives for systemic reforms to teacher evaluation. These new systems incorporate multiple measures of teacher performance, including classroom observation rubrics that define instructional improvement on a continuum (rather than by a checklist) and, most notably, measures of student performance on state standardized tests (so-called value-added measures or VAMs).2 In doing so, RTTT has brought unprecedented national attention to how teachers are evaluated and whether evidence-based measures of teacher quality provide more meaningful information about teacher effectiveness than simply years of teaching experience or educational attainment.
Recent evidence suggests that teacher evaluation systems that incorporate structured classroom observations and detailed feedback on instructional practice can generate noticeable improvements in student achievement (Taylor and Tyler 2012, Steinberg and Sartain 2015) and teacher performance (Dee and Wyckoff 2013). There is, however, little evidence on the role that newly developed teacher evaluation systems play in the removal of underperforming teachers and the compositional changes in the employed teacher labor force that may follow from any resorting of teachers across schools. The only research is on the IMPACT teacher evaluation system in the District of Columbia Public Schools, where regression discontinuity evidence suggests that the evaluation system reduced teacher retention for teachers whose performance ratings fell just below the threshold of effectiveness, suggesting that marginal lower-performing teachers responded to the threat of removal (Dee and Wyckoff 2013).
In this paper, we aim to contribute to the nascent literature on the effects of teacher evaluation reform on the exit of low-performing teachers from the classroom. Specifically, we explore a low-stakes pilot teacher evaluation system in Chicago Public Schools (CPS). The Excellence in Teaching Project (EITP) was introduced in 44 randomly selected elementary schools in 2008–2009 (Cohort 1) and then scaled up to include an additional 49 schools (Cohort 2) in 2009–10. The EITP pilot was intended to improve both the quality of professional conversations about instruction in schools as well as the quality of classroom instruction. Ultimately, the pilot aimed to improve student learning by providing teachers with better information about and guidance to improve their instructional practices.3 EITP focused on the classroom observation component of teacher evaluation and used Charlotte Danielson’s Framework for Teaching (FFT) to guide principals’ observations of and conferences with teachers. CPS also provided extensive principal training to support their use of the FFT protocol to observe and evaluate teachers. Because the EITP pilot was low stakes and not explicitly designed to increase teacher exit from the district, it lacked the accountability features of newly implemented teacher evaluation reforms in the post-RTTT policy context. However, we leverage the experimental design of the rollout to explore the extent to which the introduction of a dramatically new teacher evaluation system in Chicago impacted teacher turnover in the district, addressing the following questions:
Did the implementation of teacher evaluation reform affect teacher turnover?
Did the evaluation system induce turnover for different types of teachers (for example, low-performing teachers, nontenured teachers)?
Were there distributional consequences of teacher turnover associated with the new evaluation policy? Specifically, were teachers in the hardest-to-staff schools more likely to leave and to what extent did higher-performing teachers replace those who exited schools as a result of teacher evaluation reform?
We consider two margins of teacher turnover—teacher exit from the district and teacher transfer to other schools within the district. Although the impact of the new teacher evaluation system on teacher exit from the district may be considered a goal of a system aimed at removing underperforming teachers from the district’s labor market pool, policymakers should also be concerned with whether such a system leads to a reallocation of teacher quality through teacher transfers across schools within the district. If teacher evaluation reform induces low-performing teachers to seek transfers to schools within the same district, the overall distribution of teacher quality in the district may remain relatively unchanged, a potentially unintended consequence of newly developed teacher evaluation systems. We find that, while there was no average effect on teacher exit from CPS, teachers who had low prior evaluation ratings (Unsatisfactory or Satisfactory) were more likely to leave the district because of the evaluation pilot.
In the control schools, 13.0 percent of low-rated teachers left the district at the end of the 2008–2009 school year, compared to 23.4 percent of low-rated teachers in treatment schools after one year of implementation. This represents an 80 percent increase in the turnover of the lowest-performing teachers, providing evidence that teacher evaluation reform has the potential to impact the distribution of teacher quality. We also find that nontenured teachers, who have few contractual protections in terms of job security, were also significantly more likely to leave the district. Among nontenured teachers, we find a 46 percent increase in the exit rate. Moreover, the exit of teachers who were both low-performing and nontenured from the district drive our findings regarding differential teacher turnover, suggesting that the contract protections enjoyed by tenured teachers provided meaningful job security for those who were low-performing. At the same time, teachers who remained in treatment schools were higher-performing than those who exited, and replacement teachers (those who were new to the school in the following year) were also higher-performing. We do not find that teachers were more likely to switch schools after implementation of the pilot or that teachers disproportionately exited more disadvantaged treatment schools.
The evidence presented on the impact of teacher evaluation reform is short term in nature. Specifically, we present impact estimates of the teacher evaluation pilot at the end of the first year of implementation—the only year in which the experimental design of the pilot’s rollout was maintained. While the experimental estimates are restricted to one year, we present graphical evidence on trends in teacher turnover— both in aggregate as well as by teacher tenure—subsequent to the evaluation pilot’s experimental year to provide additional insight into the temporary or persistent effect of evaluation reform.
We begin by describing teacher evaluation in CPS under the EITP pilot and teachers’ potential labor market responses to a dramatically new teacher evaluation system. We next describe the data and empirical methods used to estimate the impact of the teacher evaluation pilot on teacher turnover. Finally, we present and discuss our findings on the impact of teacher evaluation reform in Chicago on teacher mobility and on the distribution of teacher instructional performance within schools.
II. Teacher Evaluation, Turnover, and the Chicago Context
Performance evaluations are a widely used human resources tool in education. However, little is known about teachers’ labor market responses to new information that has been revealed about their performance in the context of new teacher evaluation systems. Under these new, evidence-based systems, teachers experience increased scrutiny, principals are in their classrooms more frequently, and information about their practice and performance, including student growth on standardized tests, is more public than ever. Some teachers may react negatively to this attention and seek to switch schools or to leave the district entirely. Other teachers may welcome the increased communication and feedback about their practice, and these teachers may be less likely to leave their schools than they would be in the absence of these new evaluations. If the teachers who leave the district are the poorest performers, this is evidence that increased teacher accountability is working as intended. However, if low-rated teachers leave the district but are then replaced by other low-performing teachers, we would not expect evaluation systems to shift the distribution of teacher quality due to the removal of underperforming teachers. Moreover, if higher-rated teachers sort to higher-quality schools, teacher sorting may exacerbate the unequal distribution of teacher resources and further disadvantage students attending hard-to-staff schools.
How teachers respond in the labor market to teacher evaluation reform is of critical importance to policymakers, especially in light of evidence on the nonrandom sorting of teachers to schools. Prior work on teacher turnover shows that teachers leave low-resource schools that have the most at-risk students (Ingersoll 2001; Hanushek, Kain, and Rivkin 2004; Imazeki 2005), and high-quality teachers tend to sort into schools with other high-quality teaching staff (Feng and Sass 2012). Evidence suggests that school context and teachers’ working conditions, particularly school and principal leadership, are related to teachers’ exit decisions, independent of the observed relationship between teacher turnover and student demographic characteristics (Ladd 2011; Johnson, Kraft, and Papay 2012). Furthermore, teachers are more likely to stay in schools where the level of collaboration among teachers is high and the school leadership is supportive of teachers (Allensworth, Ponisciack, and Mazzeo 2009). Similarly, Jackson (2013) finds that nonpecuniary aspects of the teaching profession (such as school working conditions) influence teachers’ mobility decisions while also finding evidence that teachers who exit urban schools tend to be the least effective. The empirical question that we pursue is whether teacher evaluation reform promotes this type of teacher sorting.
The EITP pilot in Chicago provides a unique context for assessing questions about teacher evaluation and turnover because it was, in many ways, a precursor to the subsequent federal emphasis on teacher evaluation reform. Arne Duncan was CEO of CPS for nearly eight years before becoming Secretary of Education and reforming teacher evaluation was one of his last major initiatives in Chicago. Prior to the introduction of the EITP initiative during the 2008–2009 school year, CPS teachers were observed and evaluated based on a checklist of classroom practices.4 Under this evaluation regime, which lasted for nearly four decades, almost all CPS teachers (93 percent) received performance evaluation ratings of Superior or Excellent (based on a four-tiered rating system), while at the same time 66 percent of CPS schools failed to meet state proficiency standards under Illinois’s accountability system (The New Teacher Project 2007). With such little variation in teacher ratings, teachers had extensive private information about their own quality. This hidden information likely made it difficult for principals and district officials to assess and differentiate teacher quality.
Under the EITP pilot, teachers were evaluated multiple times per year using Charlotte Danielson’s Framework for Teaching (FFT) structured observation tool. Principals met with teachers before and after conducting a classroom observation. In postobservation conferences, teachers and their principals met to discuss the observed lesson and ways teachers could improve their instructional practice. In these conferences, principals also provided teachers with their Danielson FFT ratings, which for many teachers was the first time they had received detailed information about their instructional performance. Further, the FFT ratings generated more variation in teacher ratings than historically had been provided under traditional, checklist-based evaluation systems (Sartain, Stoelinga, and Brown 2011). But it wasn’t just teachers who had this new information; the Danielson FFT classroom observation rubric and ratings also offered principals greater insight into the instructional performance of their teachers and provided the principal with a more reliable signal about a teacher’s quality.
This revealed information had the potential to influence teacher turnover. Low-performing teachers, for example, might respond to heightened supervision and greater transparency about their performance by exiting their school and either leaving CPS or transferring to another school within the district. Since principals are limited in their ability to provide pecuniary rewards for high-quality teaching, highly rated teachers might use the more reliable rating information as a signal of their quality and sort into higher-performing schools with more collaborative professional communities and more advantaged student populations. Teacher movement across schools and out of the district has the potential to change the distribution of teacher quality across the district. Whether teacher evaluation reform in Chicago affected teacher turnover and the extent to which teacher evaluation resulted in differential turnover based on teacher quality and experience are empirical questions that we now pursue.
III. Data
Data for this paper consist of CPS administrative, personnel, and test score data from the 2005–2006 to the 2011–12 school years. Teachers in treatment (Cohort 1) schools were first exposed to the EITP pilot during the 2008–2009 school year, and control (Cohort 2) school teachers received the EITP treatment for the first time during the 2009–10 school year. In order to track teacher mobility over time, we construct a teacher-year data set with teacher and school characteristics for the 2005–2006 through 2011–12 school years (three prepolicy and four postpolicy years) for all CPS teachers, including those in both EITP and non-EITP schools.
A. Teacher Personnel Data
For all teachers in a given school year, we observe demographic information (age, race, and gender), the school in which they’re employed, years of experience in CPS, certificate area (for example, early childhood, elementary), National Board certification, tenure status, educational attainment (bachelors, masters, doctorate), and undergraduate major.
B. Teacher Evaluation Data
We observe summative evaluation ratings from the 2006–2007 and 2007–2008 school years for teachers in treatment and control schools during the 2008–2009 school year.5 These ratings were assigned prior to the implementation of teacher evaluation reform. Teachers could receive a rating of Unsatisfactory, Satisfactory, Excellent, or Superior; teachers who received either an Unsatisfactory or Satisfactory rating were considered low-performing. The distribution of teachers’ performance ratings was top heavy—in our sample of teachers with prior ratings, 94 percent of treatment and 93 percent of control teachers were rated Excellent or Superior in the year prior to implementation of EITP (see Table 2 for the full distribution of teacher ratings and Table 3 for the distribution of teacher ratings by tenure status).6
The data also include Danielson FFT ratings (Domain 2 Classroom Environment and Domain 3 Instruction) for some teachers in EITP schools (for the 2008–2009 and 2009–10 school years). In 2008–2009, we observe FFT ratings for 65 percent of nontenured teachers and 18 percent of tenured teachers in treatment (Cohort 1) schools. Of tenured teachers who had low prior evaluation ratings (Unsatisfactory or Satisfactory), 63 percent have FFT ratings. The 2008–2009 school year was an off year for tenured teacher evaluations, so CPS central office requested that principals submit FFT ratings only for nontenured teachers. Moreover, due to the low-stakes nature of EITP (FFT ratings, by union contract, could not influence a teacher’s summative evaluation rating), CPS central office could not mandate that principals enter FFT ratings, so FFT ratings data are not available for all teachers in the treatment schools during the 2008–2009 school year.
C. School Data
To characterize the schools from which teachers stay and leave, we use student test score information from the math and reading portions of the Illinois Standards Achievement Test (ISAT) and demographic (race, gender, special education, free/reduced price lunch) data. We aggregate the student data to the school level to characterize the students the school serves. See Steinberg and Sartain (2015) for a detailed discussion of the school contexts for EITP schools, specifically pretreatment balance of school characteristics across treatment and control schools.
IV. Empirical Strategy
We can answer questions concerning the impact of the new teacher evaluation policy on teacher turnover as a result of the unique rollout of the EITP pilot evaluation system. Through partnership with the University of Chicago Consortium on Chicago School Research (UChicago CCSR), CPS included an experimental design in the implementation of the evaluation pilot. CPS chose four elementary school instructional areas of the city for implementation.7 Within each of the four pilot instructional areas, UChicago CCSR randomized schools to be initial participants (treatment schools, Cohort 1) and late adopters (control schools, Cohort 2).8 In the 2008–2009 school year, Cohort 1, which consisted of 44 schools, piloted the Danielson FFT; Cohort 2, which consisted of 49 schools, did not implement the pilot until the following school year (2009–10).9 This experimental design allows us to study the impact that a newly implemented teacher evaluation system has on teacher mobility and the composition of the employed teacher labor force.
Before we address the extent to which the evaluation pilot impacted teacher turnover, we first examine the balance of pretreatment covariates across treatment and control schools. Treatment and control schools are statistically indistinguishable in terms of prior test scores, both in reading and math, as well as student composition (see Table 1). Comparing EITP schools (both treatment and control schools) to all CPS elementary schools that never implemented EITP (non-EITP schools), we find that, while non-EITP schools served fewer African American (and more Latino) students than the EITP schools, EITP and non-EITP schools serve similar proportions of students with identified special learning needs and who qualify for free/reduced-price lunch.
School Characteristics
Given that the EITP treatment targeted teachers, Table 2 summarizes the characteristics of CPS elementary school teachers. At the beginning of the 2008–2009 school year, 24 percent of treatment school teachers were nontenured (teachers with one to three years of experience) and 21 percent of control school teachers were nontenured, though this difference is not statistically significant. Over half had master’s degrees (on average, 57 percent in treatment schools and 61 percent in control schools) while very few were National Board certified. Most of the teachers were female (approximately 85 percent in both groups of schools). We also observe covariate balance for a teacher’s prior summative evaluation rating. In treatment schools, 6 percent of teachers were rated Satisfactory, while 7 percent of control teachers were. This group of teachers is of particular interest since we are concerned with how teacher evaluation reform may have differentially impacted turnover for teachers who had received low ratings in the years prior to EITP implementation.
Teacher Characteristics
We again note the pilot nature of EITP. In 2008–2009, the first year of EITP, approximately 9 percent of all CPS elementary school teachers participated in this evaluation reform effort. EITP was also a low-stakes intervention, with scores received on the Danielson observation rubric not officially incorporated into teachers’ summative evaluation ratings. To lend insight into the extent to which findings from this low-stakes pilot evaluation reform effort may be generalizable to all CPS elementary school teachers, Table 2 also presents comparisons of the observable characteristics of teachers in non-EITP schools to those teachers who were part of the EITP pilot initiative. The non-EITP teachers look remarkably similar to EITP teachers, with the share of Latino teachers the only significant difference (both in magnitude and statistical significance) between EITP and non-EITP teachers. Therefore, the covariate balance across treatment and control school teachers (and, to a large extent, EITP and non-EITP teachers) is notable since this intervention was targeted at improving teacher practice. We can therefore attribute any differences in turnover among EITP teachers at the end of 2008–2009 to the teacher evaluation intervention itself while also allowing for insight into the potential effect of evaluation reform under a districtwide rollout.
To estimate the causal effect of the teacher evaluation pilot on two margins of teacher turnover—teacher exit from the district and teacher transfer to other schools within the district—among our sample of CPS elementary school teachers, we estimate variants of the following model:
(1)
where Yi,08/09 is an indicator variable that characterizes either teacher exit from the district (Yi,08/09 equals 1 if teacher i who taught in an EITP school during 2008–2009 left the district at the end of the school year, and 0 otherwise) or teacher transfer to another school within the district (Yi,08/09 equals 1 if teacher i who taught in an EITP school during 2008–2009 taught in a different CPS school in the 2009–10 school year, and 0 otherwise). The variable Ti represents a vector of teacher characteristics, including teacher demographics (age, race, gender), experience (years teaching in CPS), certification status, and educational attainment. The variable Si represents a vector of school characteristics, including the proportion of female students, the proportion of students by race/ethnicity, the proportion of special education students, and the proportion of students receiving free or reduced-price lunch. We also include the school’s average achievement from the prior academic year (2007–2008), standardized within the EITP sample. The variable EITPi,08/09 is an indicator that equals 1 if teacher i taught in a Cohort 1 school during the 2008–2009 school year and 0 if he/she taught in a Cohort 2 school (the control group schools for the purposes of estimating the treatment effect, δ). Because the randomization was done at the instructional area level, we also include area fixed effects (ϕa) to account for the block structure of the experimental design; and εi,08/09 is a random error term.
We next explore the presence of heterogeneous treatment effects—the extent to which evaluation-induced teacher sorting is a function of observable teacher characteristics. We attend to two teacher populations of interest: teachers with low prior evaluation ratings and nontenured teachers. First, we estimate heterogeneous treatment effects by a teacher’s prior evaluation rating by including interactions of treatment status with teacher ratings, as in Equation 2.
(2)
In Equation 2, ExcellentRatingi is equal to 1 if teacher i received an Excellent rating on his/her summative evaluation at the end of the prior school year, and 0 otherwise; SuperiorRatingi is equal to 1 if teacher i received a Superior rating on his/her previous summative evaluation, and 0 otherwise; and all other variables are defined as in Equation 1. Note that the excluded group includes low-rated teachers—those teachers who received either Unsatisfactory or Satisfactory ratings just prior to the implementation of EITP. If the teacher evaluation policy induced low-rated treatment teachers to change schools or leave the district due to the policy (compared to control school teachers), the estimate of δ will be positive. If high-rated teachers (in treatment schools) are less likely to change schools or leave the district due to the policy (relative to lower-rated treatment school teachers), the coefficients θ1 and θ2 will be negative.
Distribution of Prior Evaluation Ratings, by Treatment and Tenure Status
Nontenured teachers also represent a group of interest in the context of a new teacher evaluation policy. These teachers are least protected by the teachers’ union contract in terms of job security, have spent less time in the profession, and may be less invested in teaching as a career than their more-experienced colleagues. As a result, we may observe that nontenured teachers are differentially affected by a teacher evaluation policy that provides teachers and principals with more information about teacher quality. To test for this, we estimate Equation 3.
(3)
In Equation 3, we include main effects for the EITP pilot and allow turnover to affect teachers differently by experience. Specifically, we include dummy variables for teachers who have taught in CPS for four to ten years and those who have taught in CPS for more than ten years. The excluded group includes nontenured teachers—those with one to three years of teaching experience in CPS. CPS teachers receive tenure after three years in the classroom, so all teachers with fewer than four years of CPS teaching experience are considered nontenured. Interactions between the years of experience variables and the EITP pilot variable allow for the teacher evaluation policy to differentially affect treatment school teachers with different levels of experience. The primary parameter of interest is δ, which estimates the impact of the pilot evaluation system on teacher turnover for nontenured teachers (in treatment schools) relative to nontenured teachers in control schools. Moreover, if nontenured teachers (in treatment schools) are more likely than tenured teachers (in treatment schools) to turnover as a result of the pilot, we would expect the estimates of θ1 and θ2 to be negative.
To assess the extent to which the EITP pilot differentially impacted the turnover of teachers with particular combinations of performance and experience characteristics— for example, teachers who are both low-performing and nontenured—we reestimate Model 2 separately for nontenured and tenured teachers. It is worth noting that disaggregating in this way results in modest sample sizes because very few teachers received low ratings and fewer than 25 percent of teachers are nontenured. However, this analysis allows for insight into the potential mechanism of evaluation-induced turnover; specifically, whether a teacher’s exit from an EITP school was voluntary or not. Given that tenured teachers in CPS enjoyed job protections unavailable to nontenured teachers, a differential impact of EITP on low-performing teachers by tenure status can shed light on whether the lowest-performing teachers left the school of their own volition or were induced to leave by the school’s principal.10
V. Results: The Impact of Teacher Evaluation on Turnover
A. Graphical Evidence and Main Effects
Figure 1 presents the turnover rates for all elementary school teachers in Chicago from the 2005–2006 to 2011–12 school years. The turnover rates—both the proportion of teachers who exit CPS as well as the proportion who switch schools within CPS—remain relatively stable. In a given year, approximately 80 percent of teachers return to their same school, while approximately 10 percent leave the district and 10 percent switch to another CPS school. Teacher mobility rates in Chicago are very similar to national averages. Indeed, of the approximately 3.4 million teachers nationally who were teaching during the 2007–2008 school year, 84.5 percent remained at the same school in the subsequent year (stayers), while 7.6 percent moved to a different school and 8.0 percent exited the teaching profession (Keigher 2010). Among the approximately 950,000 urban school teachers nationwide in 2007–2008, 84.5 percent were stayers, while 8.0 percent moved to a different school and 7.5 percent exited the teaching profession (Keigher 2010).
Teacher Mobility in All CPS Elementary Schools, 2005–2006 through 2010–11
Notes: Mobility is calculated as the proportion of teachers who stay in the same school, leave the district, or switch schools within the district at the end of the school year. For example, 2006 indicates the fraction of teachers who stay in their same CPS school, leave the district, or switch schools within CPS between the 2005–2006 and 2006–2007 school years. This figure includes all noncharter CPS elementary teachers, including teachers in the EITP experiment schools.
Figure 2a presents the unadjusted trends in teacher exit from the district for three groups of teachers—those in treatment schools, those in control schools, and those in all other CPS elementary schools.11 Prior to the introduction of EITP in 2008–2009, the pretreatment leave rates for teachers across all groups of schools are almost identical and they remain so after the implementation of EITP. The statistical balance among pretreatment exit rates from CPS for treatment and control schools is confirmed in Table 2, with 8 percent of treatment school teachers and 7 percent of control school teachers exiting CPS between the 2007–2008 and 2008–2009 school years, a difference that is not statistically significant.12 The graphical evidence suggests that, on average, there was no impact of EITP on teacher exit from CPS. This result is confirmed in a regression framework, adjusting for the unit of randomization (see Table 4). Table 4 presents three different specifications of Equation 1, and in all cases the coefficient on the EITP treatment indicator is neither statistically significant nor meaningful in magnitude.
Teacher Exit from CPS
Notes: Figure shows the proportion of teachers exiting CPS at the end of the school year for teachers in EITP treatment and control schools, as well as all CPS elementary school teachers in non-EITP schools. For example, 2006 indicates the fraction of teachers who exited CPS between the 2005–2006 and 2006–2007 school years. The vertical line indicates the period between the 2008–2009 and 2009–10 school years (Summer 2009)—the time period within which the first-year effect of EITP (if any) would be observed.
Impact of EITP on Teacher Turnover
Figure 2b presents the unadjusted trends in teachers switching schools within CPS by type of school—treatment, control, and non-EITP elementary schools. The pre- and post-treatment trends in switch rates appear to vary by treatment group. Just prior to the 2008–2009 school year, teachers in treatment schools were less likely to switch schools than teachers in control schools (this difference is statistically significant). We do not, however, believe this phenomenon was due to teachers reacting to the treatment before implementation. Indeed, schools were randomly assigned to EITP treatment status in July 2008, and principals were not trained until late July or early August 2008. Moreover, teachers did not receive training until in-service days the week prior to the first day of school in September. For teachers across all schools, switch rates were declining over this period, suggesting the teacher labor market in Chicago might have been changing over this time. Table 4 presents regression-adjusted estimates of the effect of EITP on teacher switching. There is no evidence of a difference between teachers in treatment and control schools in their likelihood of switching schools within CPS, conditional on area fixed effects. Teachers in treatment schools are, on average, 0.5 percentage points less likely to switch schools than control teachers, and this result is small in magnitude and not statistically significant. The result is robust to the inclusion of additional covariates. We further find that there is no differential impact of EITP on switching schools either by teacher tenure status or by a teacher’s prior evaluation rating. Therefore, we do not present the results for teachers switching schools (these results are available from the authors upon request).
Teachers Switching Schools within CPS
Notes: Figure shows the proportion of teachers switching schools within CPS between school years. For example, the end of the school year for teachers in EITP treatment and control schools, as well as all CPS elementary school teachers in non-EITP schools. For example, 2006 indicates the fraction of teachers who switch schools within CPS between the 2005–2006 and 2006–2007 school years. The vertical line indicates the period between the 2008–2009 and 2009–10 school years (Summer 2009)—the time period within which the first-year effect of EITP (if any) would be observed.
The findings on the main effects of the EITP pilot are not necessarily surprising, and in fact may be desirable in the context of new teacher evaluation systems. Specifically, policymakers and district leaders would likely be concerned if a teacher evaluation system induced turnover for the average teacher in the district, particularly in light of the aim of evaluation reform—to provide the majority of teachers with feedback to improve their practice while identifying and removing the lowest-performing part of the teacher performance distribution. Therefore, we now examine the impact of the EITP pilot on teacher turnover that evaluation reform should be designed to affect—teachers with low (prepolicy) evaluation ratings.
B. EITP Impact on Low-Performing Teachers
Table 5 summarizes the estimates of EITP on the exit rate of low-rated teachers, based on their prior year summative evaluation ratings.13 Since the excluded group in these regressions includes teachers with Unsatisfactory or Satisfactory prior-year evaluation ratings, the coefficient on the treatment indicator represents the leave rate for low-rated teachers in the treatment schools relative to low-rated teachers in the control schools. At the end of the 2008–2009 school year, 13 percent of control school teachers who received an Unsatisfactory or Satisfactory prior-year rating exited the district. For similarly low-rated teachers in the treatment schools, the regression-adjusted exit rate was approximately ten percentage points higher, representing an 80 percent increase in exit from CPS for teachers exposed to EITP in the first year of implementation. While this estimate is robust across multiple specifications (and multiple approaches for addressing missing prior year ratings data), it is only marginally statistically significant (see Model 3 in Table 5). However, while the effect of EITP on the leave rate of low-rated treatment school teachers is imprecisely estimated because very few teachers received low ratings, it is remarkably stable and large in magnitude. Although this finding should be interpreted with caution due to the small sample of low-rated teachers, it supports existing evidence that teacher evaluation reform has the potential to induce the exit of low-performing teachers (Dee and Wyckoff 2013). This result suggests that if a primary aim of teacher evaluation reform is to remove low-performing teachers from the classroom and the district, then the policy appears to have worked as intended in this context. We do not find these same effects for teachers who received high prior year ratings.
Impact of EITP on Teacher Exit from CPS, by Prior Evaluation Rating
As previously mentioned, we find no differential effects of the treatment by prior evaluation rating, on teachers switching schools. After one year of the pilot, it does not appear that teachers who had high prior evaluation ratings used the new information provided by the Danielson FFT ratings to differentiate themselves in the teacher labor market. It is possible that hiring principals (particularly those in non-EITP schools) may not have valued the information provided by the FFT ratings due to the pilot nature of the program. Further research is needed to understand the role that information generated by new teacher evaluation systems plays in intradistrict teacher transfers.
C. EITP Impact on Nontenured Teachers
In CPS, teachers in their first, second, or third year of teaching are nontenured. Once a teacher has attained tenure—those in at least their fourth year in a CPS classroom—he/she is guaranteed greater job protections, including the ability to grieve their assigned summative evaluation rating. As such, nontenured teachers are more vulnerable to layoffs that are likely involuntary in nature. They are also more likely to be the lowest performing in terms of learning gains, as evidence suggests that teachers in their first two years grow rapidly and then level off (Goldhaber 2002; Clotfelter, Ladd, and Vigdor 2007). These teachers also have less time invested in their school or CPS and may be assessing whether or not teaching is the right profession for them. For any of these reasons, we might expect that a teacher evaluation pilot aimed at providing principals and teachers with more information about instructional practice and teacher performance would differentially impact teachers at the beginning of their careers.
Table 6 summarizes the impact of EITP on exit for teachers who vary in their years of teaching service in CPS. On average, 11 percent of nontenured teachers in the control schools exited CPS at the end of the 2008–2009 school year. We find that the EITP evaluation reform increased the leave rate for nontenured teachers in the treatment schools by five to six percentage points, representing a 45–55 percent increase in the exit rate of nontenured treatment school teachers over their control school counterparts. This effect is robust and statistically significant across specifications. We do not, however, find that EITP increased the exit rate of more experienced CPS teachers. Moreover, we find no differential effects by years of experience of EITP on teachers switching schools. Generally, the more teaching experience one has, the less likely a teacher is to switch schools, and this result is consistent across teachers in treatment and control schools.
Impact of EITP on Teacher Exit from CPS, by Tenure Status
To lend further insight into the robustness of the EITP effect on the exit of nontenured teachers, we present graphical evidence on the trends in teacher exit by tenure status in Figure 3. For nontenured teachers, we find that the leave rates for those in treatment, control, and non-EITP schools are nearly identical in the 2006 and 2007 years. However, while the leave rates for treatment and non-EITP schools are the same in the prepolicy year (2008), the control school trend deviates from the non-EITP school trend. Moreover, what is particularly notable with respect to the estimated EITP effect for nontenured teachers is the control teachers’ deviation from non-EITP teachers’ leave rate trend between 2008 and 2009. Where the non-EITP leave rate is flat between 2008 and 2009, we find that exit from CPS increased between 2008 and 2009 for control teachers. This deviation among control teachers has direct implications for the counterfactual and the magnitude of the estimated EITP effect among nontenured teachers.
Teacher Exit by Tenure and Treatment Status
Notes: Figure shows the proportion of teachers in EITP treatment and control schools, as well as all CPS elementary school teachers in non-EITP schools, exiting CPS at the end of the school year. This figure includes all teachers in treatment, control, and non-EITP schools. The proportion of teachers exiting CPS is unadjusted for area fixed effects (the unit of randomization). However, the leave rates are nearly identical as those shown here when adjusting for area fixed effects.
Indeed, the graphical evidence suggests that nontenured teachers in the control schools may have reacted to the pending introduction of EITP in their schools (beginning in 2009–10) and differentially exited CPS at the end of the 2008–2009 school year. It is possible (though we do not have direct evidence of this) that control school teachers may have learned of EITP from their treatment school teacher colleagues located in the same instructional area. Much of the district’s professional development was administered through these areas, so teachers from treatment and control schools would have had opportunities to interact in the 2008–2009 school year—the first year of implementation for treatment schools and the year prior to implementation for control schools. If word-of-mouth led to nonrandom teacher exit from CPS, we would then expect to see differential leave rates between control school teachers and non-EITP teachers between 2008 and 2009. This is what we observe, and this nonrandom exit affects the counterfactual used to estimate the EITP effect for nontenured teachers. Therefore, the graphical evidence on the leave rate trends among control school teachers suggests that the estimated EITP effect on turnover among nontenured teachers may in fact be understated relative to the true effect.
D. Did EITP Differentially Affect Low-Performing Tenured and Nontenured Teachers?
As previously discussed, teacher evaluation under EITP provided both teachers and their principals with greater information about teacher performance, information that was largely private under the traditional teacher evaluation system in Chicago. This additional information about teacher performance, coupled with the fact that principals had dramatically more leverage over retention decisions for nontenured than for tenured teachers, suggests that the impact of EITP may have been centered largely on the lowest-performing nontenured teachers. Indeed, this is what we find.
Table 7 presents evidence on the impact of EITP on teacher exit from CPS for low-rated teachers by tenure status. We find no difference in the exit of low-rated tenured teachers in treatment and control schools, and this estimate is robust to multiple approaches for handling teachers who were missing prior-year evaluation ratings. In contrast, we find very large and highly significant impacts of EITP on the exit rates of low-rated nontenured teachers. Specifically, EITP increased the exit rate of low-rated nontenured teachers by 30 percentage points over their control teacher comparisons. We do note that the sample size of low-rated nontenured teachers is modest. Specifically, 4.3 percent (one of 23) of low-rated nontenured teachers in control schools exited CPS at the end of the 2008–2009 school year, while 34.8 percent (eight of 23) of low-rated nontenured teachers left treatment schools. While we acknowledge that these results should be interpreted with caution due to the modest sample sizes, the effect of EITP on the exit of low-performing nontenured teachers is highly significant and robust to multiple approaches for addressing missing prior-year evaluation ratings.14
Impact of EITP on Teacher Exit from CPS, by Prior Evaluation Rating and Tenure Status
These results not only suggest that low-rated nontenured teachers were significantly more likely to leave the district when observed and evaluated under EITP, but also provide insight into the nature of the exit decision. The exit of low-rated nontenured teachers represents some combination of voluntary and involuntary exits. Although we are unable to distinguish between voluntary and involuntary exits, the significant increase in teacher exit among nontenured teachers was likely influenced by two factors outside of a teacher’s control: (1) the additional information that EITP provided to principals about teacher performance through their careful observations of a teacher’s instructional practice, and (2) principals’ ability to fire nontenured teachers at will. Because of the administrative burdens associated with the removal of a tenured teacher from the classroom, evidence suggests that low-rated tenured teachers were not voluntarily leaving CPS as a result of teacher evaluation reform.
In light of the null findings on the main effect of teacher exit from CPS and the positive impact of EITP on teacher exit for particular subgroups—specifically, low-rated and nontenured teachers—Figures 4 and 5 summarize and decompose teacher exit from CPS across treatment and control schools, by prior evaluation rating and tenure status, respectively, at the end of the first year of the EITP pilot. In doing so, these figures illustrate where the off-setting reductions in teacher exit from CPS may be operating that, on balance, result in no main effect of EITP. Specifically, in Figure 4, there is a large difference in teacher exit among low-rated teachers with much smaller differences for teachers with Excellent or Superior prior evaluation ratings. In Figure 5, there is an approximately 50 percent difference in exit among nontenured teachers with a much smaller difference for tenured teachers. It is important to note that, given the smaller sample of low-rated (Unsatisfactory or Satisfactory) teachers relative to higher-rated (Excellent or Superior) teachers, the magnitude of the difference in teacher exit is nominal. This is also the case when examining the off-setting reduction in teacher exit by tenure status.
Teacher Exit Rates by Prior Evaluation Rating and Treatment Status, 2008–2009
Notes: Figure shows the proportion of teachers in treatment (Cohort 1) and control (Cohort 2) schools, either with available prior evaluation ratings or missing prior evaluation ratings, who exited CPS at the end of the 2008–2009 school year. The fraction of teachers exiting CPS at the end of 2008–2009 is unadjusted for teacher and school characteristics.
Teacher Exit Rates by Tenure and Treatment Status, 2008–2009
Notes: Figure shows the proportion of teachers in treatment (Cohort 1) and control (Cohort 2) schools, by tenure status, who exited CPS at the end of the 2008–2009 school year. The fraction of teachers exiting CPS at the end of 2008–2009 is unadjusted for teacher and school characteristics.
Finally, we examined the extent to which EITP led to differential exit for teachers in schools that served different student populations. Specifically, we examined whether teachers in EITP treatment schools exited the district or switched schools as a function of the achievement level of the school’s students or the share of students receiving free/reduced-price lunch. We do not find differential effects on teacher turnover by these school characteristics.15
VI. Teacher Labor Market Dynamics in the Context of Evaluation Reform
Given the significant exit of low-rated and nontenured teachers out of EITP treatment schools, we next examine the labor market implications of an evaluation policy on the distribution of teacher human capital. Specifically, we first compare the performance of teachers who exited EITP treatment schools to the performance of those teachers who remained in the EITP treatment schools. We then assess the evidence on whether EITP treatment school principals were successfully able to identify and hire higher-performing teachers to replace those teachers who had exited as a result of the EITP policy. This analysis aims to uncover the relationship between teacher evaluation and the distribution of teacher quality as measured by observer ratings of teachers’ instructional performance.
The extent to which an evaluation policy improves the overall quality of the employed teacher labor force is of critical concern to policymakers and was an important part of EITP principal training. See Sartain, Stoelinga, and Brown (2011) and Steinberg and Sartain (2015) for a detailed description of principal training under the EITP initiative. As part of their training for EITP implementation, principals were required to meet four times during the 2008–2009 school year for “half-day refreshers.” The content of the fourth of these sessions, which took place in March 2009, was dedicated to helping principals revise their interview questions for new teacher hires based on expectations of teacher instructional practice embedded in the Danielson FFT. As a result, EITP principals may have more strategically approached their hiring decisions, looking for applicants who exhibit the best instructional practices as outlined in the FFT. Tables 8–11 provide evidence that treatment schools had higher-performing teachers (with potentially more desirable observable characteristics) the year after EITP implementation.
Characteristics of Stayers, Leavers, Switchers, and Replacements in Treatment Schools
Danielson FFT Ratings for Stayers, Leavers, and Replacements in Treatment Schools
Comparing Danielson FFT Ratings of Stayers to Leavers/Switchers in Treatment Schools
Comparing Danielson FFT Ratings of Leavers/Switchers to Replacements in Treatment Schools
Table 8 summarizes characteristics of teachers who remained in EITP treatment schools (stayers) during the 2009–10 school year, those who exited the district at the end of the 2008–2009 school year (leavers), those who switched to a different CPS school for the 2009–10 school year (switchers), and those teachers who taught in an EITP treatment school for the first time during the 2009–10 school year (replacements). On average, the teachers who remained in EITP treatment schools were more likely to be Latino (and less likely to be white) than those who exited EITP treatment schools, were more likely to have majored in science as an undergraduate student, and were more likely to be nontenured, early-career teachers. Comparing the replacement teachers with those who exited EITP treatment schools, we find that the replacements are also more likely to be Latino, early-career, and have science majors. The fact that EITP schools retained and recruited more science majors is notable, given the evidence that a teacher’s content area knowledge is positively related to student performance, with greater achievement returns for teachers with coursework and certification in math and science (Goldhaber 2002, Darling-Hammond and Sykes 2003).
We next explored whether the quality of teachers who remained in and entered EITP treatment schools differed from those teachers who no longer taught in EITP treatment schools during the 2009–10 school year. We first show the principals’ ratings of teachers on the Danielson FFT as the measure of teacher quality.16 Table 9 provides descriptive evidence that the teachers who stayed in their schools were of higher quality, as measured by the Danielson FFT ratings, than those teachers who exited EITP treatment schools; this provides additional support that EITP induced lower-performing teachers to exit the treatment schools at the end of the 2008–2009 school year. The differences in principal-assigned FFT ratings by turnover status (stayers and leavers/switchers) provide a validation check on whether the evaluation ratings under the prior evaluation system (particularly teachers rated Unsatisfactory or Satisfactory compared to those with Excellent or Superior ratings) captured differences in teacher performance. Indeed, we find that those teachers who stayed in their schools had much higher Danielson FFT ratings than either those teachers who switched schools or left CPS, and these differences are statistically significant. On average, stayers had a rating of 2.98—this would be the equivalent of receiving Proficient ratings for all components (on a four-category scale of Unsatisfactory, Basic, Proficient, and Distinguished). Teachers who switched schools or left the district had an average FFT rating of 2.5. Approximately half of these teachers’ component ratings were Proficient and the other half of their component ratings were at the Basic level. This difference is equivalent to approximately 0.8 standard deviations; in terms of the language of the Danielson FFT, this is a meaningful difference in teacher quality. Table 9 also provides evidence that the teachers who replaced those who exited EITP treatment schools were, on average, higher-performing as measured by the principal-assigned Danielson FFT ratings. On average, replacement teachers in the EITP treatment schools had an average FFT rating of 2.82 compared to the average FFT rating of 2.5 for those teachers who switched schools or left the district at the end of the 2008–2009 school year. This difference is highly significant and is equivalent to a difference of approximately one-half of a standard deviation in FFT performance. Taken together, this evidence from Table 9 suggests that EITP schools improved the overall distribution of teacher quality.
Of course, it is possible that principals’ ratings of their teachers are endogenous since they may be used to exert pressure on teachers to exit their schools, and principals may be more likely to give higher ratings to the teachers they hired (the replacements). Principals responsible for observing and rating teachers also determine (to the extent possible) whether teachers return in the subsequent school year. A principal’s effort to remove a teacher certainly will influence the objectivity of the observation ratings provided to teachers. Therefore, to overcome the potential endogeneity of principal-assigned ratings while also exploring the robustness of principals’ ratings of stayers compared to leavers and switchers, we exploit the fact that a subset of treatment school teachers in 2008–2009 were randomly selected to be observed and rated by externally trained observers.17 Of the 1,248 treatment school teachers, approximately 12 percent were rated by external observers in the first year of EITP.18 Using the external observer-assigned FFT ratings, we again find that the stayers were significantly higher-performing than those teachers who exited EITP treatment schools at the end of the 2008–2009 school year, though the magnitude of this difference is smaller than the difference in principal-assigned ratings (shown in the bottom panel of Table 9).
Because Table 9 provides unadjusted mean differences in teachers’ FFT ratings, we next estimated a series of regressions to better account for the relationship between the characteristics of teachers in the EITP treatment schools and their FFT ratings. To compare the observed performance (measured by the Danielson FFT) of those teachers who remained in EITP treatment schools to those who exited, we estimate variants of the following model:
(4)
where FFTis is teacher i’s Danielson rating in school s, a composite measure of teacher performance based on multiple FFT observation ratings. The variable Ti represents a vector of teacher characteristics, including teacher demographics (age, race, gender), experience (years teaching in CPS), certification status, and educational attainment. The variable θs is a school fixed effect (so that all comparisons are made between stayers and the leavers/switchers within the same school), and εis is a random error term. The parameter of interest is α1, which estimates the difference in FFT ratings between teachers who left school s at the end of the 2008–2009 school year and those that stayed in school s for the 2009–10 school year. If the observed instructional practice of the teachers who stayed was, on average, higher than the teachers who left school s, then α1 will be positive.
Table 10 summarizes these results. We find that those teachers who stayed in EITP treatment schools had significantly higher FFT ratings than those who exited EITP treatment schools. Using the principal-assigned ratings (see Panel A), we find that the estimated difference is 0.33 FFT points, equivalent to 0.57 standard deviations in teacher performance, as measured by the Danielson FFT. This estimate is robust to the inclusion of teacher characteristics and when all comparisons are made among stayers and leavers within the same EITP treatment school. When using the ratings assigned to a random sample of EITP treatment school teachers by external observers (see Panel B), the magnitude of the difference between teachers who stay and those that exit their schools is 0.16. What is particularly notable is that 0.16 FFT points represents a 0.53 standard deviation effect size—nearly identical to the difference in performance, as measured by principals, between teachers who stayed and those who left EITP treatment schools at the end of the 2008–2009 school year.
We next compared the observed performance, measured by the Danielson FFT, between teachers who entered EITP treatment schools in 2009–10 and those who exited these schools at the end of the 2008–2009 school year. To do so, we estimate variants of the following model:
(5)
where Replacementis is equal to 1 if teacher i entered an EITP treatment school s in 2009–10, and 0 otherwise. We include the variable PrincipalChangeis to account for any changes in school leadership, particularly since principals in EITP schools were responsible for assigning Danielson FFT ratings. This variable equals 1 if teacher i’s principal was in his/her first year in school s, and 0 otherwise. All other variables are defined as in Equation 4. The parameter of interest is β1, which estimates the difference in FFT ratings between teachers who left school s at the end of the 2008–2009 school year and replacement teachers who entered school s in the 2009–10 school year. If the observed instructional practice of the replacement teachers was, on average, higher than the teachers who left school s, then β1 will be positive.
Table 11 summarizes these results. We find that those teachers who entered EITP treatment schools (and had available FFT ratings information) in the 2009–10 school year were higher-performing than those who exited EITP treatment schools (with available FFT ratings information) at the end of the 2008–2009 school year. The estimate is statistically significant and robust to the inclusion of school fixed effects and conditional on a teacher’s National Board certification status—on the order of one-third of a standard deviation.19
An important caveat regarding the use of classroom observation scores to distinguish performance differences among stayers, leavers, and their replacements relates to the selection of teachers by school principals for official submission of FFT scores. As previously described, EITP was low-stakes, and principals were not required (though encouraged) to record and submit FFT observation ratings of their teachers to CPS central office personnel. It is certainly plausible that principals were more likely to report ratings for teachers who they believed to be lower-performing. Indeed, selection of this nature would lead to a downward bias in the estimated differences in measured performance between stayers and leavers presented in Table 9 (and, to a lesser extent, Tables 10 and 11), particularly if principals are systematically less likely to report FFT scores for higher-performing stayers. If the FFT ratings of teachers who leave CPS are compared to lower-performing teachers who stayed in EITP treatment schools, we actually may be understating the true differences and relative improvements in the distribution of teacher quality among the EITP treatment schools.
To better understand the nature of this potential bias, we explore the extent to which FFT ratings were missing for treatment school teachers in 2008–2009. Figure 6 summarizes the distribution of treatment school teachers’ prior evaluation ratings by tenure status and by whether their principals submitted FFT ratings for the 2008–2009 school year. For nontenured teachers, the distribution of prior evaluation ratings for those who have FFT ratings and for those who are missing FFT ratings is almost identical. Indeed, whether or not a principal entered FFT ratings for a nontenured teacher appears to be independent of the teacher’s prior evaluation rating. However, the story is different for tenured teachers. Tenured teachers with FFT ratings were much more likely to have received a Satisfactory rating prior to EITP, while tenured teachers without FFT ratings were more likely to receive an Excellent or Superior rating prior to EITP. Specifically, 15 percent of tenured teachers with FFT ratings received a Satisfactory rating while 73 percent received either an Excellent or Superior rating; for tenured teachers without FFT ratings, only 2 percent received a Satisfactory rating while 82 percent received either an Excellent or Superior rating. The difference in the distributions of prior ratings for tenured teachers with and without FFT ratings suggests that estimates of improvements in teacher quality due to teacher exit—that is, the difference in FFT ratings between stayers and leavers—are likely biased downward.
Distribution of Prior Evaluation Ratings by Tenure and Availability of FFT Ratings
Notes: Figure shows the distribution of prior evaluation ratings for teachers in treatment (Cohort 1) schools in 2008–2009 by tenure status and the availability of FFT ratings from classroom observations conducted by school principals. No teachers in the treatment schools were rated Unsatisfactory prior to implementation of EITP. In 2008–2009, FFT ratings are available for 65 percent of nontenured teachers and 18 percent of tenured teachers in Cohort 1 schools. Of tenured teachers who had low prior evaluation ratings (Unsatisfactory or Satisfactory), 63 percent have FFT ratings. The 2008–2009 school year was an off year for tenured teacher evaluations, so central office only asked principals to submit FFT ratings for nontenured teachers. Because of the low-stakes nature of EITP, central office asked but did not require principals to enter ratings, so FFT ratings data do not cover all teachers in the sample. Two percent of tenured teachers without FFT received a Satisfactory rating.
VII. Do Specific Teacher Practices Shape Labor Market Decisions?
Principals may value certain classroom practices over others, while teachers who are more proficient at certain dimensions of classroom instruction and management may be less likely to leave teaching. Although we’re unable to distinguish between voluntary and involuntary teacher exits, particularly among the nontenured teachers, we can assess the extent to which specific teacher practices shape teacher exit from CPS. To do so, we begin by estimating a series of regressions where we regress an indicator for whether or not a treatment school teacher exited CPS at the end of the 2008–2009 school year on the principal-assigned FFT ratings from each of the ten FFT components across two domains of teacher performance—Classroom Environment (Domain 2) and Instruction (Domain 3). Domain 2 includes the following teacher practices: (a) creating an environment of respect and rapport, (b) establishing a culture for learning, (c) managing classroom procedures, (d) managing student behavior, and (e) organizing physical space. Domain 3 includes the following teacher practices: (a) communicating with students, (b) using questioning and discussion techniques, (c) engaging students in learning, (d) using assessment in instruction, and (e) demonstrating flexibility and responsiveness. See Sartain, Stoelinga, and Brown (2011) for a detailed description of the FFT observation rubric used in CPS. We then aggregate the FFT components into domain-level measures by averaging the underlying components (five each for the Classroom Environment and Instruction domains) and regress the teacher exit indicator on these two domain-level averages.
Table 12 summarizes these results, which we disaggregate by tenure status since nontenured teacher exit is more likely involuntary due to the contract protections offered to tenured teachers. Results in Columns 1–3 are for tenured teachers and Columns 4–6 are for nontenured teachers. Each column includes a different set of covariates. Columns 1 and 4 are the unadjusted regression results, while Columns 2 and 5 add school fixed effects to compare teachers within the same school. Columns 3 and 6 include controls for teacher characteristics. The coefficients can be interpreted as percentage point changes in the likelihood of exiting CPS associated with a 1-point increase in classroom observation ratings. For example, tenured teachers in treatment schools who scored 1 point higher on the Classroom Environment domain (from Basic to Proficient on a four-category scale—Unsatisfactory, Basic, Proficient, and Distinguished) were, on average, 9.1 percentage points less likely to leave CPS at the end of the 2008–2009 school year (see Column 1).
Examining the Relationship between Instructional Practices and Teacher Exit
When we restrict comparisons to teachers located in the same EITP treatment schools and thus rated by the same principals, a tenured teacher’s domain-level ratings are statistically uncorrelated with the probability of exiting CPS (see Columns 2 and 3). In addition, there is little evidence that individual teacher practices across the two domains are statistically related to the exit of tenured teachers. Given earlier evidence on the null effect of EITP on the exit of lower-rated tenured teachers, these results are not unexpected. Indeed, this evidence supports the argument that tenured teachers are less likely to leave the classroom as a result of teacher evaluation reform that is focused on improving classroom instruction and performance rather than on increasing teacher accountability.
In contrast, we find clear evidence that nontenured teachers with better instructional performance were less likely to exit an EITP treatment school (see Columns 5 and 6). Nontenured teachers rated Basic on the Instruction domain were 12–16 percentage points more likely to exit CPS than a nontenured teacher (in the same school) rated Proficient on instructional performance. Moreover, two teacher instructional practices emerge as particularly salient for shaping teacher exit decisions. First, nontenured teachers who were better able to communicate with their students (by conveying to students what they expected them to learn, providing students with clear directions for classroom activities, presenting complex concepts so that they were accessible to students, and linking concepts to students’ interests and prior knowledge) were less likely to exit their schools. Second, nontenured teachers who used better questioning and discussion techniques— techniques aimed at deepening student understanding—were also less likely to exit CPS. This evidence suggests that additional information about teacher practice revealed to principals and communicated to teachers is related to the labor market decisions of nontenured teachers. Indeed, ratings information generated from classroom observation of teacher practice is a key mechanism informing teacher exit, particularly when principals’ employment decisions are unrestricted by teacher labor contracts and novice teachers are updating their prior beliefs about their fit with the teaching profession.
VIII. Discussion
EITP represented a dramatic departure from how observations of teachers’ instructional practice were historically conducted in Chicago. Principals were trained to carefully document evidence of a teacher’s instructional quality and to supplement classroom observation notes with direct quotes from teachers and their students. Principal observations were no longer informed by a checklist of teacher behaviors but rather a detailed rubric—based on the Danielson FFT—that delineated different levels of teaching quality. These observations provided principals and teachers with more detailed information about a teacher’s instructional performance. Another important component of the intervention were pre- and post-observation conferences during which principals provided teachers with their Danielson FFT ratings and discussed avenues for improvement. As a result, EITP involved two major shifts in the way teacher performance evaluations were conducted in Chicago. First, using the Danielson FFT meant conferences between principals and teachers were structured and focused on instruction. Second, teachers received ongoing, detailed information throughout the school year about their performance in ten different areas of teaching practice rather than simply receiving a summative evaluation rating at the end of the school year.
In this paper, we examine teachers’ labor market responses in the context of teacher evaluation reform, assessing the extent to which additional information about teacher performance under EITP induced teachers to transfer out of their schools to another CPS school or to exit the district entirely. Because this paper focuses on a pilot evaluation system, we cannot explicitly estimate the impact that this pilot evaluation system may have on a districtwide level under a full-scale policy rollout. However, our findings are suggestive of the effect of evaluation reform for all CPS teachers, given how similar the EITP and non-EITP teachers are on observable characteristics. Moreover, these results provide among the first evidence on the role teacher evaluation reform can play in inducing the exit of low-performing teachers and improving the quality of the teacher labor force.
While there was no main effect of EITP on teacher exit (or transfer) for the average CPS teacher, we find that the EITP evaluation reform generated large increases in teacher exit from the district for two groups of teachers—the lowest-performing and the nontenured teachers. Specifically, these effects were concentrated among those nontenured teachers who also had low prior evaluation ratings. We do not find that low-rated tenured teachers, who were the beneficiaries of strong contract protections, exited the classroom as a result of this low-stakes evaluation system. This finding is particularly relevant in light of ongoing policy debates around teacher tenure and a recent prominent court ruling in California (Vergara v. California), which struck down that state’s near half-century-old law governing teacher tenure and other job protections. Following the California decision, advocates of tenure reform in New York are currently filing a lawsuit claiming that the state’s tenure laws protect low-performing teachers and deny students a sound, basic education (Baker 2014). The results in this paper suggest that tenure reform may be necessary to induce low-performing tenured teachers to leave the profession. In fact, as CPS implemented its new evaluation system districtwide in 2012–13, the district-union contract mandated that, when teacher layoffs occur, the lowest-performing teachers may be removed regardless of tenure status.
We also find that teachers who remained in the treatment schools after the first year of EITP implementation received higher Danielson FFT ratings than those who exited. Moreover, the teachers who replaced the exiting teachers were also higher-performing. We do not find that teachers were more likely to switch schools after implementation of EITP or that teachers disproportionately exited more disadvantaged treatment schools. Taken together, the evidence presented in this paper along with evidence from Washington D.C. (see Dee and Wyckoff 2013) suggests that teacher evaluation reform has the potential to improve the distribution of teacher quality.
EITP was officially low-stakes for teachers in the treatment schools because Danielson FFT ratings were not officially accounted for in teachers’ summative evaluation ratings. Despite that, principals were more present in teachers’ classrooms, and the extent of ongoing feedback teachers received on their instructional performance increased markedly compared to just the prior school year. As a result, the quality of a teacher’s instructional performance, which in the past had extensively been private information, now became more apparent to the school’s principal. In particular, EITP provided principals with the tools to be more savvy evaluators. The additional information about a teacher’s performance generated under EITP allowed principals to more reliably assess and differentiate teacher quality and likely informed their decisions about teacher retention. Although we are unable to determine the mechanism(s) through which teacher exit from the district operated—specifically, whether a teacher’s exit was voluntary or involuntary—our findings suggest that, in light of the newly revealed information about teacher performance, job protections guaranteed to tenured teachers appear to have sufficiently insulated them from the type of job displacement experienced by their nontenured counterparts.
Indeed, evidence on the impact of EITP on low-rated nontenured teachers suggests treatment school principals may have more actively used this additional information when making high-stakes personnel decisions. In fact, Jacob (2011) shows that after the Chicago Teachers’ Union contract changed to allow principals to fire nontenured teachers at-will, principals let go of teachers who had lower value-added measures than the nontenured teachers whose contracts were renewed, suggesting that principals use information about performance in dismissal decisions. However, a principal’s ability to fire tenured teachers was largely constrained. While nontenured teachers had very few job protections—principals could remove a nontenured teacher at the end of the school year simply by checking a box for termination in the human resources database—the dismissal of tenured teachers required a detailed and lengthy process. Despite the employment protections available to tenured teachers in Chicago, we do find that the overall quality of the teachers in treatment schools improved as a result of the EITP initiative. Notably, treatment school teachers who remained were higher performing, as measured by the Danielson FFT ratings, than those who exited, and the replacement teachers were also higher performing.
In the years since the EITP evaluation pilot was first implemented in Chicago, states and districts across the nation have embarked on ambitious efforts to reform teacher evaluation systems. In the wake of the federal RTTT initiative, these reforms have been designed to inject greater accountability into teacher evaluation systems through the use of multiple measures of teacher performance, of which classroom observation scores and VAMs are prominent features, as well as pre- and post-observation conferences between teachers and their principals (Steinberg and Donaldson 2014). In light of practical constraints limiting the use of VAMs to the minority of teachers teaching in tested grades and subjects, classroom observation scores will remain the most important measure for evaluating teachers, even among newly developed high-stakes evaluation systems. So, while the EITP pilot initiative focused entirely on structured classroom observations of teacher practice in a low-stakes context, most newly implemented evaluation systems continue to rely heavily on principal observations of teachers’ classroom practices. As a result, evidence presented herein on the effect of a low-stakes evaluation system, based entirely on observations of teacher practice, should inform policymakers and education researchers on the potential effect that newly developed and implemented evaluation systems may have on the labor market consequences of teacher evaluation reform.
Teacher evaluation reform was designed to serve two goals: improve the quality of a teacher’s instruction and remove low-performing teachers from the classroom. This paper provides evidence that teacher evaluation reform that focuses on observing and measuring the instructional performance of teachers, providing detailed and ongoing feedback to teachers on their practice, and offering principals training to improve hiring practices, has the potential to improve the overall quality of teachers and teaching in our nation’s schools.
Footnotes
Lauren Sartain is a researcher at the University of Chicago Consortium on Chicago School Research. Matthew P. Steinberg is an assistant professor at the Graduate School of Education, University of Pennsylvania. The authors thank the Chicago Public Schools for continued data access and support. They also thank two anonymous referees, workshop and conference participants at the University of Chicago, the University of Pennsylvania, the Association for Education Finance and Policy, the Association for Public Policy and Management, the Society for Research on Educational Effectiveness, researchers at the University of Chicago Consortium on Chicago School Research, Dan Black, Ofer Malamud, and Kerwin Charles for helpful comments and discussions. The authors are listed in alphabetical order. Sartain (corresponding author) can be contacted at lsartain{at}uchicago.edu; Steinberg can be contacted at steima{at}gse.upenn.edu.
↵1. Text from Arne Duncan’s September 30, 2013 speech at the National Press Club can be found at: http://www.ed.gov/news/speeches/beyond-beltway-bubble.
↵2. While much attention has been paid to the incorporation of student test score data in newly developed evaluation systems, upward of 70 percent of teachers nationwide teach in untested grades and subjects and therefore will not have a VAM as a component of their overall summative evaluation score (Prince, Schuermann, Guthrie, Witham, and Milanowski 2009).
↵3. See Steinberg and Sartain (2015) for a detailed discussion of the design and purpose of the EITP pilot.
↵4. Under the checklist system of teacher evaluation, tenured teachers rated Excellent or Superior were rated every two years (rather than annually), and probationary (nontenured) teachers were evaluated annually (The New Teacher Project 2007). See Sartain et al. (2011) for a copy of the checklist as well as the complete Danielson Framework.
↵5. We have two years of summative evaluation ratings prior to 2008–2009. To create the summative rating variable, we assign a teacher his/her 2007–2008 summative rating as the prior evaluation. If 2007–2008 is missing, we use the 2006–2007 summative evaluation rating. If a teacher has neither a 2006–2007 nor a 2007–2008 rating, then the teacher is missing prior evaluation rating. Of the 2,711 teachers in our sample, 74 percent are assigned their 2007–2008 rating, 5 percent their 2006–2007 rating, and 21 percent are missing. Moreover, our ability to examine trends in turnover by teacher performance is limited in that we do not observe teachers’ formal evaluation ratings in years after the 2007–2008 school year.
↵6. Of the 2,711 teachers in EITP treatment and control schools in the 2008–2009 school year, 566 teachers were missing evaluation ratings from the prior school years (2006–2007 and 2007–2008). We address the missingness in two ways: (a) include an indicator variable for teachers missing evaluation ratings, and (b) drop teachers with missing evaluation ratings data from the sample. Our empirical findings are robust to these approaches for handling missing prior evaluation ratings.
↵7. CPS is the third largest school district in the country. As such, the district is divided into subdistricts, which at the time of the pilot were called “instructional areas.” While not chosen at random, the four (of 17) areas were chosen to represent the broader population of CPS schools. The four areas served students on the north side of Chicago in Uptown and Edgewater, the Lawndale community and nearby neighborhoods on the west side, the State Street corridor and Englewood, and the far south side.
↵8. Schools with first-year principals and those identified for closure at the end of the 2008–2009 school year were excluded from the sample prior to randomization.
↵9. While we lose the experiment in 2009–10 when the control schools implement EITP for the first time, there is evidence that fidelity of implementation was low in the control schools due to new district leadership. Furthermore, the program was cut under the new administration in 2010. Interestingly, the observation tool discussed in this paper is now part of the state-mandated teacher performance evaluation system in Illinois as well as the teacher evaluation system being implemented in Chicago. See Steinberg and Sartain (2015) for a detailed description of the school-level implementation of EITP for both Cohort 1 and Cohort 2 schools.
↵10. The extent to which a CPS principal could fire a teacher was dictated entirely by the district-union contract. Nontenured teachers had very few job protections. Indeed, principals could remove a nontenured teacher at the end of the school year simply by “clicking them off”—checking a box for termination in the human resources database. However, dismissal of tenured teachers required a detailed and lengthy process—principals first had to assign the tenured teacher an Unsatisfactory rating, then, the principal had to identify a consulting teacher who was approved by the union within 30 days of assigning the Unsatisfactory rating, then, the principal, consulting teacher, and Unsatisfactory-rated teacher together designed a 90-day remediation plan. Over the 90-day period, the principal was responsible for conducting two observations of the teacher’s classroom practice; if the teacher’s Unsatisfactory rating remained unchanged, the teacher could appeal the Unsatisfactory rating, a process that could take many months (if not years) to resolve. (Agreement Between the Board of Education of the City of Chicago and the Chicago Teachers Union, Local Number 1, American Federation of Teachers, AFL-CIO, July 1, 2007 – June 30, 2012.)
↵11. Note that we exclude charter schools and special education or alternative schools from the sample of elementary schools.
↵12. The pretreatment (2008) leave rates are also statistically equivalent for teachers in non-EITP schools compared to EITP teachers.
↵13. We again note that summative evaluation ratings are only available from the 2006–2007 and 2007–2008 school years and only for EITP teachers who taught in CPS during the 2008–2009 school year. Since the composition of EITP teachers changes (and changes differentially for treatment and control school teachers) in school years after 2008–2009, we are unable to show trends in teacher turnover by prior performance evaluation ratings across treatment and control schools.
↵14. It is worth noting that 34 percent of nontenured teachers did not have prior evaluation ratings since many teachers in this group were in their first year in the classroom. The leave rate at the end of the 2008–2009 school year among all nontenured teachers regardless of prior evaluation rating was 11 percent.
↵15. Heterogeneous effects by school characteristics are statistically insignificant and small in magnitude. The coefficient on the interaction of treatment status and prior school-level reading score is −0.0002 (p-value = 0.836), and the coefficient on the interaction of treatment status and prior school-level math score is −0.0005 (p-value = 0.711). When interacting treatment status with the percent of students in a school qualifying for free/reduced-price lunch, the coefficient is −0.013 (p-value = 0.761).
↵16. Teachers in EITP treatment schools received multiple classroom observations, and therefore received FFT ratings for one of ten individual FFT components for multiple observations. We follow Kane, Taylor, Tyler, and Wooten (2011) and Garrett and Steinberg (2015) and create a composite measure of teacher performance based on multiple FFT observation ratings. Formally, we calculate the FFT rating score for teacher i as follows:
, where is the score (on an integer scale of 1–4) for the lth component (of 10 possible FFT components) of the jth classroom observation. The term represents the average FFT score for the jth classroom observation. Note that a simple average of all FFT components (across multiple classroom observations) will be equivalent when there are no missing component ratings.↵17. See Sartain, Stoelinga, and Brown (2011) for more detail on the random assignment of treatment school teachers to external observer/raters.
↵18. Treatment school teachers were not randomly selected for observation and rating by external observers in the 2009–10 school year. As such, we do not include the external observer-assigned Danielson FFT ratings to compare the instructional performance of replacement teachers to leavers/switchers found in Table 11.
↵19. We again note that external observers were not assigned to a random sample of treatment school teachers in the 2009–10 school year, so we omit estimates of teachers’ instructional performance using ratings provided by the external observers.
- Received May 2014.
- Accepted December 2014.
References
In this issue
Jump to section
- Article
- Abstract
- I. Introduction
- II. Teacher Evaluation, Turnover, and the Chicago Context
- III. Data
- IV. Empirical Strategy
- V. Results: The Impact of Teacher Evaluation on Turnover
- VI. Teacher Labor Market Dynamics in the Context of Evaluation Reform
- VII. Do Specific Teacher Practices Shape Labor Market Decisions?
- VIII. Discussion
- Footnotes
- References
- Figures & Data
- Info & Metrics
- References