It should be obvious even if it sometimes gets lost amidst the noise: The measure of a policy is not its intent but its impact. That’s why I’ve been tough on statewide teacher evaluation reforms. The point of these systems is sensible and seems geared toward attracting and retaining committed professionals, while helping struggling teachers to improve or find new work. In practice, though, what teacher evaluation has mostly meant is a lot of time and expense, a big hassle for teachers and principals, trivial change in how teachers are actually rated, and a raft of energy-sucking paperwork burdens. One can say all of that without questioning the motives of those who championed these systems and while continuing to believe that the idea is a fine one—on paper.
The problem is that policy doesn’t play out on paper. It plays out in the real world. As I observed last year in Letters, “Policy is a blunt tool that works best when making people do things is enough . . . Policy is far less effective when it comes to complex endeavors where how things are done matters more than whether they’re done. This is because policy can’t make schools or systems adopt reforms wisely or well.” And the consequences of reform done unwisely can ripple out in ways that we don’t fully appreciate until later.
All of which brings us to a new paper by Brown University’s Matt Kraft, UConn’s Eric Brunner and Shaun Dougherty, and Syracuse’s David Schwegman. Kraft is the guy who documented a couple years ago (along with Temple’s Allison Gilmour) that teacher evaluation reforms were having almost no impact on the share of teachers rated effective. He found that, in nearly every state, the share barely moved—because principals didn’t want to pick fights, didn’t trust that low-rated teachers would actually be removed, and so on. In the end, except in a handful of locales, these practical considerations managed to undo all the passion, energy, and political capital that had been poured into promoting these policy changes.
Now Kraft et al. have just released “Teacher Accountability Reforms and the Supply of New Teachers,” which examines the impact of these reforms on teacher recruitment. After all, for those of us who’ve long embraced rethinking teacher evaluation and pay, a key argument is that well-designed systems will make teaching more attractive to young, motivated professionals. So, have the systems that states adopted in recent years helped boost recruitment?
By leveraging the different rates at which states adopted teacher evaluation reform, Kraft et al. try to isolate the impact of these changes. After estimating that new teacher supply has declined by more than 20 percent between 2007 and 2016, they calculate that implementing high-stakes evaluation reform and repealing tenure shrunk the teacher labor supply. They report that the pool of individuals earning licenses fell, on average, “15 percent in states that adopted evaluation reforms and 16 percent in states that repealed tenure.” The researchers elaborate, explaining, “Evaluation reforms appear to result in a steady decline in new labor supply over time whereas tenure reforms result in a sharp and immediate contraction in the supply of new teachers that then slowly rebounds.”
Of course, as the researchers note, a shrinking applicant pool is just fine if it’s shrinking because the reforms are chasing away mediocre candidates (and it’s not such a big deal if it’s simply discouraging those who would apply for easy-to-fill positions). So, is that what’s been happening? Nope. The authors write, “We find no evidence that decreases in labor supply differed systematically across non-shortage versus shortage licensure areas.” The biggest reductions, for instance, were seen among secondary English (a non-shortage area) and science teachers (a shortage area). There was no clear effect on teachers in terms of where they had gone to college or what their qualifications were.
In other words, there’s no evidence that these teacher evaluation reforms have attracted talented applicants. Instead, they seem to have dissuaded new teachers—promising and less so—in equal measure. Exactly why that occurred is an important question, and one that should remind us, once again, of why we ought to spend a lot more energy examining the dynamics of how system reforms actually play out in early adopters, before rushing to insist that they “work” or to mandate them through policy.
— Frederick Hess
Frederick Hess is director of education policy studies at AEI and an executive editor at Education Next.
This post originally appeared on Rick Hess Straight Up.
