This is the fourth article in a series that looks at a recent AEI paper by Collin Hitt, Michael Q. McShane, and Patrick J. Wolf, “Do Impacts on Test Scores Even Matter? Lessons from Long-Run Outcomes in School Choice Research.” The first three essays are here, here, and here.
Over the past few blog entries, I’ve shown that the overwhelming majority of studies looking at bona fide school choice programs find positive effects on both short-term test scores and long-term student outcomes, especially college-going and college graduation. It is simply incorrect to claim, as the AEI authors did, that “a school choice program’s impact on test scores is a weak predictor of its impacts on longer-term outcomes.”
Still, there are a handful of examples of school choice programs that diminished achievement but improved high school graduation rates, including the Milwaukee voucher program and a set of Texas charter schools. I agree with Hitt, McShane, and Wolf that it would be a mistake for policymakers to prematurely kill off initiatives such as these, which may be helping students in ways not captured by reading and math scores.
But does that mean we shouldn’t close individual schools that persistently hurt student achievement? Including charter schools? The AEI authors seem to think so. Jumping from their findings on programs and extrapolating them to schools, they write: “If test score gains are neither a necessary nor a sufficient condition for producing long-term gains in crucial student outcomes, then current approaches to accountability for school choice programs are questionable at best.…Focusing on test score gains may lead regulators to favor schools whose benefits could easily fade over time and punish schools that are producing long-lasting gains.”
Let me explain what’s wrong with this logic.
1. Nobody measures high schools by their test scores alone
Hitt, McShane, and Wolf seem to think that “regulators” are closing charter schools, or turning off the public funding to private schools participating in voucher programs, based merely on low test scores. To my knowledge, that is never the case, especially for high schools, because every state accountability system and every charter school authorizer looks at high school graduation rates, too. And of course the Every Student Succeeds Act requires those graduation rates to be baked into high school ratings. So an individual high school would need both low test scores and graduation rates to be at risk of landing in a “regulator’s” bullseye. One with strong graduation rates would be fine.
2. Holding elementary or middle schools accountable for long-term impacts is not practical
But what about elementary or middle schools that diminish student achievement, yet somehow increase their pupils’ odds at graduating from high school many years later? Should we worry about regulators “punishing” such schools? Shouldn’t schools in those programs get a reprieve?
This is where we have to be careful to distinguish between programs and schools. The AEI team reviewed studies about the former, not the latter. We know from those studies that a handful of programs as a whole worsened achievement but improved graduation rates (a set of Texas’s charter schools, for example). What we don’t know is whether there was a similar mismatch at the school level. We don’t whether there were any, say, charter middle schools in Texas that reduced achievement but increased their students’ odds at graduating from high school. We simply don’t know.
Hitt, McShane, and Wolf would err on the side of leniency for schools with poor achievement, holding out hope that they might be helping their students long-term in ways not captured by test scores. This, they note, is particularly salient because these are schools of choice, and thus determined by parents to be better options than the alternatives.
There’s a case to be made that regulators should never overrule the wishes of parents, but that case is not informed by the evidence the AEI authors reviewed. There is simply nothing in the data to suggest that individual schools that diminish achievement are improving attainment. Nothing. So let’s have the debate on moral or philosophical or policy-preference terms, rather than pretending that empirical evidence signals the proper outcome.
And in my view, we shouldn’t grant low-achieving elementary and middle schools a reprieve while we wait to see whether they improve long-term outcomes, because by the time we had proof, the school in question would be years older. It might be a completely different place. We couldn’t possibly close a school based on how it affected an earlier generation of students. We must either hold elementary and middle schools accountable for the outcomes of current students or not at all.
***
Hitt, McShane, and Wolf are right to worry about promising programs and schools getting shuttered before they have a chance to prove themselves over the long term. High quality charter authorizers address this concern by doing a comprehensive review of schools before deciding whether to close them. If there are reasons to believe that test scores don’t capture all the value the school is providing, its leaders have ample opportunities to make their case to officials. As a charter authorizer myself, I can say that we are eager to be persuaded by such evidence. And as Andy Smarick has argued, voucher programs need something akin to authorizers, too, so that decisions about participating schools can be informed by nuance and human judgment, not just by test scores and other data points. (Maybe that’s what Indiana is currently stumbling into.)
But the charter schools that do get closed are not, in my view and experience, diamonds in the rough. They are campuses with depressingly low pupil achievement, poor student-level growth, and very little successful teaching and learning in their classrooms. They are places where authorizers can see with their own eyes that weak instruction, mediocre curricula, poorly prepared or demoralized teachers, and often well-meaning but ineffective leaders add up to poor educational experiences for their charges. Is there some remote chance that such schools are doing something positive for their students that might be found by a study ten years hence? I guess it’s possible. But there’s nothing in the research literature to suggest that it is so, and a lot in the literature to indicate that the kids—especially if they are poor—dare not wait that long.
***
I’ll be back with one last post tomorrow, summarizing everything we’ve learned in this series about the connection between test scores and long-term outcomes in true school choice programs. Stay tuned!
— Mike Petrilli
Mike Petrilli is president of the Thomas B. Fordham Institute, research fellow at Stanford University’s Hoover Institution, and executive editor of Education Next.
This first appeared in Flypaper.
