An NBER Working Paper made waves last week because it found large negative effects. It is the first study to find large negative effects of vouchers on achievement. It’s well done. I believe it.
The question is, why? Why are the effects so negative when prior studies have found either no effect or positive effects? Good question. Unfortunately, we know much less about reasons than some have suggested.
The libertarian Cato Institute chimed in quickly: “Although not conclusive, there is considerable evidence that the problem stemmed from poor program design. Regulations intended to guarantee quality might well have had the opposite effect.” Others on the pro-voucher side had similar responses–not surprising for those who want the government as uninvolved as possible.
These interpretations are quite premature, however. The only real “evidence” consists of this single case of Louisiana with heavier regulation and negative effects. I would make the usual warning that correlation isn’t causation, except calling this a “correlation” is a bit generous when the pattern is identified from a single example. Not only do we need more evidence on regulation effects, but we also have to consider evidence on alternative explanations.
There are two other obvious explanations for why the results are worse here in Louisiana. First, Louisiana has especially aggressive accountability for the public schools. In New Orleans, regulation and accountability seem to have contributed to large positive effects on achievement. Why does this matter? Because the voucher studies compare students who won a voucher to those who did not–and those not receiving a voucher very likely ended up in the new and improved public/charter system. Families have to be low-income to receive a voucher in Louisiana so public schools represent the main plausible alternative.
Put differently, even if the quality of voucher-participating private schools was identical in every city, they would all show different effects depending on the academic effectiveness of the publicly funded alternatives. This interpretation creates a tension within the reform community because it potentially pits public school reforms like charters and school closure against reforms like vouchers. The better one looks, the worse the other looks.
A second possibility is that the curriculum in the private schools in Louisiana simply doesn’t align with the state standards captured on the tests. Even with the “heavy regulation” on private schools that voucher supporters decry, the public schools have a much larger incentive to align their instruction than the private schools accepting vouchers. What makes Louisiana different from other cities is not just that they are regulating more heavily, but that voucher recipients take the state test, not some off-the-shelf norm-referenced test that is more in line with the ones that private schools normally use. Weaker alignment between the curriculum and the test in Louisiana would also tend to make the results look worse.
The regulation explanation is a third possibility. As I argued previously, regulation probably does reduce the number of private schools, especially the number of higher-performing private schools. On the other hand, part of the point of those regulations is to keep out low-performing private schools. In fact, these are the same kinds of regulations that have apparently generated such positive effects in the New Orleans public school reforms.
The fact that the NBER study only covers the first year is another factor. The state of Louisiana recently banned four schools from receiving new voucher students because the scores of prior voucher recipients had been so low. Eventually, this seems likely to make the results look less negative. Again, this is also what happened with the public school reforms. There were no effects of the reforms at first, but they improved quickly as low-performing schools had been closed. (We have a study coming out on school closure effects later in the spring.) The influence of the regulations on public/charter schools may be different than on private/voucher schools, but the pattern here is noteworthy.
The reform community sure has itself tied up in knots on this one. Even among those who support choice and non-governmental providers, there is disagreement over the rules of the road. Unfortunately, at this point, the Louisiana case tells us little about regulation effects. We can only speculate on the reasons why the Louisiana voucher effects have been so negative, but the fact that they are negative is important by itself.
Next month, the Education Research Alliance for New Orleans will release a series of studies that will provide a richer picture of the situation. We also just released a study on how families that are eligible for vouchers choose between public and private schools. The authors will provide a guest blog on that in the near future.
—Douglas N. Harris
This first appeared in Education Week’s Urban Education: Lessons from New Orleans on January 6, 2016. Reprinted with permission from the author.
