The First Negative Effects of Vouchers and the Predictable Misinterpretation

By 01/11/2016

2 Comments | Print | NO PDF |

An NBER Working Paper made waves last week because it found large negative effects. It is the first study to find large negative effects of vouchers on achievement. It’s well done. I believe it.

The question is, why? Why are the effects so negative when prior studies have found either no effect or positive effects? Good question. Unfortunately, we know much less about reasons than some have suggested.

The libertarian Cato Institute chimed in quickly: “Although not conclusive, there is considerable evidence that the problem stemmed from poor program design. Regulations intended to guarantee quality might well have had the opposite effect.” Others on the pro-voucher side had similar responses–not surprising for those who want the government as uninvolved as possible.

These interpretations are quite premature, however. The only real “evidence” consists of this single case of Louisiana with heavier regulation and negative effects. I would make the usual warning that correlation isn’t causation, except calling this a “correlation” is a bit generous when the pattern is identified from a single example. Not only do we need more evidence on regulation effects, but we also have to consider evidence on alternative explanations.

There are two other obvious explanations for why the results are worse here in Louisiana. First, Louisiana has especially aggressive accountability for the public schools. In New Orleans, regulation and accountability seem to have contributed to large positive effects on achievement. Why does this matter? Because the voucher studies compare students who won a voucher to those who did not–and those not receiving a voucher very likely ended up in the new and improved public/charter system. Families have to be low-income to receive a voucher in Louisiana so public schools represent the main plausible alternative.

Put differently, even if the quality of voucher-participating private schools was identical in every city, they would all show different effects depending on the academic effectiveness of the publicly funded alternatives. This interpretation creates a tension within the reform community because it potentially pits public school reforms like charters and school closure against reforms like vouchers. The better one looks, the worse the other looks.

A second possibility is that the curriculum in the private schools in Louisiana simply doesn’t align with the state standards captured on the tests. Even with the “heavy regulation” on private schools that voucher supporters decry, the public schools have a much larger incentive to align their instruction than the private schools accepting vouchers. What makes Louisiana different from other cities is not just that they are regulating more heavily, but that voucher recipients take the state test, not some off-the-shelf norm-referenced test that is more in line with the ones that private schools normally use. Weaker alignment between the curriculum and the test in Louisiana would also tend to make the results look worse.

The regulation explanation is a third possibility. As I argued previously, regulation probably does reduce the number of private schools, especially the number of higher-performing private schools. On the other hand, part of the point of those regulations is to keep out low-performing private schools. In fact, these are the same kinds of regulations that have apparently generated such positive effects in the New Orleans public school reforms.

The fact that the NBER study only covers the first year is another factor. The state of Louisiana recently banned four schools from receiving new voucher students because the scores of prior voucher recipients had been so low. Eventually, this seems likely to make the results look less negative. Again, this is also what happened with the public school reforms. There were no effects of the reforms at first, but they improved quickly as low-performing schools had been closed. (We have a study coming out on school closure effects later in the spring.) The influence of the regulations on public/charter schools may be different than on private/voucher schools, but the pattern here is noteworthy.

The reform community sure has itself tied up in knots on this one. Even among those who support choice and non-governmental providers, there is disagreement over the rules of the road. Unfortunately, at this point, the Louisiana case tells us little about regulation effects. We can only speculate on the reasons why the Louisiana voucher effects have been so negative, but the fact that they are negative is important by itself.

Next month, the Education Research Alliance for New Orleans will release a series of studies that will provide a richer picture of the situation. We also just released a study on how families that are eligible for vouchers choose between public and private schools. The authors will provide a guest blog on that in the near future.

—Douglas N. Harris

This first appeared in Education Week’s Urban Education: Lessons from New Orleans on January  6, 2016. Reprinted with permission from the author.

Comment on this article
  • Matthew Pepper says:

    There is evidence that moving to a new school can cause a significant drop in scores. A more appropriate comparison group may be waitlist students who also started the year at a brand new school, although this also brings its own bias.

    Is there variation in the starting MS grade within this sample? If so you can test this hypothesis by comparing the growth of 5th graders in a K-5 to 5th graders in a K-4.

  • Jason Bedrick says:

    I respond to Professor Harris here:

    Here’s an excerpt:

    Harris’s first theory is explicitly rejected by the NBER study. On the third page of the study, the authors write: “Negative voucher effects are not explained by the quality of public fallback options for LSP applicants: achievement levels at public schools attended by students lotteried out of the program are below the Louisiana average and comparable to scores in low- performing districts like New Orleans.” In other words, the public school alternatives are not so great and the performance of the participating private schools is considerably worse.

    That said, Harris’s second theory, which Jason Richwine also suggested, is plausible as a contributing factor. However, it is no more plausible than the Overregulation Theory. Indeed, whereas the differences in enrollment trends between voucher and non-voucher private schools provide some suggestive evidence for the Overregulation Theory, Harris provides no evidence to support the Nonaligned Test Theory. How many voucher schools were already aligned with the state curriculum and/or administered the state test? At this point, we do not know. Moreover, to the extent that testing nonalignment explains some of the very large 0.4 standard deviation difference in math scores, it is unlikely that it explains all or even most of that difference. Then again, Harris stated that he will be releasing the results of his own research on the LSP, so it’s likely he knows something that I do not.

  • Comment on this Article

    Name ()


    Sponsored Results

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform