Last week, NBER released the first random-assignment study ever to find a negative impact from a school voucher program. Previous gold standard studies had almost unanimously found modest positive effects from school choice, which raises the obvious question: what makes the Louisiana Scholarship Program (LSP) so different?

In an article for Education Next, I argued that, “although not conclusive, there is considerable evidence that the problem stemmed from poor program design.” The LSP is one of the most heavily regulated school choice programs in the nation, and that burden has led to a very low rate of private school participation. Only about one-third of Louisiana private schools accept voucher students, a considerably lower rate than in most other states. From a survey of private school leaders conducted by Brian Kisida, Patrick J. Wolf, and Evan Rhinesmith for the American Enterprise Institute, we know that the primary reason private schools opted out of the voucher program was their concerns over the regulatory burden, particularly those regulations that threatened their character and identity. For example, voucher-accepting schools in Louisiana may not set their own admissions criteria, cannot charge families more than the value of the voucher (a meager $5,311 on average in 2012), and must administer the state test.

We also know from the NBER study that the participating and non-participating private schools differ in at least one important respect. Whereas the non-participating schools experienced modest growth over the decade before the voucher program was expanded statewide (about 3 percent, on average), the participating schools had been experiencing a significant decline in enrollment (about 13 percent, on average). In other words, schools that were able to attract students tended to reject the vouchers while voucher schools tended to be those where enrollment had been dropping.

The difference in enrollment trends suggests that the LSP’s regulatory burden had the opposite of its intended effect: discouraging higher-performing schools from participating, leaving only the lower-performing schools that were so desperate to reverse their declining enrollment and increase their funding that they were willing to do whatever the voucher program required.

Several researchers and education reform advocates reached similar conclusions, including Matthew Ladner, Adam Peshek, Michael McShane, Lindsey Burke, and Jonathan Butcher. However, others expressed skepticism about what I shall call the Overregulation Theory, and proposed alternative explanations for the LSP’s poor results, while a few more raised concerns about the impact of a more free-market school choice system on equity. I will address these critiques and concerns below.

Alternative Explanations to the Overregulation Theory

Writing at Education Week, Douglas Harris of the Education Research Alliance for New Orleans concedes that “regulation probably does reduce the number of private schools, especially the number of higher-performing private schools,” but he still believes the Overregulation Theory is “premature.” Harris instead offers two potential alternatives: 1) the improved public/charter school performance in New Orleans made the performance of the private sector look relatively worse; and 2) the curriculum at most private schools may not have been aligned to the state test, so the poor performance merely reflects that lack of alignment rather than poor performance.

Harris’s first theory is explicitly rejected by the NBER study. On the third page of the study, the authors write: “Negative voucher effects are not explained by the quality of public fallback options for LSP applicants: achievement levels at public schools attended by students lotteried out of the program are below the Louisiana average and comparable to scores in low- performing districts like New Orleans.” In other words, the public school alternatives are not so great and the performance of the participating private schools is considerably worse.

That said, Harris’s second theory, which Jason Richwine also suggested, is plausible as a contributing factor. However, it is no more plausible than the Overregulation Theory. Indeed, whereas the differences in enrollment trends between voucher and non-voucher private schools provide some suggestive evidence for the Overregulation Theory, Harris provides no evidence to support the Nonaligned Test Theory. How many voucher schools were already aligned with the state curriculum and/or administered the state test? At this point, we do not know. Moreover, to the extent that testing nonalignment explains some of the very large 0.4 standard deviation difference in math scores, it is unlikely that it explains all or even most of that difference. Then again, Harris stated that he will be releasing the results of his own research on the LSP, so it’s likely he knows something that I do not.

In addition to test non-alignment issues, Louisiana state superintendent of schools John White seems to imply that the negative impact found in the NBER study was a result of the “transition cost” of students switching to new schools. White writes: “these schools (a) had never worked with these students before and (b) had never been accountable for state test results before.” Certainly, transition costs play a role, but they cannot account for the entire negative impact, which was quite large. Previous research has found that the negative impact of student mobility was only a fraction of the negative impact of Louisiana’s voucher program.

Moreover, if they are correct that test non-alignment and transition costs explain a great deal of the negative results, then merely growing accustomed to the new schools and testing regimen—and not actual improvements in learning—should account for a great deal of any improvement on the state test in the first few years.

Limits to the Overregulation Theory

Harris and White also note that the NBER study only examined the results of one year of one program. They are certainly correct that we need more data over time to draw firmer conclusions, which is one reason I presented my interpretation as “not conclusive” and wrote that “the regulations may have had the opposite of their intended effect” (emphasis added). And, indeed, there is some evidence that voucher schools improved slightly in the third year since the statewide expansion. However, while White touts that, if the voucher schools were their own district, they’d rank 9^th for performance improvement, they’d still be the fifth-worst of 76 in the state in actual performance.

Professor Joshua Goodman of Harvard University also offered a salient critique, pointing out that the NBER study’s “estimates show consistent negative impacts irrespective of previous enrollment trends.” If there had been wide variation in enrollment trends among the participating schools, that would count as evidence against the Overregulation Theory. However, if the enrollment trends among participating schools were consistently negative, with the variation mostly limited to how fast the participating schools had been losing students, then this finding would still be consistent with the Overregulation Theory.

In other words, if the participating schools are actually roughly the bottom third in terms of performance, then the lack of correlation between their enrollment trends and performance may not tell us anything about the relationship between the enrollment trends and performance of the top two-thirds of private schools that chose not to participate.

As I noted initially, the evidence for the Overregulation Theory is suggestive but not conclusive. Nevertheless, such strongly negative results should give education reform advocates great pause about the regulatory strategies employed in Louisiana. We know the regulatory burden chased away most private schools, and we have evidence that the voucher-accepting schools had been struggling with declining enrollment. If we want to better understand the LSP’s atypically disastrous performance, the program design is the logical place to start.

School Choice: Equity and Economics

Whereas Harris, Richwine, and Goodman expressed skepticism about the explanatory power of the Overregulation Theory, Neerav Kingsland expressed misgivings about its implications. (White did both.) Noting that I had raised concerns about the unintended consequences of prohibiting voucher schools from setting their own admissions criteria or charging more than the voucher, Kingsland wrote:

The logical extension of Jason’s argument is that an all voucher education system would lead to a public education system where all schools would be allowed to reject students based on wealth, academic performance, and behavior.

Is this right?

Either voucher proponents have very different views of equity than most citizens, or they don’t really view vouchers as a replacement model for the current public education.

I’m curious – which is it?

First, I should clarify that I do not believe we can know from the NBER study the extent to which any single regulation—whether the ones Kingsland listed, the mandatory state test, or others—is to blame for the LSP’s poor performance. The Overregulation Theory only suggests that the whole package of regulations in Louisiana is driving away the better private schools.

That said, there is reason to believe that each of the regulations I enumerated produces some adverse consequences.

Testing

Mandating a single test makes it easier for parents to compare performance across schools, but it also stifles diversity and innovation because tests drive what is taught, when it’s taught, and how it’s taught. If Louisiana policymakers want to ensure parents have access to valuable test data without the significant downsides of a single test, a better solution would be requiring schools to administer one of the many nationally norm-referenced exams.

Tuition and the Role of Price

The prohibition against schools charging more than the value of the voucher is intended to ensure that low-income students are not turned away due to inability to pay. However, it can have the effect of reducing access to quality schools for low-income families. For example, if the state offers a $5,000 voucher and a school charges $7,000 in tuition, some low-income families will be able to make up the $2,000 difference (whether on their own or through additional scholarships or tuition aid), although some might be not be able to afford that school. However, if the school refuses to participate in the voucher program, then even students in the first category will not have access to that school.

It’s basic economics: prices provide valuable information and price ceilings cause shortages. If we truly want all children to have access to quality schools, it is imperative that the quality schools have an incentive to expand, which means they need to be able to cover more than just the marginal cost of an additional student. However, particularly if the voucher value is relatively small, price ceilings mean that private schools will likely only fill empty seats rather than expand enrollment. As I noted in my original post, the Wolf/Kisida/Rhinesmith survey found that two-thirds of participating Louisiana private schools intended to maintain their level of voucher enrollment and 13 percent planned to decrease voucher enrollment, while only 22 percent planned to increase it. This is not a recipe for universal access.

Admissions Criteria

Mandating open admissions, like the prohibition against “topping off,” is intended to ensure all children have access to quality schools. And, similarly, it may have the opposite effect. The Wolf/Kisida/Rhinesmith survey found that most private schools that opted not to accept voucher students were very concerned about threats to their “independence, character, or identity.” These concerns may be religious in nature or relate to student behavior, motivation, or other aspects of school culture. I believe these are legitimate concerns, but even if some people disagree, all that really matters is that the schools have these concerns. If most schools aren’t willing to participate then we haven’t done much to expand access.

Now, Kingsland asked about an all-voucher education system, and of course the calculus in that situation would be different than in a voucher system that runs in tandem with a district school system. In the present system, students always have access to a school that must accept them. That would not necessarily be the case in an entirely privatized system. In that case, the argument for mandating open admissions is much more compelling, but it is still not necessary. The government could, for example, directly subsidize schools that adopted an open admissions policy or grant larger vouchers to students attending such schools, while offering smaller vouchers to students attending more selective schools. This or similar approaches (e.g., Kingsland’s proposal to grant larger vouchers for at-risk students) are more likely to yield wider private school participation—and therefore greater access to quality schools—than a strict open admissions mandate.

Moving Forward

I outlined above how several regulations intended to guarantee equity could produce adverse effects. It does not automatically follow that they are not worth doing. Perhaps, in some cases, the benefits of a particular regulation outweigh the costs. This is a legitimate debate that extends back at least to Milton Friedman’s dispute with John E. Coons and Stephen Sugarman over their competing voucher proposals in California in the 1970s.

Proponents of particular regulations should make the case that the costs are worth it to achieve some desired end, and both sides should be willing to reconsider based on compelling evidence. However, dogmatically refusing to acknowledge that regulations can have real costs or that Louisiana’s voucher program has (perhaps uniquely among choice programs) failed to deliver on its promises does nothing to serve the interests of disadvantaged children.

So what is to be done? Do policymakers throw up their hands and accept inequity? Of course not. But we must balance idealism with a clear-eyed approach to evidence and humility about what can be accomplished. Policymakers should strive for a system in which every child has access to the education that best meets his or her needs. But they should also be cognizant that intentions do not guarantee results, that there are limits to what any policy can achieve, and that a perfect system is not possible. The perfect should not become the enemy of the good.

— Jason Bedrick