One should not under-estimate the impact of the DC school voucher program on student achievement. According to the official announcement and the executive summary of the report, school vouchers lifted high school graduation rates but it could not be conclusively determined that it had a positive impact on student achievement.
Something about those findings sounds like a bell striking thirteen. Not only is the clock wrong, but the mechanism seems out of whack. How can more students graduate from private schools if they weren’t learning more? Are expectations so low in the private sector that any one can graduate?
Peering beneath the press release and the executive summary into the bowels of the study itself one can get some, if not all the answers, to these questions.
Let’s begin with the most important—and perfectly uncontested—result: If one uses a voucher to go to school, the impact on the percentage of students with a high school diploma increases by 21 percentage points (Table 3-5), an effect size of no less than 0.46 standard deviations. Seventy percent of those who were not offered a school voucher made it through high school. That is close to the national average in high school graduation rates among those entering 9th grade four years earlier. As compared to that 70 percent rate among those who wanted a voucher but didn’t get one, 91 percent of those who used vouchers to go to private school eventually received a high school diploma.
Most people would be thrilled to learn about a new way to lift the graduation rates of students from low income families by 21 percentage points—especially if it costs the taxpayer nothing at all. Indeed, the school voucher program actually saved money, because vouchers cost only about half the cost per pupil per pupil of going to District of Columbia public schools.
The results are especially exciting because they come from a Randomized Field Trial, the gold standard in social scientific and medical research. A lottery decided which applicants had the opportunity to use a voucher to go to private school, and so all the comparisons between voucher students and other students are strict ones that control for family background, parental motivation, child motivation and everything else. That is the great thing about a randomized trial. It does a better job of showing the effect of a program than any other research strategy, because it compares two groups that are essentially alike, apart from the luck that one group had on lottery day.
But how were such high graduation rates achieved, when voucher students learned no more than the other students? The answer to that riddle is that the study shows exactly the opposite: Those who went to private school scored 4.75 points higher on the reading test, an effect size of 0.13 standard deviations.
Admittedly, that is not as big an effect as is the voucher impact on graduation rates, and it is only fair to point out that statistician purists insist that any finding, before it can be declared undeniably true, must have only 5 chances in 100 of being wrong. The chances that the reading impact is in fact phony are greater than 5—in fact they are 6 in 100–and so it must be declared—by the statistician purists who supervise reports by government agencies—that “there is no conclusive evidence that [the vouchers] affected student achievement (p. xv).”
But notice the wording—there is “no conclusive evidence.” That is quite different language from saying there is “no evidence” that vouchers raised achievement. Indeed, if you invested $1,000 every time you had 94 chances in 100 of picking the right stock—and only 6 chances of getting it wrong–as is the case here, then, with modern technology, you could become richer than Bill Gates by sundown.
So the evidence is only overwhelming, not “conclusive,” that the voucher program raised student reading scores. Math results are noticeably weaker, however. It is true that there is a considerably better than even chance that those in private school also learned more in math, but still the effects are small. Private schools appear not to be teaching math as effectively as they teach reading.
One other fact to note is that the graduation data include many students who were not tested in the final stages of the study. By that time they were beyond 12th grade, and no test was possible. For these students, we don’t know—we can only suspect—that they would have tested better than their non-voucher counterparts.
But all of these considerations still do not quite explain the very large impact the voucher program had on graduation rates. For that we have to turn to the reports from parents on school safety, an orderly school climate, and overall satisfaction. Here we find effect sizes of 0.14 and 0.17 and 0.22 standard deviations, not quite 0.46 standard deviations (the size of the impact on graduation rates) but sizeable enough to suggest that there was something about the private school that supported the development of the young person in ways that went beyond the mere acquisition of reading and math skills.
In short, the various pieces of the DC voucher study hang together more nicely than one might first conclude. So the next time someone tells you there is no evidence that vouchers help kids, tell them, “I’m sorry, that just ain’t true. A darn good study by an agency of the federal government has yielded strong evidence that school vouchers can help the children of low income families who live in places like the District of Columbia.”
