Research has yielded relatively few definitive findings to guide policymakers on what educational interventions might enhance student achievement. The studies of privately financed voucher programs in Dayton, Ohio; New York City; Washington, D.C.; and Charlotte, North Carolina, however, suggest that private schooling leads to significant improvements in achievement for some students. This finding is potentially quite important, given the continuing debate over public funding of voucher programs.
The survey data the researchers gathered show that, in general, voucher recipients were more satisfied than their public-school counterparts with a variety of conditions (such as safety, teacher quality, and so forth) at their chosen schools. This finding is consistent with the literature on school satisfaction. It is not clear, however, what conclusions one should draw from parents’ perceptions. They may represent mere “buy-in effects”: the idea that simply being given a choice heightens student and parental satisfaction. Greater satisfaction with a school certainly should be regarded as a positive outcome; however, we cannot know whether these findings reflect genuine differences in school quality or simply the satisfaction of being granted options. Given this ambiguity, this review focuses on reported results on student achievement.
The effects of private schooling, as reflected in achievement data, varied across studies, grades, and racial and ethnic groups. Positive effects were consistently found for African-American students who attended private schools for at least two years. The average estimated effect for African-American students in the Dayton, New York City, and Washington programs of attending a private school rather than a public school was more than 6 percentile points in both mathematics and reading after two years. In the Charlotte program, where effects were not reported by race, the private-school effect was estimated to be an increase of approximately 6 percentile points after one year.
These gains are quite large by social science standards. For instance, the largest effects were found in Washington, D.C., where African-American students in grades 2-5 gained 10 percentile points in mathematics and 8.6 percentile points in reading after two years of private schooling. This represents a gain of about 0.5 standard deviation relative to African-American students whose applications for vouchers were unsuccessful (the control group, who remained in public schools). By comparison, a 1999 study by Jeremy Finn and Charles Achilles estimated that the well-known STAR class-size reduction program in Tennessee increased minority students’ 2nd-grade reading achievement by 0.33 standard deviation. This was with reducing class sizes by about eight students (from a class of 22-26 students to one with 13-17 students), a relatively expensive intervention. Thus, the estimates from the voucher studies reviewed here suggest that even modestly sized vouchers, which may be considerably less expensive than the marginal cost of educating students in the public sector, could provide substantial benefits for some students. This interpretation of the findings, however, deserves several cautions.
Potential for Bias
In assessments of program effects in nonexperimental settings, one concern is the potential for omitting important individual characteristics from the analysis. Many individual characteristics that influence student achievement, such as a student’s motivation, are difficult to quantify. Private- and public-school students and parents may differ from one another in both observable and unobservable ways. For instance, the very fact that parents use a voucher (and supplement it in the case of the privately funded scholarships studied here) could reveal a home environment that is conducive to academic achievement. These types of effects are difficult to measure. If unobservable variables such as parents’ commitment to education come into play, their influence may be wrongly identified as the effects of better schooling.
The strength of the studies reviewed here is that they take advantage of the fact that a lottery was used to allocate spots in these oversubscribed voucher programs. This allowed the researchers to compare the outcomes of students who were randomly assigned to either public (the control group) or private (the treatment group) schools. This strategy has intuitive appeal, is easy to understand, and is used in other contexts, such as medical research. Ideally, random assignment should mitigate concerns about the effects of unobservable characteristics. In fact, it should eliminate the need to control for any background characteristics, observable or unobservable. Of course, ideal conditions rarely, if ever, exist. Small deviations from the ideal are unlikely to result in biased findings. Larger deviations, however, can bias the results of even a carefully designed social experiment, possibly resulting in misleading conclusions.
Several findings are somewhat troubling in that they raise concerns about the quality of the control and treatment groups. First, the magnitude of the gains due to private schooling is substantially larger than that typically found in nonexperimental studies of differences between public and private schools, even those studies focusing on minority students. Second, the finding of no statistically significant gains for non-African-American students is surprising. It is not at all clear what accounts for private schools’ disproportionately benefiting a particular group of students. Finally, in some cases the estimated effects exhibited rather large swings from year one to year two. In Washington, D.C., African- American students in private schools declined by 9 percentile points in reading relative to their public school peers after the first year. The next year saw them climb to 8 percentile points above their public school peers in reading. This represented a net positive swing of 17 percentile points from one year to the next. An additional year of private schooling, in other words, is estimated to produce a staggering gain of about 0.9 standard deviation.
What might account for these anomalies? Of course, the quality of the results depends crucially on the integrity of the experiments. Differences in responses between the treatment and control groups or nonrandom placement into either group can contaminate the results.
The authors’ demographic data suggest that lottery winners and losers had similar background characteristics (although there were some statistically significant differences in Charlotte). However, many students who “won” vouchers in the random lottery chose not to use them. In Washington, D.C., 47 percent of lottery winners did not use their vouchers; in Dayton, 46 percent; and in New York, 24 percent. Voucher users and decliners may have appeared to be similar along observable lines, but may have differed in their unobservable characteristics. In fact, it is quite plausible that those parents who declined a voucher did so at least in part because they perceived that sending their children to private schools would not necessarily have been beneficial based on the specific public and private options available in their neighborhoods.
The authors use a statistical technique (known as “instrumental variables”) that explicitly accounts for potential differences, observable or unobservable, between those who use and those who decline vouchers. As a result, the findings are not likely to suffer from this potential source of bias. However, this technique cannot account for the potential bias associated with “response attrition”-that is, the potential that there are systematic differences between those who participated in the testing and survey sessions and those who did not. Some cities, such as Charlotte, encountered a fairly low participation rate in the testing sessions. The overall response rate in Charlotte was 40 percent, and only 20 percent of lottery winners who declined their vouchers participated in the study.
Similar concerns apply to attrition out of the sample over time. If students who leave the treatment sample (lottery winners) differ in important ways from students who leave the control sample (lottery losers), the results may be biased. For instance, if the students who left the treatment group tended to be of lower ability than those who left the control sample, the experiment would overstate the effect of private schooling. Given the sizable year-to-year drops in the sample sizes in all four cities, nonrandom attrition may be an issue. The attrition rates were generally similar between the treatment and control groups, and the two groups remained similar along observable dimensions (and the researchers used weights to adjust for differences in observed characteristics). Nevertheless, the concern remains that attrition over time resulted in unobservable differences between the two groups.
Findings from the similarly structured but publicly financed voucher program in Milwaukee show the potential impact of contaminating the treatment and control groups. Researchers studying this program using methodologies similar to those used in the studies reviewed here estimated relatively large effects for students who attended private schools: 6 percentile points in reading and 7 percentile points in mathematics after three years. By contrast, Princeton economist Cecilia Rouse, using a methodology that accounts for the possibility of unobservable characteristics such as greater motivation among students and parents who remained in private schools, found no statistically significant, positive effects of private schools in reading (but did find similar private-school effects in math of 1 to 2 percentage points per year).
There is no direct evidence that the results reported by William Howell et al. and Jay Greene are biased. If data collection continues on these voucher programs, it will be possible to use statistical techniques to account for the potential of bias. Regardless, if these findings accurately assess the effects of private schooling, they certainly lend credence to arguments in support of vouchers. Given this possibility, it is important to consider whether these gains can be replicated on a larger scale.
To the Next Level?
Findings from relatively small, experimental programs may not generalize to a larger scale for several reasons, a point the authors acknowledge. The evaluations did not adjust for the demographics or ability of the non-voucher students attending private schools. What is thought to be the effect of private schooling may actually be the influence on voucher users of having peers of higher ability, better behavior, and so forth. To the degree that peers influence educational outcomes, we might expect to find positive effects of private schooling even if the private schools themselves are no better than public schools. An expansion of voucher programs could diminish the positive effects of peers if the flow of students caused the demographics of private schools to look more and more like those of public schools.
Only those students and parents who expressed an interest in attending private schools had a chance to receive a voucher. The effects of private schooling on students who want to attend private schools do not necessarily predict the experiences of the general population. A similar point applies to the schools that accepted vouchers. The quality of both public and private schools varies considerably, but the authors don’t know whether they compared average public schools with average private schools. One might assume that students attending ineffective public schools are more likely to apply for a voucher. One might also argue that only the least effective private schools would have room for additional students. Without knowing what kinds of schools we’re comparing, it is not at all clear that we should expect similar results from an expanded voucher program.
Too few seats exist currently in private schools to accommodate a large migration of students into the private sector. An increase in the demand for private schooling associated with the offering of more vouchers might be expected to stimulate the expansion of existing private schools or the building of new schools, but of what quality? Economic theory suggests that the most efficient suppliers are likely to be in the marketplace already. Thus, the quality of new private schools is likely to be lower than that of existing ones. Some evidence suggests that competition from private schools makes public schools more efficient, but our knowledge of how public schools would respond to the competitive threat of losing students under a voucher system is limited. The evidence presented by Howell et al. and Greene is not sufficient to be considered decisive in the policy debate over vouchers. Nonetheless, it is encouraging with regard to students who use vouchers, particularly minority students, and much more may be learned as additional data become available.
Dan Goldhaber is a senior research associate at the Urban Institute.
