We recently released a study that shows that school vouchers in New York City had a positive impact on the college enrollment rate for African-American students but not among Hispanic students. We think the study is important because it provides the first experimental estimate of the impact of vouchers on college enrollment.
The National Education Policy Center has just released a critique of our study by Sara Goldrick-Rab of the University of Wisconsin-Madison.
Several of the issues raised by Goldrick-Rab have no merit and none undermine the primary conclusion of our study: The voucher intervention in New York City increased the college enrollment rates of African-American students. Below are responses to the primary criticisms raised in the review:
1. The review questions the equivalence of the treatment and control groups by pointing to a modest difference between the treatment and control groups in the share of African-American students’ parents who completed a bachelor’s degree. This difference is only marginally statistically significant and, as the review notes, there are other differences that favor the control group. For example, control group families are less likely to have a father absent. Because chance differences can appear for any one characteristic, statisticians have developed a test that uses information on all background characteristics to ascertain whether two groups are equivalent. The overall treatment and control groups and the African-American and Hispanic subgroups all survive this test.
2. The review says that an interpretation of the results for African-American students is not appropriate because they do not differ significantly from those observed for Hispanic students. As stated in our report, it is true that the effects for African Americans and Hispanics are both positive and do not differ from one another by an amount that is statistically significant. But we can confidently say that the effect for African-American students is positive (i.e. greater than zero), whereas we cannot say the same for Hispanic students.
3. The review asks for an interpretation of the results for the small number of white and Asian students. But the treatment and control groups for this small number of students do not survive the equivalence test mentioned in item one; interpreting the results is therefore inappropriate.
4. The review raises a technical issue related to measurement error that is incorrect. It is correct that our college attendance measure is not perfect because the process used to match students to college enrollment records is not precise. But those errors appear as part of the standard error currently reported and no further adjustment is appropriate.
5. The review makes an error in its interpretation of a null finding. It concludes that our report “convincingly demonstrates that in New York City a private voucher program failed to increase the college enrollment rates of students from low-income families.” That statement is false. The overall impact estimate is not estimated with enough precision to conclude that the voucher intervention had no effect. The overall impact is not statistically significant from zero, but it is also not statistically significant from a negative impact of 3 percentage points or a positive impact of 4 percentage points.
6. The one result that can be reached with confidence is that the impact of vouchers for African Americans was positive. None of the issues raised in this commentary compromise that conclusion.
– Matthew M. Chingos and Paul E. Peterson
