The Thomas B. Fordham Institute recently released a study on the academic impact of Ohio’s flagship school choice program authored by noted researcher Dr. David Figlio of Northwestern University. The report is noteworthy for its principal findings, namely that, not only is the sky not falling for impacted public schools, the EdChoice program has had a positive impact on the academic performance of public schools whose students are eligible for a scholarship. Surprisingly, the study also found that the students using scholarships to attend private schools who the report studied (more on that later) did not perform as well as their public school peers on the state test.

Matt Barnum of The 74 wrote an article that details some of the possible explanations for the latter finding. Based on my own experience in Ohio, I can attest that many nonpublic schools do not align their curriculum to the state test, nor do they focus much on these measures, and that is likely an important factor. However, it is important to note what the study could not address. As Dr. Figlio made clear in both his report and a presentation to The City Club, the study had significant limitations.

Ohio’s EdChoice program differs from most other school choice programs in a significant manner: student eligibility is determined solely by the performance of their assigned public schools. This has implications for how to study the program. The program creates real choice opportunities for students assigned to these schools (primarily the 10% lowest performing in the state), removes an economic incentive for middle class families to flee to the suburbs for better schools and larger properties (an important consideration for anyone knowledgeable about inner-ring suburbs and urban areas in Ohio), and creates market pressures for public schools to improve their performance, which the study confirms to be the case. This last argument is a cornerstone for many “free-market” school choice advocates and scholars, and the study’s most robust findings appear to confirm it.

The more surprising finding to both advocates and opponents was that using a scholarship to attend a private school appears to have led students to fare worse academically than had they remained in their public school. While there are many possible explanations for this, it is not worth time making excuses for why some students in private schools aren’t doing as well academically as some of their public school peers. Parents do make choices for their children based on many different factors. It may be that a child is not thriving at a particular school or is having social problems with a particular group of children; parents may disagree with teachers and/or the curriculum being taught; or they may desire a more faith-based approach to learning. Every child is unique and has their own needs. But certainly the expectation needs be that, among other benefits, choice should lead to higher levels of academic achievement.

The challenge facing researchers and policymakers is determining how these students would have performed had they stayed in their assigned public schools. As noted before, EdChoice eligibility is based on the performance of their assigned public schools. The most apt comparison of academic performance would seem to be between scholarship students and their peers who are enrolled at their assigned school. We have some data on this, and as noted in an article published by the Thomas B. Fordham Institute in 2014, they indicate that scholarship students do outperform their public-school peers in many districts, in some cases by huge margins: Consider Columbus. In 2013-14, voucher students in grades three to eight outperformed their district peers in math and reading at every grade level on Ohio Achievement Assessments (OAA’s) and Ohio Graduation Tests (OGT’s) (see table below). For instance, on the OGT’s, 96 percent of voucher students were proficient in reading compared to 72 percent of public school students. On the math portion of the OGT, 85 percent of voucher students were proficient compared to 50 percent of their public school peers. Of third graders using a scholarship to attend a private school, 96 percent were proficient in reading compared to 55 percent of students in their assigned public schools. Similarly impressive scores, with some exceptions at certain grade levels, were posted in Cincinnati, Cleveland, Dayton, Toledo, and other districts.

So why didn’t this study show the same pattern? As Dr. Figlio notes, you can’t simply compare these two groups because there must be a reason why some students chose to leave their assigned schools and others did not. In fact, his study shows that, among eligible students, those who used scholarships to move to a private school were higher-performing and less likely to be from low-income families than those who did not. Accordingly, the best available comparison group in this context consists of students attending schools where they had no access to choice because they were not designated in the lowest 10% of the state. Figlio’s study therefore compares students who left the highest-ranked schools that are eligible for EdChoice for private schools against observably similar students who remained in the lowest-ranked schools that are not. As a result it tells us little about how well students using a scholarship (who would have otherwise attended the very lowest-performing public schools) are doing—a fact Dr. Figlio acknowledges. In addition, the set of schools from which the comparison group of students is drawn may also be problematic. While these schools appear to be similar on the surface, there could nonetheless be differences between a school that has never been designated as EdChoice and a school that has been consistently so designated. In addition, the study measured the earlier years of this program. Further study may uncover more positive results—even using this methodology.

The bottom line is that we should be careful in interpreting these findings. Most importantly, the study was unable to examine the achievement of students assigned to the lowest-performing public schools. What we do know is that EdChoice has improved public schools, that parents like the choices they are provided, and that data do seem to indicate greater achievement by many students on scholarships. Despite all the creative headlines that this study has generated, a deeper dive into the report, coupled with intimate knowledge of the program and Ohio, gives us reason to believe that the program is beneficial. There are profound fiscal and equity arguments for school choice, as Dr. John C. White, Louisiana’s State Superintendent eloquently writes, and these programs take time to develop. By encouraging more private schools to participate, ensuring that parents have access to information on how their children are performing, and broadening the number of students eligible, Ohio can make this vital choice program an even greater benefit to the state and its citizens.

—Rabbi Eric “Yitz” Frank

Rabbi Frank is the Ohio director of Agudath Israel of America