June is shaping to be a busy month for private school choice advocates, critics, and opponents. The U.S. Supreme Court ruled on the Trinity Lutheran case, CREDO recently released a nationwide evaluation of charter schools, and two separate teams of researchers just released updates to their evaluations of voucher programs in Indiana and Louisiana.
A vigorous debate has surrounded the voucher programs in Indiana, the District of Columbia, Louisiana, and Ohio. Studies of these programs had shown large and negative effects on test scores by students who participate in them. School choice opponents have seized on these findings as evidence that these programs are ineffective and even harmful while advocates point out that Louisiana is heavily regulated, the first few years of an evaluation tell only the worst part of a story (i.e. there are transition effects), and that we should be careful about a heavy-handed focus on test scores.
Now on to the findings.
What the studies say
Indiana
Researchers Mark Berends from the University of Notre Dame and R. Joseph Waddington from the University of Kentucky have been studying Indiana’s voucher program since it was enacted in 2011. They employ propensity score matching methods where they compare voucher students with similar students in public schools by matching across a variety of observable background factors, including baseline test scores. A host of other model specifications are estimated as well to check the robustness of their results.
A common criticism of school choice programs is that private schools and charter schools attract better students. The study provides evidence, however, that the Choice Scholarship Program is not enrolling high-performing students compared to public school peers, aka “cream skimming.” They have found the opposite happening in Indiana where overall, “the low average achievement and diversity of low-income voucher students previously attending public schools suggests that private schools are not ‘cream-skimming’ the best students from public schools who are eligible for a voucher.”
Berends and Waddington found that math achievement by voucher students was significantly lower than their matched public school peers during their first three years in private schools, with the negative effects being larger in the first two years (-0.12 standard deviations). By year four, there was no statistically significant difference in math test scores between students who remained in private schools and the matched comparison group.
The results in reading outcomes were somewhat better for the voucher students under study, where they recovered by year three and outperformed the matched comparison group in year four (the year four estimate of 0.13 standard deviations was marginally statistically significant). An important caveat to the fourth-year point estimates is that the sample is considerably smaller than for previous years’ estimates.
The researchers also studied subgroups of students and private schools. Special education voucher students experienced a loss of 0.13 standard deviations in ELA relative to their matched comparison students. Meanwhile, Catholic schools had stronger positive effects in reading than other types of private schools.
Louisiana
Because Louisiana’s program is capped and oversubscribed, researchers Jonathan Mills and Patrick Wolf at the University of Arkansas were able to use the “gold standard” research methodology of random assignment. A randomized lottery was conducted to determine which applicants receive vouchers and which students do not. This allows researchers to make “apples-to-apples” comparisons. On average, both groups of students are similar except that one receives vouchers and the other does not. This is a different, more rigorous analytical method than what Berends-Waddington were afforded by the data for evaluating Indiana’s school voucher program.
Earlier program evaluation reports for Louisiana showed that voucher students made significantly lower gains on math and reading test scores in the first year (27 percentile points and 17 percentile points lower, respectively) than students who applied for vouchers but were not awarded them through the lottery. These effects diminished somewhat in the second year.
The new third-year evaluation shows that math and ELA achievement by LSP scholarship users was statistically similar to their counterparts. Moreover, “students initially performing in the bottom third in ELA at baseline experienced statistically significant positive effects of scholarship use on achievement in ELA after three years.”
Separate reports on the LSP also found that voucher students enrolling in private schools were less likely to be identified as requiring special education services and more likely to be de-identified as requiring special education services than students who did not participate in the LSP. In addition, the participation rate of private schools in the LSP was just 33 percent. This rate is significantly lower than participation rates in other voucher programs such as in Indiana and the District of Columbia, where 70 percent and 78 percent of private schools, respectively, signed up to participate in the voucher programs. While the researchers are careful to note that Louisiana’s tax-credit scholarship program is a potential factor (private schools participating in that program may be less likely to also enroll in the voucher program), it’s unlikely this would explain the entire participation gaps. Given that private school officials are concerned about future regulations, the regulatory structure of the program may likely offer another explanation.
What information can we draw from these studies to inform education policy?
More Than Scores
Most school choice research, and other education policies for that matter, analyzes test scores as student outcomes. Tests scores are perceived as a reasonable predictor of long-term outcomes such as attaining employment and wages. Although they provide a readily available and convenient metric for researchers to study educational programs, however, there is evidence of a disconnect between test scores and long-term outcomes. To the extent that test scores are reliable predictors of lifelong outcomes, they can provide useful information for decision making by policymakers, researchers, and stakeholders. Evidence, however, suggests they provide inconsistent indicators.
What if schools can raise test scores without any corresponding improvement in students’ life outcomes? University of Arkansas professor Jay Greene has written about the disconnect between test scores and lifelong outcomes. Greene wrote:
This concern is similar to issues that have arisen in other fields about the reliability of near-term indicators as proxies for later life outcomes. For example, as one of my colleagues noted to me, there are medicines that are able to lower cholesterol levels but do not reduce — or even may increase — mortality from heart disease. It’s important that we think carefully about whether we are making the same type of mistake in education.
We can see a disconnect in the private school choice research. For instance, evaluations of the Milwaukee Parental Choice Program and D.C. Opportunity Scholarship Program found that while students who received vouchers experienced small or no test score gains relative to their counterparts in district schools, the programs significantly increased high school graduation rates among students who used vouchers. There are numerous other studies that found evidence of similar disconnects.
For example, research on a privately funded school voucher program in New York City provides some evidence in favor of a link existing between test scores and longer-term outcomes, where vouchers raised test score gains and increased the likelihood of graduating from high school and enrolling in college. Results from other studies on private school choice, however, do not indicate a relationship. Evaluations of the D.C. Opportunity Scholarship Program and Milwaukee Parental Choice Program found little or no test score gains among voucher students compared to their peers. At the same time, these evaluations also detected large positive effects of the programs on high school graduation.
There’s nothing wrong with requiring private schools in choice programs to administer assessments. Tests can generate useful information for parents and researchers. It is a concern, however, if a state imposes a single test, especially when the test is aligned with the state’s standards and not aligned with the varied missions and objectives of private schools. This introduces risk that private schools will change how they educate children by conforming to what’s measured on the test. The policy goal for education should be expanding and diversifying educational options so all children have access to the kind of education that best suits him or her, not homogenizing schools. More importantly, policy makers, researchers, and others are not in a better position than parents to raise children and educate them. As Greene puts it in his latest piece:
Solutions imposed by distant policymakers, researchers, and pundits are no more likely to be effective in educating our children than they would be in raising them. Yes, parents, communities, and local educators will make mistakes in raising children as well as educating them. But they are better positioned to understand the context and achieve the appropriate balance for each child than are distant policymakers and experts, even if they are well-intentioned and highly knowledgeable.
Learning curves
These studies also corroborate the notion that transition effects may be in play. A meta-analysis by researchers at the University of Arkansas of school voucher programs worldwide suggests that outcomes for voucher students, as measured by math and reading test scores, tend to be negative during the programs’ first few years but improve over time. On average, the effect sizes in reading turn positive in the second year of the programs and exceed 0.1 standard deviations after three years. The effect sizes are negative in math during students’ first and second years in the program, turn positive in the third year, and increase thereafter. It appears we may be seeing similar trends in Louisiana and Indiana. Patience is warranted on all sides because these are long-term evaluations, so time will tell.
Policymakers should be cautious about drawing any conclusions based on any study that reports results for only a few years of any program or cohort of students, especially at the beginning of a school choice program, when various stakeholders, such as participating students, their parents, school leaders, and state-level administrators, are on a learning curve.
Extra caution should be exercised when making policy decisions based on test score outcomes. Schools signing up to participate in a school choice program may need time to adapt. Some schools are taking in groups of students they may not have experience educating and require time to adapt. Similarly, students who switch from district schools to private schools via a private school choice program may also need time to adapt to a new atmosphere. The Indiana research team noted that during interviews with private school teachers and leaders, some students struggled during their initial year with homework expectations. In light of such anecdotal evidence, further research will be needed to learn more about the challenges that schools in choice programs face after a program is enacted.
Although the Indiana and Louisiana studies suggest that vouchers didn’t hit it out of the park when it comes to math and reading test outcomes, the programs are trending in the right direction. Overall, these studies should be received as good news for parents, policymakers, and stakeholders. It would be a mistake, however, to use these findings to guide policymakers, researchers, and elites for identifying which schools are “good” or “bad” and which policies, education programs, or schools are working or failing. Exercising some humility by extending deference to parents will go a long way. After all, they are in the best position for deciding if a school, education program, or particular environment is working for their children. The goal of education policy should be the expansion of education options to increase the likelihood that good matches are made between students and the education they receive and deserve. Information on test scores may help, but they alone will not get us there.
–Martin F. Lueken
Martin F. Lueken, Ph.D., is Director of Fiscal Policy and Analysis at EdChoice.
