The Mathematica study of charter middle schools, just released by the U. S. Department of Education, finds no achievement gains within two years for students who won the charter lottery as compared to those who did not. Ordinarily one would take such a study seriously, because it compares similar groups of students who differ only in the luck of the draw.
But as Marci Kanstoroom pointed out some years ago (when the design of the study was released but results were yet unknown), the study was set up in such a way that it could not possibly tell us much about charter schools.
Mathematica, the firm that did the study, chose to study only those students who entered a charter middle school after having first taken a standardized test in a public school. Since standardized tests are typically not given before third grade, charter students included in the study consisted mainly of students who moved from traditional public school to a charter school in fourth grade or later.
We know from other studies that students typically lose ground when they change schools. Working with data from the State of Florida, I have regularly noticed that achievement gains by students who move from one public school to another badly trail the gains of students who remain in the same school—even after one has adjusted for many family background characteristics.
Along these same lines, the Mathematica study shows that students who change to charter schools do not perform better than those who did not change schools. .
The study. however, can tell us nothing about the experience of attending a charter school from the very beginning—nor about the longer term impact of attending a charter school such as was done by a nifty charter graduation rate study carried out by Kevin Booker, Tim R. Sass, Brian Gill, and Ron Zimmer (“The Unknown World of Charter High Schools”).
Mathematica does tell us that parents provide a positive assessment of their children’s charter school experience. Unlike achievement tests, parents seem to be able to distinguish between problems of adjustment and long-term trajectories.
Despite the setback caused by changing schools, disadvantaged students at charter schools still did better in math within two years after switching to charter schools. One wonders what further gains may take place after the adjustment to the new school is complete.
Given the fact that one could tell from the very beginning that the Mathematica study was badly designed, one wonders why the U. S. Department of Education trotted down a track bound to provide misleading information.
Mathematica’s own defense of its research design was that it could do the study more cheaply if it relied upon readily available data, even though Caroline Hoxby, facing similar data collection problems, nonetheless found a way of tracking students from first grade on (“How New York City’s Charter Schools Affect Achievement”).
