Charter Benefits Are Proven by the Best Evidence



By 05/08/2012

18 Comments | Print | NO PDF |

According to the Global Report Card, more than a third of the 30 school districts with the highest math achievement in the United States are actually charter schools.  This is particularly impressive considering that charters constitute about 5% of all schools and about 3% of all public school students.  And it is even more amazing considering that some of the highest performing charter schools, like Roxbury Prep in Boston or KIPP Infinity in New York City, serve very disadvantaged students.

As impressive and amazing as these results by charter schools may be, it would be wrong to conclude from this that charter schools improve student achievement.  The only way to know with confidence whether charters cause better outcomes is to look at randomized control trials (RCTs) in which students are assigned by lottery to attending a charter school or a traditional public school.  RCTs are like medical experiments where some subjects by chance get the treatment and others by chance do not.  Since the two groups are on average identical, any difference observed in later outcomes can be attributed to the “treatment,” and not to some pre-existing and uncontrolled difference.  We demand this type of evidence before we approve any drug, but the evidence used to justify how our children are educated is usually nowhere near as rigorous.

Happily, we have four RCTs on the effects of charter schools that allow us to know something about the effects of charter schools with high confidence.  Here is what we know:  students in urban areas do significantly better in school if they attend a charter schools than if they attend a traditional public school.  These academic benefits of urban charter schools are quite large. 

In Boston, a team of researchers from MIT, Harvard, Duke, and the University of Michigan, conducted a RCT and found:  “The charter school effects reported here are therefore large enough to reduce the black-white reading gap in middle school by two-thirds.”

A RCT of charter schools in New York City by a Stanford researcher found an even larger effect: “On average, a student who attended a charter school for all of grades kindergarten through eight would close about 86 percent of the ‘Scarsdale-Harlem achievement gap’ in math and 66 percent of the achievement gap in English.”

The same Stanford researcher conducted an RCT of charter schools in Chicago and found:  “students in charter schools outperformed a comparable group of lotteried-out students who remained in regular Chicago public schools by 5 to 6 percentile points in math and about 5 percentile points in reading…. To put the gains in perspective, it may help to know that 5 to 6 percentile points is just under half of the gap between the average disadvantaged, minority student in Chicago public schools and the average middle-income, nonminority student in a suburban district.”

And the last RCT was a national study conducted by researchers at Mathematica for the US Department of Education.  It found significant gains for disadvantaged students in charter schools but the opposite for wealthy suburban students in charter schools.  They could not determine why the benefits of charters were found only in urban, disadvantaged settings, but their findings are consistent with the three other RCTs that found significant achievement gains for charter students in Boston, Chicago, and New York City.

When you have four RCTs – studies meeting the gold standard of research design – and all four of them agree that charters are of enormous benefit to urban students, you would think everyone would agree that charters should be expanded and supported, at least in urban areas.  If we found the equivalent of halving the black-white test score gap from RCTs from a new cancer drug, everyone would be jumping for joy – even if the benefits were found only for certain types of cancer.

Unfortunately, many people’s views on charter schools are heavily influenced by their political and financial interests rather than the most rigorous evidence.  They don’t want to believe the findings of the four RCTs, so they simply ignore them or cite studies with inferior research designs in which we should have much less confidence.

Progress will be made in our application of research to charter school policies by encouraging everyone to focus on the most rigorous studies, of which we have several.  To do that, supporters of charter schools also have to refrain from citing weaker evidence, which only serves to legitimize the use of inferior studies by charter opponents.  As exciting as the outstanding performance of charter schools is in my own Global Report Card research, that evidence shouldn’t be used to endorse charter schools.  Supporters don’t need to rely on the Global Report Card to make the case for charter schools because they have four gold-standard RCTs on their side.  Opponents of charter schools have no equally rigorous evidence on their side.  And that’s the point we should all be making.

-Jay P. Greene

This blog entry originally appeared on the blog of the George W. Bush Institute for National Charter Schools Week.




Comment on this article
  • BB says:

    What these studies don’t reveal is whether the improved achievement in charter schools is caused by the differences in curriculum and instruction per se, or by the absence of students who are more challenging and who have less support at home. And we will not know that until charter operators take over regular public schools and commit to retaining as many students as regular public schools do.

  • Jay P. Greene says:

    Hi BB,

    You are mistaken when you say that the superior performance of charter school students in RCTS might be caused “by the absence of students who are more challenging and who have less support at home.” The beauty of random assignment is that the treatment group that is admitted by lottery to charter schools is identical, on average, to the control group, including their challenges and support from home. This is precisely why RCTs are so valuable.

  • BB says:

    Jay, you misunderstood my comment. Or I didn’t phrase it well. What I meant was, individual charter school students might perform better than their matched counterparts who didn’t win spots in charter schools because their education was not disrupted by the challenging behaviors (and very low academic achievement) of students who never applied for charter spots in the first place. In other words, curriculum and instruction might be identical in charter and non-charter schools, but the educational experience is different in non-charter schools entirely because of the presence of the most-challenged students (academically and behaviorally).

  • Sean says:

    Hi Jay,

    Some quick points:

    1. I’m not sure that second study’s results are as impressive as the first. If I recall, Hoxby used the 66% and 86% figures as the cumulative impact of attending a charter school for grades K-8. The Boston Foundation figure you referenced refers are per-year gains.

    2. The Boston Foundation study is a bit more nuanced than: “charter schools are better.” It is true that high-demand charters had enormous impacts. It is also true that the most important baseline characteristic- test scores from previous years- were 50% of a standard deviation higher (in math and ELA) for incoming charter high school students compared to traditional public school students. Thus, the traditional public students to whom they were compared were not, in fact, representative of traditional public school students. In their less rigorous observational study, they found less substantial gains (but could not control for unobservables).

    3. There has been some recent work done that attempts to disentangle the strategies that make these charter schools successful. Interestingly, Roland Fryer’s 2011/2012 studies suggest that what makes these schools special is not their charter-ness (or “No Excuses” label), but the strategies they employ to increase student achievement. Of course, he also found that these strategies cost $1500-$2000 per student extra, which is prohibitively expensive. The most effective strategy- by a long shot- was high dosages of individualized tutoring. It also happens to be the most expensive.

  • MLA says:

    I believe BB is implying that the schools’ population environment (not the selected sample itself) may be inherently biased in favor of the charter school. So the question remains – are charters given an unfair edge simply because the number of extremely challenging students may be fundamenatally lower in those environments.

  • Keith says:

    The children studied are identical– their peers are not. The lottery winners are in classrooms with students whose parents necessarily chose to enter the lottery; the losers are in classrooms mostly populated with students whose parents did not choose to enter the lottery.

    Which leads BB to wonder if the advantage seen in these studies is an unfair one that can never be replicated nation-wide.

  • Anne Clark says:

    I wonder why charters get results. That’s what we need to figure out. Why.

  • Jay P. Greene says:

    Now that I understand that BB’s objection has to do with peer-effects (thanks to BB and others for clarifying that), here is my response:

    First, peer-effects have generally been shown to be quite modest. They certainly aren’t large enough to explain the benefits of urban charters observed in RCTs.

    Second, even if the charter benefit were entirely explained by peer-effects, wouldn’t we still want to offer students the option to be in the positive learning environment that these critics attribute to charter schools? Would it be better to close the charters send all of the students back to traditional public schools with peers who might disrupt their learning? Even if the implausible peer-effect explanation were true, it’s not clear that our policy conclusion would be any different.

    I should add that Sean is correct in saying that the effects in Boston were more impressive than in NYC and that oversubscribed charters in Boston did better than other charters (although both groups were strongly positive in their effect). There are a lot of nuances to any study, which is why I provided the links and would encourage everyone to read these studies.

    Lastly, I’m a bit struck by the very negative and skeptical tone of some of the comments here and elsewhere to my observation that all 4 RCTs of charter effects find significant benefits for urban students. I would like those critics to honestly imagine how they would react to 4 RCTs that suggested a drug had powerful benefits in fighting one type of cancer (although showed no benefits for other types). Would these critics harp on the fact that some RCTs had stronger effects than others? Would they demand a full explanation of why the drug worked before agreeing to move forward with broader use of the treatment?

    It’s true that even RCTs are imperfect, fail to address every question, and leave some room for doubt. But if the issue were cancer rather than charter schools, would all of these people focus on the doubt or be excited about the positive results?

  • MLA says:

    JPG,

    I wouldn’t have commented if the article wasn’t interesting, thought provoking etc. That said, your response seems overly defensive.

    A couple thoughts –

    1) On peer-effects shown to be modest” – can you back this claim up? Simply writing it off is not enough, especially behind a claim that “charter benefits are proven”. If it’s modest great, tell us why, if not, well, it undermines your whole thesis – hence the questions.

    2) (on not offering alternatives) That’s a clear straw man argument – the issue is figuring out what aspects of charters are providing real replicable benefits. If we can’t ID success factors over and above simply not being around extreme needs kids (peer-effects), then the charter movement is doomed to the margins. You can’t say “hey, this works, because, well, IT DOES.”

    3) (on negativity) I reread the comments – not much negative really, just some honest questions about peer-effects. (And another straw man arg on cancer treatments.) And yes, we can all be interested/excited and still have very valid questions about the studies and conclusions. It’s all part of a healthy debate.

    Ultimately, you are right – the double-blind/RCT tests are what will provide the most insight. I REALLY like that aspect of your article, but clearly the info provided doesn’t quite back up your assertion that the benefits are “proven” without further examination. And that’s all BB and others are saying. Looking forward to more articles – keep up the good work.

    Best,

    Mike

  • BB says:

    To my mind, if the main benefits from charters come from peer effect, then there are two principal take-aways:

    1. This effect does not depend on a whole separate infrastructure of charter authorizers, charter management groups, and individual charter schools. The same effect could be attained in the public school system, as it currently is in public magnets.

    2. Alternatively, the same effect could be attained by creating a system of therapeutic schools (or therapeutic classrooms) for the students who have a negative effect on regular public school classrooms. Currently, public schools are not permitted to do this for various reasons, except for the children who are most extremely affected. But it’s not in theory impossible to improve classroom climate by doing that.

  • Ari says:

    I’m not a statistician nor am I anti-charter, but I’m disappointed that the Global Report Card makes a very basic mistake. In comparing single schools to entire districts, ther is a massive difference in the scales of each. The best school in a district gets averaged in with the rest. Individual schools will automatically have higher peaks and lower valleys in performance. 1/3 of the bottom 30 are probably also charters. I’m also willing to bet that most of the districts listed are very small. This would be more meaningful if it was either a comparison of individual schools or a comparison of operators and districts with a minimum size.

  • Jay P. Greene says:

    I would think that people trying to explain away a large significant finding by suggesting that it was produced by peer-effects would have the burden to show from rigorous research that peer-effects are large enough and likely enough to be a plausible explanation. I haven’t seen that.

    But if you would like from me an example of the modest (and sometimes mixed) effects observed from peer-effects, see:
    Does Peer Ability Affect Student Achievement? Eric A. Hanushek, John F. Kain, Jacob M. Markman, Steven G. Rivkin, October 2003, Journal of Applied Econometrics,
    pp. 527-544. (http://hanushek.stanford.edu/publications/does-peer-ability-affect-student-achievement ).

    They review the literature this way: “Hanushek (1972, 1992) finds no peer achievement effects, while Henderson et al. (1976), Summers and Wolfe (1977) and Zimmer and Toma (2000) report positive influences of higher achieving peers, at least for some students. Consideration of ability tracking in schools likewise has yielded mixed results, even though policy has presumed that tracking is generally bad for achievement (e.g. see Oakes, 1992; Argys et al., 1996). The evidence on achievement effects of racial composition has been much more voluminous, although the results are no easier to summarize or interpret (cf. Armor, 1995).” They then conduct a new analysis and find positive cumulative peer effects of about .2 sd.

  • BB says:

    Interesting. But peer ability is not the same as peer behavior — although the students who don’t apply for charter slots are often the one with the most challenging behavior.

  • MLA says:

    Wow – I’m interested and happy to help here, but this attitude is so wrong-headed it’s not even funny. No one is trying to “explain away” anything. Just some folks asking clear, reasonable questions about your assertions. The articles you cite have no relevance given that charter schools MAY fundamentally change (bias) the base population (sample) of students to the point that the data is hopelessly skewed.

  • Sean says:

    Hi Jay,

    Thanks for looking into those studies.

    Also, in a piece that EdNext published called “Pyrrhic Victories?”, the authors made a wise point that I think applies to your general thesis:

    “When they insist that ideas like school choice… boost student achievement, would-be reformers stifle creativity, encourage their allies to lock elbows and march forward rather than engage in useful debate and reflection, turn every reform proposal into an us-against-them steel-cage match, and push researchers into the awkward position of studying whether reforms “work” rather than when, why, and how they make it easier to improve schooling.”

    I think it’s misguided to headline a feature with “Charter Benefits are Proven with the Best Evidence,” especially when one of the studies you cite argues that charters (in the aggregate) are often mediocre or worse. A more useful approach might be the one that Fryer and others have taken. That is, what makes successful charters successful? And, can those strategies be adapted by different environments with a different set of political constraints? The answers, in Houston at least, are at least somewhat encouraging.

  • [...] Thus the problem is not the amount of money we spend, but how it’s spent. Charter schools typically lead to better educated kids and save us money at the same time. Inner city charter school operators like Eva Moskowitz and Geoffrey Canada and the KIPP schools do a far better job – with fewer tax dollars – than traditional public schools. Even taking the superstars of the movement out of the mix, charter schools outperform traditional public schools. As Jay Greene writes, “Charter Benefits Are Proven by the Best Evidence.” [...]

  • [...] Thus the problem is not the amount of money we spend, but how it’s spent. Charter schools typically lead to better educated kids and save us money at the same time. Inner city charter school operators like Eva Moskowitz and Geoffrey Canada and the KIPP schools do a far better job – with fewer tax dollars – than traditional public schools. Even taking the superstars of the movement out of the mix, charter schools outperform traditional public schools. As Jay Greene writes, “Charter Benefits Are Proven by the Best Evidence.” [...]

  • Michael Langdon says:

    First off, the article confuses correlative evidence with causative evidence. Charters schools are simply exhibiting a placebo effect on students. It isn’t a double-blind if the participants aren’t blind. Subjects can’t know they are receiving the treatment. It is a faulty study. Secondly, this is just another example of faux experts pretending they know what they are talking about.

  • Comment on this Article

    Name ()


    *

         18 Comments
    Sponsored Results
    Sponsors

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform

    Sponsors