Executive Summary
This November, Massachusetts voters will go to the polls to decide whether to expand the state’s quota on charter schools. The ballot initiative would allow 12 new, approved charters over the current limit to open each year.
Would the ballot proposal be good for students in Massachusetts? To address this question, we need to know whether charter schools are doing a better job than the traditional public schools in districts where the cap currently limits additional charter school seats.
There is a deep well of rigorous, relevant research on the performance of charter schools in Massachusetts. This research exploits random assignment and student-level, longitudinal data to examine the effect of charter schools in Massachusetts.
This research shows that charter schools in the urban areas of Massachusetts have large, positive effects on educational outcomes. The effects are particularly large for disadvantaged students, English learners, special education students, and children who enter charters with low test scores.
In marked contrast, we find that the effects of charters in the suburbs and rural areas of Massachusetts are not positive. Our lottery estimates indicate that students at these charter schools do the same or worse than their peers at traditional public schools. Notably, the charter cap does not currently constrain charter expansion in these areas. The ballot initiative will therefore have no effect on the rate at which these charters expand.
Massachusetts’ charter cap currently prevents expansion in precisely the urban areas where charter schools are doing their best work. Lifting the cap will allow more students to benefit from charter schools that are improving test scores, college preparation, and college attendance.
This November, Massachusetts voters will go to the polls to decide whether to expand the state’s quota on charter schools. The “Lift the Cap” referendum has generated enormous controversy, with supporters and opponents canvassing neighborhoods, running ads, and blitzing social media.
As is true with many policy debates, the back-and-forth about the referendum has generated a lot of heat but not much light.
There is a deep well of rigorous, relevant research on the performance of charter schools in Massachusetts. In fact, it is hard to think of an education policy for which the evidence is more clear.
As policies are debated, we often have to rely on research that is ill-suited to the task. Its methodology is frequently too weak to form a firm foundation for policy. Or, the population, design, and setting of the research study are so different from the policy in question that the findings cannot be easily extrapolated.
This is not one of those times. We have exactly the research we need to judge whether charter schools should be permitted to expand in Massachusetts. This research exploits random assignment and student-level, longitudinal data to examine the effect of charter schools in Massachusetts.
To preview the results: Charter schools in the urban areas in Massachusetts have large, positive effects on educational outcomes, far better than those of the traditional public schools that charter students would otherwise attend. The effects are particularly large and positive for disadvantaged students, English learners, special education students, and children who enter charters with low test scores. By contrast, the effects outside the urban areas (where the current cap does not constrain charter expansion) are zero to negative. This pattern of results accords with research at the national level, which finds positive impacts in urban areas and among disadvantaged students.[i]
Massachusetts’ charter cap currently prevents expansion in precisely the urban areas where charter schools are doing their best work. Lifting the cap will allow more students to benefit from charter schools that are improving test scores, college preparation, and college attendance.
Massachusetts’ charter school ballot question
Before we turn to a detailed discussion of the research, let’s summarize the ballot proposal and how it would alter the state’s charter law.
Current law sets a cap on the number of charter schools statewide, as well as the share of each district’s funds that can flow to charters. Massachusetts now has 78 charter schools.
Since 2010, a “smart cap” has given priority to applications from charter providers with a proven track record that seek to expand in low-performing districts.[ii] Even with the additional expansion permitted under the current smart cap, the charter cap constrains expansion in many urban areas, including Boston, Springfield, Malden, and Lawrence. Tens of thousands of students are on waiting lists for charter schools in these districts.[iii] The state’s low-income, immigrant, Hispanic, and Black students are concentrated in these cities.
The ballot initiative would raise the cap, allowing 12 new, approved charters over the current limit to open each year.[iv] New and expanding charters would have to go through the current application and review process, which is one of the most rigorous in the country.[v] An indicator of the robustness of the state’s oversight: since 1997, 17 charter schools that the state deemed ineffective or mismanaged have closed.
The state’s board of education would review any applications that seek to go above the current cap, as it does all charter applications. In contrast, in Ohio (where presidential candidate Donald Trump recently made a visit to a charter school), the state has 69 authorizers, including school districts, higher education institutions, and nonprofit organizations.[vi] Each authorizer has its own standards for approval, renewal, and revocation.
Ohio’s arrangement, in comparison to that in Massachusetts, makes it difficult for the state to set consistent, high standards for charter schools. We suspect that the robust system of accountability in Massachusetts underpins the strong performance of its charter sector.
Estimating charter school impacts
Would the ballot proposal, which allows the expansion of charter schools in low-performing districts, be good for students in Massachusetts? To address this question, we need to know whether charter schools are doing a better job than the traditional public schools in districts where the cap currently limits additional charter school seats.
In short, the answer is “Yes.” In urban, low-income districts of Massachusetts, charter students are learning more than children in the traditional public schools.
We base this statement on rigorous, peer-reviewed research. Since 2007, when we were both researchers at Harvard, we have collaborated with researchers at Harvard and MIT, including professors Joshua Angrist, Thomas Kane, Parag Pathak, and Chris Walters (who is now at Berkeley). In cooperation with the state’s department of education, which provided the student-level, longitudinal data necessary for this research, we have evaluated the effect of charter schools on student achievement, high school graduation, preparation for college, and college attendance.
Measuring the effectiveness of any school is challenging. Parents choose their kids’ schools, either by living in a certain school district or sending them to a private or charter school. As a result, some schools are filled with children of parents who are highly motivated and/or have extensive financial resources. This is selection bias, the key challenge in evaluating the effectiveness of schools.
Charters are required to run lotteries when they have more applicants than seats. And since many charter schools in Massachusetts have long waiting lists, there are many lotteries each year across the state.
The charter school lotteries are “natural experiments,” each their own randomized trial. Randomization is the gold standard for social-science research, allowing an “apples-to-apples” comparison. At the time of application, there are no differences (on average) between those who win and lose the admissions lottery. Should we observe differences in student outcomes after the lottery, we can be confident this is due to charter school attendance.[vii]
The evidence on Massachusetts charter schools
So what have we learned from our research?
Charter schools in Boston (where charter enrollment has almost reached the cap) produce very large increases in students’ academic performance.[viii] Education researchers often express test score differences in standard deviations, which allow for comparison across different tests, populations, and contexts. According to the most recent estimates, one year in a Boston charter middle school increases math test scores by 25 percent of a standard deviation. The annual increases for language arts are about 15 percent of a standard deviation.[ix] Test score gains are even larger in high school.
These differences for middle school and high school can be seen in the two graphs below, with the results disaggregated for subgroups of students. Values above zero indicate that charter school students score higher than their traditional public school counterparts. A shaded bar indicates a statistically significant positive effect.
How big are these effects? The test-score gains produced by Boston’s charters are some of the largest that have ever been documented for an at-scale educational intervention. They are larger, for example, than the effect of Head Start on the cognitive outcomes of four-year-olds (about 20 percent of a standard deviation).[x] The effect of one year in a Boston charter is larger than the cumulative effect of the Tennessee STAR experiment, which placed children in small classes for four years (17 percent of a standard deviation).[xi]
Another gauge of magnitude: the gap in test scores between Blacks and Whites nationwide (and in Boston) is roughly three-quarters of a standard deviation. One year in a Boston charter therefore erases roughly a third of the racial achievement gap.
One concern is that charter schools are just “teaching to the test.” To stay open, charter schools need to demonstrate they are effective, and performance on the MCAS (Massachusetts Comprehensive Assessment System, the statewide test) is an important part of that assessment. If the charter schools are simply coaching students on the skills they need to succeed on the MCAS, they may have little impact on real, lasting learning.
But we found positive effects of Boston’s charters beyond the MCAS test,[xii] and no evidence that they “inflate” MCAS scores.[xiii] These effects are represented in the figure below comparing the percent of charter vs. noncharter students attaining particular outcomes.[xiv] For example, the lottery studies show Boston charters substantially increase SAT scores. This is not explained by differential selection into this optional test, since charter students are just as likely as their peers in traditional public schools to take the SAT.
Boston charters double the likelihood of taking an Advanced Placement (AP) exam. They substantially increase the AP exam pass rate, with ten percent of charter students passing the AP calculus test, compared with just one percent of students in Boston’s other public schools.
Students at Boston’s charters are just as likely as their peers at traditional public schools to graduate high school, though they are more likely (by 14 percentage points) to take five years rather than four years to do so. Boston charter students enter high school with scores far below the state mean, and even further below the typical scores in the wealthy suburbs where AP courses are the norm. It is therefore unsurprising that it takes some students five years in high school to successfully complete AP courses (which are required by some Boston charters).
Boston charter students are far more likely to attend a four-year college than their counterparts in traditional public schools. This is likely due, at least in part, to their better academic preparation, as just explained. The difference is large: 59 percent attend a four-year college as compared to 41 percent for their counterparts who did not attend charters.
Reminder: All of these results are based on comparisons of applicants who randomly won or lost admission to charter schools. The estimates are therefore not biased by demographic differences between students at charters and traditional public schools.
Some might be concerned that the charter students have unusually motivated parents, as demonstrated by their willingness to apply to charters. But by this metric, all of the children in our lottery studies have motivated parents. Yet the students who don’t win admission to charters (and so are more likely to go to the traditional public schools) do far worse than those who win.
It’s also important to note here that more than a third of students in Boston Public Schools apply to charters, so any “cream skimming” goes pretty deep. As charters have expanded in Boston, differences between applicants and non-applicants in the city have narrowed considerably, and are now quite small.[xv]
Beyond Boston, charters in the other urban areas of Massachusetts also boost test scores.[xvi] Most of these schools are young compared to the Boston charters, and we have not yet evaluated their effects on long-term outcomes such as college attendance.
Across the board, we find that urban charters produce the biggest boosts for students who most need help. Score effects are largest for students who enter charters with the lowest scores. Urban charters are particularly effective for low-income and non-white students. The score gains for special education students and English learners are just as large as they are for students who are not in these specialized programs.[xvii]
In marked contrast, we find that the effects of charters in the suburbs and rural areas of Massachusetts are not positive. Our lottery estimates indicate that students at these charter schools do the same or worse than their peers at traditional public schools.
Many students in these non-urban districts have access to excellent schools, so it is not surprising that charters don’t produce better outcomes than the traditional public schools. In fact, the excellent schools are a draw for families who have the financial resources to move to high-performing, wealthy districts like Newton, Wellesley, and Weston. Low-income families can’t afford homes in these districts. Their choice is the local charter school.
Importantly, the charter cap does not constrain charters in the suburbs where they appear to have zero to negative effects. Current law allows charter schools to expand in these districts. The cap, if lifted, would expand choice in the urban areas where charters have been highly successful with disadvantaged students who most need access to better schools.
No one (including social scientists!) can predict the future. There is no guarantee that new charter schools will be as successful as existing charter schools. The research we have summarized here, and the state’s track record in carefully vetting schools, strongly suggest that if allowed to grow the charter schools in the urban areas of Massachusetts will continue to improve learning, especially among disadvantaged children.
The voters’ decision
The research we have summarized here is irrelevant to the decisions of some voters. Some oppose charter schools on principle, because they prefer the governance and structure of traditional public schools. That’s their prerogative.
What we find distressing, and intellectually dishonest, is when these preferences are confounded with evidence about the effectiveness of charter schools. The evidence is that, for disadvantaged students in urban areas of Massachusetts, charter schools do better than traditional public schools.
Voters are free to decide that the proven benefits that Massachusetts charter schools provide for disadvantaged students are outweighed by a principled opposition to charters. It’s our job as researchers to make clear the choice that voters are making.
— Sarah Cohodes and Susan Dynarski
Sarah Cohodes is an Assistant Professor of Education and Public Policy at Teachers College, Columbia University. Susan Dynarski is a professor of public policy, education and economics at the University of Michigan.
This post originally appeared as part of Evidence Speaks, a weekly series of reports and notes by a standing panel of researchers under the editorship of Russ Whitehurst.
Notes:
[i] See, for example, Gleason, Philip, Melissa Clark, Christina Clark Tuttle, Emily Dwoyer, and Marsha Silverberg. 2010. The Evaluation of Charter School Impacts: Final Report. NCEE 2010-4029. Washington, DC: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences.
[ii] Massachusetts General Laws, An Act Relative to the Achievement Gap, 2010.
[iii] Massachusetts Department of Elementary and Secondary Education. Massachusetts Charter School Waitlist Updated Report for 2015-2016 (FY16). See attached spreadsheet for location-specific waitlist numbers.
[iv] https://ballotpedia.org/Massachusetts_Authorization_of_Additional_Charter_Schools_and_Charter_School_Expansion,_Question_2_(2016)
[v] National Association of Charter School Authorizers: Massachusetts.
[vi] National Association of Charter School Authorizers: Ohio.
[vii] The lottery analyses are conducted using two-stage least-squares (2SLS). Winning the lottery is used as an instrument for attending a charter school. Throughout, when we refer to “the effect of charter school attendance,” we mean the 2SLS estimate of the effect of charter attendance, with winning the lottery used to instrument for attendance.
[viii] Abdulkadiroglu, Atila, Joshua D. Angrist, Susan M. Dynarski, Thomas J. Kane, and Parag A. Pathak. 2011. “Accountability and Flexibility in Public Schools: Evidence from Boston’s Charters and Pilots.” Quarterly Journal of Economics 126(2): 669–748.
[ix]Cohodes, Sarah R., Elizabeth M. Setren, Christopher R. Walters, Joshua D. Angrist, and Parag A. Pathak. 2013. “Charter School Demand and Effectiveness: A Boston Update.” The Boston Foundation.
[x] Puma, Mike, Stephen Bell, Ronna Cook, Camilla Heid, Pam Broene, Frank Jenkins, Andrew Mashburn, and Jason Downer. 2012. Third Grade Follow-up to the Head Start Impact Study Final Report, OPRE Report # 2012-45, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
[xi] Dynarski, Susan, Hyman, Joshua. and Schanzenbach, Diane. W. 2013. “Experimental Evidence on the Effect of Childhood Investments on Postsecondary Attainment and Degree Completion.” Journal of Policy Analysis and Management 32: 692–717.
[xii] Angrist, Joshua D., Sarah R. Cohodes, Susan M. Dynarski, Parag A. Pathak, and Christopher R. Walters. 2016. “Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice.” Journal of Labor Economics 34(2).
[xiii] Cohodes, Sarah. 2016. “Teaching to the Student: Charter School Effectiveness in Spite of Perverse Incentives.” Education Finance and Policy 11(1): 1-42.
[xiv] In the graph, the percentages for charter students are the 2SLS estimates of effect of charter attendance added to relevant noncharter mean. The percentages for the noncharter students are the proportion who attain each outcome, for students in the sample who do not attend a charter school.
[xv] Cohodes, Sarah R., Elizabeth M. Setren, Christopher R. Walters, Joshua D. Angrist, and Parag A. Pathak. 2013. “Charter School Demand and Effectiveness: A Boston Update.” The Boston Foundation.
Setren, Elizabeth. 2015. “Special Education and English Language Learners in Boston Charter Schools: Impact and Classification.” School Effectiveness and Inequality Institute (SEII) Discussion Paper 2015.05.
[xvi] Angrist, Joshua D., Parag A. Pathak, and Christopher R. Walters. 2013. “Explaining Charter School Effectiveness.” American Economic Journal: Applied Economics 5(4): 1–27.
[xvii] Setren, Elizabeth. 2015.“Special Education and English Language Learners in Boston Charter Schools: Impact and Classification.” School Effectiveness and Inequality Institute (SEII) Discussion Paper 2015.05.
