Charter schools are more popular in Arizona than in any other state. In the 2012-13 school year, 13.3 percent of Arizona students attended charter schools, almost three times the national average of 4.6 percent. That same year, Arizona’s 530 charter schools accounted for nearly a quarter of all public schools in the state. But student-level data on Arizona’s schools have not been made widely available, so the state’s charters have not been subject to the kinds of impact evaluations that have been completed in states such as Florida and North Carolina and cities such as Boston and New York.
We provide the first recent, comprehensive look at Arizona’s charter schools in a new paper released last week, which will be published in a forthcoming issue of the peer-reviewed journal Educational Evaluation and Policy Analysis. Our analysis is based on statewide, student-level longitudinal data obtained from the Arizona Department of Education (AZDOE) that contains information on test scores, school enrollment, and student characteristics for the 2005-06 through 2011-12 school years.
Many prominent studies of charter schools take advantage of admission lotteries to compare students who were equally interested in attending a charter, but only some of whom were given the opportunity. That method is the best way to measure the impact of those schools, but these studies have a potentially important shortcoming: they can only examine charters that are so popular that they have more applicants than available seats. Our study examines a wide range of charter schools in Arizona using methods that have been shown to best replicate lottery-based results.
We focus our analysis on charter middle schools, because we are able to compare charter and traditional public school students who had similar entering test scores and demographic characteristics and even attended the same elementary school. We also examine high schools, taking into account students’ academic performance at the end of middle school. It is not possible to use this methodology to examine elementary schools because testing begins in third grade, so for those schools we compare test-score growth in traditional public schools and charter schools while taking into account student characteristics such as race, age, and special education status.
On average, charter schools in Arizona do no better, and sometimes worse, than the traditional public schools. Figure 1 shows our estimates of the average impact of attending a charter school on test scores, expressed in standard deviation units. To put these units in context, the average middle school student gains about a quarter of a standard deviation per year; for elementary students, the average gain is between a third and a half of a standard deviation. For example, the negative impact of charter middle schools on math scores of 0.02 standard deviations translates into about 10 percent of a year of learning.
We find negative average impacts of attending a charter school on math scores for all three grade levels, and on science scores for middle and high schools but not for elementary schools. We don’t find any evidence that charters have much of an impact on reading scores at any grade level.
These results tell us whether a student attending a randomly selected charter school will perform better, on average, than a similar student attending a traditional public school. But only comparing averages likely misses a lot of nuance. There are good and bad charters, just as there are good and bad traditional public schools. But do charters vary more in terms of their ability to promote student achievement than comparable traditional public schools? Our data allow us to answer this question empirically, something few, if any, prior studies have done.
We find that charters vary more in their impact on student performance on state tests than traditional public schools. In other words, even though the average charter has a zero or negative impact on test scores, there are more charters with very large positive or very large negative test-score impacts than there are traditional public schools with such extreme outcomes. We also find that the negative impacts of charters are concentrated in non-urban areas (Figure 2), which is consistent with a lottery-based national study finding that charter middle schools deliver better results in urban areas.
The greater variability in the quality of charters is consistent with the idea that they are laboratories for innovation and experimentation, some of which succeed and some of which fail. We investigated further whether certain types of charters are likely to succeed or fail by separating charter schools into categories based on their mission statements. Figure 2 shows that schools with missions emphasizing academic rigor had positive effects on math scores, whereas those with a progressive (e.g., focused on the “whole child”) or more general mission statement had negative impacts. Schools focused on the arts also had negative impacts, perhaps because their focus is on an area other than core academic subjects.
Part of the deal struck by charter operators is that in return for the freedom to innovate they will be held accountable for results. Thus far, the system of accountability in place in Arizona has not produced a charter sector that produces better outcomes, on average, than the traditional sector. But we do find that the charter sector is better at weeding out poorly performing schools than the traditional sector. Figure 3 shows that the charter elementary and middle schools that closed were significantly less effective in math, reading, and science than traditional and charter schools that remained open. The same was not true for traditional public schools that closed, which barely differed from traditional schools that remained open.
Arizona’s charter school law is unique in allowing charter schools to operate for 15 years before coming up for review. Because the most rapid expansion of the Arizona charter sector occurred around the turn of the 21st century, many charters are poised to come up for review in the next few years. This provides an opportunity for rapid improvement through careful attention to quality in the reauthorization process, and the fact that lower-quality charter schools have been more likely to have their charters revoked in recent years is encouraging in this regard. But our evidence also suggests that a 15-year period with little oversight of academic quality may be too long to wait to intervene and potentially close schools that are producing subpar results. A shorter authorization period accompanied by vigorous efforts to measure quality along the way may strike a better balance between autonomy to innovate and accountability for results.
The Arizona State Board for Charter Schools, which serves as the authorizer for more than 90 percent of charter schools in the state, has already taken important steps in this direction. In October 2012, it adopted a new Academic Performance Framework that subjects each of its schools to an annual review based primarily on student achievement levels and growth and clarifies that charters may be revoked well before the 15-year contract expires. Although standardized test scores should not be the only metric to judge quality, especially among schools with certain specialized missions, the framework appears to provide a sound basis for taking timely action to address consistent under-performance.
The mediocre performance of Arizona’s charter sector as a whole should not overshadow the impressive work being done in some individual schools. The same should be said for the traditional sector, in which there is also substantial, albeit less, variation in quality. But part of what makes the charter idea compelling is that it provides opportunities for schools to innovate, while not tolerating persistent failure. For this ideal to be realized, policies that drive continuous improvement of the sector over time need to be in place.
—Matthew M. Chingos and Martin R. West
This post originally appeared on the Brookings Brown Center Chalkboard blog.
