In this paper, we use an extensive student-level data set to evaluate the impact of charter schools in North Carolina on the math and reading performance of students in grades 4 through 8. We address three main questions: Do students attending charter schools in these grades make larger or smaller gains in achievement than they would have made in traditional public schools? If so, what accounts for the quality differences between charter schools and traditional public schools? And, finally, do students who attend traditional public schools subject to competition from charter schools make larger achievement gains than they would have in the absence of charter schools?
Controlled Choice
Legislation authorizing charter schools in North Carolina was passed in 1996, and the first charter schools opened in the fall of 1997. By capping the number of charter schools statewide, limiting the annual growth in the number of schools per district, and providing for input from the local district before approval of charter applications, North Carolina has exercised more control over the establishment of charter schools than some states. Nevertheless, the North Carolina legislation is quite permissive in that it allows any individual or group to apply for a charter and does not require local district approval of a charter application. North Carolina charters operate as independent nonprofit corporations, act as their own employers, are automatically exempted from several regulations, receive operating funding at the same level as traditional public schools, and are subject to the same state testing requirements as traditional public schools. Thus North Carolina’s program includes many of the elements recommended by charter school advocates. However, the state does not provide charter schools any additional funding for facilities.
The number of charter schools in North Carolina grew steadily after 1997. By 2001–02 there were 93 charter schools and more than 18,000 charter school students; charter schools made up 4 percent of all schools and enrolled 1.4 percent of the state’s 1.3 million students. Charters can be revoked for a number of reasons, including poor student performance and financial mismanagement. Overall, about 12 percent of the charter schools that have been opened are now closed, but in no case was the decision to revoke a charter or to close due primarily to low student performance. Growth in the number of charter schools has slowed markedly since 2001–02, primarily because the state law caps the number of charter schools at one hundred. Even so, only seven states had more charter schools than North Carolina as of 2002 and of those, only five had a greater concentration of charter schools: Arizona, Florida, Wisconsin, Michigan, and California. The 93 charter schools operating in 2001–02 were spread across 46 of North Carolina’s 100 counties.
Our analysis of these schools is based on data from the North Carolina Education Research Data Center, a collaborative effort involving the North Carolina Department of Public Instruction, Duke University, and the University of North Carolina. The database contains individual-level information on test scores and background characteristics for all students in grades 3 through 8 in the state’s public schools, charter and traditional. We use these data to measure the gains in student achievement made by individual students each year between 4th and 8th grade.
The mixture of 3rd through 8th graders in North Carolina charter schools in 2001–02 differed from that in traditional public schools (see Figure 1). Compared with traditional public schools, charter schools in North Carolina enrolled a larger percentage of black students and lower percentages of Hispanic and white students. Roughly 40 percent of charter school students in grades 3–8 were black, compared with 31 percent in traditional public schools. At the same time, charter schools served a higher percentage of students whose parents are college educated and a lower percentage of students whose parents are high school dropouts.
SOURCE: Authors’ presentation of data from North Carolina Education Research Data Center |
Despite the higher average education level of their parents, charter school students exhibit lower levels of performance on end-of-grade tests in both reading and math. The gap between students in charter schools and students in traditional public schools is .12 standard deviations in reading and .22 standard deviations in math. Our analysis attempts to determine if any of this difference in performance can be attributed to the charter schools themselves.
Measuring Charter School Effectiveness
The primary challenge in determining how effective charter schools are in raising student achievement arises from the fact that charter school students are self-selected. Because they have chosen to attend a charter school, they are likely to differ in unobserved ways from otherwise similar students who choose to remain in traditional public schools.
In principle, the best way to determine how effectively charter schools raise student achievement is to conduct a randomized experiment. Studies adopting this approach take the students interested in attending a charter school, use a lottery to assign them randomly either to the charter school or to a control group of students who would not have access to that school, and then compare the achievement of the students given access to the charter school with that of the students in the control group. This approach, which is used by Caroline Hoxby and Jonah Rockoff in their study of charter schools in Chicago (see “Findings from the City of Big Shoulders”), is useful for determining if a particular charter school or the education program it offered is effective. However, the results of such experimental studies apply only to the programs offered by and the type of students who apply to the specific oversubscribed charter schools evaluated.
When studying an entire system of charter schools, including some that are not oversubscribed, it is not possible to conduct a true experiment. Thus we use a method that in effect compares the test-score gains of individual students in charter schools with the test-score gains made by the same students when they were in traditional public schools. This approach provides powerful protection against bias from self-selection. However, this protection comes at a cost. In the end, our analysis of charter school effectiveness is based on the experiences of only those students for whom we observe annual gains (whether positive or negative) in test scores at least once in a charter school and at least once in a traditional public school. Because we have information on students only in grades 3 through 8, they must have attended a traditional public school at least once between 4th and 8th grade to be included in our analysis. If such students are not representative of all students who attend charter schools, our analysis may not provide an accurate measure of the average effect of attending a charter school in these grades. As we show below, however, this limitation does not affect our basic conclusions.
The Effect of Attending a Charter School in North Carolina
Our data set includes information on five cohorts of students: those in 3rd grade in 1996, 1997, 1998, 1999, and 2000, some 495,000 students altogether. Each cohort contains all the students in 3rd grade in North Carolina public schools, including charter schools, during the specified year. Each student has a unique identifier that is consistent over time, which allows us to follow students from 3rd grade through the last year that they remain in North Carolina public schools, the year they complete 8th grade, or the 2001–02 school year, whichever comes first. To compare test-score gains of students from different grades, we first standardize their raw scores separately by grade and year to have a mean of zero and standard deviation of one and measure the change in each student’s standardized score from one year to the next. A gain score of zero indicates that a student has kept pace with the average student in the state, while a student with a gain score of 0.25 standard deviations will have improved his or her performance by enough to exceed roughly 10 percent of the state’s students.
We first compare the average gains made by all students in charter schools with the gains made by students in traditional public schools, taking into account differences in gender, ethnicity, and the highest level of education completed by their parents. Because moving between schools is known to have a negative impact on student achievement, we also control for whether the student changed schools in the current year and whether that change was structural (for instance, the student moved to a junior high school from its feeder elementary school).
The results of this initial analysis reveal that students in charter schools, on average, make annual gains that are 0.06 standard deviations less in reading and 0.08 standard deviations less in math than students in traditional public schools with similar observable characteristics. Because the students in charter schools are self-selected, however, this difference could be attributable to one or more unmeasured characteristics of the students rather than to the fact that they are in a charter school.
As explained above, we address the problem of self-selection by comparing the gains made by students the years they were in charter schools with the gains made by the same students the years they were in traditional public schools. Of the 8,745 students in our data set who attended a charter school for at least one year, 5,746 also attended a traditional public school at least once between 4th and 8th grade. The results of our analysis of these “switchers,” which continues to take into account the difficulties associated with moving between schools, again indicate that students make smaller gains while enrolled in charter schools, by nearly 0.10 standard deviations in reading and 0.16 standard deviations in math. This pattern provides strong evidence that the smaller gains made by these charter school students are indeed due to the quality of the schools they attend rather than to any unobserved differences between charter school students and students in traditional public schools.
The difference in the rate of achievement growth between students enrolled in charter schools and students in traditional public schools is substantial. It is substantially larger than differences between the growth rates for children of high-school dropouts and the children of parents with graduate degrees as well as those between blacks and whites, differences that are the focus of considerable concern. For example, our analysis indicates that the annual gains made by children of high-school dropouts lag behind those of children of parents with graduate degrees by 0.03 standard deviations in reading and 0.06 standard deviations in math. The negative effects of attending a charter school, on average, for the students in grades 4 through 8 included in our analysis, are roughly three times this large.
How Representative Are the NC Charter Students?
Analyses of the effects of attending a charter school may be misleading if the students included are not representative of all students who actually attend charter schools. It is therefore important to consider how the 5,746 “switchers” included in our final analysis, those who attended both a charter school and a traditional public school in North Carolina between grades 4 and 8, differ from the state’s full population of 8,745 charter school students in these grades. Although our data confirm that the two groups are quite similar demographically, two differences are worth noting.
The first is that students who leave charter schools before 8th grade to return to public schools are overrepresented in our analysis. Thirty-seven percent of the students for whom we observe test-score gains at least once in both sectors attended a traditional public school after they were in a charter school, while the same is true of only 30 percent of all students in charter schools. That is, charter school “exiters” are overrepresented in our analysis by nearly 25 percent. If these exiters left charter schools because they were not doing well there academically, our estimate of the effect of attending a charter school may be more negative than the true effect for all charter school students.
Looking separately at the effect of attending a charter school for exiters reveals that the effect of attending a charter school is, in fact, considerably more negative than for students who were observed first in a traditional public school and remained in a charter school throughout the study period (see Figure 2). We therefore calculated weighted averages of the effects for students observed only entering charter schools and the effects for students observed exiting charters, with the weights equal to the proportion of each group in the total population of charter school students. The corrected estimates are -0.09 standard deviations in reading and –0.15 standard deviations in math. In short, the overrepresentation of exiters matters, but it accounts for only a small fraction of the estimated negative effect of charter schools.
* Significant at the 0.05 level. SOURCE: Authors’ calculations based on data from the North Carolina Education Research Data Center |
A second key difference between switchers and the larger group of students attending charter schools in any year is that students who entered charter schools in the younger grades are underrepresented. Although our data do not allow us to address this issue directly while still accounting for the self-selection of students into charter schools, simple comparisons indicate that students who entered charter schools in the later grades made smaller gains in math (but not reading) than students who entered earlier. Given the underrepresentation of students who enter during early grades, this difference suggests that the average effects of attending a charter school across all grades, 4 through 8, may be less negative than indicated by our final analysis, at least for math. Even so, we see no reason to suspect that the true average effects are not negative.
New Schools with New Students
One potential explanation for these findings is that many of North Carolina’s charter schools were in their first years of operation and thus were grappling with the challenges of starting a new school. We therefore reran our analysis, allowing for the effect of attending a charter school to vary with the number of years the charter school had been open. The results confirm that the negative effects of attending a charter school are considerably greater for students in newly opened charter schools than for students in charter schools that are more established. After again correcting for the overrepresentation of exiters, the effects of attending a newly opened charter school were -0.17 standard deviations in reading and -0.28 standard deviations in math, or almost twice the average effect reported above for all charter schools in the state. However, it is important to note that the complications associated with being a new school cannot fully explain the poor average performance of charter schools: the negative effects of attending a charter school in North Carolina remain greater than .10 standard deviations in both subjects, even for schools that have been operating for five years (see Figure 3a and 3b).
* Significant at the 0.05 level. SOURCE: Authors’ calculations based on data from the North Carolina Education Research Data Center |
* Significant at the 0.05 level. SOURCE: Authors’ calculations based on data from the North Carolina Education Research Data Center |
We next considered whether the effects of attending a charter school also varied with the length of time the specific student had been enrolled. The results proved informative. First, the negative average effects of attending a charter school are driven largely, but not entirely, by students making unusually small gains during their first year in a charter. This is true regardless of how long the school has been operating. For some reason, transferring into a charter school seems to have a much more negative effect on achievement than transferring into a traditional public school. Second, students who choose to remain in charter schools do not continue to make smaller gains than students in traditional public schools after their initial year in a charter school. This is reassuring, in that it justifies the decision of many parents to keep their children in charter schools once they are there; the disruptive effects of moving between schools would make the return to a traditional public school counterproductive. However, it is also clear that the initial achievement hit these students take is not offset by gains in subsequent years, so that even this group, which is harmed least by attending a charter school, still has lower levels of achievement as a result of attending a charter school. Finally, we see that students who ultimately leave charter schools two or more years after they enter continue to make smaller gains relative to those they would have made in a traditional public school even after their first year in a charter school.
One reason North Carolina’s charter schools might have difficulty providing effective education programs is high rates of student turnover. Rapidly changing student populations make student grouping and scheduling more challenging, intake of new students can distract administrators from other tasks, and assessing and helping new students can place extra demands on teachers’ time. On average, the percentage of students in a school in grades 4 through 8 that made a nonstructural transfer in the previous year is far higher in charter schools than in traditional public schools. While only 14 percent of students in traditional public schools made nonstructural transfers, the same is true of more than one-quarter of students in fifth-year charter schools and of an even larger share of students in newer charter schools. Taking into account the higher rates of student turnover in charter schools reduces the magnitude of the estimated negative effect of charter schools by 29 percent in reading and by 30 percent in math. Thus high rates of student turnover may account for as much as one-third of the negative impact charter schools have on student performance. Even so, the coefficients on the charter school variable remain statistically significant, suggesting that other factors also play a role in the poor performance of charter schools.
Effects of Competition on Traditional Public Schools
Charter schools have the potential to have broader effects on student achievement if traditional public schools respond to the threat of losing students to charter schools by improving the quality of their own education programs. Although the number of charter schools in North Carolina and the nation has grown rapidly over the past decade, they still represent only a fraction of the total number of public schools and are likely to remain so for a number of years. Still, if North Carolina’s traditional public schools improved in response to their presence, the apparently negative effects of charter schools on the achievement of students who attend them could be offset by more positive statewide effects.
To estimate the effects of charter schools on students in traditional public schools, we use information on each school’s distance from the nearest charter school to develop indicators of whether or not the traditional school faces competition from charter schools. How close does a charter school have to be located to a traditional public school to provide meaningful competition? For 90 percent of the 6,576 transfers in our database, the distance between the charter school where the student enrolled and the traditional public school the student attended the previous year is less than ten miles. Our data also indicate that schools within 2.5 miles of a charter school lose a higher percentage of students to charter schools, and hence appear to face more competition, on average, than do schools 2.5 to 5 miles from the nearest charter, and that the threat of losing students to a charter school depends also on the number of charter schools within a given radius of the school. Our analysis of competitive effects therefore investigates whether the effect of charter schools on traditional public schools varies with the number of nearby charter schools as well as with the distance to the nearest charter.
First, however, we must take into account the fact that the location of charter schools is not randomly determined. If charter schools were primarily established in response to dissatisfaction with traditional public schools, they would tend to be located in areas with low-quality traditional public schools where students would tend to make below-average test-score gains. Alternatively, charter schools might be more likely to attract students in areas where parents tend to be more motivated and more informed. In those areas, gains in student test scores might be higher than in other areas, even in the absence of charter schools.
To address the problem of nonrandom location, we essentially measure the effect of charter school competition on test-score gains by comparing the gains made by students in each traditional public school before the establishment of a nearby charter with the gains those same students made in that school after the arrival of nearby charter schools. Our results suggest that traditional public schools did not respond to competition from charter schools by becoming more effective, at least as measured by the learning gains made by individual students in the years immediately following establishment of charter schools. Not only are none of the estimated effects statistically different from zero, but many point in the opposite of the expected direction. We emphasize, however, that the intensity of competition in North Carolina is not very great. Even schools located close to several charter schools are unlikely to lose a substantial percentage of students. Thus our finding should not be interpreted as a general statement about the potential of charter school competition to influence traditional public schools.
Conclusions
We set out in this research to provide a comprehensive evaluation of the impact of charter schools on the math and reading performance of North Carolina students in grades 4 through 8. Our results can only be described as discouraging for charter school supporters. Students in these grades make considerably smaller achievement gains in charter schools than they would have in traditional public schools, and the negative effects are not limited to schools in their first year of operation. Nor are the negative effects of attending a charter school substantially offset by positive effects of charter schools on traditional public schools, a finding that may reflect the fact that North Carolina charter schools provide only a limited amount of competition.
However, for students who choose to remain in charter schools, the negative effects of attending a charter school are largely limited to their first year of attending a charter school. It is also important to note that our findings apply only to students who either entered a charter school after grade 4 or exited a charter school before grade 8. Our data do not allow us to comment on the experience of students who entered charter schools before grade 4 and attended them through the end of middle school.
We also provide evidence that high student turnover rates may account for about 30 percent of the difference between test-score gains made in charter schools and what we would expect the same students to make in traditional public schools. This finding suggests that student turnover can be an unintended negative side effect of school choice. Because school-choice plans lower the costs to families of switching schools, it is plausible that such plans will increase the movement of students across schools and thereby increase student turnover rates, to the detriment of all students.
However, charter schools in North Carolina exhibit negative effects on student achievement even after controlling for student turnover rates. Further investigation to determine whether the remaining negative effects are due to peer influence, resource inadequacies, or poor management would be useful. Whatever the reason for the low performance, the public interest is not well served when charter schools are ineffective in raising student achievement.
-Robert Bifulco is assistant professor of public policy, University of Connecticut. Helen F. Ladd is professor of public policy studies and economics, Duke University.
