An influential 2013 study found that attending an urban charter school in Massachusetts boosts standardized test scores, but attending a nonurban one reduces them—despite the fact that both sets of schools are popular enough with families to hold admissions lotteries. This finding is congruent with other lottery-based research on charters, as well as observational evidence showing gains on test scores for many urban charters but few differences in other settings (see here for a review). However, schools influence much more than just test scores. Now that sufficient time has passed, we can return to the same sample of Massachusetts charter schools and study their impact on college enrollment and graduation.
In a new paper, we do just that, following applicants to 15 urban and nine nonurban charter schools from the time of application to a charter school via an admission lottery, through school enrollment, and up to six years after their expected high school graduation. We make use of the charter school lotteries to generate causal estimates of charter school impacts by comparing students who won the admissions lottery to those who did not.
At the time students in our study were enrolled in school, many of the urban charter schools in our sample employed “No Excuses” practices, including longer school days and school years, a culture of high expectations, frequent teacher observations and feedback, data-driven instruction, use of tutoring, and strict disciplinary practices. Nonurban charter schools in Massachusetts operated with a greater range of educational models, including themed schools and schools emphasizing project-based learning.
As in prior work, we found that urban charter schools raised state standardized test scores, SAT scores, and Advanced Placement (AP) test-taking and scores, though they also lengthened the amount of time students took to graduate high school. The urban charters also increased four-year college enrollment and decreased enrollment in two-year institutions. When we turn to college completion, urban charter students performed as expected given their increased college enrollment. Ultimately, they were about 5 percentage points more likely to receive any college degree, with the bulk of that gain (4 percentage points) coming from increases in BA attainment (see Figure 1a).
Our findings for nonurban charter schools were more surprising. As before, attending one of these schools led students to perform worse on state standardized tests than comparable students who were not admitted. In other markers of academic achievement, students at nonurban charters performed about the same as their counterparts in traditional public schools. They did take fewer AP exams, but this was because the nonurban charters (mostly small schools) were less likely to offer APs than nearby traditional schools. Educational trajectories diverged when we turn to college: attending a nonurban charter school increased the likelihood that students enrolled in a four-year college by 9 percentage points and increased the likelihood that they graduated with a BA within six years by 10 percentage points. This is a very large difference (See Figure 1b).
Despite varying test score impacts, urban and nonurban charter schools in Massachusetts both boost college completion. (Figure 1)
Note: Figure reports mean outcomes for admitted charter applicants who attended charter schools vs. comparable applicants who did not win the admission lottery. Test scores are measured two years after the admission lottery. College enrollment is measured two years after expected high school graduation. College completion is measured six years after expected high school graduation.
In percentage terms, the urban and nonurban college gains are not that different, but that is because a relatively small proportion of the urban sample graduated college. By the sixth year, 23 percent of urban comparison students graduated with a BA, with urban charter attendance increasing this to 27 percent—an increase of 18 percent. In nonurban areas, 50 percent of comparison students graduated college, with the charter effect boosting this outcome to close to 61 percent—21 percent increase.
We draw two main conclusions from these findings. First, multiple charter school models can induce gains in college completion. While many policymakers, advocates, and funders have focused on No Excuses practices as a path forward for charter school success, the nonurban schools in this sample operating on alternative models deliver a large boost to BA attainment. Second, although test scores and longer-term outcomes are typically positively correlated, the relationship between test scores and college outcomes does not hold in all contexts. Researchers and policymakers should therefore be wary of evaluating educational interventions solely on standardized test results, especially when participants are relatively high achieving.
Sarah Cohodes is an associate professor of public policy at the Gerald R. Ford School of Public Policy. Astrid Pineda is a Ph.D. candidate in economics and education at Teachers College, Columbia University.
