Strict attention to detail, long school days, and a singular focus on college are the hallmarks of “no excuses” charter schools. Families in cities across the United States have flocked to them as academic game changers, and research shows that many of their students beat the odds on standardized tests.
But critics allege that such gains are hollow. The “no excuses” approach, they say, amounts to a paternalistic culture of test preparation that detracts from real learning and comes at a steep cost to social and emotional health. Successfully navigating adult life, including the risks and rigors of college, will take much more.
Do “no excuses” charter high schools merely help students succeed on standardized tests? Or are their students more likely to succeed after they leave school behind? Is it test prep, or true learning? Little prior research is available on this question. And although there is a robust positive correlation between test performance and college enrollment, there is little existing evidence as to whether schools that increase test scores the most also help their students succeed at the next level.
To shed light on these questions, we studied Noble Street College Prep, a high-performing no-excuses charter high school in Chicago where admission is granted via randomized lottery. We analyzed student records to estimate the effect of attending Noble on college enrollment, persistence, and quality, using success in postsecondary studies as a proxy for success in young adulthood.
Overall, our results suggest that the benefits of attending a no-excuses charter high school extend beyond graduation and into early adulthood. Students who attended Noble Street College Prep were not only more likely to enroll in college, but also far more likely to enroll in a competitive four-year school. They were also more likely to persist in college, trends that continued for several years after high school graduation.
We were only able to obtain randomized lottery information from the College Prep campus, but data from a broader group of Noble high schools indicate they have higher college enrollment rates than other schools with similar student populations. This result suggests that the Noble Network of Charter Schools has continued to produce positive results as it has expanded.
These findings provide strong evidence that Noble measurably improves students’ preparation for college as opposed to just pushing marginal students into low-quality institutions. That is in keeping with its stated mission, “to prepare low-income students with the scholarship, dedication, and honor necessary to succeed in college and lead exemplary lives, and be a catalyst for education reform in Chicago.”
A College-Centered Culture
Noble Street College Prep was founded in 1999 as Noble Street Charter High School. In 2006, it was renamed and the network began to expand. The Noble Network currently includes 17 campuses with more than 11,000 students in the west and south sides of Chicago (Figure 1). The schools attract a predominantly poor, minority student body: 98 percent of students are black or Hispanic and 89 percent are eligible for free or reduced-price school meals.
Noble network schools follow key practices and principles typically associated with the no-excuses approach: frequent teacher feedback, data-driven instruction, high-dosage tutoring, increased instructional time, and high expectations. All students are expected to take college entrance exams and win acceptance to college. Acceptances are celebrated publicly, and counselors assist students in applying for grants and scholarships.
On average, students spent 7.5 hours per day and 185 days per year in school, compared to an average of 6.9 hours per day and 170 days per year in Chicago Public Schools (CPS) during the period this analysis covers. This implies that Noble students spent 18 percent more time in school than their peers in Chicago, on average, amounting to 858 hours over four years, or nearly three-quarters of a year of additional instruction.
The school day is structured to ensure that all students receive differentiated instruction in smaller-group settings, organized by performance on interim assessments. During morning and afternoon meetings, teachers track individual academic progress, mark behavioral infractions, and hold students accountable as a group for maintaining academic and behavioral standards. Each afternoon, teachers maintain office hours for optional academic support, which becomes mandatory if a student’s performance falls below a certain threshold. Most campuses also feature some form of afterschool tutoring provided by outside organizations.
Noble aggressively recruits teachers with a demonstrated track record of success and rewards teachers whose students demonstrate above-average academic growth with performance bonuses. Teachers receive regular feedback on their performance and attend campuswide professional development sessions each Friday. They meet regularly to analyze student data and collaboratively plan how to use it to drive instruction.
Looking at a Lottery
We used lottery data to compare Noble students to a comparison group of their peers throughout Chicago in order to estimate the effect of attending Noble on college enrollment and persistence. Our analysis focuses on the three cohorts of Noble students who enrolled between 2003 and 2005, as those were the only years in which students were enrolled via randomized lottery and for which school records were complete.
Our lottery-based results are only for Noble Street College Prep. However, we also report nonexperimental results on a wider group of schools in the Noble network based on school-average data from later years, through 2013.
We gathered data on student characteristics from Noble’s lottery and enrollment files. A typical lottery file included a student’s name, gender, date of birth, address, 8th-grade school, sibling indicators, lottery result (accepted or waitlist), and waitlist position. We reviewed records for both lottery losers and winners, which we defined as students immediately accepted at Noble or offered one of the first 10 waitlist positions.
Noble also provided us with enrollment records, which we merged with the lottery data to identify which students eventually enrolled. To ensure that all students in our analysis had an equal chance of being accepted, we exclude students who were automatically accepted because they had an older sibling enrolled in a Noble school. If students entered multiple lotteries over a series of years, we only included the first entry.
We linked the lottery data to information on college enrollment and persistence from the National Student Clearinghouse (NSC), a nonprofit organization that maintains a database of students’ college enrollment and graduation records. The link was made using each student’s name, date of birth, and expected high-school graduation date. At the time of our submission, the NSC database included enrollment records at more than 90 percent of colleges and universities in the United States. By necessity, we assumed that students who did not match any records in the NSC database never enrolled in college.
Noble also provided us with internally collected data on college enrollment of their graduating seniors, which were consistent with the NSC match for 93 percent of students. We use the NSC data throughout our analysis because they are available for both lottery winners and losers.
We also linked the lottery data to publicly available CPS data on students’ middle schools, including the percentage of 8th graders who scored proficient or better on the math section of the Illinois Standards Achievement Test (ISAT), the percentage scoring proficient or better on the reading section, and the percentage of black or Hispanic students. In total, we were able to match 72 percent of our sample of lottery applicants to their 8th-grade school’s CPS summary records.
Like students in most Chicago schools during the period studied, students who won the lottery and enrolled at Noble were roughly 90 percent black or Hispanic, though the school enrolled a much larger share of Hispanic students than the average charter or traditional public school at that time. In more recent years, the network’s collective student body has been quite similar to the rest of the district on nonracial dimensions. Noble students are marginally less likely to qualify for special education, but slightly more likely to be eligible for a free or subsidized lunch.
Summary data also show that Noble students perform extremely well on standardized tests. More precisely, Noble students enter high school with slightly lower test performance than the average public school student, though significantly higher than the average student at a Chicago charter school. However, by 11th grade, Noble students score markedly higher than the CPS average and the charter average on all sections of the ACT. In the 2013–14 school year, all 10 Noble campuses with enrolled 11th graders ranked in the top quarter of all Chicago high schools for overall ACT performance, scoring in the top 33 out of 156 schools with reported results. In addition, Noble students were also more likely to earn a high school diploma. Students entering 9th grade at a Noble campus were 21 percentage points more likely to graduate within five years of enrollment than their peers in traditional public schools, and 20 percentage points less likely to drop out within five years of enrollment.
The crux of our analysis is a comparison of lottery winners and losers, so it is crucial that the process used to determine who was offered a seat at Noble was truly random. A comparison of students’ data from before the lottery confirms that the two groups are similar with respect to all characteristics we can observe. We found no significant differences between lottery winners and losers on characteristics, including gender, age at high school entry, and the math and reading scores and racial composition of their middle schools.
Effect on College Enrollment
We found that students who won the Noble lottery were far more likely to enroll and persist in college than their peers who lost the lottery and thus attended high school elsewhere. Lottery winners were 10 percentage points more likely to enroll in college than students who did not win the lottery, a 17 percent increase compared to the losers’ college enrollment rate of 59 percent. When we adjusted the results for the fact that not all lottery winners attended Noble, we find that actually enrolling in Noble for any length of time increased college enrollment by 13 percentage points, or 22 percent (Figure 2). Our main analyses control for students’ age, gender, and the average test scores at their middle schools, but we obtain similar results from a simple comparison of lottery winners and losers, as we would expect given the use of the lottery.
A natural concern is that the increase in college enrollment might come at the expense of quality. What if Noble merely pushes students who are on the fence about whether or not to attend college into lower-quality schools? We found that this was not the case by examining the type of degrees offered at the schools students attended, as well as the test scores of incoming students, as a proxy for rigor and quality.
Noble students were 15 percentage points more likely to attend a four-year school and 14 percentage points more likely to attend a college where the median two-subject SAT score was above 1000—increases of 50 and 78 percent, respectively. By comparison, they were 5 percentage points more likely to attend a two-year school after graduation, a difference that was not statistically significant.
Enrolling in school is one thing. But did Noble students stick with their studies? We found that Noble students were 17 percentage points more likely to persist for two semesters or more, and 12 percentage points more likely to persist for four semesters or more. These differences in persistence were driven mainly by students enrolled at four-year colleges.
We reviewed the data to determine if the impacts of attending Noble varied based on students’ gender, middle school quality, and neighborhood poverty rate. We found few consistent patterns in the results to suggest that Noble was most effective for specific groups of students. We also investigated whether college enrollment patterns shifted over time, given the network’s rapid expansion while the students in our sample were enrolled, and found no evidence that Noble’s effectiveness declined appreciably over this period.
Success at Scale?
The experimental results clearly demonstrate that early cohorts attending Noble Street College Prep were more likely to enroll in college, enroll in selective four-year institutions, and remain enrolled for at least four semesters. But those results are for just one school. Can we interpret them to mean that the academic practices and policies implemented throughout the Noble network lead to improved college outcomes? What about at other no-excuses charter high schools with similar practices? Or was there something unique about the original campus that accounts for its students’ success?
To provide evidence on the generalizability of our main results, we carried out a secondary, nonexperimental analysis using school-average data reported by CPS to compare a broader group of Noble high schools to other Chicago high schools in later years. We examined average college enrollment rates for the graduating class of 2013 at 104 schools, seven of which were part of the Noble network. Our analysis takes into account the scores of incoming students on the ACT Explore exam administered to Chicago students in the fall of 9th grade, as well as the racial composition and percentage of students eligible for a subsidized lunch or for special education.
We found that, at the seven Noble high schools with graduating seniors in 2013, students were 19 percentage points more likely to enroll in college than one would predict based on their incoming ability (Figure 3). Noble’s college enrollment rates were among the best in the district, and rates at all seven schools surpassed expectations by a wide margin. Comparing Noble high schools only to other charter schools and adjusting for other differences in students’ background produces an estimated Noble effect of 13 percentage points on college enrollment—a large, significant difference. While complete data were not available for any other year, we repeated this analysis with the Class of 2012 using 10th-grade test scores to control for differences in student ability and found, reassuringly, a similar pattern.
We should be careful when interpreting these nonexperimental results. The research design does not benefit from the random variation used in our earlier analysis, so we cannot rule out the possibility that the students who enrolled in Noble network schools, despite their below-average test scores, would have been more likely to attend college anyway. Nevertheless, we find it reassuring that the best evidence we can muster indicates that Noble students continue to outperform expectations even during the network’s rapid expansion.
Implications
As no-excuses charter schools continue to expand, it is critical to understand whether the short-term academic gains they typically produce translate into long-term improvements in their students’ quality of life. We believe that our findings present the strongest evidence to date of long-lasting academic benefits, and should be a cause for cautious optimism. We see three elements of this analysis that should be of particular interest.
First, Noble’s educational model is broadly consistent with the practices of high-performing charter schools, and our secondary analysis suggests that scaling and reproducing these results is feasible. The effects we estimate are large, persistent, and not driven by any particular subgroup of students. While evidence is strongest in the lottery-based randomized analysis, our estimates for a larger group of students reaffirm those initial estimates.
Second, to our knowledge, our results are the first to demonstrate conclusively that a high school intervention can simultaneously improve overall college enrollment, persistence, and quality. Other studies linking high school quality to college, including evaluations of a public school-choice program in Charlotte-Mecklenburg Schools in North Carolina and of Harlem Children’s Zone Promise Academy in New York City have found impacts that are either transitory or not statistically significant.
Finally, we demonstrate the effectiveness of an intervention that occurs relatively late in students’ academic lives. Much of the public conversation around school improvement focuses on early childhood and the elementary years, in an effort to prevent or lessen inequitable outcomes for poor children. Yet Noble’s intensive academic program does not start until 9th grade, after many of its students have spent their formative years in low-performing schools. Relatively soon after, they are prepared to enroll and succeed in college, a critical step to success in adulthood. It is clear from their experience that such efforts are never too late.
Matthew Davis is a doctoral student in Applied Economics at the University of Pennsylvania. Blake Heller is a doctoral student in Public Policy at Harvard University.
This article appeared in the Winter 2017 issue of Education Next. Suggested citation format:
Davis, M., and Heller, B. (2017). Raising More than Test Scores: Does attending a “no excuses” charter high school help students succeed in college? Education Next, 17(1), 64-70.
