Despite mixed reviews from many educators–and some researchers–Chicago’s policy to end social promotion has turned out to be a popular program. Surprisingly, perhaps, its most avid fans are the people most affected by it: teachers and students. The Consortium on Chicago School Research (CCSR), an independent research group founded in 1990, has asked many questions about this pioneering education reform. Among them: Does retention improve academic outcomes? What are untoward–and unforeseen–side effects? Did it contribute to test-score gains? Did high-stakes testing policies produce only one-time impacts on behavior or are there long-term impacts? One of the more interesting questions the CCSR asked was, Did high-stakes accountability cause the teachers, parents, and students of the Chicago Public Schools (CPS) to change their behavior in ways that would lead to higher achievement, or does the evidence suggest that the CPS’s initiatives resulted in simply more focus on testing?
Chicago’s ending of social promotion was intended to make educators pay more attention to the lowest-performing students, encourage parents to become more involved in their children’s education, and send strong messages to students that achievement mattered. However, opponents of high-stakes testing worried that if students with low skills felt that promotional test cutoffs were out of their reach and that they could not do well on standardized tests based on their past experience, they might react to the pressure of high-stakes testing by becoming less engaged with school (But see Figure 1).
The Study
Fortunately, the Chicago experiment–now in its eighth year–became part of an ongoing study of the city’s public school system begun by the CCSR in 1994. Since 1999 the CCSR has published several studies of Chicago’s attempt to end social promotion that help to provide an extensive, empirical, and longitudinal look at the impact of the high-stakes testing policies on the Chicago school system. This article is adapted from one of those reports, “Ending Social Promotion: Response of Students and Teachers” (February 2004, CCSR).
Our study drew on four basic sources of data: teacher surveys, principal surveys, student surveys, and personal interviews with a sample of teachers from five low-performing schools. The CCSR began its comprehensive biannual survey of Chicago schools two years before the policy to end social promotion was introduced, polling all CPS teachers, principals, and students in 6th, 8th, and 10th grades about a range of education-related topics, including time spent on test-preparation activities, the content of reading and mathematics instruction, and students’ experiences in the classroom. In 1999 and 2001 survey questions were added asking teachers and principals specifically about the new social promotion policy. While no survey data were collected about social promotion per se before 1999, the biannual survey begun in 1994 served to monitor changes in teachers’ reports of their teaching behavior and students’ reports of their classroom experiences before and after the program was rolled out.
Of the 16,895 elementary school teachers in the system, 7,900, about 47 percent, responded to the survey in 1999 (see Figure 2). In addition, some 315 of 450 principals responded to the 1999 survey (see Figure 3); 30,000 students (about 50 percent of all 6th and 8th graders) also responded. Response rates in 1994, 1997, and 2001 were similar. There was no evidence of response bias at the school level: the proportion of respondent teachers in low-income, minority, and low-performing schools was the same as in the entire CPS system.
In addition to the survey data collected, in-depth interviews were conducted with 43 teachers who taught in the promotion-gate grades (3rd, 6th, and 8th, where students faced their test-score Rubicons) at five K-8 schools in the system. These five schools were located in neighborhoods with some of the highest retention rates in the city (after the promotion policy took effect), and they had large percentages of minority and poor students.
The interviews with teachers were especially valuable in providing insight into how, exactly, teachers spent their time “preparing” for the Iowa Tests of Basic Skills (ITBS). Half of the teachers, for instance, said they had aligned the content of their curriculum with the content of the test, but only 4 percent included curriculum content in what they considered “test preparation.”
Overall, though our study confirms some of the fears expressed by opponents of the program (see “Retaining Retention“), the student and teacher survey responses and personal interviews also suggest that Chicago’s bold experiment had a positive impact. Although time spent on test preparation increased substantially after the institution of high-stakes accountability, it declined in subsequent years. Teachers shifted instructional emphases in reading and math, increasing student exposure to grade-level material, and devoted more time to reading skills relevant to the test. And low-achieving 6th- and 8th-grade students received greater academic support and reported greater academic engagement.
What Was the Policy?
From the beginning, the centerpiece of Chicago’s high-stakes testing program for students was a set of minimum test-score standards on the reading and mathematics sections of the ITBS for students in the 3rd, 6th, and 8th grades. Students who did not meet the test-score cutoffs at the end of the school year were required to participate in a special summer program, called Summer Bridge, and to retake the test in August. Those who again failed to meet minimum test scores were retained in their grade or, if they were 15 years or older, were sent to alternative schools. In the first several years under the policy, more than one-third of all 3rd, 6th, and 8th graders failed to meet the promotion test-score cutoffs by the end of the school year. Though the result varied by grade and year, the summer program succeeded in cutting the test failure rate by a third.
Beginning in the 2000-2001 school year, the district began using a range of variables around the promotion cutoff to make promotion decisions; recommendations from teachers and principals, for example, were then incorporated into the promotion decisions. Our study, however, focuses on the impact of the CPS’s policy before these 2000-2001 policy changes. In the first several years of the policy, the CPS retained 20 percent of eligible 3rd graders and approximately 10 percent of 6th- and 8th-grade students–compared with an almost negligible retention rate before the ending of social promotion.
During this same period a system of school accountability was also implemented in the CPS. If 85 percent of a school’s students fell below national norms in reading, the school would be placed on academic probation. Because the CPS began both student and school accountability programs simultaneously, it is difficult to disentangle the effects of one initiative from the other conclusively. However, we do scrutinize the potential differential effects of school and student accountability by examining how trends differ across schools of different achievement levels and across students of different achievement levels, regardless of the school they attend.
Responses of Teachers and Principals
There is no doubt that the introduction of high-stakes accountability for schools and the end of social promotion for students changed the context of teaching in the Chicago Public Schools. Teachers in low-performing schools faced substantial pressure to raise test scores, and all teachers faced the challenge of improving the achievement of their lowest-performing students. Would they view district efforts as supportive of their own work in the classroom: helping to motivate students, sending the message that achievement matters, and ensuring that students have the basic skills they need before they advance to the next grade? Or would they think that high-stakes testing was directed at them: limiting autonomy in the classroom, placing excessive pressure on students and teachers, and undermining professionalism by assuming that teachers’ own judgments of students’ performance were wrong or inadequate?
We examined the teacher and principal surveys as well as the interview transcripts from the 43 teachers in low-performing schools to assess how Chicago educators viewed the initiative. Our results suggest that despite some reservations, cautions, and concerns, the majority of teachers and principals supported and endorsed the general education goals of the policy ending social promotion.
The 1999 results showed that more than 75 percent of teachers and almost 90 percent of principals indicated that they “felt supported in their efforts to help students who had been retained and who were at risk of retention.” They believed they had adequate information, resources, and training to deal with these students. This was, perhaps, a remarkable show of support given the fact that, in 1998, over one-third of Chicago’s elementary schools had retention rates between 6 and 16 percent and another one-third between 16 and 33 percent. Four percent of Chicago’s schools retained more than one-third of 3rd, 6th, and 8th graders that year. Again, this compared with retention rates that were less than 3 percent before the policy initiative.
Better Motivation and Involvement
Teachers and principals were also positive about the influence the policy had on student motivation. In the 1999 surveys, 67 percent of teachers and 72 percent of principals agreed or strongly agreed that the threat of retention motivated students to work harder in school.
“Because the students know,” reported one 8th-grade teacher during an interview with one of the CCSR’s field staff, “I’m not the one failing you, I’m not the one holding you back. And so then the students are much more motivated to do their work, especially as the year progresses.” Said another teacher: “We have pupils who have become students. That is, they actually do some studying because they have a goal now, whereas before they knew that they were going to be pushed on [promoted] no matter what they did.”
Teachers and principals felt that the policy had positive effects on parental involvement as well. Almost 90 percent of principals and 75 percent of teachers surveyed agreed or strongly agreed that the policy had made parents more concerned about their child’s progress.
What teacher would not be happy with more highly motivated students, greater parental involvement in education, and additional program supports to help struggling students? Indeed, some have suggested that ending social promotion in Chicago simply transferred responsibility for poor student performance to the students, parents, and after school and summer programs, in effect removing the burden from teachers (see Figure 1).
However, survey results do not support this hypothesis. The majority of CPS teachers and principals reported that they believed the policy of ending social promotion positively influenced their behavior and their school’s instructional efforts (see Figure 2 and Figure 3). Some 85 percent of teachers and almost 90 percent of principals agreed that, as a result of the new criteria for promotion, “Nearly all teachers [in this school] feel extra responsibility to help students meet standards.” More than 80 percent of teachers and principals agreed that the policy had made them more sensitive to individual students’ needs and problems.
The policy, said a math teacher, “made me more accountable. It has . . . kept me on my toes the entire year. There’s not one day gone by that I haven’t thought about what they need to know and that if they don’t pick up these skills they will not pass. And I feel it is my responsibility to get them to pass math.”
Finally, teachers and principals believed that the policy affected instruction positively. More than 80 percent of teachers and 87 percent of principals surveyed agreed or strongly agreed that the promotion standards had “focused the school’s instructional efforts in positive ways.”
Changed Instructional Practices
Even if educators viewed the initiative positively and reading and mathematics test scores were improving, questions remained. How did teachers change instruction as a result of the pressure? Did they focus more on test preparation, or did they change how they taught or what they taught so that students learned more? Could the test-score increases be sustained? To answer these questions we used survey data to trace trends in teachers’ reports of the time they spent on test preparation and on the content they emphasized in mathematics and language arts instruction before and after the implementation of the new promotion policy.
From the survey data we estimated that in 1994 teachers were spending an average of about 10.5 hours a year on test preparation activities such as giving practice tests and teaching test-taking strategies. By 1999 that average had increased to 21 hours. Since the CPS’s accountability policy provided strong incentives for the lowest-performing schools, these schools faced significant pressure to change behavior in order to get off, or to avoid being placed on, academic probation. Among teachers in the lowest-performing schools, we estimated that the amount of time spent on test preparation increased from 14 hours in 1994 to 32 hours in 1999.
Not surprisingly, for teachers and students in promotion-gate grades, time on test preparation increased substantially. Before the implementation of high-stakes testing, an equal proportion of teachers in each grade (about 30 percent) reported spending 20 hours or more on test preparation each year. By 1999 almost two-thirds of all teachers in the 3rd and 8th grades reported spending more than 20 hours a year on test preparation. In no other grades were such high levels of test preparation reported.
However, despite the sharp increases observed between 1994 and 1999, in 2001 the proportion of teachers who reported spending more than 20 hours a year on test preparation declined somewhat: from nearly 50 percent in 1999 to 44 percent in 2001.
Trends in Content Emphasis
Though the increased emphasis on the mechanics of taking tests should be considered a factor in the increase of mathematics and reading scores throughout this period, survey results also found signs of significant changes in teachers’ emphasis on content in language arts and in the time devoted to content appropriate to grade level in mathematics.
In 1994 the average 8th-grade teacher reported spending 38 percent of her time on mathematics topics that could be considered at grade level on the ITBS. But by 2001 that average 8th-grade teacher spent 44 percent of her time on grade-level material. Similarly, the average proportion of time that 8th-grade teachers spent on the most basic mathematics concepts–those associated with 1st- through 3rd-grade material (such as simple addition, reading a clock, and multiplication facts) declined from nearly one-quarter of their mathematics instructional time in 1994 to 18 percent in 2001.
Similarly, in almost all grades except 5th, we saw significant increases in the proportion of time language arts teachers reported spending on reading. Language arts teachers devoted more time to reading comprehension than to other language arts topics such as study skills, penmanship, public speaking, and listening skills after the adoption of the promotion policy. As with mathematics pacing, the largest increases occurred among 7th- and 8th-grade teachers.
In survey responses, some teachers and principals reported that they spent less time on other subjects as a result of the promotion policy–some 40 percent said that they were spending less time on social studies and science–although 57 percent said they did not spend less time on those subjects because of the new policy. Interviews suggested that many teachers welcomed the increased emphasis on basic reading and mathematics skills. As one 8th-grade teacher commented, “[Reading is] the most important thing that they’re going to get out of here. It’s got to be the most important. They can’t function if they can’t read. . . . To me, all the other things should be put aside until they can do this.”
Is this content alignment positive? To the extent that the allocation of instructional effort in reading and mathematics was undesirable before policy implementation, this shift in content emphasis can be viewed positively.
Student Expectations and Academic Support
If we are to believe teachers’ reports, a lot changed in the Chicago Public Schools between 1994 and 2001. Teachers reported that ending social promotion made them more sensitive to their students’ needs and influenced instruction in positive ways. To what extent did the change in policy influence students’ experiences of school?
The CCSR’s 6th- and 8th-grade surveys ask students to respond to a series of questions about the personal support they receive from their teachers for their schoolwork, questions that addressed whether they believe their teacher was willing to give them extra help or noticed if they were having trouble learning something. In 1994 students with the lowest achievement test scores reported significantly less personal support for their schoolwork from their teachers than students performing at higher levels. That gap had closed significantly by 2001. In 2001, 6th graders of all levels of performance reported higher levels of personal support from teachers, with the most dramatic increases occurring among students with the lowest skills. The same trend was reported among 8th graders with the lowest test scores. Low-achieving 6th and 8th graders also reported much higher levels of parental support for their schoolwork in 2001 than in 1994. While parental monitoring and support of schoolwork differed dramatically across students of different achievement levels in 1994, by 2001 there were significantly fewer differences.
Finally, afterschool participation, another key indicator of student school engagement, showed a marked increase following implementation of the high-stakes policies and related programs (see Figure 1). In 1994 only 16 percent of 8th graders at moderate risk of retention reported attending an afterschool program on a regular basis. Even though it is difficult to attribute the gains to the new retention policy, since many schools began requiring it, by 2001, 43 percent of 8th graders at moderate risk reported such attendance. High-risk 8th graders showed similar improvements: their participation in afterschool programs increased from 18 percent to 34 percent in the same period.
The New Focus
For educators and policymakers across the country struggling to implement effective education reforms in the context of increased accountability, the lesson of Chicago is that accountability can and does encourage teachers and principals to pay greater attention to the lowest-performing students in their classrooms. At the same time, school districts must place a high priority on finding ways to minimize the costs associated with time spent on test preparation while maximizing the potential benefits of increased accountability. Such policies do help focus teachers’ energy and attention on the appropriate content, but teachers may need help in learning to change their instructional practice. Accountability policies that are accompanied by significant investments in building teachers’ capacity and skills will likely meet with the greatest success.
Robin Tepper Jacob is an associate in the Education and Family Support area at Abt Associates, Inc., Cambridge, Massachusetts. Susan Stone is an assistant professor at the School of Social Welfare at the University of California, Berkeley.
