Proponents of later start times, who have received considerable media attention in recent years, argue that many students who have to wake up early for school do not get enough sleep and that beginning the school day at a later time would boost their achievement. A number of school districts have responded by delaying the start of their school day, and a 2005 congressional resolution introduced by Rep. Zoe Lofgren (D-CA) recommended that secondary schools nationwide start at 9:00 or later. Despite this attention, there is little rigorous evidence directly linking school start times and academic performance.

In this study, I use data from Wake County, North Carolina, to examine how start times affect the performance of middle school students on standardized tests. I find that delaying school start times by one hour, from roughly 7:30 to 8:30, increases standardized test scores by at least 2 percentile points in math and 1 percentile point in reading. The effect is largest for students with below-average test scores, suggesting that later start times would narrow gaps in student achievement.

The primary rationale given for start times affecting academic performance is biological. Numerous studies, including those published by Elizabeth Baroni and her colleagues in 2004 and by Fred Danner and Barbara Phillips in 2008, have found that earlier start times may result in fewer hours of sleep, as students may not fully compensate for earlier rising times with earlier bedtimes. Activities such as sports and work, along with family and social schedules, may make it difficult for students to adjust the time they go to bed. In addition, the onset of puberty brings two factors that can make this adjustment particularly difficult for adolescents: an increase in the amount of sleep needed and a change in the natural timing of the sleep cycle. Hormonal changes, in particular, the secretion of melatonin, shift the natural circadian rhythm of adolescents, making it increasingly difficult for them to fall asleep early in the evening. Lack of sleep, in turn, can interfere with learning. A 1996 survey of research studies found substantial evidence that less sleep is associated with a decrease in cognitive performance, both in laboratory settings and through self-reported sleep habits. Researchers have likewise reported a negative correlation between self-reported hours of sleep and school grades among both middle- and high-school students.

I find evidence consistent with this explanation: among middle school students, the impact of start times is greater for older students (who are more likely to have entered adolescence). However, I also find evidence of other potential mechanisms; later start times are associated with reduced television viewing, increased time spent on homework, and fewer absences. Regardless of the precise mechanism at work, my results from Wake County suggest that later start times have the potential to be a more cost-effective method of increasing student achievement than other common educational interventions such as reducing class size.

Wake County

The Wake County Public School System (WCPSS) is the 16th-largest district in the United States, with 146,687 students in all grades for the 2011–12 school year. It encompasses all public schools in Wake County, a mostly urban and suburban county that includes the cities of Raleigh and Wake Forest. Start times for schools in the district are proposed by the transportation department (which also determines bus schedules) and approved by the school board.

Wake County is uniquely suited for this study because there are considerable differences in start times both across schools and for the same schools at different points in time. Since 1995, WCPSS has operated under a three-tiered system. While there are some minor differences in the exact start times, most Tier I schools begin at 7:30, Tier II schools at 8:15, and Tier III at 9:15. Tiers I and II are composed primarily of middle and high schools, and Tier III is composed entirely of elementary schools. Just over half of middle schools begin at 7:30, with substantial numbers of schools beginning at 8:00 and 8:15 as well. The school day at all schools is the same length. But as the student population has grown, the school district has changed the start times for many individual schools in order to maintain a balanced bus schedule, generating differences in start times for the same school in different years.

The only nationally representative dataset that records school start times indicates that, as of 2001, the median middle-school student in the U.S. began school at 8:00. More than one-quarter of students begin school at 8:30 or later, while more than 20 percent begin at 7:45 or earlier. In other words, middle school start times are somewhat earlier in Wake County than in most districts nationwide. The typical Wake County student begins school earlier than more than 90 percent of American middle-school students.

Data and Methods

The data used in this study come from two sources. First, administrative data for every student in North Carolina between 2000 and 2006 were provided by the North Carolina Education Research Data Center. The data contain detailed demographic variables for each student as well as end-of-grade test scores in reading and math. I standardize the raw test scores by assigning each student a percentile score, which indicates performance relative to all North Carolina students who took the test in the same grade and year. The second source of data is the start times for each Wake County public school, which are recorded annually and were provided by the WCPSS transportation department.

About 39 percent of WCPSS students attended magnet schools between 2000 and 2006. Since buses serving magnet schools must cover a larger geographic area, ride times tend to be longer for magnet school students. As a result, almost all magnet schools during the study period began at the earliest start time. Because magnet schools start earlier and enroll students who tend to have higher test scores, I exclude magnet schools from my main analysis. My results are very similar if magnet school students are included.

The data allow me to use several different methods to analyze the effect of start times on student achievement. First, I compare the reading and math scores of students in schools that start earlier to the scores of similar students at later-starting schools. Specifically, I control for the student’s race, limited English status, free or reduced-price lunch eligibility, years of parents’ education, and whether the student is academically gifted or has a learning disability. I also control for the characteristics of the school, including total enrollment, pupil-to-teacher ratio, racial composition, percentage of students eligible for free lunch, and percentage of returning students. This approach compares students with similar characteristics who attend schools that are similar, except for the fact that some schools start earlier and others start later.

The results produced by this first approach could be misleading, however, if middle schools with later start times differ from other schools in unmeasured ways. For example, it could be the case that more-motivated principals lobby the district to receive a later start time and also employ other strategies that boost student achievement. If that were the case, then I might find that schools with later start times have higher test scores, even if start times themselves had no causal effect.

To deal with this potential problem, my second approach focuses on schools that changed their start times during the study period. Fourteen of the district’s middle schools changed their start times, including seven schools that changed their start times by 30 minutes or more. This enables me to compare the test scores of students who attended a particular school to the test scores of students who attended the same school in a different year, when it had an earlier or later start time. For example, this method would compare the test scores of students at a middle school that had a 7:30 start time from 1999 to 2003 to the scores of students at the same school when it had an 8:00 start time from 2004 to 2006. I still control for all of the student and school characteristics mentioned earlier.

As a final check on the accuracy of my results, I perform analyses that compare the achievement of individual students to their own achievement in a different year in which the middle school they attended started at a different time. For example, this method would compare the scores of 7th graders at a school with a 7:30 start time in 2003 to the scores of the same students as 8th graders in 2004, when the school had a start time of 8:00. As this suggests, this method can only be used for the roughly 28 percent of students in my sample whose middle school changed its start time while they were enrolled.

Results

My first method compares students with similar characteristics who attend schools that are similar except for having different start times. The results indicate that a one-hour delay in start time increases standardized test scores on both math and reading tests by roughly 3 percentile points. As noted above, however, these results could be biased by unmeasured differences between early- and late-starting schools (or the students who attend them).

Using my second method, which mitigates this bias by following the same schools over time as they change their start times, I find a 2.2-percentile-point improvement in math scores and a 1.5-point improvement in reading scores associated with a one-hour change in start time.

My second method controls for all school-level characteristics that do not change over time. However, a remaining concern is that the student composition of schools may change. For example, high-achieving students in a school that changed to an earlier start time might transfer to private schools. To address this issue, I estimate the impact of later start times using only data from students who experience a change in start time while remaining in the same school. Among these students, the effect of a one-hour later start time is 1.8 percentile points in math and 1.0 point in reading (see Figure 1).

These estimated effects of changes in start times are large enough to be substantively important. For example, the effect of a one-hour later start time on math scores is roughly 14 percent of the black-white test-score gap, 40 percent of the gap between those eligible and those not eligible for free or reduced-price lunch, and 85 percent of the gain associated with an additional year of parents’ education.

The benefits of a later start time in middle school appear to persist through at least the 10th grade. All students in North Carolina are required to take the High School Comprehensive Test at the end of 10th grade. The comprehensive exam measures growth in reading and math since the end of grade 8 and is similar in format to the end-of-grade tests taken in grades 3–8. Controlling for the start time of their high school, I find that students whose middle school started one hour later when they were in 8th grade continue to score 2 percentile points higher in both math and reading when tested in grade 10.

I also looked separately at the effect of later start times for lower-scoring and higher-scoring students. The results indicate that the effect of a later start time in both math and reading is more than twice as large for students in the bottom third of the test-score distribution than for students in the top third. The larger effect of start times on low-scoring students suggests that delaying school start times may be an especially relevant policy change for school districts trying to meet minimum competency requirements (such as those mandated in the No Child Left Behind Act).

Why Do Start Times Matter?

The typical explanation for why later start times might increase academic achievement is that by starting school later, students will get more sleep. As students enter adolescence, hormonal changes make it difficult for them to compensate for early school start times by going to bed earlier. Because students enter adolescence during their middle-school years, examining the effect of start times as students age allows me to test this theory. If the adolescent hormone explanation is true, the effect of school start times should be larger for older students, who are more likely to have begun puberty.

I therefore separate the students in my sample by years of age and estimate the effect of start time on test scores separately for each group. In both math and reading, the start-time effect is roughly the same for students age 11 and 12, but increases for those age 13 and is largest for students age 14 (see Figure 2). This pattern is consistent with the adolescent hormone theory.

To further investigate how the effect of later start times varies with age, I estimate the effect of start times on upper elementary students (grades 3–5). If adolescent hormones are the mechanism through which start times affect academic performance, preadolescent elementary students should not be affected by early start times. I find that start times in fact had no effect on elementary students. However, elementary schools start much later than middle schools (more than half of elementary schools begin at 9:15, and almost all of the rest begin at 8:15). As a result, it is not clear if there is no effect because start times are not a factor in the academic performance of prepubescent students, or because the schools start much later and only very early start times affect performance.

Of course, increased sleep is not the only possible reason later-starting middle-school students have higher test scores. Students in early-starting schools could be more likely to skip breakfast. Because they also get out of school earlier, they could spend more (or less) time playing sports, watching television, or doing homework. They could be more likely to be absent, tardy, or have behavioral problems in school. Other explanations are possible as well. While my data do not allow me to explore all possible mechanisms, I am able to test several of them.

I find that students who start school one hour later watch 12 fewer minutes of television per day and spend 9 minutes more on homework per week, perhaps because students who start school later spend less time at home alone. Students who start school earlier come home from school earlier and may, as a result, spend more time at home alone and less time at home with their parents. If students watch television when they are home alone and do their homework when their parents are home, this behavior could explain why students who start school later have higher test scores. In other words, it may be that it is not so much early start times that matter but rather early end times.

Previous research tends to find that students in early-starting schools are more likely to be tardy to school and to be absent. In Wake County, students who start school one hour later have 1.3 fewer absences than the typical student—a reduction of about 25 percent. Fewer absences therefore may also explain why later-starting students have higher test scores: students who have an early start time miss more school and could perform worse on standardized tests as a result.

Conclusion

Later school start times have been touted as a way to increase student performance. There has not, however, been much empirical evidence supporting this claim or calculating how large an effect later start times might have. My results indicate that delaying the start times of middle schools that currently open at 7:30 by one hour would increase math and reading scores by 2 to 3 percentile points, an impact that persists into at least the 10th grade.

These results suggest that delaying start times may be a cost-effective method of increasing student performance. Since the effect of later start times is stronger for the lower end of the distribution of test scores, later start times may be particularly effective in meeting accountability standards that require a minimum level of competency.

If elementary students are not affected by later start times, as my data suggest (albeit not definitively), it may be possible to increase test scores for middle school students at no cost by having elementary schools start first. Alternatively, the entire schedule could be shifted later into the day. However, these changes may pose other difficulties due to child-care constraints for younger students and jobs and afterschool activities for older students.

Another option would be to eliminate tiered busing schedules and have all schools begin at the same time. A reasonable estimate of the cost of moving start times later is the additional cost of running a single-tier bus system. The WCPSS Transportation Department estimates that over the 10-year period from 1993 to 2003, using a three-tiered bus system saved roughly $100 million in transportation costs. With approximately 100,000 students per year divided into three tiers, it would cost roughly $150 per student each year to move each student in the two earliest start-time tiers to the latest start time. In comparison, an experimental study of class sizes in Tennessee finds that reducing class size by one-third increases test scores by 4 percentile points in the first year at a cost of $2,151 per student per year (in 1996 dollars). These calculations, while very rough, suggest that delaying the beginning of the school day may produce a comparable improvement in test scores at a fraction of the cost.

Finley Edwards is visiting assistant professor of economics at Colby College.

Birdette Hughey is the 2011 Mississippi Teacher of the Year.

In February 2012, the New York Times took the unusual step of publishing performance ratings for nearly 18,000 New York City teachers based on their students’ test-score gains, commonly called value-added (VA) measures. This action, which followed a similar release of ratings in Los Angeles last year, drew new attention to the growing use of VA analysis as a tool for teacher evaluation. After decades of relying on often-perfunctory classroom observations to assess teacher performance, districts from Washington, D.C., to Los Angeles now evaluate many of their teachers based in part on VA measures and, in some cases, use these measures as a basis for differences in compensation.

Newspapers that publish value added measures no doubt relish the attention they generate, but the bigger question in our view is whether VA should play any role in the evaluation of teachers. Advocates argue that the use of VA measures in decisions regarding teacher selection, retraining, and dismissal will boost student achievement, while critics contend that the measures are a poor indicator of teacher quality and should play little if any role in high-stakes decisions. The Obama administration has thrown its weight squarely behind the advocates, launching a series of programs that encourage states to develop evaluation systems based substantially on VA measures.

The debate over the merits of using value added to evaluate teachers stems primarily from two questions. First, do VA measures work? In other words, do they accurately capture the effects teachers have on their students’ test scores? One concern is that VA measures will incorrectly reward or penalize teachers for the mix of students they get if students are assigned to teachers based on characteristics that VA analysis typically ignores.

Second, do VA measures matter in the long run? For example, do teachers who raise test scores also improve their students’ outcomes in adulthood or are they simply better at teaching to the test? Recent research has shown that high-quality early-childhood education has large impacts on outcomes such as college completion and adult earnings, but no study has identified the long-term impacts of teacher quality as measured by value added.

We address these two questions by analyzing school-district data from grades 3–8 for 2.5 million children, linked to information on their outcomes as young adults and the characteristics of their parents. We find that teacher VA measures both work and matter. First, we find that VA measures accurately predict teachers’ impacts on test scores once we control for the student characteristics that are typically accounted for when creating VA measures. Second, we find that students assigned to high-VA teachers are more likely to attend college, attend higher-quality colleges, earn more, live in higher socioeconomic status (SES) neighborhoods, and save more for retirement. They are also less likely to have children during their teenage years.

Teachers in all grades from 4 to 8 have large impacts on their students’ adult lives. On average, a 1-standard-deviation improvement in teacher value added (equivalent to having a teacher in the 84th percentile rather than one at the median) in a single grade raises a student’s earnings at age 28 by about 1 percent. Replacing a teacher whose value added is in the bottom 5 percent with an average teacher would increase students’ total lifetime incomes by more than $1.4 million for a typical classroom (equivalent to $250,000 in present value). In short, good teachers create substantial economic value, and VA measures are useful in identifying them.

Our findings address the three main critiques of VA measures raised in a recent Phi Delta Kappan article by Stanford education professor Linda Darling-Hammond and her colleagues. We show directly using quasi-experimental tests that standard VA measures are not biased by the students assigned to each teacher. Hence, value-added metrics successfully disentangle teachers’ impacts from the many other influences on student progress. We also show that although VA measures fluctuate across years, they are sufficiently stable that selecting teachers even based on a few years of data would have substantial impacts on student outcomes such as earnings.

Students assigned to high-VA teachers are more likely to attend college, earn more, live in higher socioeconomic status neighborhoods, and save more for retirement.

Data

We draw information from two sources: school-district records on students and teachers, and information on the same students and their parents from administrative data sources such as tax records. The school-district data contain student enrollment history, test scores, and teacher assignments from the administrative records of a large urban school district. These data span the school years 1988–89 through 2008–09 and cover roughly 2.5 million children in grades 3 through 8.

The school-district data include approximately 18 million test scores. Test scores are available for English language arts and math for students in grades 3–8 from the spring of 1989 to 2009. In the early part of the sample period, these tests were specific to the district, but by 2005–06 all tests were statewide, as required under the No Child Left Behind law. In order to calculate results that combine scores from different tests, we standardize test scores by subject, year, and grade. The district data also contain other information on students, such as race or ethnicity, gender, and eligibility for free or reduced-price lunch (a standard measure of poverty).

Our data on students’ adult outcomes include earnings, college attendance, college quality (measured by the earnings of previous graduates of the same college), neighborhood quality (measured by the percentage of college graduates in their zip code), teenage birth rates for females (measured by claiming a dependent born when the woman was still a teenager), and retirement savings (measured by contributions to 401[k] plans). Parent characteristics include household income, marital status, home ownership, 401(k) savings, and mother’s age at child’s birth.

Having spent a single year in the classroom of a teacher with value added that is 1 standard deviation higher increases earnings at age 28 by about 1 percent. If that 1 percent advantage were to remain stable throughout an individual's career, it would add up to about $25,000 in total earnings.

Do Value-Added Measures Work?

Value-added analysis aims to isolate the causal effects teachers have on student achievement by comparing how well their students perform on end-of-year tests relative to similar students taught by other teachers. These comparisons take into account students’ test scores in the prior year as well as their race or ethnicity, gender, age, suspensions and absences in the previous year, whether they repeated a grade, special education status, and limited English status. We also control for teacher experience as well as for class and school characteristics, including class size and the academic performance and demographic characteristics of all students in the relevant classroom and school.

Many other researchers use methods for measuring teacher value added that are similar to ours, so it is not surprising that we obtain similar results. For example, we find that a 1-standard-deviation increase in teacher value added corresponds to increases in student math and English scores of 12 and 8 percent of a standard deviation, respectively. In both subjects, this difference is equivalent to approximately three months of additional instruction.

Can we take this as evidence of teachers’ causal impact on student test scores? Recent studies by economists Thomas Kane, Doug Staiger, and Jesse Rothstein, among others, have reached divergent conclusions about whether VA measures should be interpreted in this way. In particular, critics contend that VA measures are likely to be biased as a result of the way that students are assigned to teachers. For example, some teachers might be consistently assigned students with higher-income parents (which typically cannot be accounted for by school districts when generating VA measures because they do not collect precise data on family income). We implement two new tests to determine whether VA estimates are biased.

Our first test examines whether in fact high-VA teachers tend to be assigned students from more-advantaged families. We calculate an overall measure of parents’ socioeconomic status, combining the parental characteristics listed above. Not surprisingly, parent socioeconomic status is strongly predictive of student test scores, and, looking at simple correlations, we find that less-advantaged students do tend to be assigned to teachers with lower VA measures. However, controlling for the limited set of student characteristics available in school-district databases, such as test scores in the previous grade, is sufficient to account for the assignment of students to teachers based on parent characteristics. That is, if we take two students who have the same 4th-grade test scores, demographics, classroom characteristics, and so forth, the student assigned to a teacher with higher VA in grade 5 does not systematically have different parental income or other characteristics.

This first test shows that any bias in VA estimates due to the omission of parent characteristics that we are able to observe is minimal. The possibility remains, however, that students are assigned to teachers based on unmeasured characteristics unrelated to parent socioeconomic status. For example, principals may consistently assign their most-disruptive students to teachers whom they believe are up to the challenge. Alternatively, principals might assign these same students to their least-effective teachers, whom they are not worried about losing. Our second test seeks to determine the amount of bias introduced by this kind of sorting.

To do so, we exploit the fact that adjacent grades of students within the same school are frequently assigned to teachers with very different levels of value added because of idiosyncrasies in teacher assignments and turnover. During our analysis period, roughly 15 percent of teachers in our data switched to a different grade within the same school from one year to the next, 6 percent of teachers moved to a different school within the same district, and another 6 percent left the district entirely. These year-to-year changes in the teaching staff at a given school generate differences in value added that are unlikely to be related to student characteristics.

To illustrate, suppose a high-VA 4th-grade teacher enters a school at the beginning of a school year. If VA estimates capture teachers’ true impact on their students, students entering grade 4 in that school should have higher year-end test scores than those of the previous cohort. And the size of the change in test scores across these consecutive cohorts should correspond to the change in the average value added across all teachers in the grade. For example, in a school with three equal-sized 4th-grade classrooms, the replacement of a teacher with a VA estimate of 0.05 standard deviations with one with a VA estimate of 0.35 standard deviations should increase average test scores among 4th-grade students by 0.1 standard deviations.

In fact, that is exactly what we find, as shown in Figure 1. To construct this figure, we first define the top 5 percent of teachers as “high VA” and the bottom 5 percent as “low VA.” Figure 1 displays average test scores for cohorts of students in the years before and after a high-VA teacher arrives. We see that end-of-year test scores in the subject and grade taught by that teacher rise immediately by about 4 percent of a standard deviation. This impact on average test scores is commensurate in magnitude with what we would have predicted given the increase in average teacher value added for the students in that grade.

We obtain parallel findings when we examine the departure of high-VA teachers and the entry and exit of low-VA teachers. When a high-VA teacher leaves a given subject-grade-school combination, test scores of subsequent students in that subject, grade, and school fall. Likewise, students benefit from the departure of a low-VA teacher and are harmed by the arrival of a low-VA teacher.

Together, these results provide direct evidence that removing low-VA teachers (bottom 5 percent) and retaining high-VA teachers (top 5 percent) improves the academic achievement of students. But what about the remaining 90 percent of teachers? When we perform a similar analysis for all teachers, we again find that changes in the quality of the teaching staff strongly predict changes in test scores across consecutive cohorts of students in the same school, grade, and subject. Moreover, in middle schools, where students usually learn math and English from different teachers, we confirm that the arrival or departure of math teachers affects math scores but not English scores (and vice versa).

Using these techniques, we can calculate the amount of bias in our VA estimates. We find that the degree of bias is, on average, less than 2 percent. We therefore conclude that standard VA estimates accurately capture the impact that teachers have on their students’ test scores. Although the results could differ in other settings, our method of using natural teacher turnover to evaluate bias in VA estimates can be easily implemented by school districts to evaluate the accuracy of their VA models.

Do Value-Added Measures Matter?

Even though value-added measures accurately gauge teachers’ impacts on test scores, it could still be the case that high-VA teachers simply “teach to the test,” either by narrowing the subject matter in the curriculum or by having students learn test-taking strategies that consistently increase test scores but do not benefit students later in their lives. To address this issue, we measure the relationship between teachers’ value added and their students’ outcomes in adulthood. We compare students who were assigned high-VA vs. low-VA teachers in grades 4–8 and study their outcomes in adulthood.

We find that high-VA teachers raise students’ chances of attending college at age 20 (see Figure 2a). A student assigned to a teacher with a VA 1 standard deviation higher is 0.5 percentage points more likely to attend college at age 20 (an increase of 1.3 percent). Students of higher-VA teachers also attend higher-quality colleges, as measured by the average earnings of previous graduates of those colleges.

A person’s income doesn’t begin to stabilize until their late twenties, so our analysis of earnings focuses on the year when students were 28, the oldest age at which we observe a sufficiently large number of students. We find that having spent a single year in the classroom of a teacher with value added that is 1 standard deviation higher increases earnings at age 28 by $182, or about 1 percent (see Figure 2b). If that 1 percent advantage were to remain stable throughout an individual’s career, it would add up to about $25,000 in total earnings.

In addition to improved earnings, we also find that improvements in teacher value added significantly reduce the likelihood that female students will have a child during their teenage years, increase the socioeconomic status of the neighborhoods in which students live in adulthood, and raise 401(k) retirement savings rates. Moreover, it is likely that improved education would yield benefits that we are not able to measure but have been shown by other studies, such as reduced crime and improved citizenship.

To sum up, our evidence confirms that the students of high-VA teachers benefit not just by scoring higher on math and reading tests at the end of the school year, but also through improved outcomes later in life. The size of these effects may seem small, but recall that they reflect the impact of a higher-VA teacher for a single year and could compound over time to the extent that students are exposed to multiple high-VA teachers. As important, a single high-VA teacher has this effect not only on a single student but rather on an entire classroom—and often on many classrooms of students over the course of a career.

Wanda Booth, Florida's 2011 Charter School Teacher of the Year, works with students. Teachers in all grades have large impacts on their students' adult lives.

Policy Implications

In a recent article (see “Valuing Teachers,” features, Summer 2011), Eric Hanushek argues in favor of dismissing the bottom 5 percent of teachers based on their VA scores. While such a policy would have many costs and benefits that are beyond the scope of our study, we can illustrate the magnitudes implied by our analysis by calculating its impacts on students’ earnings. Our estimates imply that replacing a teacher whose value added is in the bottom 5 percent with an average teacher would increase students’ cumulative lifetime income by a total of $1.4 million per classroom taught. This gain is equivalent to $267,000 in present value at age 12, discounting at a 5 percent interest rate. However, it is important to realize there is uncertainty in VA measures, which are estimates that may be based on only a few classrooms of students, so the gains from removing teachers identified as ineffective based on a limited number of years of data are smaller. We estimate the gains from “deselecting” the bottom 5 percent of teachers to be approximately $135,000 in present value based on one year of data and $190,000 based on three years of data. These benefits, while still large, would have to be weighed against any costs associated with the policy, such as teachers demanding higher pay to compensate them for the risk of dismissal.

We also measure the expected gains from policies that pay higher salaries or bonuses to high-VA teachers in order to increase retention rates. The gains from such policies appear to be only somewhat larger than their costs. Although the benefit from retaining a teacher whose value added is at the 95th percentile after three years is nearly $200,000 per year, most bonus payments end up going to high-VA teachers who would have stayed even without the additional payment. Replacing low-VA teachers is therefore likely to be a more cost-effective strategy to increase teacher quality in the short run than paying to retain high-VA teachers. In the long run, higher salaries could attract more high-VA teachers to the teaching profession, a potentially important benefit that we do not measure here.

While these calculations illustrate the magnitudes of teachers’ impacts on students, they do not by themselves offer a blueprint for the design of optimal teacher evaluations, salaries, or merit-pay policies. Teachers were not evaluated based on test scores in the school district and time period we study. VA measures may not be as useful for identifying teachers with positive long-term impacts on their students if teachers respond to their use in evaluation systems by engaging in practices such as teaching to the test or even outright cheating. In addition, our analysis does not compare value added with other measures of teacher quality, like evaluations based on classroom observation, which might be even better predictors of teachers’ long-term impacts than VA scores.

In summary, our research demonstrates that good teachers are of great value to their students, and that VA measures are a potentially valuable tool for measuring teacher performance. The most important lesson we draw is that finding policies to raise the quality of teaching is likely to yield substantial economic and social benefits.

Raj Chetty is professor of economics at Harvard University. John N. Friedman is assistant professor of public policy at Harvard Kennedy School. Jonah E. Rockoff is associate professor of business at Columbia University’s Graduate School of Business. For further information on the study, see http://obs.rc.fas.harvard.edu/chetty/value_added.html.

Commentary

In light of the widespread attention given to the Chetty, Friedman, and Rockoff research, Education Next asked four experts to comment on the study’s implications for teacher policy.

Implications for Policy Are Not So Clear - By Douglas Harris
Profound Implications for State Policy - By Chris Cerf and Peter Shulman
More Evidence Would Be Welcome - By Dale Ballou
Low-Performing Teachers Have High Costs - By Eric A. Hanushek

Click to enlarge

Not all school systems are moving in that direction. In Cambridge, Massachusetts, a district with surprisingly low student performance given the substantial per-pupil resources at its command, the school committee has decided to try to boost student achievement by abandoning its K–8 model in favor of having separate middle schools that serve grades 6 through 8 (though, in an unusual twist, each of the latter will be housed in the same facility as an elementary school).

In short, policymakers nationwide continue to wrestle with a basic question: At what grade level should students move to a new school? In the most common grade configuration in American school districts, public school students make two school transitions, entering a middle school in grade 6 or 7 and a high school in grade 9. This pattern reflects the influence of enrollment pressures and pedagogical theories that, over the past half century, all but eliminated the K–8 school from the American education landscape. A small fraction of students do attend public schools encompassing grades K–8, 6–12, or even K–12, however. We exploit this variation by comparing the achievement trajectories of Florida students entering a middle school or a high school to those of their peers who do not make those transitions.

Our study extends research conducted in New York City (see “Stuck in the Middle,” research, Fall 2010), in which Jonah Rockoff and Benjamin Lockwood found that entering a middle school causes a sharp drop in student achievement relative to the performance of those remaining in K–8 schools. It is hard to know whether one can generalize from results from the nation’s largest city (and school district), however, especially when it employs a complex procedure for assigning students to middle schools. Also, the New York City study was unable to follow students after 8th grade, making it impossible to know whether the negative impacts that were observed were temporary or extended into high school. This is a critical question inasmuch as a key rationale for middle school is its potential for easing the transition to high school. What is lost at the first transition may be more than gained at the second, which is presumably less abrupt for the middle-school child than for the one entering high school directly from an elementary-school environment.

To explore these issues, we use statewide data covering all students in Florida public schools who were in grades 3 to 10 between 2000 and 2009. Although a large majority of Florida students enter a middle school in grade 6, some do so in grade 7. Still others attend K–8 schools and avoid the middle-school transition altogether. To determine whether entering a middle school in grade 6 or grade 7 has any effect on achievement, we examine whether students experience a drop in test scores relative to students in K–8 schools that coincides with their transition to the new school. In the same way, we compare the learning trajectories of students entering high school in grade 9 to those of students who attend K–12, 6–12, or 7–12 schools in order to determine whether high-school transitions affect achievement.

Our results cast serious doubt on the wisdom of the middle-school experiment that has become such a prominent feature of American education. We find that moving to a middle school causes a substantial drop in student test scores (relative to that of students who remain in K–8 schools) the first year in which the transition takes place, not just in New York City but also in the big cities, suburbs, and small-town and rural areas of Florida. Further, we find that the relative achievement of middle-school students continues to decline in the subsequent years they spend in such schools. Nor do we find any sign that the middle-school students catch up with those who remained in the K–8 environment once all of them have entered high school. On the contrary, students entering a middle school in grade 6 are more likely not to be enrolled in any Florida public school as 10th graders (despite having been enrolled in grade 9), a strong indication that they have dropped out of school by that time.

We also find that the transition to high school causes a small drop in student achievement for all students who make this transition (as distinct from those in schools with 6–12 grade configurations). However, this drop holds far less policy significance both because of its size and because the decline does not appear to persist beyond grade 9.

The achievement drops we observe as students move to both middle and high schools suggest that moving from one school to another (or simply being in the youngest grade in a school) adversely affects student performance. The size and persistence of the effect of entering a middle school, however, suggests that such transitions are particularly damaging for adolescent students or that middle schools provide lower-quality education than K–8 schools provide for students at the same point in their education.

Data and Method

We draw the data for our analysis from the Florida Department of Education’s PK-20 Education Data Warehouse. The data contain state math and reading test scores for all Florida students attending public schools in grades 3 to 10 from the 2000–01 through 2008–09 school years. They also include information on the school each student attends and its location as well as student characteristics such as ethnicity, gender, special education classification, and eligibility for a free or reduced-price lunch.

We use different samples of students for different parts of our analysis. First, to estimate the effect of entering a middle school in grade 6 or 7, we examine only students enrolled in grade 3 between 2001 and 2004 who completed the state test in both math and reading in each of the subsequent five years. Second, to investigate whether the effects of middle-school entry persist through grades 9 and 10, we examine only students enrolled in grade 3 in 2001 or 2002 who were tested in both subjects each of the following seven years. Finally, to estimate the effect of entering high school in grade 9, we examine students enrolled in grade 6 between 2001 and 2005 who were tested in both math and reading in the following four years.

Our strategy for identifying the effects of alternative grade configurations on student achievement parallels and extends that of Rockoff and Lockwood’s study of New York City middle schools mentioned above. Specifically, we examine changes in individual students’ achievement over time, focusing on differences in the timing of students’ entry into middle school that result from the grade configuration of the school the student attended in 3rd grade. For example, we are interested in whether students who attended a K–6 school in 3rd grade experience a drop in their achievement in 7th grade relative to students who attended a K–8 school in 3rd grade and thus did not switch schools between grades 6 and 7.

The key assumption of our methodology is that there are no unobserved differences between students who in 3rd grade attended schools that had these different grade configurations that affect achievement precisely in the year when students enter middle school. In other words, we are assuming that the negative effect of a transition is not anticipated by parents and reflected in the choice of a school with a particular grade configuration in grade 3. We conduct an analogous analysis of high-school entry, taking advantage of the different grade configurations of the schools students attended in 6th grade.

Because we compare the achievement of individual students to themselves over time, our analysis takes into account all student characteristics (both observed and unobserved) that do not change over time. In addition, we also control for whether the individual student had been retained in a grade, whether the student had ever been retained, and whether the student attends a charter school (which in Florida are more likely than traditional public schools to have K–8 configurations).

The Middle-School Cliff

We find that students who will enter a middle school in 6th or 7th grade have positive achievement trajectories in math and reading from 3rd grade to 5th, relative to their counterparts who will never enter a middle school because they attend a school that continues through 8th grade. Achievement in both subjects falls dramatically in 6th grade for students who enter middle school in that grade. Students who will enter middle school in grade 7 continue to improve relative to their K–8 peers through grade 6, but experience a sharp drop in achievement upon entering middle school in grade 7.

Specifically, we find math achievement falls by 0.12 standard deviations and reading achievement falls by 0.09 standard deviations for transitions at grade 6 (see Figure 1). Students who make the transition at grade 7 experience even larger drops in their achievement of 0.22 and 0.15 standard deviations in math and reading, respectively. National data indicate that student achievement increases by roughly 0.30 standard deviations in math and 0.25 standard deviations in reading each year for typical 6th- and 7th-grade students. The drops in achievement we observe for students entering middle schools therefore amount to between 3.5 and 7 months of expected learning over the course of a 10-month school year.

Just as troubling is the fact that these students’ relative performance in both subjects continues to decline in subsequent middle-school grades. After three years in a middle school, students who entered in 6th grade score 0.23 standard deviations in math and 0.14 standard deviations in reading worse than we would have expected had they attended a K–8 school. After two years in a middle school, students who entered in 7th grade underperform by 0.31 standard deviations in math and 0.15 standard deviations in reading.

We also find little evidence that students who attend middle school make larger achievement gains than their peers in grades 9 and 10, by which time most Florida students have entered high school. In addressing this issue we must limit our attention to the two cohorts of students entering 3rd grade prior to 2001 or 2002, whose progress we are able to follow through the 10th grade. Although the math achievement of students who entered middle school in 7th grade improves by 0.05 standard deviations in 9th grade relative to students who attended K–8 schools, the same pattern is not evident in reading or in either subject for the much larger group of students who entered middle school in 6th grade (see Figure 2). In other words, we can safely reject the hypothesis that students who attend middle schools benefit at the transition to high school from their previous experience with school transition or from the specific educational programs available in middle schools.

Investigating the transition to high school, we find that students moving to a new high school between grades 8 and 9 suffer a small drop in achievement of 0.03 standard deviations in math and 0.04 standard deviations in reading (relative to those in grade 6–12 schools or schools with another configuration that requires no transition at this point). However, their relative achievement trajectories become positive again after this drop at the transition point.

We supplement our analysis on math and reading achievement with similar analyses of the effects of entering a middle school on the probability of students’ not being enrolled in a Florida public school in 10th grade (a proxy for dropping out of high school by this time) and on being retained in 9th grade (often a strong predictor that a student will leave school prior to graduation). Our results suggest that entering a middle school in 6th grade increases the probability of early dropout by 1.4 percentage points (or 18 percent). Although entering a middle school in 7th grade does not appear to increase early dropout, it increases the probability that a student will be retained in 9th grade by 1 percentage point. Both results provide additional cause for concern with the middle-school model.

Is it possible that our results reflect differences across school districts that employ alternative grade configurations? We explore this question by conducting our test-score analysis separately for schools in Miami-Dade County. With more than 345,000 students, Miami-Dade is the largest district in Florida and offers a wide range of grade configurations for students up through grade 8. We find that the negative effects of entering a middle school for grade 6 or grade 7 are, if anything, even more pronounced in Miami-Dade County than they are statewide.

Not Just an Urban Problem

This result for Miami-Dade County raises the possibility that the negative effects of middle-school entry are only notable in urban settings. We address this issue by looking separately at the effects of entering a middle or high school across communities of varying sizes. Using Census Bureau classifications, we group students into three categories according to the location of the school they attended in 3rd grade: 1) a large or midsize city, 2) suburbia (specifically, the urban fringe of a large or midsize city), and 3) towns and rural areas. The results suggest that the negative effects of entering a middle school are most pronounced in cities, but they remain sizable even in rural areas, confirming that the negative effects of configurations that separate the middle-school grades are by no means limited to urban school districts.

We also examine whether the middle-school effect varies across subgroups of students defined in terms of prior test performance, ethnicity, and gender. Students whose 3rd-grade scores were below the statewide median saw substantially larger declines in math scores at both the middle- and high-school transition points than higher-achieving students. These patterns are consistent with the theory that lower-achieving students have access to fewer educational resources outside of school and may therefore be at higher risk of being adversely affected by school transitions. We find no clear indication that the negative effect differs in size for higher- and lower-achieving students in reading, however.

Results for students of different ethnicities follow a similar pattern. Grade configuration has a larger effect on the math scores of traditionally disadvantaged subgroups than on other students. Black students in particular demonstrate large relative gains in math achievement prior to entering a middle school but then suffer larger drops both at and following the transition. Again, however, we find only small and statistically insignificant differences between the effects estimated for students of different ethnicities in reading. We find no differences in the effects for girls and boys.

Potential Explanations

Our results confirm that transitions into both middle schools and high schools cause drops in student achievement but that these effects are far larger for students entering middle schools. One possible interpretation of this pattern is that school transitions are more disruptive for younger students, perhaps because they are more susceptible to the negative influence of older students. Yet our estimates suggest that the effect of middle-school entry on student achievement is larger for students entering in grade 7 than for students entering in grade 6. Moreover, the fact that relative achievement continues to decline after students’ initial entry into middle schools suggests that average educational quality in Florida is lower in stand-alone middle schools than in schools serving grades K–8.

To explore why this might be the case, we first examine several characteristics of Florida elementary, middle, and K–8 schools. The most striking difference across school types involves cohort sizes (the average number of students in each grade). Although middle schools offer far fewer grades than K–8 schools, Florida middle schools on average enroll 146 more students than their K–8 counterparts; as a result, typical grade cohorts are almost three times as large. Florida middle schools also spend 11 percent less per student and have higher student-teacher ratios than K–8 schools, suggesting a potential role for differences in available resources. In contrast, we find no evidence that differences in observed teacher characteristics could explain our findings. Average teacher experience and average teacher salaries are similar across school types, while the share of the school’s instructional staff without prior experience is modestly higher in K–8 schools.

We conduct two analyses to shed light on whether these observed differences between middle schools and K–8 schools are likely to contribute to differences in school quality. First, we rerun our test-score analysis while controlling for these differences and find a similar pattern of results. Second, we examine whether the size of the drop in relative achievement suffered by students entering middle school in grade 6 varied with the characteristics of the middle school they attended. The results of this analysis again provide little evidence that low middle-school quality stems from differences in the school characteristics we can observe.

Middle schools could also differ from K–8 schools in their educational practices in ways that lead to lower student-achievement gains. To explore this possibility, we draw on a unique survey of Florida school principals conducted in 2003–04 to document responses to the state’s high-stakes accountability system. Confidentiality requirements preclude us from linking survey responses to specific schools, but we can document any differences in the average responses offered by principals of different school types.

We find few significant differences in the educational practices of the two groups of schools in our study. In particular, we observe no differences in the length of the school day or in measures of the extent to which schools had adopted specific policies to help low-performing students, policies to improve the performance of ineffective teachers, and incentives to reward highly effective teachers. If anything, these measures suggest that middle schools are more likely to have policies aimed at improving student achievement. We also find no differences across school types when we measure the degree of teacher autonomy.

A final set of survey items asked not about specific policies or practices but about the school’s overall climate. On these items, middle-school principals expressed significantly lower levels of agreement with statements indicating that their new and veteran teachers were excellent. This suggests that teachers in these schools may be less well equipped to deal with the challenges presented by their students. More middle-school principals also agreed with the statement that parents are worried about violence in the school. Although differences on the remaining items were statistically insignificant, they consistently point in the direction of middle schools having less-favorable school climates than K–8 schools.

In short, we find little evidence that the negative effects of attending a middle school are attributable to differences in resources, cohort sizes, or educational practices. We do, however, find suggestive evidence that the overall climate for student learning is worse in middle schools than in schools that serve students from elementary school through the 8th grade. This suggests a final potential interpretation of our results that is directly related to the choice of grade configuration: students may benefit from being among the oldest students in a school setting that includes very young students, perhaps because they have greater opportunity to take on leadership roles. This interpretation could account for both the gains in relative achievement made by students in K–5 and K–6 schools prior to entering middle schools and the superior performance of K–8 students relative to their peers in middle schools. A possible, if unlikely, alternative explanation is that students entering schools with different grade configurations have different growth trajectories for reasons having nothing to do with their schooling environment.

Taken as a whole, our results suggest that school transitions lower student achievement but that attending middle schools in particular has adverse consequences for American students. Especially when considered along with those of other recent studies, our findings clearly support ongoing efforts in urban school districts to convert stand-alone elementary and middle schools into schools with K–8 configurations. They are also relevant to the expanding charter-school sector, which has the opportunity to choose grade configurations without the disruption caused by school closures. More research is needed to see whether policy or pedagogical innovations can mitigate the effects of middle school. In the meantime, policymakers should exercise caution before extending the middle-school experiment to school districts that still enjoy the K–8 configuration.

Martin R. West is assistant professor of education at the Harvard Graduate School of Education and deputy director of the Program on Education Policy and Governance (PEPG) at Harvard’s Kennedy School. Guido Schwerdt is a researcher at the Ifo Institute for Economic Research in Munich, Germany.

In this study, I investigate whether the opportunity to attend a school other than a student’s assigned neighborhood school reduces criminal activity, especially among disadvantaged youth. Many of the schools chosen by the students were “better” on traditional indicators, such as student test scores and teacher characteristics. All of them, however, were preferred by the applicant over the default option. The analysis therefore sheds light on whether efforts to expand school choice can be an effective crime-prevention strategy, particularly when disadvantaged students can gain access to “better” schools.

We know that criminal offenders often have low levels of education: only 35 percent of inmates in U.S. correctional facilities have earned a high school diploma, compared to 82 percent of the general population. Criminal activity is concentrated among minority males; it begins in early adolescence and peaks when most youth should still be enrolled in secondary school. The schools these young men would attend are typically in high-poverty urban neighborhoods, have high rates of violence and school dropout, and struggle to retain effective teachers. Such schools may be a particularly fertile environment for the onset of criminal behavior. Yet little research has been conducted to determine the effect of school quality on crime.

In this study I explore this question using data from the Charlotte-Mecklenburg (North Carolina) school district (CMS) to measure the impact of school quality on arrest and incarceration rates. I take advantage of the CMS districtwide open-enrollment school-choice plan, which until recently let students choose where they wanted to go to school and employed lotteries to admit students to oversubscribed schools. I compare the criminal activity of students who won the lottery to attend their first-choice school to that of students who lost the lottery.

I find consistent evidence that attending a better school reduces crime among those age 16 and older, across various schools, and for both middle and high school students. The effect is largest for African American males and youth who are at highest risk for criminal involvement. In general, high-risk male youth commit about 50 percent less crime as a result of winning the school-choice lottery. They are also more likely to remain enrolled in school, and they show modest improvements on measures of behavior such as absences and suspensions. Yet there is no detectable impact on test scores for any youth in the sample.

School Choice in CMS

With more than 150,000 students enrolled in 2008–09, Charlotte-Mecklenburg is the 20th largest school district in the nation. The CMS attendance area encompasses all of Mecklenburg County, including Charlotte and several surrounding cities. Overall, CMS is racially and demographically diverse. About 45 percent of the students in CMS middle and high schools in 2003 were African American, less than 10 percent were Hispanic (although the Hispanic population was growing rapidly over this period), and about 50 percent were eligible for free or reduced-price lunches. Individual CMS schools vary widely in demographic composition: CMS high schools in 2003 ranged from less than 10 percent to close to 90 percent nonwhite, and were also dissimilar in average test scores and rates of high school graduation.

From 1971 until 2001, CMS schools were forcibly desegregated under a court order. Students were bused all around the district to preserve racial balance in schools. After several years of legal challenges, the court order was overturned, and CMS was instructed that it could no longer determine student assignments based on race. In December 2001, the CMS school board instituted a policy of districtwide open enrollment for the 2002–03 school year. School boundaries were redrawn as contiguous neighborhood zones, and children who lived in each zone were guaranteed access to their neighborhood school. Under busing, schools were racially balanced, but the surrounding neighborhoods remained highly segregated. Thus the redrawing of school boundaries led to concentrations of minority students in some schools.

The first open-enrollment lottery took place in the spring of 2002. CMS conducted an extensive outreach campaign to ensure that choice was broad-based, and 95 percent of parents submitted at least one preferred school; parents could submit up to three (not including their neighborhood school). Admission for all students from outside the neighborhood zone was subject to grade-specific limits. The lottery process for oversubscribed grades gave preference first to students who previously attended the school and their siblings, then to low-income students applying to schools that previously did not have a majority of low-income students, and finally to students applying to a school within their “choice zone” (which would guarantee them access to district-provided transportation). I study the effects of winning a seat at a preferred school in the 2002 lotteries on student outcomes through 2009, seven years after the lotteries were conducted.

Because nearly all rising 12th graders received their first choice, I restrict my study to students in grades 6 through 11. I also exclude the 5 percent of students who were not enrolled in any CMS school in the previous year. About 60 percent of the remaining students chose (and were automatically admitted to) their neighborhood school. About 75 percent of applicants to nonguaranteed schools were in lottery priority groups in which the probability of admission was either zero or one. Even though these students chose a nonguaranteed school, there is no randomness in whether they were admitted, so I do not use them in the study. The resulting sample consists of 1,891 high-school students (grades 9–11) and 2,320 middle-school students (grades 6–8). Compared to all students in CMS, these students were more likely to be African American and eligible for free lunch; they also had lower test scores and higher rates of absence and out-of-school suspensions (see Figure 1).

Data

Since the mid-1990s, the North Carolina Department of Public Instruction (NCDPI) has required all districts to submit data that include demographic information, attendance rates, and behavioral outcomes, yearly test scores in math and reading for grades 3 through 8, and subject-specific tests for higher grades. I used these data, along with internal CMS files that contain student-identifying information such as name, date of birth, and exact address in every year. This information enabled me to match CMS students to arrest records from the Mecklenburg County Sheriff’s Office, which include all arrests of adults (age 16 and over in North Carolina) that occurred in the county.

I measure crime severity in two ways, both of which are intended to capture the idea that not all crimes are equal. First, I use estimates that economists have developed of the social cost of crimes, which include tangible costs, such as lost productivity and medical care, as well as intangible costs, such as impact on quality of life; these estimates are extremely high for fatal crimes. (The estimated social cost of murder is $4.3 million in 2009 dollars. The next costliest crime is rape, which is estimated at $125,000.) To avoid the results being driven entirely by a few murders, in my main analysis I limit the cost of murder to twice the cost of rape. The second measure of severity weighs crimes by the expected punishment resulting from a successful conviction. Neither measure accounts for justice system costs such as police or prisons.

Methodology

If the school lottery is truly random, the winners and losers will on average have identical observed and unobserved characteristics. With a large enough sample, a simple comparison of outcomes between winners and losers would identify the causal effect of winning the lottery. In reality, CMS conducted many lotteries (for each school and grade). The number of students in each lottery is relatively small, so my analysis combines data from all of the middle-school and all of the high-school lotteries. My results reflect the average difference in outcomes between winners and losers across all of the lotteries conducted at each level.

The result is the “intent-to-treat” effect of winning a lottery; it is an intent because students offered a place in their first-choice school did not always take it (for example, they may have moved out of the district). Students who won the lottery are more than 55 percentage points more likely than losers to attend their first-choice school in the first year, and on average spend an additional 1 to 1.5 years enrolled in that school overall. One can therefore obtain a rough estimate of the effect of actually attending the first-choice school (as a result of winning the lottery) by doubling the results presented below.

I examine the impact of winning the lottery on crime separately for groups of students with different propensities to commit crimes, with a focus on the highest-risk group. Because students with adult arrest records can be tracked all the way back to kindergarten in some cases, I use all of the potential predictors of criminal behavior—test scores, demographics, behavior, and neighborhood characteristics—to calculate an index of crime risk. The students in the top 20 percent of this crime-risk index are disproportionately African American males and eligible for free lunch (see Figure 1a). Their test scores are on average one standard deviation below the North Carolina state average, and they are absent and suspended many more days than the average student (see Figure 1b). Because high-risk students are overwhelmingly male, I exclude females from all of the analyses. The results comprise a final sample of 1,014 high-school students and 1,081 middle-school students.

High-school lottery winners attend schools that are demographically very similar to the schools attended by lottery losers, while middle-school winners attend schools that are less African American and higher income on average. All lottery winners travel farther to attend their first-choice school, but the distance is greater for high school students than for middle school students.

High-school lottery winners in the high-risk group and all middle-school lottery winners experience modest increases in standard measures of school quality. Their peers’ average test scores are about 0.15 standard deviations higher, and the new schools have higher-quality teachers, measured in terms of the fraction of teachers with less than three years’ experience, the fraction that are new to the school that year, the percentage of teachers with an advanced degree, and the share of teachers who attended a “highly competitive” college as defined by the Barron’s rankings. For youth in the high-risk group, the gain as measured by these quality indicators is roughly equivalent to moving from one of the lowest-ranked schools to one around the district average.

Results

I find that winning a lottery for admission to a preferred school at the high school level reduces the total number of felony arrests and the social cost of crime. Among middle school students, winning a school-choice lottery reduces the social cost of crime and the number of days incarcerated. Importantly, I find that these overall reductions in criminal activity are concentrated among students in the highest-risk group. Indeed, I find little impact either positive or negative of winning a school-choice lottery on criminal activity for the 80 percent of students outside of this group.

Consider first the results for high school students in the high-risk group. Among these students, winning admission to a preferred school reduces the average number of felony arrests over the study period from 0.77 to 0.43, a pattern driven largely by a reduction of 0.23 in the average number of arrests for drug felonies (see Figure 2). The average social cost of the crimes committed by high-risk lottery winners (after adjusting the cost of murders downward) is $3,916 lower than for lottery losers, a decrease of more than 35 percent. (Without adjusting for the cost of murder, I estimate the reduction in the social cost of crimes committed by lottery winners at $14,106.) High-risk lottery winners on average commit crimes with a total expected sentence of 35 months, compared to 59 months among lottery losers.

Among high-risk middle-school students, I find no effect of winning a school-choice lottery on the average number of felony arrests. Although the number arrests for violent felonies falls, this is offset by an increase in the number of property arrests. Because violent crimes carry greater social costs, however, winning a school-choice lottery reduces the average social cost of the crimes committed by middle school students by $7,843, or 63 percent. It also reduces the total expected sentence of crimes committed by each student by 31 months (64 percent).

An important limitation of this analysis is that I do not have access to data on juvenile crime. Especially for students in the middle school sample, this could mask big differences in juvenile offending in the early years after the lotteries were conducted. As an alternative, I examine the effect of winning the lottery on school disciplinary outcomes such as absences and suspensions, as well as on test scores. Among the high-risk group, lottery winners are absent slightly less than the lottery losers are. The effect on high school suspensions in 2003 is relatively large, but the other school discipline effects are small and statistically insignificant.

In contrast to the results for crime and disciplinary outcomes, I find no evidence that winning admission to a preferred school leads to test-score gains. But I do find some impacts on enrollment, grade progression, and grade attainment for high-risk youth. For example, high-risk middle-school lottery winners are 18 percentage points more likely than lottery losers to be enrolled in CMS in their 10th-grade year. The effect on 11th-grade enrollment is about half the size (9 percentage points), and there is no impact on persistence into 12th grade.

Despite the impacts on enrollment and progression, there is no detectable increase in high school graduation rates. Because I am limited to CMS administrative data, it is difficult to distinguish dropouts from subsequent GED recipients or transfers who may have graduated elsewhere. Administrative records are particularly problematic for high-risk youth, who sometimes disappear from CMS well before they are old enough to do so legally. The graduation rate is only about 25 percent among high-risk high-school students, and currently only about 10 percent among high-risk middle-school students, although some who are still enrolled may yet graduate. Additionally, a bit less than 10 percent of the high-risk middle-school sample never appears in any high school grade but subsequently appears in the arrest data. Because any intervention aimed at high school students would miss this group altogether, this suggests that high school might be too late for the youth at highest risk of criminal activity.

Explanations and Policy Implications

Overall, I find that winning the lottery to attend a first-choice school has a large impact on crime for high-risk youth. High-risk lottery winners experienced roughly a 50 percent reduction in the measures of criminal activity that weight crimes by their severity.

I consider four possible explanations for the reduction in crime among high-risk lottery winners. The first is incapacitation, which advances that winning the lottery entails longer bus rides to and from school, thus occupying youth during high-crime hours. The second is contagion, in which winning the lottery prevents crime by removing high-risk youth from crime-prone peers or neighborhoods, thereby reducing contemporaneous exposure of high-risk youth to criminogenic influences. These first two explanations would predict a strong initial effect that fades over time. If, for example, drug-market activity is concentrated within a few schools, we might expect large differences in criminality in the high school years that diminish as enrollment in the chosen school ends and lottery winners and losers return to the same neighborhoods. When I examine the effect of winning a school lottery separately at different points in time after the lotteries were conducted, however, I find larger effects in later years. I therefore conclude that there is little support for the incapacitation and contagion explanations since they do not fit the pattern of results over time.

A third possibility is that the reduction in crime comes from the skills students gain by attending a higher-quality school. If the schools attended by lottery winners do a better of job of teaching skills that increase students’ ability to find employment, they will stay enrolled in school longer, delaying the onset of criminality through the peak period of offending behaviors. Moreover, youth with more and better schooling will gain access to more and better opportunities for paid work, making crime less attractive. Based on a back-of-the-envelope calculation of the relationship between enrollment and criminal activity in my sample, I estimate that the effects of winning a school lottery on enrollment could potentially explain about 45 percent of the impact on criminal activity in the high school sample, but only about 10 percent in the middle school sample.

Alternatively, peer networks formed in middle or high school could have a persistent influence on adult criminality without affecting skills directly. In my own data, I find relatively little evidence that the propensity of a student’s peers to engage in criminal activity influences the degree to which he commits violent crimes. This may be due in part to the high rate of early dropout among violent felons. However, having crime-prone peers in middle school substantially increases the likelihood of committing a violent crime, especially for youth in the high-risk group. Based on this relationship, I estimate that changes in peers can explain roughly 9 percent of the impact on violent arrests in the middle school sample.

Regardless of the mechanisms by which admittance to a preferred school influences criminal activity, the fact that these impacts are concentrated among high-risk students has important implications for the design of school-choice programs. It may make sense for oversubscribed schools of choice to give preferential admission to students at greatest risk of criminal activity. To illustrate this point, I use my results to evaluate the consequences of two different types of lotteries: 1) those giving priority to the highest-risk students and 2) a simple lottery similar to those virtually all charter schools nationwide are required to use to admit students when the schools are oversubscribed. The actual CMS lottery system gave preferences to low-income students who applied to schools with a low fraction of low-income students. As a consequence, many poor (and high-crime risk) students were automatically admitted to schools while other students had to win the lottery.

If slots in oversubscribed schools were systematically allocated to the highest-risk students, the social cost of crime would fall by an additional 27 percent relative to the actual CMS assignment mechanism. A more realistic form of targeting is the method actually pursued by CMS, giving preference to low-income students within the lottery system. I estimate that this policy choice lowered the social cost of crime by about 12 percent, relative to a simple charter-style lottery with no preferential treatment. Although this analysis does not consider the possibility that a greater concentration of high-risk students could have adverse effects on other students, it nonetheless highlights the likely beneficial consequences of giving preference to disadvantaged students in the admissions process for oversubscribed schools.

Conclusion

In this study, I find that winning a lottery for admission to the school of choice greatly reduces criminal activity, and that the greatest reduction occurs among youth at the highest risk for committing crimes. The impacts persist beyond the initial years of school enrollment, seven years after the school-choice lottery was held. The findings suggest that schools may be an opportune setting for the prevention of future crime. Many high-risk youth drop out of school at a young age and are incarcerated for serious crimes prior to the age of high school graduation. For these youth, who are on the margins of society, public schools may present the best opportunity for intervention.

The end of busing and the implementation of open enrollment in CMS was a significant policy change. The four neighborhood high schools to which most of the lottery applicants were assigned lost more than 20 percent of their enrollment in a single year. In subsequent years, two of these schools were restructured as magnet schools offering specialized programs in a small school setting. Two middle schools that lost significant numbers of students were subsequently closed. The open enrollment policy thus sent a strong signal of parental demand to CMS that may have resulted in the shutting down or restructuring of low-performing schools. The No Child Left Behind Act of 2001 included a provision that allowed parents to transfer students from “persistently dangerous” public schools, but many states have set the legal threshold so high that very few schools qualify. The results here suggest that, to the extent that low-quality schools are also persistently dangerous, allowing students to leave them might benefit individual students as well as society as a whole.

David J. Deming is assistant professor of education at the Harvard Graduate School of Education. This article is adapted from a study in the November 2011 issue of the Quarterly Journal of Economics.

For nearly a decade, the No Child Left Behind Act (NCLB) has focused the attention of policymakers and researchers squarely on the achievement of low-performing students, with some apparent success. The math and reading scores on the National Assessment of Educational Progress of the nation’s lowest-achieving 10 percent of 4th and 8th graders have risen sharply since 2000, continuing a trend that began in the 1990s. Yet some may wonder about the potential cost of this focus on higher-achieving students, for whom improvements over the same time period have been modest. Among the questions related to this debate is whether additional programs and resources should be devoted to students on the higher end of the spectrum, those considered gifted.

Three million students in the United States are classified as gifted, yet little is known about the effectiveness of traditional gifted and talented (G&T) programs. In theory, G&T programs might help high-achieving students because they group them with other high achievers and typically offer specially trained teachers and a more advanced curriculum. While previous research indicates that ability grouping is in fact correlated with higher achievement, these findings could be misleading if students placed in high-ability classrooms were likely to be successful for reasons that researchers are unable to measure, such as stronger motivation. To our knowledge, no existing studies offer convincing evidence on the causal effect of G&T programs on student achievement.

Our research begins to fill this gap with two studies of the G&T programs available to high-achieving middle-school students in a large urban school district in the southwestern United States which, to preserve anonymity we shall refer to as LUSD. Since 2007, all 5th-grade students in LUSD have been evaluated to determine eligibility for gifted and talented programs starting in 6th grade. Those students who are deemed eligible often are grouped in classes with other gifted students. They are also permitted to apply for admission to two middle schools that have oversubscribed magnet G&T programs.

The two studies use different methods to ask distinct but closely related questions. The first exploits the fact that eligibility for G&T programming in LUSD is determined by a well-defined cutoff in students’ evaluation scores. By comparing students who score just above the cutoff to those who score just below, the study provides evidence on the effect of enrollment in a G&T program on achievement for those students on the margin of eligibility. The second study takes advantage of the randomized lotteries that determine admission to the district’s two premier magnet G&T programs. By comparing students who win the lottery and attend the magnet G&T schools to those who lose the lottery and attend other “neighborhood” programs, the research provides evidence on whether the magnet G&T programs provide any additional benefits.

The results of both studies will be discouraging for those hopeful that current G&T programs provide a means to accelerate the progress of our most capable students. The first shows that barely eligible students who participated in LUSD’s G&T curriculum for all of 6th grade and half of 7th grade exhibit no significant improvement in test scores across a range of subjects, despite their being surrounded by higher-achieving peers and taking more advanced courses. The lottery study corroborates these results, as students admitted to the G&T magnet schools show little improvement in test scores by 7th grade, despite having higher-achieving peers and being taught by more effective teachers. The lone exception is in science, where students admitted to G&T magnet schools performed at substantially higher levels.

It is difficult to know what accounts for these puzzling results. Our best guess, which we discuss in detail below, is that being placed with higher-achieving peers is not all that it is cracked up to be. Students admitted to both types of G&T programs suffer a large drop in their relative rank in terms of grades within their classes, which could have adverse consequences that offset any benefits of improvements in their educational environment. But we are getting ahead of ourselves. Let’s first take a closer look at the programs and the evidence on their effects.

Gifted Students in LUSD

LUSD is a large school district, with more than 200,000 students. The district is heavily minority and very low income; the minority population is more heavily Hispanic than African American. All LUSD students are evaluated for placement in middle-school G&T programs during 5th grade, including those who participated in the district’s G&T program in elementary school. In order to be deemed eligible for the middle school G&T program, a student must meet the eligibility criteria set forth in the “gifted and talented identification matrix.” The matrix converts scores on standardized tests—the Stanford Achievement Test for English-speaking students and the Aprenda exam for Spanish-speaking students with limited English proficiency—scores on the Naglieri Nonverbal Ability Test (NNAT), average course grades, teacher recommendations, and indicators for socioeconomic status into an overall index score.

While all students who meet these requirements qualify for the G&T program, not all end up being classified as G&T, because parents are allowed to opt out. Some students also enroll in the program initially but later withdraw. Schools in LUSD have a monetary incentive for attracting gifted students, as LUSD provides a funding boost of 12 percent over the average allotment for a regular student.

Gifted students in LUSD are far less likely to be economically disadvantaged and more likely to be white or Asian than other students in the district. They also perform at far higher levels on the Stanford Achievement Tests, which the district administers annually in five subjects: math, reading, language, social science, and science. Their advantage in math and reading test scores in 5th grade is roughly 0.7 of a standard deviation, which amounts to well over two years of academic progress (see Figure 1). By the time the same students have reached 7th grade, these gaps have widened to 1.5 standard deviations in math and 1.25 standard deviations in reading. While this pattern suggests that the students enrolled in the district’s G&T programs learn at a faster rate between 5th and 7th grade, it does not necessarily mean that the G&T programs are the cause. It is to that question we now turn.

Effects on Barely Eligible Students

Our first study examines the effects of participation in a G&T program on students who were just barely eligible to participate based on their overall index scores. We focus on students who were evaluated for G&T eligibility as 5th graders in the spring of 2008 for whom we are able to observe outcomes as 7th graders in the 2009–10 school year. Our outcome measures include Stanford Achievement Test scores and attendance rates, both of which are drawn from administrative data provided by the district. After restricting the sample to students near the G&T eligibility cutoff, we are able to examine these outcomes for roughly 2,600 students.

The method used in the study, known as regression discontinuity analysis, takes advantage of the fact that the district uses a strict numerical cutoff in the index score assigned to students as 5th graders in order to determine their eligibility to participate in the G&T program the following year. Because the students are unable to precisely manipulate their index scores, those scoring just below the eligibility cutoff should be very similar to those scoring just above the cutoff. We can therefore attribute any differences in student outcomes on either side of the cutoff to the effect of having being deemed eligible.

As noted above, not all eligible students end up participating in G&T programs due to factors such as a parent’s decision to opt out. Similarly, some students who do not initially qualify later become eligible through an appeals process that allows parents to submit an alternative standardized test score or through additional evaluations conducted in 6th grade. As a result, we use standard statistical techniques to account for the fact that the cutoff our regression discontinuity analysis exploits is “fuzzy” rather than sharp. This allows us to provide evidence on the effects of actual participation in the G&T program, not simply eligibility for it.

Before looking at student outcomes, we first used the same method to confirm that participation in the district’s standard G&T programs led to measurable differences in students’ educational experiences. Clearly, it did. The average achievement of the peers in G&T students’ classrooms were between 0.25 and 0.33 of a standard deviation higher in each core academic subject. Participation in the G&T program also increased the number of advanced courses in which students enrolled in 6th and 7th grade. We found no evidence, however, that the teachers to whom students in the G&T program were assigned were any more effective, as measured by their impact on student test scores.

Did these improvements in peer characteristics and curricular rigor translate into improved outcomes? Our results indicate that they did not (see Figure 2). Our estimates of the effects of G&T participation for barely eligible students are close to zero in all five subjects and are sufficiently precise to allow us to rule out with 90 percent confidence effects as small as 0.04 standard deviations (sd) in math, 0.07 sd in reading, 0.12 sd in language, 0.10 sd in social studies, and 0.19 sd in science. We also looked at the impact of G&T participation for specific student subgroups defined by gender, race/ethnicity, socioeconomic status, and whether the students had been classified as gifted in elementary school. We found little evidence of differential impacts for students in any of these groups.

The Effects of G&T Magnet Programs

Why does the G&T program in LUSD not yield benefits for students on the margin of eligibility? One reason could be that the qualification boundary is set so low that such students are not able to take advantage of the programs’ purported benefits. Our second analysis, which uses experimental research methods to study the effects of enrollment in the district’s G&T magnet programs, is intended to shed light on this concern.

LUSD has 41 middle schools, of which 8 have G&T magnet programs, and 2 of these are oversubscribed. As a result, the district uses lotteries to determine which students will be admitted as 6th graders. Our analysis compares the performance of students who win the lottery and attend one of the G&T magnet programs to those who lose the lottery and either attend a neighborhood G&T program in the district, a magnet school based on a different specialty, or a charter school. Because the lottery is random, any differences in outcomes between lottery winners and losers can be attributed to the effect of enrolling in the G&T magnet program rather than one of these alternatives. Moreover, the results of this analysis will apply to the entire population of students who chose to apply.

Our lottery analysis is based on the sample of LUSD 5th-grade students determined to be eligible for G&T programs in 2007–08 who applied for admission to one of the two middle schools with an oversubscribed G&T magnet program. This group includes 542 students, 394 of whom were offered admission and 148 of whom were not. We find no statistically significant differences in the observed characteristics of lottery winners and losers, suggesting that the lotteries were in fact conducted in a random way.

The students in the lottery differ both academically and demographically from the students who were included in the regression discontinuity study. Not only do the lottery students have higher test scores than students at the eligibility cutoff, but their test scores exceed those of the average G&T student in the district. Lottery participants are also less likely to be on subsidized lunch, and less likely to be minority.

Of the 542 lottery participants, only 440 students, including 331 winners (84 percent) and 109 losers (74 percent), remain in LUSD by 7th grade. Fortunately, the observed characteristics of lottery winners and losers who remain in the district continue to be very similar. Even so, when analyzing the data we control for students’ demographic characteristics and prior achievement, and use weights designed to make the final sample comparable in terms of its observed characteristics to the set of students that initially applied for the lottery.

One disadvantage of this second study is that the lottery losers have a range of alternative experiences and most participate in standard G&T programs, so the comparison group’s educational experience is less clear than it was in the regression discontinuity analysis. Nonetheless, our data confirm that students admitted to the G&T magnet schools with lotteries seem to have experienced large improvements in their educational environment. Winning the lottery increased the average achievement of students’ classroom peers by as much as a full standard deviation in some subjects. And in contrast to the G&T program as a whole, students admitted by lottery to G&T magnet program were assigned to more effective teachers.

Turning to student outcomes, however, our results provide little evidence that attending a G&T magnet program leads to improvements in student achievement (see Figure 3). The one exception is science test scores, for which we estimate a positive effect of 0.28 standard deviations. Due to the relatively small sample sizes, all of the effects are imprecisely estimated and do not allow us to definitively rule out reasonably large positive effects. Even so, the estimated effects for math, reading, and social studies are negative, and the estimated effect for language is effectively zero.

Discussion

It is difficult to understand why we find little evidence that G&T programs positively affect achievement. A common concern with studies of high-achieving students is that the available achievement measures may not be well suited to discern improvements for this group. This would be particularly worrisome if we were using a state accountability exam targeted toward low-achieving students, but it is less of an issue with the Stanford Achievement Tests. Indeed, we found little evidence of students performing near the maximum levels on these tests in either the regression discontinuity or lottery samples. Although it is possible that the additional course material taught in G&T classes is poorly aligned with topics covered in the achievement test, research documenting the benefits of being placed with higher-ability peers suggests that we should see improvements, even if that were the case.

The effect of being placed in a higher-ability classroom may not necessarily be positive, however, especially for a marginal G&T student. In particular, the drop in ranking relative to one’s peers may have a negative effect: a marginal G&T student is likely to go from being near the top of the regular class to being near the bottom of the G&T class. Even students in the middle of the G&T distribution are likely to experience a loss of ranking in the magnet G&T schools as compared to their neighborhood schools. It may be that students are demoralized by the drop in their relative rankings or that teachers provide more resources to students at the top of the class.

Substantial evidence from educational psychology indicates that students who are placed in higher-achieving groups can suffer psychological harm. A commonly used measure is a student’s “self-concept,” how a student perceives her abilities relative to an objective measure such as achievement. A 1995 study by Herbert Marsh and colleagues compared G&T students to observably similar students in mixed G&T and non-G&T classes and found that G&T students show declines in their math and reading self-concept. More recent research has documented lower self-concept and greater test anxiety among gifted students in ability-segregated classrooms.

Although we do not have direct evidence on student confidence, we can make use of student course grades and rank within the class to probe for evidence consistent with this kind of effect. We evaluate the impact of G&T program enrollment in the regression discontinuity study and of attending a G&T magnet in the lottery analysis. In both cases, we find clear reductions in student grades. For the regression discontinuity sample, grades fall by a statistically significant 4 points out of 100 (3 points changes a grade from a B+ to a B, for example) in math and by 2 to 3 points in other subjects, although these effects are not statistically significant for 7th grade. For the lottery analysis, the grade reductions are even more dramatic, with drops of 7 points in math, 8 in science, and 4 in social studies.

It is also useful to consider how students’ rankings within their peer groups differ by treatment status, as this provides a direct measure of how a student may perceive his position in the overall distribution of student ability. We assume that students mostly compare themselves to their schoolmates who take the same courses in the same grade. Thus, we rank students within each school, grade, and course by their final course grades and then convert these rankings to percentiles. The rankings based on 7th-grade courses exhibit notable drops when students cross the G&T eligibility threshold. Controlling for race, gender, economic disadvantage, LEP (Limited English Proficiency), and prior gifted status, marginal G&T students have a relative rank in 7th grade that is 13 to 21 percentiles lower than similar students who were not admitted. Attending a premier G&T magnet in 7th grade generates a nearly 30 percentile ranking drop in all four of the courses examined.

In short, the necessary conditions are clearly met for a drop in relative ranking to play a role in offsetting the expected positive impact of more rigorous courses, more effective teachers, and higher-achieving peers. The possibility that G&T students are subject to such a mechanism suggests potential constraints on the benefits of programs that provide more similar peers and an increase in traditional education inputs.

One should not conclude from the lack of achievement results, however, that the G&T programs should be scuttled. Our analysis occurs in a district with a large number of relatively high-quality magnet programs, and thus the alternatives to the G&T programs may be strong. There may also be benefits that we are not able to capture, such as impacts on SAT scores, graduation rates, and college attendance. Further, our study examines a G&T program in one district. Certainly, districts vary in the approaches they take to educating gifted students, so it may be that similar studies of programs in other districts would yield different results. Nonetheless, this study does raise questions about the efficacy of G&T programs and the traditional model of ability-segregated classrooms.

Sa Bui is a doctoral candidate in economics at the University of Houston, where Steven Craig is professor of economics and Scott Imberman is assistant professor of economics.

If principals have the authority to dismiss teachers, will they dismiss the less effective ones, or will they instead make perverse decisions by letting the good teachers go? Evidence from low-stakes surveys suggests that principals are able to identify the most and least effective teachers in their schools, as measured by their impact on student achievement (see “When Principals Rate Teachers,” research, Spring 2006). But would that ability influence their dismissal decisions?

On this topic, debate has been vigorous but research almost nil, in good part because teachers with tenure are not easily dismissed and principals take on that task only if they have a strong backbone or face an extremely urgent situation, or both. In some instances, however, principals have considerable latitude when it comes to dismissing teachers who have not been in service long enough to have earned tenure.

One such situation developed in Chicago in July 2004 when the Chicago Public Schools (CPS) and the Chicago Teachers Union signed a new collective-bargaining agreement that gave principals the flexibility to dismiss probationary (nontenured) teachers beginning in the 2004–05 school year for any reason and without the documentation and hearing process that is typically required for dismissals in other districts. Since CPS provided information that allowed me to link information on CPS teacher dismissals to several measures of teacher performance, I was able to study whether principals exercise their authority wisely. The procedures were fairly straightforward. By comparing the characteristics of dismissed versus nondismissed probationary teachers within the same school and year, I was able to determine just how much weight school administrators place on a variety of teacher characteristics, including their performance in the classroom.

I find that principals in Chicago do exercise their authority in  sensible ways. Principals are more likely to dismiss teachers who are frequently absent and who have previously received poor evaluations. They dismiss elementary school teachers who are less effective in raising student achievement. Principals are also less likely to dismiss teachers who attended competitive undergraduate colleges. It is interesting to note that dismissed teachers who were subsequently hired by a different school are much more likely than other first-year teachers in their new school to be dismissed again.

These results suggest that other school districts could possibly improve student achievement if they adopted policies similar to those applied in Chicago. To be clear, however, the analysis presented in this paper does not seek to evaluate the educational impact of this new policy. Instead, it uses the existence of the policy, in conjunction with detailed data on teachers and principals, to provide descriptive evidence on the relationship between the exercise of dismissal authority and teacher effectiveness.

Teacher Dismissals in Chicago

As in many public school districts, teacher layoffs and dismissals in CPS are highly regulated. Prior to 2004, virtually no teachers—not even probationary teachers—were dismissed for cause in CPS. Of course, it is likely that some teachers who switched schools or left CPS entirely were informally “counseled out” by school administrators. But it was impossible to distinguish these “involuntary” separations from truly voluntary attrition.

This situation changed with the signing of a new collective-bargaining agreement in 2004. Each February, principals are able to log into a district computer system that has a list of all of the probationary teachers in their school (i.e., those who have been teaching for fewer than five consecutive years during the period of my analysis). The principal can then check one of two boxes: renew or nonrenew. Although principals are required to provide district officials with at least one reason for the nonrenewal decision, they are not required to justify or explain their decision and they do not need to provide teachers with this reason. If a principal chooses nonrenew, the teacher may reapply to positions in other Chicago public schools. However, nonrenewed teachers are not guaranteed another job in CPS. The ease with which administrators can dismiss a probationary teacher, with a simple “click” of a button, is noteworthy. This policy change made Chicago the only large school district in the country to provide principals with this degree of flexibility over personnel decisions. Already since the conclusion of the analysis period for this study (2005 through 2007), this flexibility has diminished in several ways. For example, the probationary period has been reduced from 4 to 3 years, and principals who choose to nonrenew a teacher now must have conducted at least one formal observation of the teacher prior to nonrenewal.

Data

The data for my study of this policy change come from several sources. Teacher personnel files provide information on teacher background, current assignment, and, for probationary teachers, whether or not they were renewed. I supplement this with information on school demographics, principal characteristics from personnel files, and student test-score information.

I examine dismissal among probationary teachers in CPS in three consecutive school years: 2004–05, 2005–06, and 2006–07. The sample excludes individuals who were employed by the central office, including speech pathologists, nurses, counselors, and teachers working in administrative or professional development capacities. Moreover, I exclude teachers in a handful of “alternative” schools that serve severely disabled students or other special populations, as well as teachers on leave or who were employed less than half time. For a small number of teachers who taught subjects such as art or music in multiple schools, I include only the observation in the school that is listed as their “primary” appointment. The final sample consists of 16,246 elementary school teachers and 7,764 high school teachers spread across 588 schools.

Measures of Teacher Quality

This analysis incorporates three proxies for teacher performance. First, I use teacher absences because they are well measured, are easy to interpret, and impose substantial nonfinancial and financial costs on the school. The second measure is the formal performance rating that the principal gave the teacher in prior years. Traditionally, principals rate teachers every one to three years (depending on the tenure status of the teacher) on a four-point scale that indicates superior, excellent, satisfactory, or unsatisfactory performance. While there are no high stakes associated with these ratings (virtually no teachers receive an unsatisfactory rating), there is considerable variation across teachers in the top rating categories, and they arguably provide a sense of how the principal views the teacher. The third measure is a value-added estimate of teacher effectiveness. This measure is meant to capture the extent to which each teacher contributes to student achievement growth from one year to the next, as measured by the standardized tests taken by students in CPS. While this is an objective and direct measure of one important dimension of teacher effectiveness, only a fraction of teachers work in grades and subjects in which students take standardized tests. It is not possible to calculate value-added measures for many teachers in our sample, including teachers in grade 2 or below, most teachers in grades 10 or above, and any teacher in a noncore subject. Unlike some school districts, Chicago traditionally has not maintained reliable data linking teachers to classrooms, particularly at the elementary level. Working with CPS officials, however, I was able to obtain such links for a limited sample of teachers and years, thus allowing me to create value-added measures for part of my sample.

Methods

The primary goal of my analysis is to determine which teacher, principal, and school characteristics are associated with the likelihood that a teacher will be dismissed. I first compare the probability that a teacher is dismissed across schools and years in order to discern any differences related to school characteristics. Then to examine the influence of teacher characteristics on the likelihood of dismissal, I compare teachers within the same year and school to account for unobserved school-level factors that might be correlated with teacher characteristics and the probability of dismissal.

A concern with this approach is that if the analysis fails to include a teacher characteristic that a) principals consider in the dismissal decision and b) is correlated with one of the included variables, the estimate for the included characteristic may be biased. One potentially important variant of this concern involves the supply of teachers. If it is more difficult to find qualified teachers in certain subjects or grade levels, then the principal may be less likely to dismiss teachers in these areas. To the extent that teachers in harder-to-staff areas are concentrated among particular demographic groups, or tend to graduate from particular institutions, the results for these teacher characteristics could be misleading. Also, schools fund teachers from a variety of revenue streams, and it may be difficult for principals to reallocate positions across funds. For this reason, if a school experiences a decline in a particular revenue fund, the principal may be more inclined to dismiss teachers funded by this source.

To address these concerns, I account in all analyses for the teacher’s program area (for example, regular education grades 1 to 3, regular education grades 4 to 8, secondary math, secondary science, bilingual education, vocational education, etc.) and for the revenue source from which each teacher position is funded.

Of course, it is still possible that my results concerning specific teacher characteristics suffer from a standard omitted variable bias. For example, it may be the case that high rates of absenteeism are associated with a bad attitude or shirking in other dimensions, and it is these factors, rather than the absences per se, that the principal is reacting to in dismissing teachers with more absences. In this case, one may not be able to say anything definitive about principal views regarding teacher absenteeism itself, but rather about behaviors and characteristics associated with absenteeism, all of which presumably speak to performance in some form or another.

Dismissal Policy Impact

Each year under the new policy, roughly 11 percent of probationary teachers were dismissed, despite the fact that more than one-third of schools did not dismiss any teachers. The numbers of teachers who were nonrenewed in any given year likely overstates the impact of the policy because a number of young teachers would likely have left CPS in the absence of the policy, either voluntarily or due to subtle “encouragement” on the part of the principals. If the dismissal policy merely formalized previously informal dismissals, however, then one would not necessarily expect to find a substantial change in separations.

Comparing dismissal rates before and after implementation of the new policy provides insight on this issue. In the three years prior to the introduction of the policy, roughly 10 to 15 percent of first-year probationary teachers left CPS and an additional 4 percent moved to a different CPS school. In the years after the policy was in place, the corresponding rates were roughly 18 and 10 percent, respectively. Comparing the year immediately prior to establishment of the policy (2004) with the first two years of the policy’s implementation (2005 and 2006), it appears that the separation rate increased by roughly 9 percentage points (see Figure 1). In contrast, there was virtually no change among more-experienced teachers (i.e., those with 6 to 15 years of experience), who were not subject to the policy. The dismissal policy therefore appears to have had at least a modest impact on the number of teacher separations, although the impact is not as large as the overall nonrenewal numbers would suggest.

It is worth noting that more than half of the dismissed teachers were rehired the following year by another school in the district. For example, 50.6 percent and 56.4 percent of first-year probationary elementary and high school teachers, respectively, who were dismissed in spring 2005 were rehired by a CPS school in the fall. At least some of the dismissals under the policy were the result of position cuts, in which case the teacher’s former principal may have provided the teacher with a good recommendation; it is therefore not surprising that some fraction of dismissed teachers were rehired. It is also likely that some fraction of teachers dismissed due to poor performance were also rehired by other CPS schools.

Which school and principal characteristics are related to dismissal? In both elementary and secondary schools, principals in the district’s larger schools dismissed a smaller fraction of probationary teachers. In elementary schools, higher student achievement at the school is associated with a smaller fraction of probationary teachers being dismissed. Among high schools, however, schools with higher-achieving students dismissed a larger fraction of their probationary teachers. Principals who attended more competitive colleges and principals who were older dismissed a smaller proportion of teachers in both elementary and high schools. Male high-school principals dismissed a significantly smaller percentage of their teachers, while principal gender did not play as important a role at the elementary level. Finally, principals new to the building dismissed a substantially larger fraction of teachers in elementary schools, but not in high schools.

Teacher Characteristics

Turning to the characteristics of individual teachers, I find that prior-year principal evaluations and current-year teacher absences both influence the likelihood of dismissal (see Figure 2). Teachers who were rated satisfactory in the prior academic year were 22.1 percentage points more likely to be nonrenewed than teachers in the same school who were rated superior. Teachers rated excellent were 4.3 percentage points more likely to be dismissed than those rated superior. Given an average dismissal rate of roughly 11 percent, these results suggest that teacher performance as reflected in prior evaluations is strongly associated with dismissal. Teachers who were absent 11 to 20 times between September and March of the current year were also 11.3 percentage points more likely to be nonrenewed than their colleagues who were never absent. Teachers absent 6 to 10 days were 3.5 percentage points more likely to be dismissed.

The results also indicate that principals value teachers with stronger educational backgrounds as measured by college quality. For example, a teacher who attended a highly competitive college (with a Barron’s ranking of four) is nearly 3 percentage points (roughly 15 percent) less likely to be dismissed than a teacher who attended a least-competitive (unrated) college. On the other hand, on average, principals do not seem to value certification exam performance or advanced degrees, at least after taking into account the other available measures of teacher performance.

Interestingly, probationary teachers who were dismissed from another school in the prior year, and rehired by the current school, are substantially more likely to be dismissed a second time. For example, elementary school teachers who were dismissed from another school in the prior year were 4.9 percentage points (about 45 percent) more likely to be let go relative to first-year teachers in the school. In high school, previously dismissed teachers were 13.4 percentage points (more than 130 percent) more likely to be dismissed than first-year teachers. These results suggest that many of the initial nonrenewal decisions were not idiosyncratic, stemming from a particularly bad match, or based on temporary difficulties experienced by the teacher. Rather, they suggest that, at least in many cases, the initial nonrenewal decision reflected a concern with the teacher’s general productivity.

These results provide evidence that principals consider some measures of teacher performance and qualifications in making their dismissal decisions. To the extent that one views student achievement as the primary outcome of interest, however, one should directly assess how a teacher’s ability to improve student achievement influences the likelihood of dismissal. I provide some evidence on this issue by focusing on the relationship between teacher value-added and dismissal for the subsample of 803 elementary school and 1,134 high school teachers for which value-added measures are available.

For elementary schools, a one-standard-deviation increase in teacher value-added is associated with a 7.1-percentage-point (over 100 percent) decrease in the likelihood of dismissal (see Figure 3). In contrast, I find that teacher value-added has zero association with dismissal among the sample of 9th-grade core-subject teachers in high schools. One possible reason for the difference across grade levels is that the assessment used for the 9th-grade value-added measure is the PLAN test, which is given in the fall of a student’s 10th-grade year. PLAN is developed by ACT and is not tightly linked to any particular curriculum. Hence, because of both the timing of the exam and its content, the 9th-grade value-added measures may not capture teacher effectiveness as well as the elementary value-added measures.

Do Principals Discriminate?

One potential concern about policies like Chicago’s that provide principals with greater discretion in personnel decisions is that principals would dismiss teachers capriciously or on the basis of criteria unrelated to performance. Indeed, I find that several teacher demographics, including age, gender, and race, are associated with the likelihood of dismissal, even after controlling for the measures of teacher performance and qualifications described above. Principals are 3.8 percentage points more likely to dismiss male teachers than female teachers, an effect of more than 25 percent given the baseline dismissal rate of 10 to 12 percent. Principals are considerably more likely to dismiss older teachers. For example, teachers 36 to 50 years of age are 4 percentage points (33 percent) more likely to be dismissed than teachers age 22 to 28. The relatively small number of probationary teachers over age 50 is 10 percentage points (nearly 100 percent) more likely to face dismissal than their youngest counterparts. And black teachers are 2.1 percentage points less likely to be dismissed than their colleagues.

While these results raise some concerns, it would be incorrect to conclude on the basis of this evidence alone that principals in Chicago were acting in a discriminatory manner. The analysis reported here cannot control for many direct measures of teacher qualities that principals could legitimately consider in making a dismissal decision (e.g., energy, enthusiasm, ability to relate to children, familiarity with the best instructional practices). Moreover, the sample selection introduced by nonrandom hiring may lead to biased estimates of the relationship between dismissal and any easily observable, predetermined teacher characteristic such as age or gender. If, for example, male teachers were less productive on average than female teachers (or even if the principal believed this to be the case), then the marginal male teacher who was hired must be more attractive on some other, likely unobservable, dimension relative to the marginal female teacher hired.

In order to shed light on the issue of principal discrimination, I examine whether principals are more likely to dismiss teachers of a different gender, age, or race from their own. Although principals are no more likely to dismiss a teacher of the opposite gender, they are somewhat more likely to dismiss teachers of a different race. While these patterns could indicate discrimination, it is possible that they are explained by other factors. Given the widespread belief that same-race role models are crucial for low-income students, it would not be surprising if principals took into account the composition of their student body when making dismissal decisions. Indeed, insofar as prior research has demonstrated that, all else equal, students learn more when taught by a teacher of the same race, this might be a legitimate determination on the part of the principal. My results provide support for this hypothesis. I find that as the fraction of students in the school that share the race of the teacher rises, the likelihood that the teacher will be dismissed declines. Specifically, an increase of 50 percentage points in the fraction of students who share the teacher’s race decreases the likelihood that the teacher will be dismissed by slightly more than 1 percentage point, or 10 percent. More importantly, the evidence that principals are more likely to dismiss a teacher of a different race becomes statistically insignificant after controlling for this variable.

Finally, I find evidence that younger principals are more likely to dismiss older teachers than they are to dismiss younger teachers. There are no obvious explanations for this pattern, although one might speculate that younger principals may value different characteristics in a teacher than older principals. Regardless, this pattern does seem to warrant further exploration.

Conclusions

By comparing the characteristics of dismissed versus nondismissed probationary teachers within the same school and year, the analysis presented above provides a unique source of evidence on which teacher characteristics principals value most highly. I find that principals do consider teacher performance in determining which teachers to dismiss. Principals are significantly more likely to dismiss teachers who are frequently absent and who have received unsatisfactory evaluations in the past. Perhaps most telling, elementary school teachers who were dismissed had significantly lower impacts on student achievement in prior years than their peers who were not dismissed.

These results suggest that reforms along the lines of the Chicago policy could improve student achievement by providing principals with the tools to manage the quality of personnel in their classrooms. It should be noted, however, that many principals—including those in some of the worst-performing schools in the district—did not dismiss any teachers despite the new policy. The apparent reluctance of some Chicago principals to utilize the additional flexibility granted under the new contract may indicate that issues such as teacher supply and/or social norms governing employment relations are more important factors than policymakers have realized.

Brian Jacob is professor of education policy and economics at the University of Michigan. This article is based on a study that is forthcoming in Educational Evaluation and Policy Analysis.

Calls to reform teacher layoff policies have begun to appear with regularity in newspaper editorials, policy briefs, and statehouses—and for good reason. A growing body of research confirms that teacher quality is the most influential in-school factor driv­ing student achievement. That being the case, teacher dismissal policies and procedures can have profound implications for how much students learn.

Newly available data on “reduction-in-force” (RIF) notices received by teachers in Washington State shed light on the consequences of existing layoff policies for student achievement as well as the consequences of adopting alternatives. Our analysis of these data provides strong evidence that seniority plays an out­sized role in determining which teachers are targeted for layoffs, likely in part because collective bargaining agreements ordinarily require that the teachers last hired are the first to be fired. Those in subject areas with teacher shortages, such as mathematics and sci­ence, are less likely than other teachers to receive a lay­off notice, suggesting that districts have some degree of flexibility in their dismissal procedures. However, were districts to adopt policies that allowed admin­istrators to dismiss teachers according to their effec­tiveness rather than their seniority, they could lay off fewer teachers, achieve the same budgetary savings, and increase the overall efficacy of their teaching force.

Seniority-Based Layoff Policies

In the overwhelming majority of school-district collective bargaining agreements, “last in, first out” provisions make seniority the determining factor in which teachers are laid off. All of the 75 largest school districts in the nation use seniority as a factor in layoff decisions, and seniority is the sole factor determining the order of layoffs in more than 70 percent of these districts.

The situation in Washington State—the focus of this study—looks similar. A review of the collective bargaining agreements operating in Washington’s 10 largest school districts shows that all use seniority as a basis for determining layoffs, and 8 of these districts use seniority as the only determinant of which teachers get laid off.

There are notable examples of districts that do not rely solely on seniority. In 2004, the Chi­cago Public Schools changed its policies to allow principals’ evaluations of untenured teachers to influence layoff decisions (see “Principled Principals” research). And the Los Angeles Unified School District recently agreed to limit the use of seniority in layoff determinations as part of a settlement in a lawsuit brought by the American Civil Liberties Union (ACLU). Over the past two years, more than a dozen states have sought to change laws that make seniority the determining factor in layoff decisions; so far, Florida, Idaho, Utah, and Ohio have succeeded.

Driving these changes is a belief that seniority-based layoff policies may have negative consequences for student achieve­ment. First, to achieve a targeted budget reduction, school districts need to lay off a greater number of junior teachers than senior teachers (as junior teachers have lower salaries), meaning that a seniority-based layoff policy will cause class sizes to rise more than they would under an alternate arrange­ment. Second, the most-senior teachers may not be the most effective teachers. With a seniority-based layoff policy, school systems may be forced to cut some of their most promising new talent rather than dismiss more-senior teachers, who may not be terribly effective in raising student achievement. A final way in which seniority-based systems may have consequences for student achievement is that strict adherence to seniority would require at least some districts to lay off teachers in subject areas with teacher shortages, such as math and special education.

Beyond the effects of seniority-based layoffs on the teacher workforce as a whole are potential distributional conse­quences. In many districts, schools with high proportions of at-risk students tend to employ the most first- and second-year teachers. Under a seniority-based layoff policy, these schools stand to lose the largest share of their teachers.

Data

This study relies on a unique dataset from Washington State that links teachers to their schools and, in some cases, to their students; the dataset also includes information on those teachers who received RIF notices in the 2008–09 and 2009–10 school years. In the 2008–09 school year, 2,144 employees received a layoff notice and in 2009–10, some 450 employees received a notice.

Employees who received these notices can be linked with administrative records of their credentials, school assignments, academic degrees, and compensation. The administrative database we used provides a record of employees working in Washington State’s school districts and includes information such as their places of employ­ment, experience and degree, gender and race, and annual compensation levels.

We restrict our analysis to employees who were in a teach­ing position the year they received a layoff notice. Our final sample includes 1,717 teachers who received a layoff notice in 2008–09 and 407 teachers who received one in 2009–10, with 130 teachers who received a layoff notice in both school years. Overall, about 2 percent of teachers in the state received a layoff notice in either year. It is important to stress that not all these teachers were ultimately laid off, largely due to the influx of federal stimulus money. Of the 1,717 teach­ers who received a RIF notice in 2008-09, for example, 1,457 returned to the same district in 2009-10. We still focus on all RIF notices because they indicate the teachers who were targeted for layoffs, and thus tell us about the likely effects of the system that governs layoffs.

The database does not include a direct measure of a teach­er’s seniority in the current district, so we estimate seniority based on how many years the teacher has been employed by the same district. The credentials data include where each teacher was trained and in what areas each teacher holds endorsements. We create a measure of the selectivity of each teacher’s college and code each endorsement a teacher holds in any of 10 subject areas.

Information about the schools in which teachers are employed comes from two sources. Washington State Report Card data provide measures of racial composition, student-teacher ratios, the percentages of students enrolled in the free or reduced-price meals program, total enrollment, and the percentage of students who passed the reading and math Washington Assessment of Student Learning exams in each teacher’s school. We use the Common Core of Data to iden­tify teachers in urban areas, the grade level of each teacher’s school, and the per-pupil expenditure on instruction by each teacher’s district.

We can also link a subset of teachers to their students’ test-score performance, which allows us to use value-added models to estimate their teaching effectiveness. Our data on student achievement come from the Washington State Assessment of Student Learning, a statewide test given annually in 3rd through 8th grade as well as in 10th grade. The student database also includes information on race and ethnicity, free or reduced-price meal eligibility, and status in the following programs: Learning Assistance Program reading/math, Title I reading/math, Title I Migrant, Gifted/Highly Capable, State Transitional Bilingual Program, and Special Education.

Methods

We first examine the simple associations between the various teacher and school characteristics listed above and the likelihood of receiving a layoff notice. In order to provide a more detailed picture of the factors that are associated with teacher layoff notices, we then examine the effects of each of these various factors on the prob­ability that a teacher received a layoff notice, while con­trolling for the others. Of course, these relationships are correlations only and in theory may not represent causal relationships. However, we are confident that, despite the nonexperimental nature of this study, its findings none­theless provide an accurate picture of the causal impact of, for instance, a teacher’s credential on the likelihood of receiving a layoff notice.

The teacher characteristics that we examine include senior­ity in district, degree level (master’s or higher vs. bachelor’s), gender, race, college selectivity, and endorsement area. The school characteristics include whether it is in an urban area, grade level (e.g., high school), the number of students enrolled, student-teacher ratio, the percentage of students who are eli­gible for the free or reduced-price lunch program, the percent­age of minority students, and measures of student achievement in reading and math. In addition, we control for district-level characteristics, including total enrollment, per-pupil expendi­tures, and percentage of funding that comes from local, state, and federal sources.

These analyses identify the teacher, school, and district characteristics that are associated with layoff notices, but perhaps of greater interest is the relative effectiveness of teachers who receive layoff notices. For the subset of teach­ers who can be linked to students, we are able to estimate value-added measures of classroom performance for each teacher in each year. These indicate how well a teacher’s students did relative to other teachers’ students, controlling for prior student achievement and for student and fam­ily background characteristics (for example, age, race and ethnicity, disability, free or reduced-price lunch status, and parental education level).

Who Gets RIFed?

Not surprisingly, we find that most of the teachers receiving layoff notices are relatively junior. Approximately 60 percent of teachers receiving layoff notices have two or fewer years of experience, and approximately 80 percent have two or fewer years of seniority within their current district. It is interesting to note, however, that some teachers who receive layoff notices are well into their careers, implying that at least some districts in the state are making judgments about which teachers should be laid off based on criteria other than seniority.

Teachers who received layoff notices are also far less likely to hold an advanced degree. Consequently, there is an aver­age difference of about $15,000 in salary between teachers who did and did not receive notices. Had all 1,717 teachers who received layoff notices in 2008–09 actually been laid off, the salary savings in the state would have been $5,521,238. As noted earlier, one of the prevail­ing critiques of seniority-based layoffs is that it is necessary to lay off more teachers in order to attain a specified budget objective than it would be if districts used alternative criteria. If teach­ers were laid off at random (so that the laid-off teachers made the average salary in their dis­trict), we estimate that it would only be neces­sary to lay off 1,349 teachers in order to attain the same budgetary savings. This is roughly 20 percent less than the actual number of teachers who received layoff notices.

According to the 2006 report “Educator Supply and Demand in Washington State,” there are 14 endorsement areas for which there are “high degrees of shortage,” all of which fall into math, science, or special education. We classify any teacher with an endorsement in one of these areas accordingly. There is some evidence to suggest that school districts are choosing to retain teachers in subject areas with teacher shortages, with 13.3 per­cent of teachers that received layoff notices falling into such a category compared to 15.1 percent of teachers who did not receive a notice.

Teachers receiving a notice tended to be in smaller schools, but were not, in general, more likely to be teaching in schools with high proportions of minority students or lower test-score levels. However, school-level measures can mask a significant degree of teacher sorting across classrooms within schools. For the subset of teachers who can be linked to their students, we find that teachers who received a layoff notice are more likely to be teaching poor, non-white, and lower-scoring students than other teachers.

We next examine our value-added measures of teacher effectiveness and find that teachers who received layoff notices were about 5 percent of a standard deviation less effective on average than the average teacher who did not receive a notice. This result is not surprising given that teach­ers who received layoff notices included many first- and second-year teachers, and numerous studies show that, on average, effectiveness improves substantially over a teacher’s first few years of teaching.

Explaining RIFs

Our analysis of multiple factors indicates that, as expected, seniority plays an important role in determining whether teachers receive a layoff notice. We find additional evi­dence that districts are choosing to retain teachers thought to have advanced or atypical skills. On average, teachers with a master’s degree or an endorsement in a subject area with teacher shortages are about 0.6 percentage points less likely to receive a RIF notice. Conversely, teachers with endorsements in health, physical education, or the arts are far more likely to receive a layoff notice. Finally, we find evidence that school districts behave strategically by retaining teachers who have endorsements in multiple areas and therefore provide flexibility in terms of the classes they can teach. Perhaps surprisingly, controlling for district and school characteristics does not noticeably change the results reported above, and few of the school-level vari­ables identifying student demographics are predictors of which teachers receive layoff notices.

Finally, we ran our analysis including value-added measures of teacher effective­ness for the subset of teachers we are able to link to individual students. It is first worth noting that the inclusion of the teacher effec­tiveness measures does little to change the estimated effects of the teacher, school, and district characteristics discussed above. More importantly, the effects of the value-added measures (based on both math and read­ing scores) are close to zero, suggesting that effectiveness plays little or no role in deter­mining which teachers are targeted for lay­offs. And, these results were robust to a vari­ety of different ways of measuring teacher value added. In other words, the fact that teachers who received layoff notices were, on average, somewhat less effective than their peers is an artifact of the relationship between effectiveness and seniority.

Policy Implications

Our findings largely comport with what one would expect given seniority provisions in collective bargaining agree­ments. The surprise is that factors other than seniority do appear to influence which teachers are targeted for layoffs.

To get a more concrete sense of the extent to which various factors play into the targeting of teachers for layoffs, we ran simulations based on the effects calculated by our statistical model. First, we calculate the expected probability of a teacher with each combination of endorsement area and seniority level receiving a layoff notice. Although a teacher’s endorse­ment area does affect the likelihood of being laid off, the effect is far smaller than the influence of seniority. For instance, we estimate the probability that a first-year special education teacher receives a layoff notice is 6.2 percent, compared to 17 percent for a first-year health/physical education teacher. This difference is statistically significant, but it pales in com­parison to the difference in probability for a first-year teacher compared to a teacher with 12 or more years of seniority: The estimated probability of a teacher with 12 or more years of seniority receiving a layoff notice is less than one-quarter of 1 percent for every endorsement area (see Figure 1).

Next we examine the implications of employing an effec­tiveness-based layoff policy rather than the seniority-driven system currently in place. First, we calculate a value-added measure of effectiveness that com­bines data from all available years and both sub­jects (averaging math and reading). Teachers in each school district are then ranked accord­ing to this value-added score. Finally, starting with the least effective teachers in each district and moving up the effectiveness ladder, enough teachers are assigned to a hypothetical layoff pool to achieve a budgetary savings for each district that is at least as great as the budgetary savings each district would have seen had all the teachers who received a layoff notice in 2008–09 actually been laid off.

The overlap between the subgroup of teach­ers who received a layoff notice and the sub­group of teachers who received one in our simu­lation is relatively small—only 23 teachers (or 16 percent of the teachers for whom we could estimate value-added who received a layoff notice). Moreover, because the teachers who received layoff notices in our simulation were more senior (and had higher salaries) than the teachers who actually received layoff notices, the simulation results in far fewer layoffs. We calcu­late that districts would only have to lay off 132 teachers under an effectiveness-based system in order to achieve the same budgetary savings they would achieve with 145 layoff notices under today’s seniority-driven system, a difference of about 10 percent.

As expected, there are large differences in classroom effec­tiveness between teachers who actually received layoff notices and those who would have received them in our effectiveness-based simulation. The two groups differ by about 20 percent of a standard deviation in students’ math and reading achieve­ment (see Figure 2). The magnitude of the difference is strik­ing, roughly equivalent to having a teacher who is at the 16th percentile of effectiveness rather than at the 50th percentile. This difference corresponds to roughly 2.5 to 3.5 months of student learning.

Since there is little overlap between the samples under these different scenarios, we investigate the likelihood that different types of students might be disproportion­ally affected by one type of layoff system. For the subset of teachers who can be linked to student-level data, we consider the characteristics of the students whose teachers received a layoff notice under the actual system and in our simulation. We find that the probability that students in a particular subgroup have a teacher who received a layoff notice varies considerably from one subgroup to the next. In particular, black students are far more likely than other students to have been in a classroom of a teacher who received a layoff notice. The effectiveness-based layoffs result in fewer layoff notices and are much more equita­bly distributed across student subgroups; black students in particular are only marginally more likely to have been in a classroom with a teacher who received a layoff notice under this system.

Districts across the country are rethinking layoff strate­gies. This is sensible, because although the simplicity and transparency of a seniority-based system certainly has advantages, it is hard to argue that it is in the best interest of students. The effectiveness-based system in our simulation would result in a very different group of teachers targeted for layoffs than does the current system and in layoffs that affect different segments of the student population. Most importantly, the differences in the effectiveness of teach­ers laid off under each type of system have implications for student achievement.

Dan Goldhaber is director of the Center for Education Data and Research at the University of Washington Bothell and a co-editor of Education Finance and Policy. Roddy Theobald is a researcher at the Center for Education Data and Research and doctoral student in statistics at the University of Washington.

The working paper on which this article is based is available here.

An unabridged version of this article is available here.

In recent years, a consensus has emerged among researchers that teacher quality matters enormously for student performance. Students taught by more-effective teachers learn substantially more over the course of the year than students taught by less-effective teachers. Yet little is known about what makes for a more-effective teacher.

Most research on teacher effectiveness has focused on teacher attributes, finding that readily measurable characteristics such as experience, certification, and graduate degrees generally have little impact on student achievement. Relatively few rigorous studies look inside the classroom to see what kinds of teaching styles are the most effective. We tackle this underexplored area by investigating the relative effects of two teacher practices—lecture-style presentations and in-class problem solving—on the achievement of middle-school students in math and science.

Ever since John Dewey explored hands-on learning at the University of Chicago Laboratory School more than a century ago, lecture-style presentations have been criticized as old-fashioned and ineffective. It is said, for example, that lectures presume that all students learn at the same pace and fail to provide instructors with feedback about which aspects of a lesson students have mastered. Students’ attention may wander during lectures, and they may more easily forget information they encountered in this passive manner. Lectures also emphasize learning by listening, which may disadvantage students who favor other learning styles.

Alternative instructional practices based on active and problem-oriented learning presumably do not suffer from these disadvantages. But they may have their own shortcomings. Learning by problem-solving may be less efficient, as discovery and problem-solving often take more time than mastering information received from an authority figure. And incorrect or misleading information may be conveyed in conversations among students in middle schools.

Nonetheless, a number of small-scale studies have identified positive impacts of interactive teaching styles on student learning. As a consequence, prominent organizations such as the National Research Council and the National Council of Teachers of Mathematics, since at least 1980, have called for teachers to engage students in constructing their own new knowledge through more hands-on learning and group work. By the mid-1990s, in a study for the National Institute for Science Education, Iris Weiss could identify “some encouraging signs. The majority of elementary, middle, and high school science and mathematics classes worked in small groups at least once a week, and roughly one in four classes did so every day. Moreover, the use of hands-on activities had increased since the mid-1980s.” Even so, more than a decade later, traditional lecture and textbook methodologies continue to be a significant component of science and mathematics instruction in U.S. middle schools. A rigorous, large-scale study has yet to resolve a question that has divided pedagogical thinking for generations.

In our study, we examine whether student achievement in the United States is affected by the share of teaching time devoted to lecture-style presentations as distinct from problem-solving activities. Employing information on in-class time use provided by a nationally representative sample of U.S. teachers in the 2003 Trends in International Mathematics and Science Study (TIMSS), we estimate the impact of teaching practices on student achievement by looking at the differential effects on the same student of two different teachers, using two different teaching strategies. We find that teaching style matters for student achievement, but in the opposite direction than anticipated by conventional wisdom: an emphasis on lecture-style presentations (rather than problem-solving activities) is associated with an increase—not a decrease—in student achievement. This result implies that a shift to problem-solving instruction is more likely to adversely affect student learning than to improve it.

Data and Methodology

Our research draws on data from the 2003 Trends in International Mathematics and Science Study (TIMSS). The TIMSS data comprise information on students in two grades in a number of countries, but we utilize only information on 8th-grade students in the United States. Our sample includes 6,310 students in 205 schools with 639 teachers (303 math teachers and 355 science teachers, of which 19 teach both subjects). In addition to test scores in math and science, the TIMSS data include background information on students’ home and family life as well as data on teacher characteristics, qualifications, and classroom practices. School principals provide information on school characteristics.

Most important for our purpose, teachers were asked what proportion of time in a typical week students spent on each of eight in-class activities. The overall time in class apportioned to three of these activities—listening to lecture-style presentation, working on problems with the teacher’s guidance, and working on problems without guidance—likely provides a good proxy for the time in class in which students are taught new material. We divided the amount of time spent listening to lecture-style presentations by the total amount of time spent on each of these three activities to generate a single measure of how much time the teacher devoted to lecturing relative to how much time was devoted to problem-solving activities.

A change in our measure of teaching style can be interpreted as a shift from spending time on one practice to spending time on the other, holding constant the total time spent on both practices. For example, an increase of 0.1 indicates that 10 percentage points of total time devoted to teaching new material are shifted from teaching based on problem solving to giving lecture-style presentations. We combined the other teaching activities (besides lecturing and problem solving) into a separate measure of the share of total teaching time devoted to other activities and control for this measure throughout our analysis. We also control for the total number of minutes per week that the teacher reported teaching the math or science class, as more total instructional time could have an independent effect on student learning.

Although it is difficult to determine from the TIMSS data exactly how much time is spent on lecturing as distinct from problem-solving activities, it appears that teachers generally follow the advice given by progressive educators. On average, they allocate twice as much time to problem-solving activities as to direct instruction. Specifically, teachers devote about 40 percent of class time to problem-solving activities (with or without teacher guidance); during roughly 20 percent of class time, students listen to the initial presentation of material to be learned. The remainder of the class time is allocated to such tasks as class management, reviewing homework, re-teaching the material, and clarifying content (see Figure 1).

Teachers who spent more time lecturing were more likely to be male and under age 50. Interestingly, they were also less likely to have the maximum number of years of teacher training registered by the background survey or to have taken pedagogical or content knowledge classes in the prior two years (see Figure 2).

A key challenge in studying the effects of teaching practices is that teachers may adjust their methods in response to the ability or behavior of their students. If teachers tend to rely more on lectures when assigned more capable or attentive students, this would generate a positive relationship between the amount of time spent lecturing and student achievement, even in the absence of a true causal effect. Similarly, there could be unobserved differences between students whose teachers rely more and less heavily on lecturing if, for example, teachers in schools serving low-income students adopt different practices than teachers in other types of schools.

To address these concerns, we exploit the fact that the TIMSS study tested each student in both mathematics and science. This allows us to compare the math and science test scores of individual students whose teacher in one subject tended to emphasize a different teaching style than their teacher in the other subject. In other words, we ask, if a given student’s math teacher spent more (or less) time lecturing than his or her science teacher, does the student perform better or worse on the math test than on the science test?

Results

Contrary to contemporary pedagogical thinking, we find that students score higher on standardized tests in the subject in which their teachers spent more time on lecture-style presentations than in the subject in which the teacher devoted more time to problem-solving activities. For both math and science, a shift of 10 percentage points of time from problem solving to lecture-style presentations (e.g., increasing the share of time spent lecturing from 20 to 30 percent) is associated with an increase in student test scores of 1 percent of a standard deviation. Another way to state the same finding is that students learn less in the classes in which their teachers spend more time on in-class problem solving.

Importantly, the strength of the relationship increases when we restrict our analysis to the roughly one-third of students in the TIMSS sample who had the exact same peers in both their math and science classes. Among this group of students, a shift of 10 percentage points of time from problem solving to lecturing is associated with an increase in test scores of almost 4 percent of a standard deviation—or between one and two months’ worth of learning in a typical school year (see Figure 3). This pattern increases our confidence that the overall result does not reflect differences in the peer composition of students’ math or science classes. In fact, it suggests that peer effects may actually be leading us to understate the strength of the relationship between lecturing and student learning.

Do certain types of students benefit more from lectures than others? We find suggestive evidence that the relationship between lecture-style teaching and achievement is strongest among higher-achieving and more-advantaged students. For example, the positive effect is largest for students who report having more than one bookcase in the home, a rough indicator of the quality of their home environment. There is no evidence, however, that lower-achieving students or students from less-advantaged backgrounds learn less when their teachers emphasize lectures.

These patterns are consistent with the findings of a 1997 study by Dominic Brewer and Dan Goldhaber, which found that more in-class problem solving for American 10th-grade students in math is related to lower test scores on a standardized test. Because our results are based on comparisons of the same student in two different classes, however, they are less subject to the concern that teachers adjust their practices based on the students to which they are assigned. Furthermore, the other commonly investigated teacher characteristics (e.g., gender, experience, and credentials) do not show significant effects on student achievement in our analysis. This is in line with previous findings in the literature and underscores the importance of the statistical relationship between more lecture-style teaching and student achievement.

While the richness of the TIMSS data enables us to control for an unusually large set of teacher characteristics, our results could still be biased if teachers with different effectiveness levels are more likely to choose different teaching styles. For example, if more-effective teachers tend to spend more time lecturing because they are good at it and enjoy it, then our results could show a positive effect of lecture-style presentations, even if those teachers would have been even more effective had they devoted more time on problem-solving activities. Given the pedagogical emphasis on the use of problem-solving activities, it seems unlikely that the very best teachers would be using the less-effective teaching style (the only alternative explanation for our finding).

Still, it is important to keep in mind that our results are limited to student achievement as measured by the 2003 TIMSS test scores in 8th-grade math and science in the United States. Different results might be found for different subjects, grades, or tests. Depending on the teacher, the students, the content taught, or other factors, problem-solving activities could turn out to be the more effective style. Even though lecture-style teaching seems to be a more effective method in middle-school math and science, that does not mean it would be the preferable approach to elementary-school reading.

Also, our findings are based on student performance on the TIMSS math and science exams, which are designed to measure mastery of factual knowledge of the curricula that schools expect students to learn. Other tests intended to measure problem-solving ability and the competence to apply mathematical and scientific concepts in real-world settings (such as the Programme for International Student Assessment [PISA] administered by the Organization of Economic Cooperation and Development) might yield different results. Unfortunately, we are unable to ascertain whether this might be the case, as PISA did not ask teachers about their pedagogical approach.

Finally, our information on teaching practices, which is based on in-class time use reported by teachers, does not allow us to distinguish between different implementations of teaching practices. In other words, a certain teaching technique may be very effective if implemented in the optimal way. But the strength of our approach is that it examines which teaching style turns out to be effective, on average, for teachers in general. Optimal teaching methods that cannot be executed by teachers in general may do more harm than good.

Conclusion

Given the limitations of the data, our finding that spending increased time on lecture-style teaching improves student test scores results should not be translated into a call for more lecture-style teaching in general. But the results do suggest that traditional lecture-style teaching in U.S. middle schools is less of a problem than is often believed.

Newer teaching methods might be beneficial for student achievement if implemented in the proper way, but our findings imply that simply inducing teachers to shift time in class from lecture-style presentations to problem solving without ensuring effective implementation is unlikely to raise overall student achievement in math and science. On the contrary, our results indicate that there might even be an adverse impact on student learning.

Guido Schwerdt is a postdoctoral fellow at the Program on Education Policy and Governance (PEPG) at Harvard University and a researcher at the Ifo Institute for Economic Research in Munich, Germany. Amelie C. Wuppermann is a postdoctoral researcher at the University of Mainz, Germany.

An unabridged version of this article is available here.

The ubiquity of “satisfactory” ratings stands in contrast to a rapidly growing body of research that examines differences in teachers’ effectiveness at raising student achievement. In recent years, school districts and states have compiled datasets that make it possible to track the achievement of individual students from one year to the next, and to compare the progress made by similar students assigned to different teachers. Careful statistical analysis of these new datasets confirms the long-held intuition of most teachers, students, and parents: teachers vary substantially in their ability to promote student achievement growth.

The quantification of differences has generated a flurry of policy proposals to promote teacher quality over the past decade, and the Obama administration’s recent Race to the Top program only accelerated interest. Yet, so far, little has changed in the way that teachers are evaluated, in the content of pre-service training, or in the types of professional development offered. A primary stumbling block has been a lack of agreement on how best to identify and measure effective teaching.

A handful of school districts and states—including Dallas, Houston, Denver, New York, and Washington, D.C.—have begun using student achievement gains as indicated by annual test scores (adjusted for prior achievement and other student characteristics) as a direct measure of individual teacher performance. These student-test-based measures are often referred to as “value-added” measures. Yet even supporters of policies that make use of value-added measures recognize the limitations of those measures. Among the limitations are, first, that these performance measures can only be generated in the handful of grades and subjects in which there is mandated annual testing. Roughly one-quarter of K–12 teachers typically teach in grades and subjects where obtaining such measures is currently possible. Second, test-based measures by themselves offer little guidance for redesigning teacher training or targeting professional development; they allow one to identify particularly effective teachers, but not to determine the specific practices responsible for their success. Third, there is the danger that a reliance on test-based measures will lead teachers to focus narrowly on test-taking skills at the cost of more valuable academic content, especially if administrators do not provide them with clear and proven ways to improve their practice.

Student-test-based measures of teacher performance are receiving increasing attention in part because there are, as yet, few complementary or alternative measures that can provide reliable and valid information on the effectiveness of a teacher’s classroom practice. The approach most commonly in use is to evaluate effectiveness through direct observation of teachers in the act of teaching. But as “The Widget Effect” reports, such evaluations are a largely perfunctory exercise.

In this article, we report a few results from an ongoing study of teacher classroom observation in the Cincinnati Public Schools. The motivating research question was whether classroom observations—when performed by trained professionals external to the school, using an extensive set of standards—could identify teaching practices likely to raise achievement.

We find that evaluations based on well-executed classroom observations do identify effective teachers and teaching practices. Teachers’ scores on the classroom observation components of Cincinnati’s evaluation system reliably predict the achievement gains made by their students in both math and reading. These findings support the idea that teacher evaluation systems need not be based on test scores alone in order to provide useful information about which teachers are most effective in raising student achievement.

The Cincinnati Evaluation System

Jointly developed by the local teachers union and district more than a decade ago, the Cincinnati Public Schools’ Teacher Evaluation System (TES) is often cited as a rare example of a high-quality evaluation program based on classroom observations. At a minimum, it is a system to which the district has devoted considerable resources. During the yearlong TES process, teachers are typically observed and scored four times: three times by a peer evaluator external to the school and once by a local school administrator. The peer evaluators are experienced classroom teachers chosen partly based on their own TES performance. They serve as full-time evaluators for three years before they return to the classroom. Both peer evaluators and administrators must complete an intensive training course and accurately score videotaped teaching examples.

The system requires that all new teachers participate in TES during their first year in the district, again to receive tenure (usually in their fourth year), and every fifth year thereafter. Teachers tenured before 2000–01 were gradually phased into the five-year rotation. Additionally, teachers may volunteer to be evaluated; most volunteers do so to post the high scores necessary to apply for selective positions in the district (for example, lead teacher or TES evaluator).

The TES scoring rubric used by the evaluators, which is based on the work of educator Charlotte Danielson, describes the practices, skills, and characteristics that effective teachers should possess and employ. We focus our analysis on the two (out of four total) domains of TES evaluations that directly address classroom practices: “Creating an Environment for Student Learning” and “Teaching for Student Learning.” (The other two TES domains assess teachers’ planning and professional contributions outside of the classroom; scores in these areas are based on lesson plans and other documents included in a portfolio reviewed by evaluators.) These two domains, with scores based on classroom observations, contain more than two dozen specific elements of practice that are grouped into eight “standards” of teaching. Table 1 provides an example of two elements that comprise one standard. For each element, the rubric provides language describing what performance looks like at each scoring level: Distinguished (a score of 4), Proficient (3), Basic (2), or Unsatisfactory (1).

Data and Methodology

Cincinnati provided us with records of each classroom observation conducted between the 2000–01 and 2008–09 school years, including the scores that evaluators assigned for each specific practice element as a result of that observation. Using these data, we calculated a score for each teacher on the eight TES “standards” by averaging the ratings assigned during the different observations of that teacher in a given year on each element included under the standard. We then collapsed these eight standard-level scores into three summary indexes that measure different aspects of a teacher’s practice:

• The first, which we call Overall Classroom Practices, is simply the teacher’s average score across all eight standards. This index captures the general importance of the full set of teaching practices measured by the evaluation.

• The second, Classroom Management vs. Instructional Practices, measures the difference in a teacher’s rating on standards that evaluate classroom management and that same teacher’s rating on standards that assess instructional practices. A teacher who is more skilled at managing the classroom environment, as compared to her ability to engage in desired instructional activities, will receive a higher score on this index than a teacher who engages in these instructional practices but who is less skilled at managing the classroom.

• The third, Questions/Discussion vs. Standards/Content, measures the difference between a teacher’s rating on a single standard that evaluates the use of questions and classroom discussion as an instructional strategy, and that same teacher’s average rating on three standards that assess teaching practices that focus on classroom management routines, on conveying standards-based instructional objectives to students, and on demonstrating content-specific knowledge in teaching these objectives.

Our main analysis below examines the degree to which these summary indices predict a teacher’s effectiveness in raising student achievement. Note, however, that we did not construct the indices based on any hypotheses of our own about which aspects of teaching practice measured by TES were most likely to influence student achievement. Rather, we used a statistical technique known as principal components analysis, which identifies the smaller number of underlying constructs that the eight different dimensions of practice are trying to capture. As it turns out, scores on these three indices explain 87 percent of the total variation in teacher performance across all eight standards.

For all teachers in our sample, the average score on the Overall Classroom Practices index was 3.21, or between the “Proficient” and “Distinguished” categories. Yet one-quarter of teachers received an overall score higher than 3.53 and one-quarter received a score lower than 2.94. In other words, despite the fact that TES evaluators tended to assign relatively high scores on average, there is a fair amount of variation from teacher to teacher that we can use to examine the relationship between TES ratings and classroom effectiveness.

In addition to TES observation results, Cincinnati provided student data for the 2003–04 through 2008–09 school years, including information on each student’s gender, race/ethnicity, English proficiency status, participation in special education or gifted and talented programs, class and teacher assignments by subject, and state test scores in math and reading. This rich dataset allows us to study students’ math and reading test-score growth from year to year in grades four through eight (where end of year and prior year tests are available), while also taking account of differences in student backgrounds.

Our primary goal was to examine the relationship between teachers’ TES ratings and their assigned students’ test-score growth. This task is complicated, however, by the possibility that factors not measured in our data, such as the level of social cohesion among the students or unmeasured differences in parental engagement, could independently affect both a TES observer’s rating and student achievement. To address this concern, we use observations of student achievement from teachers’ classes in the one or two school years prior to and following TES measurement, but we do not use student achievement gains from the year in which the observations were conducted. (If some teachers are assigned particularly engaged or cohesive classrooms year after year, the results could still be biased; this approach, however, does eliminate bias due to year-to-year differences in unmeasured classroom traits being related to classroom observation scores.)

We restrict our comparisons to teachers and students within the same schools in order to eliminate any potential influence of differences between schools on both TES ratings and student achievement. In other words, we ask whether teachers who receive higher TES ratings than other teachers in their school produce larger gains in student achievement than their same-school colleagues.

Results

We find that teachers’ classroom practices, as measured by TES scores, do predict differences in student achievement growth. Our main results, which are based on a sample of 365 teachers in reading and 200 teachers in math, indicate that improving a teacher’s Overall Classroom Practices score by one point (e.g., moving from an overall rating of “Proficient” [3] to “Distinguished” [4]) is associated with one-seventh of a standard deviation increase in reading achievement, and one-tenth of a standard deviation increase in math (see Figure 1).

The specific point system that TES uses to rate teachers as Proficient and Distinguished is somewhat arbitrary. For a better sense of the magnitude of these estimates, consider a student who begins the year at the 50th percentile and is assigned to a top-quartile teacher as measured by the Overall Classroom Practices score; by the end of the school year, that student, on average, will score about three percentile points higher in reading and about two points higher in math than a peer who began the year at the same achievement level but was assigned to a bottom-quartile teacher.

This difference might not seem large but, of course, a teacher is just one influence on student achievement scores (and classroom observations are only one way to assess the quality of a teacher’s instruction). By way of comparison, we can estimate the total effect a given teacher has on her students’ achievement growth; that total effect includes the practices measured by the TES process along with everything else a teacher does. The difference between being taught by a top-quartile total-effect teacher versus a bottom-quartile total-effect teacher would be about seven percentile points in reading and about six points in math (see Figure 2). This total-effect measure is one example of the kind of “value-added” approach taken in current policy proposals.

From these data, we can also discern relationships between more specific teaching practices and student outcomes across academic subjects (see Figure 1). Among students assigned to different teachers with the same Overall Classroom Practices score, math achievement will grow more for students whose teacher is better than his peers at classroom management (i.e., has a higher score on our Classroom Management vs. Instructional Practices measure). We also find that reading scores increase more among students whose teacher is relatively better than his peers at engaging students in questioning and discussion (i.e., has a high score on Questions/Discussion vs. Standards/Content). This does not mean, however, that students’ math achievement would rise if their teachers were to become worse at a few carefully selected instructional practices. Although this might raise their Classroom Environment vs. Instructional Practices score it would also lower the Overall Classroom Practices score, and any real teacher is the combination of these three scores.

Do these statistics provide any insight that teachers can use to focus their efforts? First, our finding that Overall Classroom Practices is the strongest predictor of student achievement in both subjects indicates that improved practice in any of the areas considered in the TES process should be encouraged. In other words, the practices captured by the TES rubric do predict better outcomes for students. If, however, teachers must choose a smaller number of practices on which to focus their improvement efforts (for example, because of limited time or professional development opportunities), our results suggest that math achievement would likely benefit most from improvements in classroom management skills before turning to instructional issues. Meanwhile, reading achievement would benefit most from time spent improving the practice of asking thought-provoking questions and engaging students in discussion.

Can we be confident that the various elements of practice measured by TES are the reasons that students assigned to highly rated teachers make larger achievement gains? Skeptical readers may worry that better teachers engage in more of the practices encouraged by TES, but that these practices are not what make the teacher more effective. To address this concern, we take advantage of the fact that some teachers were evaluated by TES multiple times. For these teachers, we can test whether improvement over time in the practices measured by TES is related to improvement in the achievement gains made by the teachers’ students. This is exactly what we find. Since this exercise compares each teacher only to his own prior performance, we can be more confident that it is differences in the use of the TES practices themselves that promote student achievement growth, not just the teachers who employ these strategies.

Conclusion

Is TES worth the considerable effort and cost? Does the intensive TES process (with its multiple observations and trained peer evaluators) produce more accurate information on teachers’ effectiveness in raising student achievement gains than do more-subjective evaluations? In fact, studies of informal surveys of principals (see “When Principals Rate Teachers,” research, Spring 2006) and teacher ratings by mentor teachers find that these more-subjective evaluation methods have similar power to detect differences in teacher effectiveness as the TES ratings. These studies may lead some to question the need for the more detailed TES process. We contend, however, that evaluations based on observations of classroom practice are valuable, even if they do not predict student achievement gains considerably better than more subjective methods like principal ratings of teachers.

The additional information the TES system provides can be used in several important ways. First, the data gleaned from the observations allow researchers to connect specific teaching practices with student achievement outcomes, providing evidence of effective teaching practices that can be widely shared.

The TES program also has the advantage of furnishing teachers and administrators with details about the specific practices that contributed to each teacher’s score. The descriptions of practices, and different performance levels for each practice, that comprise the TES rubric can help teachers and administrators map out professional development plans. A school administrator who desires to differentiate the support she provides to individual teachers would benefit from knowing the components of each teacher’s overall scores. A teacher who would like to improve his classroom management skills may find that he has scored relatively low in a particular standard, and then take steps to improve his practice in response to that information.

Finally, scoring individual practices allows for understanding of more fine-grained variations in skill among teachers with similar overall ratings. It is notable, especially given “The Widget Effect” study, that nearly 90 percent of teachers in our sample received an overall “Satisfactory” rating (i.e., “Distinguished” or “Proficient” in Cincinnati’s terms). Still, there are readily discernible differences in mastery of specific skills within that 90 percent, and those differences in skills predict differences in student achievement.

There are other aspects of the Cincinnati system that may or may not account for the results we observed. First, the observers were external to the school and, in most cases, had no personal relationship with the person they were observing. Second, the observers were trained beforehand and were required to demonstrate their ability to score some sample videos in a manner consistent with expert scores. Simply handing principals a checklist with the same set of standards may not lead to a similar outcome.

The results presented here constitute the strongest evidence to date on the relationship between teachers’ observed classroom practices and the achievement gains made by their students. The nature of the relationship between practices and achievement supports teacher evaluation and development systems that make use of multiple measures. Even if one is solely interested in raising student achievement, effectiveness measures based on classroom practice provide critical information to teachers and administrators on what actions they can take to achieve this goal.

Thomas J. Kane is professor of education and economics at the Harvard Graduate School of Education. Eric S. Taylor is a doctoral student at the Stanford University School of Education. John H. Tyler is associate professor of education, economics, and public policy at Brown University. Amy L. Wooten is a doctoral student at the Harvard Graduate School of Education. Reflecting equal contributions to this work, authors are listed alphabetically. This article is based in part on a larger study which is forthcoming in the Journal of Human Resources.

American 15-year-olds continue to perform no better than at the industrial-world average in reading and science, and below that in mathematics. According to the results of the 2009 Program for International Student Assessment (PISA) tests, released in December 2010 by the Organisation for Economic Co-operation and Development (OECD), the United States performed only at the international average in reading, and trailed 18 and 23 other countries in science and math, respectively. Students in China’s Shanghai province outscored everyone.

Many have identified variations in teacher quality as a key factor in international differences in student performance and have urged policies that will lift the quality of the U.S. teaching force. To that end, President Barack Obama has called for a national effort to improve the quality of classroom teaching and repeatedly indicated his support for policies that would provide financial rewards for outstanding teachers.

In a March 2009 speech to the Hispanic Chamber of Commerce, he explained,

Good teachers will be rewarded with more money for improved student achievement, and asked to accept more responsibilities for lifting up their schools. Teachers throughout a school will benefit from guidance and support to help them improve.

In the administration’s Race to the Top initiative, the U.S. Department of Education encouraged states to devise performance pay plans for teachers in the hope that such an intervention could have a significant impact on student performance.

But is there anything in the data the OECD has accumulated to give policymakers reason to believe that merit pay works? Do the countries that pay teachers based on their performance score higher on PISA tests? Based on my new analysis, the answer is yes. A little-used survey conducted by the OECD in 2005 makes it possible to identify the developed countries participating in PISA that appear to have some kind of performance pay plan. Linking that information to a country’s test performance, one finds that students in countries with performance pay perform at higher levels in math, science, and reading. Specifically, students in countries that permit teacher salaries to be adjusted for outstanding performance score approximately one-quarter of a standard deviation higher on the international math and reading tests, and about 15 percent higher on the science test, than students in countries without performance pay. These findings are obtained after adjustments for levels of economic development across countries, student background characteristics, and features of national school systems.

I draw these conclusions cautiously, as my study is based on information on students in just 27 countries, and the available information on the extent of performance pay in a country is far from perfect. Further, the analysis is based on what researchers refer to as observational rather than experimental data, making it more difficult to make confident statements regarding causality.

It is possible that what I have observed is the opposite of what it seems: countries with high student achievement may find it easier to persuade teachers to accept pay for performance, thereby making it appear that merit pay is lifting achievement. More generally, both performance pay and higher levels of achievement could be produced by some set of factors other than all of those taken into account in the analysis. For example, performance pay could be more widely used in places where, as in Asia, cultural expectations for student performance are high, making it appear that performance pay systems are effective, when in fact both performance pay plans and student achievement are the result of underlying cultural characteristics. But even if my findings are not indisputable, I did carry out a variety of checks to see if any observable factor, such as Asian-European differences, could account for the conclusion. Thus far, I have been unable to find any convincing evidence that the findings are incorrect. Given that, let us take a closer look at what can be learned about the impact of performance pay from PISA data.

Prior Research

Standard economic theory predicts that workers will exert more effort when monetary rewards are tied to the amount of the product they produce. Not only does performance pay stimulate individual effort on the job, it is theorized, but jobs where rewards are tied to effort attract energetic, risk-taking employees who are likely to be more productive. This latter consideration, says Stanford economist Edward Lazear, “is perhaps the most important” way in which a merit pay plan can influence worker performance. But if economists expect positive results from merit pay, many educators believe that teachers are motivated primarily by the substantive mission of the teaching profession and that they do not respond to—indeed, they may resent and resist—monetary incentives that tie salary levels to performance indicators.

To see whether the education sector is an exception to general economic theory, a number of performance pay experiments have been carried out, and in Israel and India such studies have shown positive impacts on student achievement. Experimental studies have tracked only the short-term impact of merit pay, however, and so have not identified any long-term effects that might come from changes in the kinds of people who choose to go into this line of work. Conceivably, a merit pay system could discourage entry into the profession of potentially excellent teachers reluctant to subject themselves to the requirements of a pay-for-performance scheme. Alternatively, if performance pay makes teaching more attractive to talented workers, short-term evaluations could understate its benefits.

One way to capture the long-term effects of teacher performance pay, including changes in the characteristics of those choosing to become a teacher, is to compare countries with performance pay systems to those without. This is now possible because the OECD in 2005 administered a separate survey to each of its member countries concerning the teacher compensation systems in place during the 2003–04 school year, when the 2003 PISA study was conducted. The 2003 PISA provides test score results in math, reading, and science for representative samples of 15-year-olds within each country, or nearly 200,000 students altogether. (The relative performance of countries on the PISA changed only slightly between the 2003 and 2009 tests. For example, in no subject did the scores for the United States differ significantly between 2003 and 2009.)

The PISA study is particularly useful, because it also includes information on a wide variety of family, school, and institutional factors that are likely determinants of student achievement. My analysis adjusts, at the level of the individual student, for such characteristics as the student’s gender and age, preprimary education, immigration status, household composition, parent occupation, and parent employment status. Nine measures of school resources and location are available, including class size, availability of materials, instruction time, teacher education, and size of community. Country-level variables included in the analysis were per capita GDP, teacher salary levels, average expenditure per student, external exit exams, school autonomy in budget and staffing decisions, the share of privately operated schools, and the portion of government funding for schools.

The PISA sampling procedure ensured that a representative sample of 15-year-old students was tested in each country. The student sample sizes in the OECD countries range from 3,350 students in 129 schools in Iceland to 29,983 students in 1,124 schools in Mexico. I therefore use weights when conducting my analysis so that each country contributes equally to the estimated effect of performance pay on student achievement.

Measuring Teacher Performance Pay

The measure of performance pay available from the OECD survey is less precise than one would prefer. It simply asks officials in participating countries whether the base salary for public-school teachers could be adjusted to reward teachers who had an “outstanding performance in teaching.” While the survey asked about many other forms of salary adjustments, the study protocol reports that this was the only one that “could be classified as a performance incentive.”Among the 27 OECD countries for which the necessary PISA data are also available, 12 countries reported having adjustments of teacher salaries based on outstanding performance in teaching. The form of the monetary incentive and the method for identifying outstanding performance varies across countries. For example, in Finland, according to the national labor agreement for teachers, local authorities and education providers have an opportunity to encourage individual teachers in their work by personal cash bonuses on the basis of professional proficiency and performance at work. Outstanding performance may also be measured based on the assessment of the head teacher (Portugal), assessments performed by education administrators (Turkey), or the measured learning achievements of students (Mexico). Unfortunately, the coding of the measure does not allow my analysis to consider variation in the scope, structure, and incentives of performance-related pay schemes.

As an example of the limitation of this measure, note that the United States is coded as a country where teacher salaries can be adjusted for outstanding performance in teaching on the grounds that salary adjustments are possible for achieving the National Board for Professional Teaching Standards certification or for increases in student achievement test scores. That policy, however, affects only a few teachers in selected parts of the country. Given such weaknesses in the survey measure, it is all the more remarkable that I was able to detect impacts on student achievement.

Main Results

As noted above, my main analysis indicates that student achievement is significantly higher in countries that make use of teacher performance pay than in countries that do not use it. On average, students in countries with performance-related pay score 24.8 percent of a standard deviation higher on the PISA math test; in reading the effect is 24.3 percent of a standard deviation; and in science it is 15.4 percent (see Figure 1). These effects are similar to the impact identified in the experimental study conducted in India and about twice as large as the one found in a similarly designed Israeli study.

Figure 2 depicts the math result graphically. The figure’s vertical axis displays the average math test scores of students in each country after adjusting for all of the control variables in the model, with the exception of the variable measuring the use of performance pay. The horizontal axis in turn shows the performance pay variable, also after adjusting for those same control variables. The solid line on the figure shows the estimated relationship between these two variables across the 27 countries included in the analysis. It shows a clear positive association between the variation in country-average test scores and the variation in teacher performance pay that cannot be attributed to the other factors included in the analysis.

A lingering concern, however, is that the analysis may be contaminated by the fact that the very cultures that introduce merit pay are those that set high expectations for student achievement. The countries represent widely different cultures, including Asian ones, where expectations for students are often much higher than in Europe and North America. The best way to account for cultural differences among the continents of the world is to control in the analysis for the average effect of living on a particular continent, a strategy known to statisticians as continental fixed effects. Figure 1 thus also shows results based on models that include a fixed effect for each of the four continents with OECD countries: Europe, North America, Oceania, and East Asia. In these models, the effects of pay for performance are shown to be even larger than the results based on comparisons across continents. In other words, the findings cannot be attributed to cultural differences among the major regions of the world, because they are even larger when one looks only at patterns within these regions.

As a further test, I estimated the impact of performance pay for only the 21 participating European countries. Once again, the results showed even larger positive effects than those obtained for the full sample.

Other Sensitivity Tests

When findings are based on small samples, it is important to ascertain whether a conclusion is sensitive to the particular analysis being conducted. Even after conducting a preferred analysis that maximizes use of the information available and best conforms to underlying economic theory, it is important to make sure that the pattern that one has identified is not a statistical accident that readily disappears if a slightly different analysis is conducted. For this reason, I performed a variety of sensitivity tests for math achievement because the reliability of the math test across countries and cultures is usually considered higher than it is for reading or science. Remarkably, the relationship between performance pay and math achievement remained essentially unchanged, regardless of the sensitivity test that I ran.

My first sensitivity check focused on cultural differences among countries that were not captured by the continental fixed effects analysis. In this sensitivity check, I excluded two countries, Mexico and Turkey, which have particularly low levels of GDP per capita. Since it is known that the level of GDP is strongly correlated with educational performance, it may be that the inclusion of these two countries is producing misleading results. But dropping these countries hardly affects results.

The second sensitivity test excluded the level of educational attainment of the teachers, on the grounds that teacher quality might itself be affected by a country’s performance pay policies and therefore should not be used as a control variable. Excluding this variable did not materially change the results from those reported in Figure 1. In a third series of sensitivity tests, I excluded from the analysis one country at a time to make sure that the situation in no one country was driving the overall pattern of results. I found no evidence that that was happening.

The incidence of performance pay is, to some extent, clustered in two regions: Scandinavia (Denmark, Finland, Norway, and Sweden) and Eastern Europe (Czech Republic and Hungary). In a fourth set of sensitivity tests, I separately excluded the countries from these two regions from the analysis to see whether results were highly dependent on one or the other cluster. The results remained unchanged, indicating that neither of these regional clusters is solely responsible for the main result.

A fifth set of sensitivity tests was possible because I have information on other policies that lead to differential pay among teachers. Salaries may vary depending on 1) the teaching conditions and responsibilities (such as taking on management responsibilities, teaching additional classes, and teaching in particular areas or subjects), 2) teacher qualifications and training, or 3) a teacher’s family status and/or age. Since it is possible that student achievement is higher whenever pay schedules are flexible, regardless of the connection to teacher classroom effectiveness, I estimated the impact of each of these three sets of factors on math achievement. None showed a significant impact on performance, and the effect of performance pay remained large and significant, even when these other possible salary adjustments were included in the analysis.

In sum, the main results shown in Figure 1 survive a wide variety of sensitivity tests. That the results are robust to multiple model specifications provides strong evidence that performance pay helps to explain the variation in student performance on the PISA tests.

Differential Effects

With one exception (immigrants benefited less than native-born students from a performance pay regime), I found only small differences in the impact of performance pay on the math achievement of subgroups in the population. Since important differential effects were identified for only one subgroup, one cannot infer that the impact of performance pay on student math learning is concentrated on any particular group of students.

I did, however, find a surprising difference in the way in which a teacher’s education background affects math learning, depending on the presence of a pay-for-performance system. In countries with performance pay, teachers who have an advanced degree in pedagogy do not outperform those without such a degree (the only measure of a teacher’s education available in the PISA data base). However, in countries without performance pay, students learn more in math if they have a pedagogically trained teacher. Perhaps an incentive system washes out any differences that may be caused by variations in teacher training.

Conclusions

The analysis presented above represents the first evidence that, all other observable things equal, students in countries with teacher performance pay plans perform at a higher level in math, reading, and science. The differences in performance are large, ranging from 15 percent (in science) to 25 percent (in math and reading) of a standard deviation. Since one-quarter of a standard deviation is roughly a year’s worth of learning, it might reasonably be concluded that by the age of 15, students taught under a policy regime that includes a performance pay plan will learn an additional year of math and reading and over half a year more in science. However, this conclusion depends on the many assumptions underlying an analysis based on observational data.

Although these are impressive results, before drawing strong policy conclusions it is important to confirm the results through experimental or quasi-experimental studies carried out in advanced industrialized countries. Nothing in the PISA data allows us to identify crucial aspects of performance pay schemes, such as the way in which teacher performance is measured, the size of the incremental earnings received by higher-performing teachers, or very much about the level of government at which or the manner in which decisions on merit pay are made. Studies of such matters are probably better performed within countries, taking advantage of variation in policies within those countries. The study design also does not allow one to tease out the relative importance of the incentive to existing teachers of a performance pay plan as compared to the changes that may take place in teacher recruitment when compensation depends in part on merit rather than just on a standardized pay schedule. Since much more work needs to be done on all of these questions, a wit might insist that performance pay apply to scholars as well.

Ludger Woessmann is professor of economics at the University of Munich and head of the department of Human Capital and Innovation at the Ifo Institute for Economic Research.

An unabridged version of this article is available here.

Merit pay proponents argue that monetary incentives for better teaching can improve the quality of instruction in our nation’s classrooms. Yet only a handful of studies have evaluated the impact of teacher merit pay on student achievement. These studies offer no conclusive recommendations regarding the optimal role of merit pay in U.S. school systems, leaving policymakers largely dependent on studies on other countries for information about how best to implement merit pay programs.

Recently, the New York City Department of Education (DOE) conducted a policy experiment to test whether merit pay given to all teachers at an effective school could increase student achievement. The city’s School-Wide Performance Bonus Program, launched in 2007 and endorsed by both the DOE and the teachers union, was implemented in a randomly selected subset of the city’s most disadvantaged schools. The randomized design of school selection makes it possible to separate out the causal effect of this form of merit pay from myriad other influences on student learning.

Our analysis is based on data from the first two years of the bonus program. In interpreting our findings, it is important to appreciate the key features of the program’s structure. Teachers received bonuses based on the overall performance of all tested students in their school, rather than just on the performance of students in their own classrooms. According to proponents of group incentives, this design can minimize conflicts and foster a spirit of cooperation among teachers at participating schools. However, under group incentive schemes, individual teachers may not have sufficient motivation to improve their own performance if they know that their success in attaining a bonus depends heavily on the efforts made by other teachers. Especially in schools with a large number of teachers, it may be difficult to sustain a school-wide push to mobilize the efforts of most teachers. The New York City bonus program thus provides valuable information on the effects of a school-wide bonus plan.

Other specific characteristics of the bonus plan and the New York City context may also have influenced its effectiveness. If a school won a bonus, money was distributed among teachers and other school personnel by a committee consisting of two administrators and two teachers union representatives at the school. The bonus program was implemented alongside a new citywide accountability system that provided strong incentives to improve student achievement, regardless of whether a school was participating in the bonus program. Also, over the period we examine, all schools experienced increases in student achievement on the New York state test, leading some to suggest that the exam had grown easier (or at least easier to teach to). Roughly 90 percent of participating schools received a bonus in the second year of the program.

Did the group bonus program operating in this policy environment have an impact on student achievement? We find very little effect overall, positive or negative. There is some evidence, however, that the program had a positive impact in schools where teachers were few in number, an environment in which it may be easier for teachers to cooperate in pursuit of a common reward. This study leaves open the question of whether a bonus program that rewards teachers for their own specific effectiveness would be more successful.

The Program

In November 2007, the New York City DOE launched the School-Wide Performance Bonus Program, randomly selecting 181 schools serving kindergarten through 8th grade to participate from a group of 309 high-need schools. (Disadvantaged high schools were also randomly selected into the program; we focus only on elementary and middle schools since these are the grades for which we can measure math and reading achievement.) The remaining 128 schools that were not selected serve as the control group for the purposes of our evaluation. The 309 schools included in the study differed from other city schools in the following ways: They had a higher proportion of English Language Learners (ELL), special education, minority students, and students eligible for the Title I free or reduced-price lunch program, as well as lower average math and reading scores. Teachers in these schools had slightly less experience and slightly more absences than teachers in other schools. The schools were smaller and had fewer teaching staff than other New York City schools.

The bonus program was the product of lengthy negotiations between district administrators and the teachers union. As a result of these negotiations, schools had to gain the support of 55 percent of their full-time United Federation of Teachers (UFT) staff each year in order to participate. Out of the 181 schools selected for the program, 25 schools voted not to participate in the first year of implementation or withdrew from the program following an initial vote of approval, and three more schools pulled out before the second year. Additionally, at the discretion of the DOE, two schools initially assigned to the treatment group were moved to the control group, and four schools initially designated as control schools were moved to the treatment group and subsequently voted to participate in the program. Of course, the schools that elected not to take part in the program and those moved by the DOE may differ in important ways from schools that chose to participate. We therefore consider the treatment group to include all 181 schools originally deemed eligible for bonus payments and take into account the fact that not all of them were actually participating in the program when interpreting our results.

Schools that implemented the program could earn a lump-sum bonus for meeting school-wide goals. These goals were tied to the New York City accountability system and were mainly determined by student performance on state math and reading exams. Under this accountability system, schools receive scores and grades that summarize their overall performance on three sets of measures: school environment, student performance, and student progress. The school environment measure incorporates student attendance and the results from surveys of parents, teachers, and students. Student performance measures include average student achievement on reading and math exams, along with median proficiency and the percentage of students achieving proficiency. The student progress measure considers the average change in test scores from year to year and the percentage of students who made progress from one year to the next. The accountability system also gives “extra credit” for exemplary progress among high-need students. Schools received target scores based on their accountability grades, and schools with lower accountability grades needed to make larger improvements to reach their targets.

Schools participating in the bonus program received awards based on their progress toward meeting target scores. Schools that achieved their goals received bonuses equal to $3,000 per union teacher. Schools that fell short but manage to meet 75 percent of their goal received $1,500 per union teacher. Schools that did not achieve their target faced no consequences from the bonus program beyond the absence of incentive pay. For a sense of the strength of the incentive provided by the bonuses, the full $3,000 award represents a 7 percent increase in the salary of teachers at the bottom of the pay scale and a 3 percent increase for the most experienced teachers. In other words, these bonuses provided a substantial monetary benefit to most recipients.

Each participating school was required to develop a plan for distributing any lump-sum bonus awarded to the school. In the first year of the program, plans had to be submitted to the DOE after students took the state math and reading exams but before exam results were released and, thus, before schools knew whether they would receive a bonus. In every school, a four-member compensation committee, consisting of the principal, a second administrator, and two teachers elected by the school’s UFT members, determined how bonuses would be distributed. The DOE program guidelines placed only two restrictions on the schools’ bonus distribution plans: all union teachers had to receive a portion of the bonus payment and bonuses could not be distributed based on seniority. Otherwise, the committees had full discretion over bonus amounts and over whether other school employees would also receive funds. About half of the school committees chose to divide the award roughly equally among all recipients. In these schools, the difference between the highest and lowest bonus payment was less than $100. In the rest of the schools, the difference between the highest and the lowest bonus ranged from a low of $200 to a high of $5,000.

Of the 158 schools that voted to participate in the first year of the program, 87 (55 percent) received bonus payments. The bonus pool totaled $14.0 million in the first year and averaged $160,500 per school. In the second year of the program, the 2008–09 school year, 139 participating schools (91 percent) earned bonus awards, averaging $195,100 per school and totaling $27.1 million.

Little Difference for Students

Before we get to the detailed findings of our study, it is important to make clear the nature of the incentives NYC teachers and administrators faced over the period we examine. First, the 2007–08 school year was the first year of both the bonus program and a new citywide accountability system. The accountability system provided strong incentives to improve student achievement, regardless of whether a school was participating in the bonus program. For example, schools that earned A or B accountability grades were eligible for principal bonuses and additional funds when students transferred from schools receiving a poor grade. Schools that received D and F grades faced potential consequences, including principal removal and school closure. With this in mind, we see the results of our study as representing the effect of group-based teacher merit pay for schools that are already under accountability pressure. However, given that all school districts in the United States are subject to No Child Left Behind and many states have implemented their own accountability systems, this may be the most appropriate context in which to study the consequences of merit pay.

The second thing to keep in mind is that the power of the bonus program incentives was likely muted in the first year because of the timing of the program announcement. Eligible schools were notified in November of 2007, leaving relatively little time for teachers and administrators to alter their educational plans before accountability exams were administered in January for reading and March for math. As noted above, the percentage of schools that hit their achievement targets increased between the first, truncated year of the program and the second, when schools had more time to respond to the program incentives. But we caution readers to remember that this leap in bonus payouts is not, by itself, evidence that merit pay worked. It may instead reflect citywide performance improvements or, more pessimistically, that the New York state tests decreased in difficulty over this period. The most important comparison to make is between the treatment group schools eligible for the bonus program (most of which actually participated in the program) and the schools in the control group. Treatment-group schools need to at least outpace their counterparts in the control group over these two years for us to say that merit pay made a real difference for student achievement. It is this comparison that is at the heart of our analysis.

How did bonus program schools fare compared to schools in the control group? Both groups of schools saw an increase in the average math and reading scores during the first two years of the bonus program; treatment-group schools, however, did not experience a statistically significant improvement in average test scores relative to the schools in the control group. Nor did these results change notably when we 1) made adjustments for the small differences in treatment and control school characteristics that existed despite randomization between treatment-group and control-group schools, or 2) took into account whether treatment-group schools elected to participate in the bonus program. It is possible, of course, that looking at average student achievement could divert our attention from changes for particular groups of students. Were teachers, we wanted to know, focusing their attention on either high-achieving or low-achieving students in an effort to meet target scores? We used statistical techniques similar to the one we employed to examine changes in average scores to assess the effect of the bonus program on the percentage of students achieving proficiency on math and reading exams. Once again, we found no evidence that the bonus program led to changes in this measure of student achievement. Participation in the bonus program did not, for example, boost the percentage of students who scored at or above the level designated as “proficient” under New York state accountability standards. Bonus-program schools fared no better than schools in the control group, and in the second year of the program, treatment schools experienced a statistically significant, although quite small, decrease in math proficiency.

On a related note, the New York City accountability system and, as a byproduct, the bonus program, contain incentives to focus on particular groups of students, since improvements for some student groups matter more in the calculations of a school’s accountability grade. In addition to calculating overall achievement for all students in a school, components of the New York City accountability system take into account changes in the achievement of students who were in the lowest third of their grade in the prior year, those on the cusp of proficiency, and those close to the school’s median score, along with students who are designated as ELL and students who are enrolled in special education programs. Again, we found no evidence to suggest that the bonus program led to achievement gains for any of these groups of students. On average, students in these groups fared just as well whether they attended a school that was participating in the bonus program or one in the control group.

Limitations of Group Bonuses

Does evidence that the New York City bonus program did not lead to marked gains in student achievement, at least in the program’s first two years, mean that merit pay for teachers in general does not work? That is certainly one possible conclusion to be drawn from our findings. Another possibility is that this particular type of merit pay program, where bonuses are based on school-wide performance and teachers expect to receive bonus payments regardless of their effort, does not work in all schools. Group bonuses may weaken the incentives for individual teachers to increase effort devoted to raising student achievement to the point that the programs become ineffective. And perhaps this problem would be mitigated in programs in which rewards are more tightly coupled to the effort an individual makes in the classroom.

Think about two schools, one with many more teachers than the other, both participating in a school-wide merit pay program. In each school, the impact of an individual teacher’s effort on the expected bonus is determined by the number of other teachers with tested students, since bonus receipt is primarily based on student performance on math and reading exams. Because of this, a very good teacher with a large number of teaching colleagues can do less to raise school-wide student performance than a teacher of the same quality in a school with fewer teachers. In the school with more teachers, the diffusion of responsibility for test-score gains across many teachers may erode the incentive that any individual teacher has to increase effort in the classroom. Some teachers may conclude that exerting additional effort will produce little difference in the overall performance of the school. The central idea here is that teachers could face relatively strong or weak incentives under the same merit pay program as a result of the number of teachers at their school. With this logic in mind, we examined the effect of the New York City school-wide merit pay program at schools with different numbers of teachers with test-taking students. Did schools with fewer teachers show signs that teachers were responding to merit pay incentives?

We conducted a statistical analysis similar to our method for estimating the average effect of the bonus program across all New York City schools in the experiment. But this time, we looked for different effects on math scores in schools with more and fewer math teachers and different effects on reading scores on schools with larger and smaller cohorts of reading teachers.

It turns out that the effectiveness of school-wide bonus programs may, in fact, depend on the number of teachers with tested students in a school (see Figure 1). For schools in the bottom quartile of the number of teachers with tested students, that is, schools with approximately 10 or fewer such teachers in elementary and K–8 schools and five or fewer in middle schools, school-wide merit pay did lead to improved student achievement. We estimate that the New York City bonus program had a positive effect on student math achievement in these schools in both program years, although the estimated effect in the second year fell just short of conventional levels of statistical significance. Conversely, this analysis also indicates that the program may have slightly lowered student achievement in schools with larger teaching staffs. Math achievement gains attributable to the bonus program in schools with smaller teaching staffs were modest in size but meaningful. In the first year of the program, the bonus program boost to math scores was, by our estimates, 3.2 points on the New York state test, or 0.08 student-level standard deviations. To benchmark this effect against the magnitude of other familiar results, it is slightly smaller than the estimated 0.1 standard deviation gain in achievement that results from being assigned to a teacher at the 85th percentile of the effectiveness distribution rather than a teacher at the median.

The Devil in the Details

The New York City bonus-pay program provides us with a valuable opportunity to study the effect of merit pay for teachers in an experimental setting. We are a long way from amassing a convincing body of research on either side of the debate over merit pay in education, but what this experiment makes frustratingly clear for merit pay proponents is that the structure of the payment scheme can make a large difference. For merit pay to improve student outcomes, teachers must face strong incentives to improve their performance. Our study indicates that school-wide bonus programs may be able to provide those incentives in schools with relatively small teaching staffs. They may also be appropriate for schools characterized by a high degree of staff cohesion, in which teachers work collaboratively to improve student learning and it is difficult to isolate the performance of a single teacher. The early experience with the New York City School-Wide Performance Bonus Program suggests, however, that a heavy reliance on school-wide rewards may hamper the effectiveness of merit pay programs in schools with large teaching staffs that are not highly collaborative.

Sarena Goodman and Lesley Turner are PhD candidates in Columbia University’s Department of Economics. The randomization of schools participating in the School-Wide Performance Bonus Program was designed and conducted by Harvard University economist Roland Fryer.

An unabridged version of this article is available here.

An unabridged version of this article is available here.

Programs that enable students to attend private schools, including both vouchers and scholarships funded with tax credits, have become increasingly common in recent years. This study examines the impact of the nation’s largest private school scholarship program on the performance of students who remain in the public schools. The Florida Tax Credit Scholarship Program (FTC) was signed into law in 2001 and opened to students from low-income families in the 2002–03 school year. FTC provides corporations with tax credits for donations they make to scholarship funding organizations, the nonprofits that determine student eligibility for the program and issue scholarships. Corporations can receive dollar-for-dollar tax credits for up to 75 percent of their total state tax obligation each year.

Although little noticed, tax credit scholarship programs now send many more low-income students to private schools than do traditional school voucher programs. More than 104,000 students nationwide attended private schools through tax credit programs in the 2008–09 school year, while only 60,000 students used private school vouchers. Scholarship programs similar to Florida’s now operate in several states, including Arizona, Pennsylvania, Rhode Island, and Indiana, and are being considered in Maryland and New Jersey. The Florida program is set to expand dramatically in the coming years (see sidebar), making evidence of its consequences all the more timely.

One popular argument for expanding private school choice is that public schools will improve their own performance when faced with competition for students. Because state school funding is tied to student enrollment, losing students to private schools means losing revenue. The threat of losing students to private schools may give schools greater incentive to cultivate parental satisfaction by operating more efficiently and improving the outcomes valued by students and parents. Alternatively, private school vouchers and scholarships may have unintended negative effects on public schools: they may draw away the most involved families from public schools, community monitoring of those schools may diminish, and schools may reduce the effort they put into educating students.

It is notoriously difficult to gauge the competitive effects of private schools on public school performance. Private schools may be disproportionately located in communities with low-quality public schools, causing the relationship between private school competition and public school performance to appear weaker than it actually is. If, however, private schools are located in areas where citizens care a lot about educational quality, the relationship will appear stronger than it truly is.

This study takes advantage of the introduction of the FTC to provide new evidence on the effects of increased competition on student achievement in public schools. Before the program began in 2002, roughly 11 percent of students in Florida were enrolled in private schools. FTC targeted students from low-income families, only 5 percent of whom had been attending private schools.

We examine whether students in schools that face a greater threat of losing students to private schools as a result of the introduction of tax-credit funded scholarships improve their test scores more than do students in schools that face less-pronounced threats. We find that they do, and that this improvement occurs before any students have actually used a scholarship to switch schools. In other words, it occurs from the threat of competition alone.

Florida Tax Credit Scholarship Program

In order to be eligible for an FTC scholarship, students must meet the income guidelines (until recently, family incomes below 185 percent of the federal poverty line for new applicants) and either must have attended a Florida public school for the full school year before program entry or be entering kindergarten or first grade. With the exception of these early-grade private school students, students already attending private schools in Florida are not eligible for first-time scholarships. Students who enter a private school on a scholarship are eligible to retain their scholarships in future years, so long as their family income remains within the stated limits (until recent changes, below 200 percent of the federal poverty line).

Scholarships need not cover all of the costs of attending private schools, and parents are free to send their children to any private school regardless of the share of tuition and fees the scholarship covers. The scholarship is quite generous; it covers approximately 90 percent of tuition and fees at a typical religious elementary school in Florida and two-thirds of tuition and fees at a typical religious high school. As a result, the program greatly increased the accessibility of private schools to low-income families. In the first year, some 15,585 scholarships were awarded, increasing the number of low-income students attending private schools by more than 50 percent. For the 2009–10 school year, the FTC program awarded scholarships to 28,927 students (see Figure 1).

Data

Our analysis draws on several data sources. The Florida Department of Education’s Education Data Warehouse provides test scores from the Florida Comprehensive Achievement Test (FCAT) and demographic characteristics for all students in the public schools. We use test-score data from the 1999–2000 school year through 2006–07. Students classified as learning disabled were excluded from the analysis, as they are eligible for a more generous voucher through the McKay Scholarship Program, and the FTC program should therefore have had no effect on schools’ efforts to retain these students (see “The Case for Special Education Vouchers,” features, Winter 2010).

The Florida Department of Education publishes public and private school addresses, including latitude and longitude information for the public schools. The address information was geocoded to generate measures of the pressure that public schools face from private competitors. We first limited our attention to the 92 percent of public school students in the state attending schools with a private competitor within a five-mile radius. Because it is not obvious how best to gauge the amount of competition faced by the remaining schools, we constructed four different measures:

• Distance: the crow’s-flight distance between the physical addresses of each public school and the nearest private competitor. A private school qualifies as a competitor to a public school if it serves any of the grades taught in that public school.

• Density: the number of private competitors within a five-mile radius of the public school.

• Diversity: the number of different types of private schools within a five-mile radius of the public school. To generate this measure, we first identified 10 distinct types of private schools, defined by their stated religious (or secular) affiliation.

• Concentration: an index of how varied the private school competitors are for a given public school, based on the counts of different types of schools within a five-mile radius.

The distance and density measures gauge whether easier access to a private school of any type increased the competitive pressure on public schools when the new policy lowered the effective cost of attending private school for eligible students. The diversity and concentration measures capture the variety of options available to students; public schools in areas with more varied options should feel more competitive pressure in the wake of the policy change.

Methods

In order to determine the effect of scholarship-induced private school competition on public school performance, we examine whether students in schools that face a greater threat of losing students to private schools as a result of the introduction of tax-credit funded scholarships improve their test scores more than do students in schools that face a less-pronounced threat. Specifically, we look to see whether test scores showed greater improvement in the wake of the new policy for students attending public schools with more (or more varied) nearby private options that suddenly became more affordable for low-income students than did scores for students attending schools with fewer (or less varied) potential competitors.

This analysis is possible because of the considerable variation in potential competition faced by schools across the state of Florida. Prior to the introduction of the program, some communities in Florida had a much richer and more diverse set of private school options than did other communities. The overall share of low-income students attending private schools ranged from 1.4 percent in Punta Gorda to 7.9 percent in the Melbourne-Titusville-Cocoa-Palm Bay area. More importantly, by our measures, the amount of competition that specific public schools faced on the eve of the program also varied widely. Our density measure, for example, ranges from one private school within five miles to 60. The average Florida public school had roughly 14 private schools nearby, but more than 30 percent of schools had fewer than 2 or more than 30.

We isolate the effect of competition introduced by the program by comparing the performance of each school’s students to the performance of its students the year before the program was enacted. When making these comparisons, we take into account student characteristics that are associated with test scores, including gender and race/ethnicity, English-language-learner status, and eligibility for free or reduced-price lunch.

We also control for some characteristics of schools that could affect the degree of competitive pressure. Most importantly, we control for the letter grades that schools received from the state’s accountability system; schools with lower grades may feel particular pressure to increase their scores to avoid accountability sanctions, independent of the effects of the FTC program. We also control for the percentage of the school’s student body that was eligible for free or reduced-price lunch, as only these students were eligible for FTC scholarships.

There are three main ways in which a program that expands access to private schools could affect public school performance. Public schools could react to private school competition by altering their policies, practices, or effort; this is the direct competitive effect. Such a program could also affect public schools by changing the mix of students who attend them. A third possibility is that, so long as only a few students leave a public school with scholarships, the program could have effects on resources. Resource effects could be either negative (as total state aid decreases with the loss of students), or positive (as per-pupil resources might actually increase following small losses of students, due to the indivisibility of classroom teachers). We can eliminate the possibility of student-body composition and resource effects by concentrating solely on the FTC program’s effects during the 2001–02 school year, after the program’s announcement but before students could actually leave the public schools with a scholarship. During this academic year, students in the public schools were applying for private school scholarships for the following year.

Results

We find that all four measures of competition (distance, density, diversity, and concentration) are positively related to student performance on state math and reading tests. (Because we obtained similar results looking at performance in each subject separately, we focus our discussion on the average score across both subjects.) Each of our competition measures uses different units. We therefore report the estimated effects of a one standard deviation increase in the amount of competition faced by a given public school by each measure.

For every 1.1 miles closer to the nearest private school, public school math and reading performance increases by 1.5 percent of a standard deviation in the first year following the announcement of the scholarship program. Likewise, having 12 additional private schools nearby boosts public school test scores by almost 3 percent of a standard deviation. The presence of two additional types of private schools nearby raises test scores by about 2 percent of a standard deviation. Finally, an increase of one standard deviation in the concentration of private schools nearby is associated with an increase of about 1 percent of a standard deviation in test scores.

Although these effects are relatively small, they consistently indicate a positive relationship between private school competition and student performance in the public schools, even before any students leave for the private sector. That is, these results provide evidence that public schools responded to the increased threat of losing students to the private schools. The fact that we obtain quite similar results regardless of the specific measure used makes us confident that the findings are not driven by other factors that might distinguish public schools facing more or less competition based on a given measure. Indeed, in ongoing work we have also considered measures of competition based on the number of available slots in nearby private schools and on the number of nearby churches, and again find very similar results.

Moreover, it is important to recognize that the results reported above represent lower-bound estimates of the effects of competition on public school performance. They are based only on comparisons of schools with different levels of competition. If all public schools improved their performance in response to the scholarship program, this improvement would not be detected by our analysis.

One might expect that some public schools have a greater incentive to respond to potential competition associated with the availability of private school scholarships for low-income students than others. We consider two major reasons schools may face different incentives to react to competitive pressure. First, elementary and middle schools may have more of an incentive to respond to competitive pressure than high schools because the scholarships cover a greater share of private school tuition and fees in the early grades than they do in the high school years. Although the differences in the share covered might not matter for higher-income families, for many low-income families the difference in out-of-pocket expenses between an elementary or middle school and a high school is likely to be significant. Knowing this, public high schools might not react as strongly to competition from a private school scholarship program as would public elementary and middle schools. Consistent with this hypothesis, we find that the effect of competition is more than twice as large for elementary and middle schools as it is for high schools (see Figure 2).

Second, public schools that stand to lose the largest amounts of revenue if many of their scholarship-eligible students leave may be more responsive than those schools less likely to lose large amounts of revenue. All public schools may experience resource effects as a consequence of losing students to private schools through a scholarship program. However, those that are on the margin of receiving federal Title I aid have the largest incentive to retain students from low-income families. These federal resources, which average more than $500 per pupil, are directed to school districts, which then allocate them to the elementary and middle schools that low-income students attend. We find that public schools that are likely to receive Title I aid in the next year if they retain their low-income students, but not if they don’t, tend to improve disproportionately in the year following the program announcement, whereas schools whose Title I aid is unlikely to change respond much less noticeably or not at all.

We also investigate whether the estimated effects of the scholarship program persist in later years. After the first year of the analysis, resource and composition effects may occur as students who receive scholarships leave the public schools for private schools. We find that the effects of the voucher program grow stronger over time (see Figure 3), resulting perhaps from increased knowledge of the program, which might contribute to greater competitive pressure, or from the advent of composition and resource effects. While it is difficult to disentangle the reasons for this strengthening over time of the program’s estimated effects, these findings nonetheless suggest that our first-year results may understate the positive effect of the FTC program on public school performance

Conclusion

Our results indicate that the increased competitive pressure public schools faced following the introduction of Florida’s Tax Credit Scholarship Program led to general improvements in their performance. Both expanded access to private school options and greater variety of options that students have in terms of the religious (or secular) affiliations of private schools are positively associated with public-school students’ test scores following the introduction of the FTC program. The gains occur immediately, before any students leave the public schools with a scholarship, implying that competitive threats are responsible for at least some of the estimated effects. And the gains appear to be much more pronounced in the schools most at risk to lose students (elementary and middle schools, where the cost of private school attendance with a scholarship is much lower) and in the schools that are on the margin of Title I funding.

To be sure, our study has several limitations. First, our measures of competition reflect the state of the private school market in 2001, before private schools had a chance to respond to the FTC scholarship program. Although that ensures that the competition measure is not itself affected by postpolicy test scores, it does give a less accurate view of the competitive pressures faced by schools in subsequent years.

Second, our study is based on data from a single state. It is possible that the dynamics between competitive pressures and public-school students’ test scores are systematically different in Florida than they are in the rest of the nation. In particular, more than 90 percent of Florida’s students live in the state’s top 20 most populous metropolitan areas. In states with a greater share of the population in rural areas, the effects of a scholarship program may not exert the same degree of competitive pressure on public schools. It may also be the case that Florida’s diverse range of private school options and accompanying greater competition among the private schools limit the study’s generalizability. Nonetheless, our results indicate that private school competition, brought about by the creation of scholarships for students from low-income families, is likely to have positive effects on the performance of traditional public schools.

David Figlio is professor of education, social policy and economics at Northwestern University and research
associate at the National Bureau of Economic Research. Cassandra M.D. Hart is a doctoral student in the
Department of Human Development and Social Policy at Northwestern University.

An unabridged version of this article is available here.

Middle school. The very words are enough to make many Americans shudder with memories of social anxiety, peer pressure, bad haircuts, and acne. But could middle schools also be bad for student learning? Could something as simple as changing the grade configuration of schools improve academic outcomes? That’s what some educators have come to believe.

States and school districts across the country are reevaluating the practice of educating young adolescents in stand-alone middle schools, which typically span grades 6 through 8 or 5 through 8, rather than keeping them in K–8 schools. The middle-school model began to be widely adopted almost 40 years ago. Now, reformers in Massachusetts, Pennsylvania, Ohio, Tennessee, Oklahoma, Maryland, and New York, and the large urban districts of Cincinnati, Cleveland, Philadelphia, and Baltimore, are challenging the notion that grouping students in the middle grades in their own school buildings is the right approach.

Why the turn against middle schools? For more than three decades, American public education embraced this organizational model. Between 1970 and 2000, the number of public middle schools in the U.S. grew more than sevenfold, from just over 1,500 to 11,500. These new middle schools displaced both traditional K–8 primary schools and junior high schools (which first appeared a century  ago and served grades 7–8 or 7–9). From 1987 to 2007, the percentage of public-school 6th graders in K–6 schools fell from roughly 45 percent to 20 percent.

Neither the middle school nor the junior high has ever been popular among private schools, which educated only 2 percent of their 6th and 7th graders in these types of schools in 2007. And maybe the private schools have had it right all along. For the last two decades, education researchers and developmental psychologists have been documenting changes in attitudes and motivation as children enter adolescence, changes that some hypothesize are exacerbated by middle-school curricula and practices.

These findings are cause for concern, but there is reason to doubt their conclusions. Because the studies use data from a single school year to contrast students in middle schools and K–8 schools, most of the available research cannot reject the possibility that differences between the groups of students, rather than in the grade configuration of their schools, are actually responsible for the differences in behavior and achievement.

To provide more rigorous evidence on the effect of middle schools on student achievement, we turned to a richly detailed administrative dataset from New York City that allowed us to follow students from grade 3 through grade 8. Some of these children attended middle schools and some did not. Because we could follow the same children over a period of time, we could do a better job of ruling out the role of influences other than middle-school attendance on educational outcomes.

What we found bolsters the case for middle-school reform: in the specific year when students move to a middle school (or to a junior high), their academic achievement, as measured by standardized tests, falls substantially in both math and English relative to that of their counterparts who continue to attend a K–8 elementary school. What’s more, their achievement continues to decline throughout middle school. This negative effect persists at least through 8th grade, the highest grade for which we could obtain test scores.

We found that the middle-school achievement gap cannot be explained by a scarcity of financial resources for the schools. Instead, the cause is more likely to be related to other school characteristics, especially the fact that middle schools in New York City educate far more students in each grade. Although our conclusions about the reasons for the middle-school gap are tentative, we are quite confident that the evidence shows that middle schools are not the best way to educate students—at least in places like New York City.

Data and Methods

Our study was based on data for New York City school children who were in grades 3 though 8 during the 1998–99 through 2007–08 school years. We were able to follow students who entered 3rd grade between the fall of 1998 and the fall of 2002 for six years, until most had completed the 8th grade. We have data about the grade configuration and other characteristics of their schools, individual academic achievement as measured by annual standardized test scores in math and English, and a variety of personal characteristics. In particular, we know each student’s gender, ethnicity, whether they received free or reduced-price lunch through the federal lunch program, whether they were English language learners or received special education services, and their record of suspensions and absences from school.

Elementary schools in New York City typically serve students until grade 5 or grade 6, while a smaller portion of elementary schools run through grade 8. This means that most students move to a middle school in either grade 6 or grade 7, while some never move to a middle school. Of the 3rd graders in our initial sample of students, 62 percent were in a K–5 school, 24 percent were in a K–6 school, and 7 percent were enrolled in a K–8 school. The small fraction of remaining students attended K–3, K–4, or K–7 schools and are excluded from our analysis.

To isolate the impact of attending a middle school from the many other factors that influence student achievement, we combined two basic strategies. Most importantly, we tracked the performance of individual students over time to see how their performance evolved relative to that of their peers as they progressed from grades 3 to 8, in essence, using each student as his or her own control group. This step alone provides much stronger grounds for conclusions about the effects of attending a middle school than previous research.

A lingering concern, however, is the possibility that different types of students choose to attend middle schools than choose to continue in a K–8 school. If students do sort themselves into middle schools because of some unobserved characteristic that causes changes in academic achievement over time, we would incorrectly attribute differences in achievement to the middle schools instead of to characteristics of the students themselves. We reduced the likelihood of making this mistake by using a statistical technique that effectively takes the choice to switch schools out of the students’ (or parents’) hands. Specifically, we ran a statistical model that used the last grade served by the school that a student attended in grade 3 to predict whether the student attended a middle school. We then used that prediction to place each student into one of the two groups we are comparing, that is, students who attend middle schools and those who do not. Our key assumption in taking this approach is that there are no unobservable factors that cause a drop in student achievement at precisely the same time as students must leave the elementary schools they attended in grade 3. While we cannot definitively rule out the existence of such factors, we do not know of any plausible alternatives that would explain our findings.

The Middle-School Disadvantage

What determines a student’s level of academic achievement is complex. But the simple fact is that students who enter public middle schools in New York City fall behind their peers in K–8 schools.  This is true both for math and English achievement. Even more troubling, the middle-school disadvantage grows larger over the course of the middle-school years. With the transition into a middle school, students set out on a trajectory of lower achievement gains.

The achievement gap between middle-school students and K–8 students is put in stark relief in Figure 1, which displays our estimates of the impact of attending a middle school on student achievement as measured by standardized tests in math and English Language Arts. The graphs show how well students who attend a middle school perform relative to how we would expect them to perform if they attended a K–8 school. We report those differences, in standard deviations of student achievement in math and reading, for the 3rd through 8th grades. We separate students who enter a middle school in grade 6 from those students who enter a year later, in grade 7.

No matter whether students enter a middle school in the 6th or the 7th grade, middle-school students experience, on average, a large initial drop in their test scores. Even after accounting for a host of other factors that influence student achievement, students who eventually attend middle schools go from scoring better than their counterparts in K–8 schools in the year prior to transitioning to middle school to scoring below where we would expect if they were not attending a middle school. Math achievement for 6th graders transitioning to middle school falls by 0.18 standard deviations, and English achievement falls by 0.16 standard deviations. Contrast that decline with the 6th-grade test scores for students who will enter middle school the following year, in the 7th grade. Their test scores in both subjects continue to improve relative to their peers in K–8 schools. When these 6th graders move to a middle school in the 7th grade, however, we see the same dramatic fall in academic achievement: math scores decline by 0.17 standard deviations and English achievement falls by 0.14 standard deviations. Just how large are these effects? Consider that decrease in achievement associated with middle school entry—between 0.14 and 0.18 standard deviations—is roughly 20 to 25 percent of the achievement gap between poor and non-poor students (as measured by free lunch receipt) in New York City (about 0.7 standard deviations).

Moreover, these are not temporary dips followed by rebounds in learning. Throughout the middle-school years, students fall further behind. After two years in a middle school, on average a student who entered in the 7th grade will score 0.10 standard deviations in math and 0.09 standard deviations in English below what we would expect if he had gone to a K–8 school. After three years in a middle school, a student who entered in the 6th grade will underperform on 8th-grade assessments by 0.17 standard deviations in math and by 0.14 standard deviations in English.

A particularly distressing finding from our study is that students with lower initial levels of academic achievement fare especially poorly in middle school. To investigate the possibility of different effects on students with higher and lower initial achievement levels, we separated students into two groups: one group had grade 3 test scores above the citywide median, the other group scored below the median. Although we found substantial drops in achievement during middle school for both groups of students, the first-year drop and cumulative deficit were, respectively, 50 percent and more than 200 percent greater for students who start at the lower end of the achievement distribution.

We also found evidence that student absence rates increased when students entered middle schools and were significantly higher in grade 8 than for students who never entered a middle school (see Figure 2). More specifically, our estimates indicate that students were missing almost two additional days of school per year than would have been the case had they attended a K–8 school. Thus, increased absences may be one mechanism through which middle schools lower student achievement. There is little chance, however, that absences could explain a large share of the overall effect of attending a middle school.

To be sure, the population of public school children in New York City is different from that of many other school districts around the country. These differences might mean that middle-school attendance would have smaller or larger effects on other students than we estimate it to have on New York City’s public school children. For example, students with fewer educational resources at home may be more strongly affected by changes in their school environment. If that is the case, studying New York City students, who arguably come from less advantaged backgrounds than, say, the students in New York City suburbs, may have led us to find a larger middle-school effect than had we followed a more-affluent student population. While we encourage readers to be cautious about applying our findings without qualification to all public schools, we also encourage school districts to support research that can identify middle-school effects in other settings, especially since we find the consequences of attending a middle school for student achievement to be substantial and troubling.

Explaining the Trouble with Middle Schools

Why might New York City’s middle schools be detrimental to academic achievement? We find little support for the notion that differences in resources, such as per-pupil expenditures and class size, could explain the middle-school achievement gap. In middle schools serving grades 6–8 and grades 7–8, average per-pupil expenditures were $10,094 and $11,082, respectively, while per-pupil expenditures in K–8 schools were roughly equivalent, at $10,950. Nor do students experience a large decline in per-pupil spending when they move to a middle school. Average per-pupil expenditure in K–5 schools was $10,144 (compared to the $10,094 for grade 6–8 middle schools) and $9,680 in K–6 schools (compared to $11,082 in grade 7–8 middle schools).

Nor can we attribute the disparity we see to differences in class size. The average class size is slightly smaller for 5th graders in K–5 schools than for 6th graders in 6–8 schools (24 vs. 25 students); students in K–8 schools see similar growth in class size between grades 5 and 6. Class size is actually larger for grade 6 students in K–6 schools than for grade 7 students in 7–8 schools (24 vs. 23 students).

What about the possibility that the relative age of students in a school, especially during adolescence, can influence how students learn? In other words, does being the youngest students in a school have negative effects on the educational experience of those students? We could not find evidence in our data to support this explanation for the initial drop in test scores upon transitioning to a middle school. In our study sample, about one-third of new 7th graders moved out of a school serving grades K–6 and entered a middle school for 7th and 8th graders, becoming the youngest cohort in the school, while roughly half of new 7th graders entered a grade 6–8 middle school as part of the school’s middle cohort of students. We find that the effect of entering a middle school was essentially the same for both of these groups.

At least part of the problem with middle schools may be that they usually combine students from multiple elementary schools. In the New York City schools we studied, the average cohort size was 75 students in K–8 schools, 100 students in K–5 and K–6 schools, and over 200 students in middle schools for grades 6–8 and 7–8 (see Figure 3). We went back to our data and analyzed the effect of these cohort size differences on test scores. What we found was that cohort size has a pronounced influence on student achievement during these school years. We estimate that an 8th grader who attends school with 200 other 8th-grade students will score 0.04 standard deviations lower in both math and English than he would if he attended a school with 75 other 8th graders, the average cohort size for a K–8 school. This 0.04 standard deviation deficit represents roughly one-quarter of the largest test-score declines we attribute to middle-school attendance.

Given the data we have, we can only speculate about why it is harder to educate middle school–aged students in large groups. Developmental psychologists have shown that adolescent children commonly exhibit traits such as negativity, low self-esteem, and an inability to judge the risks and consequences of their actions, which may make them especially difficult to educate in large groups. The combining of multiple elementary schools and their students also disrupts a student’s immediate peer group. And middle schools often serve a more diverse student population than many students encountered in elementary school. Yet while it seems plausible that these changes in environment would matter, we could not find any evidence in our data that any one hypothesis can explain the drop in learning among students moving to middle schools.

Even though a full explanation of the middle-school achievement gap eludes us, there does seem to be a consensus among New York City students and their parents that educational quality in the city’s public middle schools is lower than in the boroughs’ K–8 schools. We reached this conclusion after examining responses to a citywide survey of parents of children in grades K–8 and students in grades 6 and higher, which was conducted at the end of the 2006–07 and 2007–08 school years as a part of the city’s new school accountability system.

On average, New York City parents of students in middle schools gave their schools lower marks on measures related to education quality than parents whose children attend K–8 schools. Figure 4 shows that parent evaluations of school safety, academic rigor, and overall educational quality was much lower among those whose children attended middle schools than among parents with children in K–5, K–6, and K–8 schools. It is important to note that this is not simply a product of the challenges of educating adolescents. There is little perceptible decline in satisfaction among parents in K–8 schools as their children age, a consistency we would not expect if educational quality simply cannot withstand the onslaught of puberty.

The students’ opinions are consistent with their parents’ assessments, although the lack of data on students below grade 6 prohibits us from more direct measurement of the degradation of education quality in middle schools. The clearest pattern that emerges from student reports is that 6th and 7th graders in middle schools think their schools have less academic rigor, less mature social behavior among the students, are less safe, and provide lower-quality education than do 6th graders in K–6 or K–8 schools.

The Longer View

We don’t yet know whether the troubling slide in test scores for middle-school students persists through the end of high school, a question that is certainly worth studying. Unfortunately, our data do not allow us to follow the students in our study further than grade 8. If the decline does continue, middle schools not only hurt student achievement in the short term but set students up for unnecessary longer-term disadvantages.

Of course, it is possible that transitioning to high school could be more difficult for students who come from K–8 schools than for middle school students. If K–8 students experience a larger drop in achievement upon entering high school, that could bring the two groups of adolescents back into parity. But it is hard to recommend closing the middle-school achievement gap by bringing everybody down. The better option is to address the trouble with middle schools—or do away with them altogether.

Jonah E. Rockoff is associate professor of business at the Columbia Graduate School of Business. Benjamin B. Lockwood is research coordinator at the Paul Milstein Center for Real Estate at the Columbia Graduate School of Business.

An unabridged version of this article is available here.

Never before have Americans had greater access to information about school quality. Under the federal No Child Left Behind Act (NCLB), all school districts are required to distribute annual report cards detailing student achievement levels at each of their schools. Local newspapers frequently cover the release of state test results, emphasizing the relative standing of their community’s schools. Meanwhile, new organizations like GreatSchools and SchoolMatters aggregate this information and make it readily available to parents online.

But do all these performance data inform perceptions of school quality? Or do citizens base their evaluations instead on such indicators as the racial or class makeup of schools, regardless of their relationship with actual school performance?

In discussions of parental choice in education, researchers have frequently speculated that parents would base their evaluations of schools primarily on the characteristics of their student bodies. Columbia University professor Amy Stuart Wells, for example, concluded that the decisions of St. Louis parents participating in a voluntary desegregation program were based “on a perception that county is better than city and white is better than black, not on factual information about the schools.” And even if some parents base their decisions on educational quality, many observers worry that low-income and minority parents will be less informed about or interested in school quality, placing their children at a disadvantage in the education marketplace.

The evidence on these questions available to date comes from small-scale studies of specific school districts, making it difficult to reach general conclusions about the degree to which parents and the public at large are well informed about the performance of local schools. We are now able to supplement that research with data from a nationally representative survey of parents and other adults conducted in 2009 under the auspices of Education Next and the Program on Education Policy and Governance (PEPG) at Harvard University. Because we knew the addresses of respondents in advance of the survey, we were able to link individual respondents to specific public schools in their community and to obtain their subjective ratings of those schools. We also gathered publicly available data on student achievement in the same schools, making it possible to compare respondents’ subjective ratings to objective measures of school quality.

Our results indicate that citizens’ perceptions of the quality of their local schools do in fact reflect the schools’ performance as measured by student proficiency rates in core academic subjects. Although citizens also appear to take into account the share of a school’s students who are poor when evaluating its quality, those considerations do not overwhelm judgments based on information about academic achievement.

Public Perception and Objective Quality Measures

The 2009 Education Next–PEPG Survey was administered to a nationally representative sample of 3,251 American adults, including an oversample of 948 residents of the state of Florida. The Florida oversample was conducted in order to link perceptions of school quality to the unusually rich information about school performance available in that state. The survey was administered over the Internet by the polling firm Knowledge Networks in February and March of 2009. (For methodological details and complete survey results, see “The Persuadable Public,” features, Fall 2009.)

Before conducting the survey, we geo-coded the address of each respondent to latitude-longitude coordinates and a census block. We also obtained latitude-longitude coordinates for every U.S. public school from the National Center for Education Statistics. Using census blocks to place respondents within school districts, we then linked each respondent to the closest elementary, middle, and high schools (up to five schools of each type) operated by the local school district.

The survey asked all respondents this question: “Each of the following schools in your area serves elementary-school students. Which one, if any, do you consider your local elementary school?” It then offered each respondent a personalized list of the five closest elementary schools from which to pick; respondents were also allowed to specify a school that did not appear on the list. After a specific elementary school had been identified, the survey asked the respondent to grade this school on a scale from A to F. This same process was then repeated for middle and high schools.

We converted the A to F grades that respondents assigned to the schools into a standard grade-point-average (GPA) scale (A=4 and F=0). Of the elementary and middle schools our survey respondents rated, 41 percent received a B grade, while 36 percent received a C. In contrast, only 14 percent of schools received an A grade, 7 percent a D, and 2 percent an F. This distribution corresponds to an overall GPA of 2.57, or just below a B-minus average. Interestingly, respondents assigned their local middle schools grades that were, on average, one-quarter of a letter grade lower than the grades they assigned their local elementary schools (see Figure 1).

We measured actual school quality as the percentage of students in a school who achieved “proficiency” in math and reading on the state’s accountability exams (taking the average proficiency rate across the two subjects). School-level data on student proficiency were drawn from SchoolDataDirect.org for the 2007–08 school year, the most recent year for which test-score data would have been publicly available when the survey was conducted. Although the rigor of state content standards and definitions of math and reading proficiency vary widely (see “State Standards Rise in Reading, Fall in Math,” features), we are able to adjust for these differences by limiting our comparisons to respondents within the same state when examining the relationship between proficiency levels and school ratings.

To be sure, the percentage of students achieving proficiency in core academic subjects is an imperfect measure of quality, even when comparing schools in the same state. Given the strong influence of out-of-school factors on student achievement, any quality measure based on the level of student performance at a single point in time will be heavily influenced by characteristics of a school’s student body. At the same time, proficiency rates are the only quality measure available for a national sample of schools. They are determined in part by the amount students learn in school, and research suggests that moving to a school with higher proficiency rates does produce achievement gains.

Nor do we wish to claim that any judgment of school quality that does not correspond to test-score performance is uninformed or irrational. The ability to promote math and reading achievement is hardly the only dimension along which citizens are likely to evaluate their local schools. But we suspect that high test scores go along with other aspects of school quality that citizens value in their schools, so that evidence of a connection between student achievement and public opinion likely indicates that parents and other members of the public have the information they need to make reasonable judgments about their schools.

National Evidence

These data enable us to provide the first evidence on the extent to which citizens’ subjective ratings of specific schools correspond to publicly available information on their actual performance. Because other school characteristics may also influence perceptions of school quality, we incorporated into our analysis data from the National Center for Education Statistics on the racial/ethnic composition of each school, the percentage of students eligible for free or reduced-price lunch (an indicator of poverty), average cohort size (our preferred measure of school size), and pupil-teacher ratio (a proxy measure of class size) in the 2007–08 school year. We exclude high schools when analyzing the data for the nation as a whole because proficiency data are unavailable for many of them, and when available, typically reflect the performance of only a single cohort of students. We also adjust for whether the respondent was evaluating an elementary or a middle school to account for the fact that middle schools received systematically lower grades from survey respondents.

Figure 2 presents the strength of the relationship between citizen ratings of school quality and each of these school characteristics after taking into account the other key variables built into our analysis. The values of each variable except the one identifying elementary schools have been standardized to illustrate their relative importance. (In technical terms, the relationships presented for these variables reflect the effect of an increase of one standard deviation in the value of the characteristic in question.) The figure confirms that student proficiency rates are a significant predictor of citizen ratings of school quality. An increase of 18 percentage points in percent proficient (i.e., one standard deviation) is associated with a rating that is on average 0.16 grade points higher, or about one-sixth of a letter grade.

Examining the racial/ethnic and class makeup of a school’s student body in isolation would suggest that both are important predictors of citizen ratings, a fact that may explain the common perception that this is the case. In particular, schools with 25 percentage points more African American students received ratings that were 15 percent of a letter grade lower, while schools with 24 percentage points more Hispanic students received ratings that were 16 percent of a letter grade lower. Schools with 26 percentage points more poor students received ratings that were one-quarter of a letter grade lower.

However, when these variables are considered simultaneously and alongside school performance and resource measures, only the poverty indicator retains predictive power. Neither the percentage of students who are African American nor the percentage who are Hispanic is systematically related to perceptions of school quality. The percentage of students who are poor remains an important predictor of citizen ratings, with a relationship essentially as strong as that for proficiency rates.

Even after controlling for proficiency rates and other school characteristics, middle schools receive ratings that are, on average, 18 percent of a letter grade lower than comparable elementary schools. In other words, proficiency rates explain some, but by no means all, of the lower perceived quality of middle schools. This finding is of interest given recent research suggesting that middle schools have adverse consequences for student achievement (see “Stuck in the Middle,” research). In contrast, neither school size nor pupil-teacher ratio are important determinants of perceptions of school quality. In fact, the weak relationship between pupil-teacher ratio and school ratings is in the opposite of the expected direction: schools with larger classes receive somewhat higher grades, perhaps because effective schools attract more families to the neighborhood.

As noted above, it has often been speculated that disadvantaged groups are less informed about school quality than more-advantaged groups. But we find that the relationship between school performance and citizen ratings is as strong for African American and Hispanic respondents as it is for whites. The relationship between school quality and citizen ratings is also essentially the same for high-income and more-educated respondents as it is for low-income and less-educated respondents.

We also consider whether the relationship between school performance and citizen ratings is stronger for parents of school-age children, who are arguably the most connected to their local schools, or for homeowners, whose property values are influenced by school quality. Perhaps surprisingly, homeowners are no more sensitive to differences in school quality than are other citizens. However, the relationship between proficiency rates and school ratings is more than twice as strong for parents of school-age children than for other respondents (see Figure 2). An increase of one standard deviation in percent proficient is associated with a rating from parents that is one-third of a letter grade higher, as compared with 16 percent of a letter grade higher for the public as a whole. Parents also give low-scoring schools far lower ratings than do other local residents, but this difference narrows and eventually reverses direction as proficiency rates increase (see Figure 3). Like those of other citizens, parents’ ratings of local schools are not influenced by the schools’ racial/ethnic composition, school size, or pupil-teacher ratios. However, parents do appear to be somewhat more responsive than other citizens to school poverty rates and take an especially dim view of middle schools, assigning them grades that are 39 percent of a letter grade lower than otherwise similar elementary schools.

Finally, we consider the issue of differences in school quality across states. Because NCLB allows each state to set its own standards for proficiency, schools in different states with the same percentage of students achieving proficiency may be of markedly different quality if one state has high standards and the other low. The national sample allows us to examine the degree to which citizen ratings of school quality are responsive to performance levels relative to the nation or simply to differences in performance within specific states. The National Assessment of Educational Progress (NAEP) conducted every two years by the U.S. Department of Education provides evidence on the average performance of 4th- and 8th-grade students in each state in mathematics and reading. We use data from the 2007 NAEP to see whether respondents in states with higher-scoring students rate their schools higher, on average, than respondents in states with lower NAEP scores. That is, if we compare respondents whose local schools have the same proficiency rate as measured by their state test, do the respondents in states with better schools, as measured by student performance on the NAEP, assign their school higher grades? We find no evidence that respondents in general, or even parents, have information about school quality beyond the information provided on the state assessments. In other words, citizens appear to be taking cues about school quality from local comparisons or from information provided by their state testing system without taking into account the relative rigor of state standards.

Levels or Growth?

Our analysis yields strong evidence that citizens, and especially parents of school-age children, rate schools in a way that lines up with publicly available information about school quality. As discussed previously, however, the percentage of students scoring at the proficient level on state tests is an imperfect indicator of school quality, contaminated as it is by the fact that student achievement is influenced by a host of factors outside of a school’s control. A better, if still an imperfect, measure of school quality is the amount of growth in student achievement from one year to the next. To examine the correspondence of citizen perceptions of school quality and measures of test-score growth, we turn to our representative sample of residents of Florida, where the state accountability system evaluates schools based on both test-score levels and test-score growth. Because high-school performance data are widely available in Florida, we are able to include high schools in this portion of the analysis.

Florida assigns schools letter grades based on a point system with eight main components, which we divide into two categories: level-related points (percentage proficient in math, English, writing, and science) and growth-related points (percentage making learning gains in math and reading and the percentage of the lowest 25 percent of students making gains in math and reading). The level variable is highly correlated with the school quality measure (percent proficient) used in the national analysis, but the correlation between the growth variable and percent proficient is considerably weaker.

Our basic strategy is to compare the ratings Florida residents assigned to their schools both to test-score levels and to test-score growth at those schools. Because measures of test-score growth are less stable over time than measures of test-score levels, we average the points awarded to each school based on levels and growth over the previous three years. Adjustments are also made for the same demographic and school characteristics as in the national analysis. To make the results as comparable as possible to those reported for the national sample, we also scale the point variables so that a one-unit increase in each variable corresponds to a shift of one standard deviation in the performance distribution of Florida public schools.

The results indicate that Florida residents’ perceptions of school quality are even more responsive to differences in student achievement levels than are those of the national public. An increase of one standard deviation in the level variable is associated with ratings that are almost one-third of a letter grade higher after taking into account other school characteristics. We also find that perceptions of school quality in Florida are unrelated to student demographic characteristics, including the percentage of students who are poor, once we take into account levels of student achievement. Although we cannot be sure, both Floridians’ greater responsiveness to test performance and their lack of responsiveness to student demographic characteristics could reflect the transparency and salience of the state’s high-profile school accountability system.

When both the test-score level and growth variables are examined simultaneously, however, the relationship between level-related points and citizen evaluations of schools is almost twice as strong as for growth-related points. This suggests that citizen ratings do reflect differences in the growth in student achievement across schools, but that this is primarily because of the correlation between achievement levels and achievement growth.

The Role of Accountability Systems

So far we have shown that citizens’ assessments of schools are strongly related to objective measures of performance made available by state accountability systems. Yet it is difficult to determine whether respondents’ apparent sensitivity to actual quality is the result of publicly available information or simply direct experience with schools. The fact that parental perceptions track actual school quality more closely than those of other citizens, but the perceptions of homeowners do not, suggests that direct interactions with a school may be a more important factor than simply having a vested interest in acquiring information about local school quality. But do accountability systems also play a role in shaping citizen perceptions?

Again, Florida provides an ideal case for more detailed analysis. As noted above, the Florida Department of Education uses the total number of points received (i.e., the sum of level- and growth-related points) to assign each school a letter grade between A and F. These grades receive considerable media attention in Florida, so we might expect citizen ratings to be correlated with them. This expectation is confirmed in the data: a school grade that is one point higher (again measured on a standard GPA scale) is associated with a respondent rating that is 0.2 grades higher.

To test the hypothesis that publicly available information has an impact over and above direct observation of school performance, we can compare the ratings given by respondents whose schools were very close to the cutoffs in the point system used by Florida to assign school grades. We know that schools with more points received higher ratings on average, but might also expect to see a “jump” in the average rating at these cutoffs. Because schools on either side of the cutoff should be of essentially the same quality, we can interpret any jump in the rating observed at the cutoff as the pure effect of information provided by the school grade on citizen perceptions of school quality.

We focus our attention on the B/C cutoff, because that is the only one for which we have enough respondents assigned to schools near the cutoff to yield results with a reasonable degree of precision. Comparing respondents’ ratings of schools on either side of this cutoff suggests a large positive effect of receiving the higher (B) grade, with an increase in the grades assigned to schools in the range of of 36 to 57 percent of a letter grade. That the publicized school grades have a direct effect on respondent ratings over and above the relationship between ratings and the underlying point variables suggests that the signals provided by the state’s school accountability system do in fact affect citizen perceptions of their local schools.

Implications

The findings reported above represent the first systematic evidence that Americans’ perceptions of the quality of their local public schools reflect publicly available information about the academic achievement of the students who attend them. Importantly, disadvantaged segments of the population are no less informed about school quality than other citizens. Although the mechanisms explaining this responsiveness are not entirely clear, our evidence suggests that both direct experience with schools and the public dissemination of performance data may play a role.

It is worth emphasizing several limitations on this evidence of responsiveness. First, the relationship between actual and perceived quality is modest for citizens as a whole, although it is quite strong for parents, who have the most opportunities to observe schools and arguably have the strongest incentives to be informed. Second, both parents and the public appear to be more responsive to the level of student achievement at a school than to the amount students learn from one year to the next. Finally, citizens appear sensitive to relative differences in school quality within their state (as reflected in school performance on state tests) but insensitive to information on school quality in the state as a whole (as measured by statewide performance on a national assessment).

Even so, at least two policy implications emerge from our results. First, our finding that accountability ratings influence citizens’ assessments of their local schools coupled with the fact that citizen ratings are more strongly associated with achievement levels than with achievement growth suggest that featuring growth measures more prominently in school accountability ratings could cause citizens to pay more attention to this barometer of school quality. Second, our finding that citizen ratings are associated with student performance on state tests but not with performance on a national assessment suggests that a closer alignment of state standards (or a move toward common standards across states) might help citizens form more accurate perceptions of their schools. In particular, it could lower perceptions of school quality in states where many students perform poorly relative to national norms but are deemed proficient by the state.

Matthew M. Chingos is a postdoctoral fellow at Harvard University’s Program on Education Policy and Governance. Michael Henderson is a doctoral candidate in Harvard’s Department of Government. Martin R. West is assistant professor of education at the Harvard Graduate School of Education and executive editor of Education Next.

How has the No Child Left Behind (NCLB) Act affected student achievement? This is no idle question, as the landmark federal law is long overdue for reauthorization. The Obama administration has recently urged Congress to add the issue to its already crowded 2010 agenda, even going so far as to include an additional $1 billion for K–12 education in its budget proposal if the law is reauthorized this year (a wholly symbolic gesture, given that it is Congress that sets spending levels, but one that indicates the administration’s priorities).

Yet heightened attention to NCLB has not produced consensus over its consequences for students. No Child Left Behind was a reauthorization of the Elementary and Secondary Education Act (ESEA), the central federal legislation relevant to K–12 schooling. NCLB dramatically expanded the law’s scope by requiring that states introduce school-accountability systems that applied to all public schools and students in the state. NCLB requires annual testing of students in reading and mathematics in grades 3 through 8 (and at least once in grades 10 through 12) and that states rate schools, both as a whole and for key subgroups, with regard to whether they are making adequate yearly progress (AYP) toward their state’s proficiency goals. Supporters and critics, in their various approaches to discerning NCLB’s impact, share a significant problem: because NCLB applies to all public school students, researchers lack a suitable comparison group and so have been unable to distinguish the law’s effects from the myriad other factors at work over the past eight years.

The new research we present below takes on this challenge. Our basic insight is that the test-based accountability provisions that are the defining characteristic of NCLB did not come from nowhere, but rather were modeled quite closely on reforms adopted by many states in the 1990s. For states with such accountability systems in place before 2002, NCLB’s most important components may have created some logistical headaches but were largely irrelevant. In contrast, NCLB forced the remaining states to enact accountability systems for the first time. We can therefore estimate the impact of NCLB’s accountability mandates by comparing test-score changes in states that did not have NCLB-style accountability policies in place when the law was implemented to test-score changes in those that did.

We find that the accountability provisions of NCLB generated large and statistically significant increases in the math achievement of 4th graders and that these gains were concentrated among African American and Hispanic students and among students who were eligible for subsidized lunch. We find smaller positive effects on 8th-grade math achievement. These effects are concentrated at lower achievement levels and among students who were eligible for subsidized lunch. We do not, however, find evidence that NCLB accountability had any impact on reading achievement among either 4th or 8th graders.

Assessing NCLB

The broad interest in understanding whether NCLB has influenced student achievement, both overall and for key subgroups, has motivated careful scrutiny of trend data from the National Assessment of Educational Progress (NAEP) and other sources. For example, the authors of a report commissioned by the U.S. Department of Education’s Institute of Education Sciences (IES) note that achievement trends on both state assessments and NAEP were “positive overall and for key subgroups” through 2005. Using more recent data, a report by the Center on Education Policy concludes that reading and math achievement as measured by state assessments has increased in most states since 2002 and that there have been smaller but similar patterns in NAEP scores. Both reports were careful to stress that these national gains are not necessarily attributable to the effects of NCLB.

Other studies have taken a less sanguine view of these achievement gains, arguing that they are misleading because states have made their assessment systems less rigorous over time. University of California scholar Bruce Fuller and colleagues, for example, document a growing disparity between student performance on state assessments and NAEP since the introduction of NCLB and conclude that “it is important to focus on the historical patterns informed by the NAEP.” Using NAEP data on 4th graders, they conclude that the growth in student achievement has actually slowed since the introduction of NCLB.

Turning to the broader literature on school accountability, several researchers have evaluated the achievement consequences of the accountability systems states developed during the 1990s. One study by Martin Carnoy and Susanna Loeb of Stanford, which was based on state-level NAEP data, found that the within-state growth in math performance between 1996 and 2000 was larger in states with higher values on an accountability index, particularly for African American and Hispanic students in 8th grade. Another study, by Eric Hanushek and Margaret Raymond, both also at Stanford, evaluated the impact of school-accountability policies on state-level NAEP math and reading achievement measured by the difference between the performance of a state’s 8th graders and that of 4th graders in the same state four years earlier. They classified states as having either “report-card accountability” or “consequential accountability.” Report-card states provided a public report of school-level test performance. States with consequential accountability both publicized school-level performance and attached consequences to that performance. Hanushek and Raymond found that the introduction of consequential accountability within a state was associated with increases in NAEP scores.

Both of these studies suggest that NCLB-style accountability provisions may increase student achievement and also demonstrate how state-level NAEP data can be used to evaluate accountability systems. The analysis described below effectively extends this important work to cover the more recent state accountability reforms that were compelled by NCLB.

Research Design

Given the various social, economic, and educational factors at work before and after NCLB was implemented, it is difficult to draw strong conclusions about the policy’s impact from a simple comparison of achievement trends before and after enactment of the law. For example, the nation was suffering from a recession around the time NCLB was implemented, which one might expect would have reduced student achievement in the absence of other forces. At the same time, other national education policies and programs were in place that may also have influenced student achievement.

Perhaps the central challenge in evaluation research is to identify a plausible comparison group that was unaffected by the intervention under study. In the case of NCLB, this is particularly difficult, as the policy simultaneously applied to all public schools in the United States.

We address this issue by comparing trends in student achievement across states that had varying degrees of prior experience with state school-accountability policies similar to those brought about by NCLB. The intuition behind this approach is that NCLB represented less of a “treatment” in states that had already adopted NCLB-like school-accountability policies prior to 2002. To the extent that NCLB-like accountability had either positive or negative effects on measured student achievement, we would expect, once NCLB had been implemented, to observe those effects most distinctly in states that had not previously introduced similar policies.

This strategy relies on the assertion that pre-NCLB school-accountability policies were comparable to NCLB—that is, that the two types of accountability regimes are similar in the most relevant respects. The fact that many state officials criticized NCLB, arguing that it duplicated their prior accountability systems, suggests the functional equivalence of the two sets of policies. To ensure that this is the case and relying on a number of different sources, we evaluated the comparison states according to whether the features of their pre-NCLB accountability policies closely resembled the key aspects of NCLB. We found that they were in fact quite similar.

As an additional check on the validity of our treatment and comparison groups, we used our research design to estimate the impact of NCLB accountability on outcomes that we would not expect to be affected, such as the state-level average poverty rate and median household income. The fact that our method does not find any “effect” of NCLB on such outcomes suggests that these states can serve as a plausible comparison group for isolating the impact of NCLB accountability.

We implement our research design in a more fine-grained manner than simply comparing achievement trends in the treatment and comparison states. We define the treatment as the number of years without prior school accountability between the 1991–92 academic year and the onset of NCLB. Hence, states with no prior accountability have a value of 11. Illinois, which adopted its policy in the 1992–93 school year, would have a value of 2. Texas would have a value of 4 since its policy started in 1994–95, and Vermont would have a value of 9 since its program began in 1999–2000. This method implies that the larger the value of this treatment variable, the greater potential impact of NCLB. The total effect we report is the impact of NCLB accountability in 2007 for states with no prior accountability relative to states that adopted school accountability in 1997 (the mean adoption year among states that adopted accountability prior to NCLB).

It is important to note that this research design will capture the impact of the accountability provisions of NCLB, but not the impact of other NCLB components such as the Reading First program or its Highly Qualified Teacher provisions. Additionally, our estimates will identify the impact of NCLB-induced school-accountability provisions on states without prior accountability policies. To the extent that one believes that states that expected to gain the most from accountability policies adopted them prior to NCLB, one might view the results we present as an underestimate of the average effect of school accountability.

Data

This analysis uses data on math and reading achievement from the state NAEP, which offers a representative sample of student achievement in each state at regular intervals. Participation in the state NAEP was voluntary prior to NCLB, although roughly 40 states did participate. NCLB made participation mandatory. The main advantage to using NAEP data for our analysis is that it is a low-stakes exam that is not directly tied to any state’s standards or assessments. Instead, NAEP aims to assess a broad range of skills and knowledge within each subject area. Consequently, NAEP data should be relatively immune to concerns about accountability-driven test-score inflation, such as may result from “teaching to the test.”

Because our research design depends on measuring achievement trends prior to NCLB, we limit our sample to states that administered the state NAEP at least twice prior to the implementation of NCLB. We include 2002 as a pre-NCLB data point in our analysis because, given the timing of the passage and implementation of the law, it seems unlikely that spring 2002 scores could have been substantially influenced by NCLB (see sidebar). All states administered NAEP in 2003, 2005, and 2007.

Our sample includes 39 states for 4th-grade math, 38 states for 8th-grade math, 37 states for 4th-grade reading, and 34 states for 8th-grade reading (see Figure 1). With a few exceptions, our analysis sample closely resembles the nation in terms of student demographics (e.g., percentage African American and percentage Hispanic), observed socioeconomic traits (e.g., the poverty rate), and measures of the levels and pre-NCLB trends in NAEP test scores.

Results

We find that the accountability provisions of NCLB increased 4th-grade math achievement by roughly 7.2 scale points (0.23 standard deviations) by 2007 in states with no prior accountability policies relative to states that adopted accountability systems in 1997. How large is this effect? As one point of reference, consider that the difference between the average scores of 4th and 8th graders in our sample suggests that students gain roughly 12 scale points per year. By this measure, the NCLB impact is equivalent to roughly two-thirds of the average annual gain in scale points. Consider also that the achievement gap between black and white 4th graders on the NAEP math exam is roughly 30 scale points (1 standard deviation), which means that the impact of NCLB is equivalent to about one-quarter of this difference. The effect for 8th-grade math is smaller (0.10 standard deviations) and falls just shy of achieving conventional levels of statistical significance. We find no effects for 4th- and 8th-grade reading.

The design of NCLB necessarily focused the attention of schools on helping students attain proficiency. Figure 2 presents our estimates of the effects of NCLB accountability on the percentage of students achieving at or above the basic and proficient performance levels on NAEP. Although states’ definitions of proficient vary widely, very few set the proficiency bar as high as NAEP and most correspond more closely to NAEP’s basic performance level. We find that NCLB accountability increased the share of students performing at or above basic in math by 10 percentage points among 4th graders and 6 percentage points among 8th graders. Math proficiency rates among 4th graders also increased by 6 percentage points. Again, however, we do not find consistent evidence that NCLB increased reading performance at either grade level.

Given NCLB’s focus on proficiency, one would expect the law to disproportionately influence achievement among previously low-achieving students. Our results showing larger increases in the percentage of students reaching the performance level of basic on the NAEP are broadly consistent with this theory. However, in contrast with some previous research and commonly voiced concerns, we do not find that the introduction of NCLB harmed students at higher points on the achievement distribution. Indeed, NCLB accountability seemed to increase achievement among higher-achieving students, if by a smaller amount than it did among their low-achieving peers. For example, in 4th-grade math, we find that NCLB increased scores at the 10th percentile by roughly 0.29 standard deviations compared with an increase of only 0.17 standard deviations at the 90th percentile (see Figure 3).

One of the primary objectives of NCLB was to reduce inequities in student performance by race and socioeconomic status. Indeed, this concern drove the requirement that, under the statute, accountability ratings be determined by subgroup performance in addition to aggregate school performance. Hence, it is of particular interest to understand the effect of NCLB accountability on specific student subgroups.

In 4th-grade math, these estimated effects are somewhat larger for Hispanic students relative to white students. Similarly, the effects were substantially larger among students who were eligible for subsidized lunch (regardless of race) relative to students who were not eligible. We also found relatively large effects for black students but only when our analysis weighted the state-year NAEP data by the corresponding enrollments of black students. This pattern suggests that NCLB generated more meaningful improvements in the achievement of black students in states where public schools served larger numbers of black students. The effects were roughly comparable for boys and girls.

In 8th-grade math, we find extremely large positive effects for Hispanic students and small, only marginally significant effects for white students. Unfortunately, the results for black students are too imprecisely estimated to warrant interpretation. The effects for students eligible for subsidized lunch are large and statistically significant. Interestingly, for 8th graders the effects are substantially larger for girls, with boys experiencing little if any benefit of accountability.

Unintended Consequences?

One concern about NCLB and most other test-based school-accountability policies is that they may cause schools to neglect subjects other than math and reading. NAEP data offer some opportunity to test this hypothesis in the context of NCLB. A sizable number of states administered state-representative NAEP tests in science. Unfortunately, during our analysis period, the 4th-grade science exam was only administered in 2000 and 2005 and the 8th-grade science exam was administered in 1996, 2000 and 2005. The lack of multiple pre- and post-NCLB measures of student achievement limit the power of our research design. Nonetheless, when we apply our research design to these data, we find no statistically significant effects at either grade level at any point on the achievement distribution. Moreover, we are able to rule out effects larger than roughly 0.10 standard deviations. While these results should be taken with a grain of salt, they cast doubt on some claims that NCLB accountability has had an adverse impact on student performance in science.

Another major concern with test-based accountability, including NCLB, is that it provides teachers an incentive to direct energy toward the types of questions that appear most commonly on the high-stakes test and away from other topics within the tested domain. As noted above, one of the benefits of the analysis presented here is that it relies on student performance on NAEP, which should be relatively immune from such test-score “inflation” since it is not used as a high-stakes test under NCLB or any other accountability system. It is nonetheless interesting to examine whether NCLB accountability has improved student achievement in any particular topic within math or reading. The NAEP math exam measures student performance in five specific topic areas: algebra, geometry, measurement, number properties and operations, and data analysis, statistics, and probability. Our results suggest that NCLB had a positive impact in all math topic areas for the 4th-grade sample. Among 8th graders, NCLB had a moderately large and statistically significant impact in data analysis and marginally significant effects in number properties and geometry.

The NAEP reading exam measures student competency in several skills related to comprehension: reading for information (i.e., primarily nonfiction reading), reading for literary experience (i.e., primarily fiction reading), and (for 8th grade only) the ability to perform a task (e.g., students apply knowledge from reading bus schedules or directions for repairing something). We find no significant differences in student achievement effects by topic area in reading; that is, NCLB accountability did not appear to have significant effects on student achievement in any of the three reading competencies. Keep in mind, however, that our research design does not allow us to comment on the effects of other aspects of the law, such as the Reading First program, that were explicitly designed to boost reading performance.

Summing Up

So how has NCLB accountability affected student achievement? Our results suggest that its consequences have been mixed. Specifically, we find that the accountability provisions of NCLB generated large and broad gains in the math achievement of 4th graders and somewhat smaller gains for 8th graders. Our results suggest that NCLB accountability had no impact on reading achievement for either group.

The mixed results presented here pose difficult but important questions for policymakers considering whether to “end” or “mend” NCLB. The evidence of substantial and almost universal gains in math is undoubtedly good news for advocates of NCLB. But the lack of any effect in reading, and the fact that the policy appears to have generated only modestly larger impacts among disadvantaged subgroups in math (and thus made only minimal headway in closing achievement gaps), suggests that the impact of NCLB has fallen short of its extraordinarily ambitious goals. Some commentators have argued that the failure of NCLB and earlier accountability reforms to close achievement gaps reflects a flawed, implicit assumption that schools alone can overcome the achievement consequences of dramatic socioeconomic disparities.

An effective redesign of accountability policies like NCLB may need to pay more specific attention to the processes and practices operating within schools. Along those lines, it is interesting to note that our evidence of differential effects by grade and subject is broadly similar to the results from evaluations of earlier state-level school-accountability policies. Understanding the sources of these differences is likely to be particularly useful as policymakers discuss the future design and implementation of school-accountability systems. For example, the unique effectiveness of NCLB in improving the math skills of younger students could be related to the biological evidence that cognitive skills are more malleable at early ages. These outcomes may also result from the specific ways in which schools and teachers have adjusted their instructional practices, perhaps differently for mathematics and reading. Much evidence suggests that school decisions about curricula (e.g., textbooks, instructional software, and the corresponding pedagogy) can have comparatively large effects on student achievement. Further research that can credibly and specifically examine how school and teacher responses have contributed to the achievement effects documented here would be a useful next step in identifying effective policies and practices that can reliably improve student outcomes.

Thomas Dee is associate professor of economics at Swarthmore College. Brian Jacob is professor of education policy and economics at the University of Michigan.

The constitutionality of state school-finance systems has been under attack for nearly 40 years. Since the California Supreme Court’s 1971 ruling in Serrano v. Priest, finance-reform advocates have filed 139 separate lawsuits in 45 states. The specific language varies from state to state, but virtually all state constitutions contain education clauses that require the state legislature to provide an “adequate,” “basic,” or “thorough and efficient” education for all children. Plaintiffs have relied on these provisions to seek increases in the financial resources devoted to public schools, especially those serving disadvantaged students. Courts have in turn deemed school-finance systems unconstitutional in 28 states.

While school-finance lawsuits have attracted significant attention in the legal community and generated numerous state-specific case studies, nationwide analyses of the effects of school-finance judgments (SFJs) have been relatively few. This small pool of studies has produced some common conclusions, namely, that such judgments reduce funding inequality between districts by increasing spending in the poorest districts and that they do so by transferring responsibility for education funding from local to state governments. Some questions remain unanswered, however, such as why SFJs have substantially different effects in different states.

A court’s ruling that an existing school-finance system is unconstitutional is only the first step toward funding reform. Some court orders provide instruction for how the legislature should fix the system, but most simply instruct state politicians to redesign the finance system themselves. In either case, the new finance system must garner the approval of the state legislature and governor. In other words, after the court ruling, the reform must pass through the state’s usual lawmaking process. States with similar court rulings may end up with very different reforms, depending on how the legislature and governor respond.

With this political process in mind, we decided to investigate how politics might influence the way an SFJ alters a state’s school-finance system. Our starting point was estimating the change in per-pupil funding that could be confidently attributed to an SFJ. We did this by comparing changes in funding in school districts where the state’s school-finance system has been ruled unconstitutional in a court challenge to funding changes in comparable districts in states where no SFJ has been issued. We studied district-level changes in school funding following 23 school-finance judgments issued between 1988 and 2005. The lawsuits were all related to general education funding, and each was the first SFJ in a state during our period of study. In total, we studied funding outcomes in more than 13,000 districts over 18 years.

What we were most interested to know is whether the change in funding differs if a state has unified Democratic control of the state legislature and the governorship at the time of the court decision, unified Republican control, or when control is divided between the two parties as, for example, when the governor is a Republican and the Democrats control one or both of the houses of the legislature. To find out, we compared the outcomes of SFJs issued in each of these circumstances.

We found that court-ordered finance reform alters district funding levels under each type of partisan regime. On balance, Democratic control results in across-the-board increases in state funding to local school districts, while Republican and divided-government regimes tend to produce funding increases targeted to poorer districts. SFJs in all three types of political environments lead to a shift in funding responsibility from local to state governments, although to differing degrees.

Which Party is Responsible?

As we began our study, we had to decide how to assign responsibility for school funding changes produced by an SFJ in the years following the judgment, especially when the party that controls the state government changed. We decided to focus on partisan control at the time of the court decision because the government at the time of the ruling is obligated to craft the policy response. Our approach, then, attributes the effect of the SFJ in subsequent years to the party in power when the judgment is made, even if there is a subsequent change in partisan control. We checked the validity of this decision by rerunning our analysis, attributing the funding associated with an SFJ in any given year to the party in control of the state government in that same year. With this method, our estimates of the relationship between partisan control and the effect of an SFJ, in dollars, were much less precise than when we used our preferred approach, although the substantive conclusions of our analysis remained the same. The better estimates lead us to conclude that party control at the time of the court decision has, on average, the most important role in determining the political response to an SFJ.

Table 1 lists the cases used in our analysis and the configuration of partisan control of the state government at the time of the court decision. Only three SFJs were issued during periods of unified Republican government: in New Hampshire, Ohio, and Wyoming. This suggests the need for caution in interpreting our results, especially about the patterns in school finance we see under Republican governments. There were seven judgments handed down during unified Democratic government (in Alabama, Kentucky, Maryland, Missouri, Tennessee, Vermont, and West Virginia) and 11 delivered when government was divided (in Connecticut, Idaho, Kansas, Massachusetts, Minnesota, Montana, North Carolina, New Jersey, New York, South Caroline, and Texas).

State, Local, and Federal Funding

While SFJs require a policy response from the state government, and therefore are expected to have a direct impact on state funding, they may also have an indirect effect on funding from local sources. Indeed, one concern over the efficacy of court-induced reforms is that local districts may reduce their own contribution to the schools in response to increases in state aid, thereby undermining efforts to increase total school spending. To provide a more comprehensive picture of the effect of SFJs, we look at the impact on both state and local funding.

Of course, because spending on schools also includes a small amount of federal aid, total funding is not simply the sum of state and local funding. Federal funds, which make up about 10 percent of total education funding, have until recently been limited to specific programs, such as the National School Lunch Program and special education. Thus, we would not expect a state court decision  to influence federal funding, an assumption that is borne out in the data.

Gauging the Effects

Our basic strategy was to compare changes in funding levels in districts where the state’s school-finance system has been ruled unconstitutional to funding changes in comparable districts in states where an SFJ has not been issued. We make these comparisons with groups of districts that had Democratic, Republican, or split-party control of the state government at the time the SFJ was issued. We allow for a one-year delay for the judgment to take effect because we assume that any changes in policy made as a result of the decision will be reflected in the next year’s budget, at the earliest.

Because most school-finance lawsuits are aimed at increasing funding for poor districts specifically, we designed our analysis to measure how the effects of SFJs, and of the party in control of the state government at the time of the decision, might be different for school districts with high rates of students in poverty and for districts where the students are better-off financially. To look for these differences, we divided each state’s districts into four quartiles based on the proportion of students living in poverty and allowed for the possibility that the effect of an SFJ, and of one under Democratic, Republican, or divided government, could be different in each quartile.

To isolate the effects of an SFJ on districts within each poverty quartile, we focus on changes in spending over time within specific school districts after taking into account changes from year to year in average education spending across all of the nation’s school districts. Thus we effectively control for unmeasured attributes of each school district that are constant over time and for national trends that affect all districts, such as economic conditions or changes in federal education policy that could have an impact on funding even in the absence of an SFJ. We adjust for inflation by converting all per-pupil funding figures to constant 2007 dollars.

Of course, there are other factors that likely influence changes over time in the level of per-pupil funding in a school district, including characteristics that change over time and influence either their receipt of state funding or the propensity of school districts to raise their own local revenue. We account for the variation in funding that should be directly attributed to the percentage of the student population living in poverty, independent of any change produced by an SFJ. We also include the total number of students in the district, to allow for the possibility that large districts operate differently from small districts. And we estimate the impact on per-pupil expenditure of the proportion of students in a district with Individualized Education Plans (IEPs), as students with IEPs generally have special needs that result in higher spending. Finally, we include the proportion of the student population that is African American and the proportion Hispanic. Although we have no reason to believe that these two variables directly cause changes in education funding, they may be correlated with other relevant factors, such as property values or population growth, for which we lack direct information.

In addition to district-specific characteristics, we take into account state-level characteristics that could influence state funding of education. In particular, we control for the fraction of the state’s population over age 65 to account for the possibility that the elderly oppose increases in school spending. We also control for the fraction of the population that is of school age, which captures aggregate demand for educational services. The final control variable in our analysis is per-capita income in the state, as the demand for government services may increase with income.

Annual district-level financial and demographic information comes from the Common Core of Data (CCD), available from the National Center for Education Statistics (NCES). For years in which CCD data are not available (1988–1992 and 2005), we use data from U.S. Census Bureau Elementary-Secondary Education Finance Survey (F-33). Our analysis considers only local school districts and parts of local supervisory unions with at least 100 students, as identified by the CCD. We exclude Hawaii and Washington, D.C., because each has only one school district.

Additional district demographic information, including the proportion of the population aged 5 to 17 and the proportion of school-aged children living in poverty, comes from the U.S. Census Small Area Income and Poverty Estimates for most years. For 1989 and 2005, the district demographic information comes from the School District Demographics System. Because district poverty information is not available for every year, we use the poverty estimates from the closest available survey year. For example, the district poverty estimates for 1996, 1997, and 1998 all use the data from 1997.

Partisan Patterns

A new and very clear picture about the impact of politics on SFJs emerged from our findings. The school-finance reforms implemented by Democratic state governments have substantially different effects on district funding than reforms produced by Republican or divided governments. When a Democratic state government implements an SFJ-induced reform, all districts, poor and non-poor alike, see increases in total funding. Under Republican and divided governments, districts with different levels of poverty fare quite differently.

Figure 1 represents our findings graphically. Each bar in the graph represents the effect of an SFJ, that is, the within-district change in spending after the decision, for each category of partisan control and district poverty. We present separate estimates for the change in total funding, in funding from state sources, and in funding from local revenues.

In Democrat-led reforms, our estimates show, districts in every poverty quartile see a shift from local to state funding after an SFJ. Local funding decreases, while state funding increases. This pattern of centralization of school funding is consistent with evidence from earlier studies, which also shows that localities partially offset state efforts to increase overall education spending after SFJs.

The upshot is a net increase in total funding ranging from roughly $750 to $1,000 per pupil—a sizable impact, given that total per-pupil funding in our sample is a little over $9,750 on average. While a few of the differences between quartiles are statistically significant, they are substantively small relative to the overall level of the funding increases. Indeed, if anything, the results indicate that the most affluent districts fare better than the poorest districts, in terms of total funding, when Democrats are in power, although this difference is not statistically significant. We should note here that high poverty does not necessarily imply low spending (in many states, high-poverty districts have the highest spending levels), so our findings here do not bear directly on spending inequality.

School-finance rulings handed down to divided governments produce decidedly different results. State funding increases across the board, but the changes in state funding differ markedly across the levels of district poverty: The poorer the district, the larger the increase in state funding. But, as in states with Democrat-led reforms, SFJs are not unmitigated wins for school-district budgets. All four quartiles see sizable reductions in funding from local sources. These reductions are large enough that the poorest quartile is the only one to see positive net changes in total funding. Overall, divided government reforms appear to represent a more or less straightforward redistribution of funding toward the poorest districts. The net effects on state education funding appear to be budget-neutral, as we estimate that there is little change in total education funding after an SFJ under a divided government. That said, the net increase of roughly $175 per student in total funding for poor districts is fairly modest when compared to total per-pupil funding.

Republican-controlled reforms present yet a third pattern of funding changes. Under Republican governments, funding shifts from local to state only for the poorest districts. Districts in the most affluent quartile face cuts in state funding, but they are able to more than compensate for these reductions by increasing local funding. In other words, Republican-led reforms involve centralization of funding for the poorest districts and decentralization of funding for the richest districts. The middle two quartiles are essentially unaffected. On net, both the poorest and the richest districts see increases in total funding, the former courtesy of state aid and the latter financed from their own tax base. Indeed, the richest half of districts in Republican states are the only group under any partisan regime to experience an increase in local funding following an SFJ.

Alternative Explanations

A lingering concern with our results may be that party control of the state government is related to the decision to file a school-reform lawsuit. Finance reform advocates may time the filing of their lawsuits to take advantage of what they view as particularly favorable political conditions. Another possibility is that advocates might resort to litigation only when the legislative and political process fails to provide reform. Either of these possibilities means that SFJs might have effects that appear to be associated with party control but are not actually caused by the response of the party in power.

We answer by first noting that because nearly all states—45 of 50—were subject to at least one education-finance lawsuit, the central issue is not whether a state would face a suit but when. Beyond that point, we believe that this is not a major concern for three reasons: 1) the amount of time between lawsuit filing and the court decision is often long and always unpredictable; 2) the party in control often changes between the lawsuit filing and decision; and 3) lawsuits do not appear to be precipitated by changes in political regime. Among the 23 cases included in our study, the length of time from the initial filing through the final appellate court decision ranged from less than a year to nine years. On average, the process took four years. Due to the length of time the suits take and the variability of the speed of the adjudication process, advocates could not effectively time their lawsuits to specific political circumstances. In almost half of the cases (11 out of 23), the party in control changed between the time of filing and the time of decision. Further, school-finance lawsuits do not appear to be triggered by changes in party control. On average, the party in control in the state was stable for six years prior to the filing of a case. In only three cases did the party in control change in the year of the lawsuit filing, and for each of those three cases, the party in control changed again before the lawsuit was decided.

Conclusion

Which partisan arrangement leads to the best results for poor districts after a school-finance judgment? That question requires stepping into the debate about the relationship between student outcomes and school funding and goes beyond the evidence we present here. What our study does show is one of the many possible ways that politics can influence the implementation of court-ordered school-finance reform. Clearly, reforms implemented by Democrats produce the largest net increases in funding for all students. However, by delivering roughly equivalent funding increases to districts at all income levels, Democrat-led reforms do not target new resources to districts serving poor students. Reforms implemented by divided or Republican governments deliver concentrated benefits to districts serving poor students. In these instances, however, the actual flow of new dollars into poor districts is more meager than when Democrats are in control.

Christopher Berry is assistant professor at the Harris School of Public Policy at the University of Chicago. Charles Wysong is a student at Stanford Law School.

It is unclear whether and how public funds should be mobilized to close those gaps. Some advocate expanding existing programs that target disadvantaged children on the grounds that limited public resources should be directed toward the families and children most in need. Others consider the perennial underfunding of targeted programs like Head Start as evidence of a lack of political support for this approach, and argue that providing universal access is needed to ensure adequate public funding over the long run. In other words, any new funding for preschool education must benefit middle-class children if it is to gain their parents’ political backing. Or so it is argued.

Existing research provides little insight into the relative merits of universal programs and those targeted to specific groups. While there have been several recent studies of the short-term effects of universal preschool programs in the U.S., there is no evidence to date on long-term consequences. Some studies suggest that Head Start has lasting effects in reducing criminal behavior and increasing educational attainment, but this program is much more intensive than any universal program is likely to be and serves a very disadvantaged population.

In the absence of direct evidence on the types of preschool programs now under consideration, this study attempts to shed light on the likely consequences of a new universal program by estimating the impact of earlier state interventions to introduce kindergarten into public schools. In the 1960s and 1970s, many states, particularly in the southern and western parts of the country, for the first time began offering grants to school districts operating kindergarten programs. Districts were quick to respond. The average state experienced a 30 percentage point increase in its kindergarten enrollment rate within two years after an initiative, contributing to dramatic increases in kindergarten enrollment (see Figure 1). These interventions present an unusual opportunity to study the long-term effects of large state investments in universal preschool education.

My results indicate that state funding of universal kindergarten had no discernible impact on many of the long-term outcomes desired by policymakers, including grade retention, public assistance receipt, employment, and earnings. White children were 2.5 percent less likely to be high school dropouts and 22 percent less likely to be incarcerated or otherwise institutionalized as adults following state funding initiatives, but no other effects could be discerned. Also, I find no positive effects for African Americans, despite comparable increases in their enrollment in public kindergartens after implementation of the initiatives. These findings suggest that even large investments in universal early-childhood education programs do not necessarily yield clear benefits, especially for more disadvantaged students.

Kindergarten in the U.S.

Many state governments have only recently introduced state grants for school districts that operate kindergarten programs. Kindergartens began outside of the public school system, funded largely through philanthropic organizations or private tuition. Over the first half of the 20th century, kindergartens slowly became incorporated into urban schools, at the same time gaining partial funding through local taxes. As late as the mid-1960s, however, such programs continued to rely heavily on local resources, as only 26 states and the District of Columbia helped fund kindergarten costs. There were remarkable changes over the next decade, however: Between 1966 and 1975, 19 states began funding kindergarten for the first time. The majority of these states were in the South, but the West was also well represented. By the late 1970s, only two states—Mississippi and North Dakota—did not fund kindergarten programs (see Figure 2).

Initially, states channeled their funding to districts in one of two ways. Some states revised existing funding formulas to include financial support for kindergartens on a basis equivalent to support for all other grades in a state’s public school system. Other states appropriated separate monies for kindergarten, an approach that made kindergarten funding more vulnerable to budget cuts. Eventually, however, all states made kindergarten a part of the basic state school program.

The initiatives were introduced during a period of rising labor-force participation among women with young children, so kindergarten’s popularity may have been due to the fact that it provided families with subsidized child care. The stated purpose, however, was to improve children’s educational outcomes. In particular, it was claimed that kindergarten would provide the preparation children need to succeed in the elementary school years. Greater success in school would, in turn, reduce state spending not only on special education and “re-education” of children who failed, but also on public assistance and incarceration over the long term.

Whether state funding of kindergarten was capable of achieving these goals is open to question. Kindergartens have historically maintained a curriculum focused more on children’s social development and less on academic training. While a focus on socialization does not preclude long-term effects, kindergarten programs lacked features of some targeted interventions—such as parental involvement and health services—that may be critical to their success. State-funded kindergartens for five-year-olds may also have reduced enrollment in private kindergartens and in education programs funded through Head Start and Title I. My study seeks to shed light on these important policy questions of relevance to the current conversations concerning early childhood education.

Method and Data

To find out the long-term impacts of the introduction of universal kindergarten, I take advantage of the staggered introduction of state funding for kindergarten from the 1960s forward, combined with the fact that children generally attend kindergarten at age five. More specifically, I calculate the average difference in outcomes between individuals who were age five before the introduction of kindergarten funding and children born in the same state who were five years old after the initiative was introduced. I further adjust these comparisons to take into account the fact that kindergarten enrollment was increasing gradually in many states prior to the adoption of state funding. The remaining differences should reflect the long-run effects of the typical state-funded kindergarten program.

I restrict my analysis to the 24 states that introduced state funding for universal kindergarten after 1960 because the data needed for the analysis are not available for earlier years. I also limit attention to the 1954 to 1978 birth cohorts because they span the period over which most of these funding initiatives were passed, and doing so provides me with data both before and after the introduction of these initiatives necessary to estimate the effects of kindergarten funding on long-term outcomes.

I combine data from several sources. I measure the kindergarten enrollment rate with the state kindergarten-to-first-grade enrollment ratio, calculated from the federal Common Core of Data and earlier published data. Data for the analysis of the initiatives’ long-term effects were drawn from Public Use Microdata Samples (PUMS) of the Decennial Census. In particular, I examine 1) whether a child was below grade for age while still of school age (a proxy for grade retention); 2) three indicators of adult educational attainment (high school dropout, high school degree only, and some college); 3) adult wage and salary earnings and indicators of employment and receipt of public assistance income; and 4) an indicator for residence in institutionalized group quarters, a widely used proxy for incarceration.

Limited Impact

I begin the empirical analysis by examining how the funding initiatives affected kindergarten enrollment. The results confirm that funding had a large, immediate impact on kindergarten participation. In the first year in which funding was available, the kindergarten enrollment rate in the typical state was about 15 percentage points higher than would have been the case in the absence of state funding. Two years out, it was 33 percentage points higher, and the lion’s share of gains in kindergarten enrollment from the funding initiative had been achieved. Anecdotal evidence suggests that the take-up of kindergarten was not completely immediate because of shortages of classrooms and teachers rather than because of a gradual increase in local demand. On net, the public school kindergarten enrollment rate of children turning five after an initiative was about 30 percentage points higher than it would otherwise have been.

I next investigate whether these developments were matched by changes in child well-being. Because grade retention and educational attainment were arguably the prime targets of policymakers, I first consider the effects of kindergarten funding on those indicators. Whites had more education as adults as a result of the initiatives, but the effect was quite small: only a 2.5 percent reduction in the dropout rate (see Figure 3). Because the dropout rate among whites prior to kindergarten funding was roughly 15 percent, this reduction amounts to less than half of 1 percentage point, which is a small effect even if we take into account that kindergarten enrollment rose 30 percentage points (not a full 100 percentage points) as a result of the initiatives. College attendance also increased among whites, but by an even smaller amount. The analogous estimates for African Americans suggest that affected children attained lower levels of education. While not statistically significant, the estimates are sufficiently precise to rule out the possibility that African Americans experienced even the small positive gains in educational attainment evident among whites. The apparent gains in educational attainment for whites occurred without significant reductions in grade repetition, either in absolute terms or relative to African Americans.

I then turn to an investigation of the impacts on earnings, employment, public assistance receipt, and the proxy for incarceration described earlier. I again find little evidence that kindergarten funding affected these outcomes. The most notable exception is that whites of kindergarten age after passage of a funding initiative were less likely to reside in prisons or institutionalized group quarters as adults. The effect is relatively large, at 22 percent. Once again, however, no such effects were observed for African Americans. Moreover, I find no evidence of an impact of state kindergarten funding on earnings for individuals of either race. The estimated effects on earnings are imprecise, however, and leave open the possibility that kindergarten attendance had effects on earnings comparable to any other year of education for African Americans and whites alike. In general, the earnings estimates should be viewed with caution, as they could be distorted by the fact that the sample includes some individuals who are young and could still be enrolled in school.

These results remain essentially unchanged when estimated using to a series of alternative approaches, including adding controls for state demographic and labor market conditions. I also perform the analysis separately by gender, which reveals that the effect of kindergarten funding on institutionalization for whites is primarily due to its effect on men, for whom the institutionalization rate is much higher. The magnitude of the effect for white men is similar to that observed for whites overall (a reduction of 23 percent). Among African Americans, there are no effects on institutionalization rates for men or women. The gender-specific results also reveal that kindergarten funding was associated with significantly lower earnings for African American women. To the extent that kindergarten funding displaced African American enrollment in more intensive early education, a possibility that I explore below, these findings would be consistent with recent findings that girls are more responsive to intensive preschool interventions.

Why Did African Americans Not Benefit?

My main results imply that there were some positive impacts of state subsidization of kindergarten, particularly on incarceration rates. What is potentially unexpected, however, is that the funding initiatives appear to have had positive effects only for whites. What might explain these findings? I explore three broad hypotheses for why African Americans might not have benefited as much as whites from the funding initiatives: 1) kindergarten funding disproportionately drew African Americans out of higher-quality education settings; 2) instead of raising additional revenue to fund local kindergarten programs fully, school districts offered lower-quality kindergarten programs to African Americans or moved funds from existing school programs from which African Americans may have disproportionately benefited; and 3) African Americans were more adversely affected by any subsequent “upgrading” of school curricula as more students entered elementary grades having attended kindergarten. The first of these hypotheses receives the most support in the available data.

Data from the Panel Study of Income Dynamics suggest that the introduction of state funding for kindergarten prompted a reduction in Head Start participation among African Americans. The existence of kindergarten funding among all states in a region (relative to none) was associated with a statistically significant 25-percentage-point-reduction in the likelihood that an African American child attended Head Start at age five. Given an enrollment rate of 26 percentage points across the observed cohorts, this estimate implies that state funding for kindergartens essentially eliminated enrollment of African American five-year-olds in Head Start (see Figure 4). By comparison, enrollment of whites in Head Start at age five was much lower (2 percent), and the change in enrollment after the average funding initiative close to zero.

Together with historical accounts of the importance of Head Start in providing education for five-year-olds in the absence of state-funded kindergartens, these estimates strengthen support for the hypothesis that state funding for kindergartens decreased enrollment of African American five-year-olds in federally funded early education for the poor. It is difficult to gauge the extent to which the movement of African American five-year-olds from Head Start to kindergarten might have offset positive impacts of kindergarten attendance elsewhere in the African American population. However, a back-of-the-envelope calculation suggests that the reduction in Head Start attendance among African Americans may account for at least 16 percent of the 1-percentage-point increase in the African American-white gap in high school dropout rates after the initiatives were passed. Head Start has also been found to reduce criminal behavior among African-American males.

I uncover no support for the hypothesis that school districts failing to supplement the state grants placed African American students in lower-quality programs, either in kindergarten or in later grades. I also detect no evidence that the establishment of kindergarten programs as a result of the funding initiatives prompted an increase in academic expectations of students in the early grades, which would have adversely affected children with low levels of achievement. Because the data available to test these alternative hypotheses are not ideal, however, these conclusions must be viewed with caution.

Back to the Future

Although there is great interest among policymakers in extending free early education to disadvantaged children, evidence to date on long-run effects of preschool has been limited to experimental evaluations of model preschools and nonexperimental studies of Head Start. This study has attempted to expand this literature by measuring the long-term effects of a historical episode of public investment in universal early education—the introduction of state funding for public school kindergarten in the 1960s and 1970s. I find evidence that state funding of universal kindergarten lowered high-school dropout and institutionalization rates among whites, but not among African Americans, and detect no impact of state funding for children of either race on grade retention, public assistance receipt, employment or earnings. Why the positive effects for whites occur for dropout and incarceration only is not entirely clear and should be grounds for future research.

These findings complement those of existing research on the long-term effects of targeted programs. First, they suggest that, in the absence of higher-quality alternatives, participation in a low-intensity preschool program may have some limited positive long-term effects. In other words, even a weak program may be better than no program at all, as can be seen in the results for whites. Second, when alternatives already exist for many disadvantaged children, universal programs may not yield additional benefits for that group.

Though there are clear limits to the generalizability of these findings, they do provide some tentative lessons for policymakers. On one hand, the higher rates of preschool participation among children today suggest that any positive long-term effects of extending universal public schooling to four-year-olds may be even smaller than those estimated here for kindergarten. On the other hand, the universal preschool programs being proposed today have a more academic orientation than kindergarten has had, and may therefore have larger impacts on long-term well-being despite significantly “crowding-out” enrollment in other programs. The truth will only be discovered in the years to come.

Elizabeth U. Cascio is assistant professor of economics at Dartmouth College.

Charter schools have become a popular alternative to traditional public schools, with some 5,000 schools now serving more than 1.5 million students, and they have received considerable attention among researchers as a result.

Most studies focus on the effects of charter attendance on short-term student achievement (test scores), using either data sets that follow students over time (see “Results from the Tar Heel State,” research, Fall 2005) or random assignment via school admission lotteries (see “New York City Charter Schools,” research, Summer 2008) to control for differences between students in charter and traditional public schools. Beyond measuring achievement effects, however, there has been only limited analysis of the impacts of charters on the students who attend them. Even less research has been conducted on the effects of charter high schools specifically, though a large portion of all charter schools in the U.S. serve some or all of the high school grades.

Developing a high school model suited to the 21st-century student has been the Holy Grail of education reform in recent years, absorbing governors, task forces, and vast sums spent on small schools, university-based schools, and concept schools (see “High School 2.0,” features). With roughly 30 percent of American students dropping out before receiving a diploma—a rate that has been stable for several decades—assessing existing alternatives to the traditional high school is an urgent task.

In this study we use data from Chicago and Florida to estimate the effects of attending a charter high school on the likelihood that a student will complete high school and attend college. Given the impact of educational attainment on a variety of economic and social outcomes, a positive result could have significant implications for the value of school-choice programs that include charter high schools. We find evidence that charter high schools in both locations have substantial positive effects on both high school completion and college attendance. Controlling for key student characteristics (including demographics, prior test scores, and the prior choice to enroll in a charter middle school), students who attend a charter high school are 7 to 15 percentage points more likely to earn a standard diploma than students who attend a traditional public high school. Similarly, those attending a charter high school are 8 to 10 percentage points more likely to attend college (see Figure 1). Results using an alternative method designed to address concerns about unmeasured differences between students attending charter and traditional public high schools suggest even larger positive effects. Our main results are comparable to those of some studies which find that attending a Catholic high school boosts the likelihood of high school graduation and college attendance by 10 to 18 percentage points.

Methods

Determining the influence of charter school attendance on educational attainment is difficult because students who choose to attend charter high schools may be different from students who choose to attend traditional public high schools in ways that are not readily observable. The fact that the charter students and their parents actively sought out an alternative to traditional public schools suggests the students may be more motivated or their parents more involved in their child’s education than is the case for students attending traditional public schools. Since these traits are not easily measured, the estimated impact of charter high schools on educational attainment could be biased.

Our main analysis uses two methods to address students’ self selection into charter schools. First, we control for any observable differences between charter and non-charter high school students prior to high school entry. These include factors such as race/ethnicity, gender, disability status, and family income. The most important characteristic included among our statistical controls is 8th-grade test score, which aims to capture differences in student ability and students’ educational experiences prior to high school.

Second, we limit our analysis to students who attended a charter school in 8th grade, just prior to beginning high school. That is, we compare high school and postsecondary outcomes for 8th-grade charter students who entered charter high schools (the treatment group) with outcomes for 8th-grade charter students who entered conventional public high schools (the comparison group). If there are unmeasured student or family characteristics that lead to the selection of charter schools in general, these unmeasured characteristics should be relatively constant among students and families who choose charter middle schools. Unlike other nonexperimental studies of charter school impacts, our study therefore addresses student self-selection into charter schools directly by ensuring that the comparison students as well as the treatment students were once charter choosers.

Charter school 8th graders who went on to attend a charter high school differed from their peers who subsequently attended a traditional public high school in several respects, particularly in Florida, which suggests the importance of taking such differences into account when assessing the effects of charter attendance (see Figure 2). However, there may still be unmeasured differences that explain why one charter 8th grader attends a charter high school while another charter 8th grader attends a traditional public high school. For this reason, we estimate charter school effects by comparing students who are more likely to attend a charter school because they live closer to one to those less likely to attend a charter school because it is less convenient. For many charter middle-school students, attending a charter high school may be infeasible due to the lack of a charter high school within a reasonable distance. Such students make different choices not because of unmeasured characteristics, but because of a factor out of their control: the distance from home to the nearest charter school.

Data

The data required to analyze the impact of charter high schools on educational attainment are substantial. One must have data on school type (charter or public) and test scores of individual students prior to high school, individual-level high school attendance records and exit information, and college attendance after high school. Finally, the jurisdiction studied must have a sufficient enrollment of students in charter high schools to provide reliable results. The areas we analyze, the state of Florida and the city of Chicago, are two of just a handful of places where all of the necessary data elements are currently in place.

The Florida data, which cover the four cohorts of 8th-grade students from the school years 1997–98 to 2000–01, come from a variety of sources. The primary source for student-level information is the Florida Department of Education’s K-20 Education Data Warehouse (K-20 EDW), an integrated longitudinal database covering all public school students in the state of Florida. The K-20 EDW includes detailed enrollment, demographic, and program participation information for each student, as well as reading and math achievement test scores.

As the name implies, the K-20 EDW includes student records for both K–12 public school students and students enrolled in community colleges or four-year public universities in Florida. The K-20 EDW also contains information that allows us to follow students who attend private institutions of higher education within Florida. Data from the National Student Clearinghouse, a national database that includes enrollment data on 3,300 colleges from throughout the United States, is used to track college attendance outside the state of Florida. Any individual who does not show up as enrolled in a two- or four-year college or university is classified as a non-attendee.

High school graduation is measured using withdrawal information and student award data from the K-20 EDW. Only students who receive a standard high school diploma are considered to be high school graduates. Students earning a GED or special education diploma are counted as not graduating. Similarly, students who withdrew with no intention of returning or left for other reasons, such as nonattendance, court action, joining the military, marriage, pregnancy, and medical problems, but did not later graduate, are counted as not graduating.

The Chicago data, which cover the five cohorts of students who were in 8th grade during the school years 1997–98 to 2001–02, were obtained from the Chicago Public Schools. The data include 8th-grade math and reading test scores and information on student gender, race/ethnicity, bilingual status, free or reduced-price lunch status, and special education status. This data set is also linked to the National Student Clearinghouse. High school graduation is determined by withdrawal information from the Chicago Public Schools data. As in Florida, only students who receive a standard high school diploma are considered to be high school graduates.

Results

The raw data on our study population of students who were in charter schools in 8th grade reveal substantial differences in educational attainment between attendees of charter high schools and those of traditional public high schools. In Florida, 57 percent of students who went from a charter school in 8th grade to a traditional public school in 9th grade received a standard high school diploma within four years, compared to 77 percent of charter 8th graders who attended a charter high school. In Chicago, the corresponding high school graduation rates were 68 and 75 percent. Similar differences are found for college attendance. In Florida, among the study population of charter 8th graders, 57 percent of students attending a charter school in 9th grade went to either a two- or four-year college within five years of starting high school, whereas among students who started high school in a traditional public school the college attendance rate was only 40 percent. In Chicago, the gap in college attendance is smaller but still sizable: among the study population of charter 8th graders, 49 percent of students at charter high schools attended college, compared to 38 percent of students at traditional public high schools.

Controlling for student demographics, 8th-grade test scores, English language skills, special education program participation, free or reduced-price lunch status (a measure of family income), and mobility during middle school does not alter the basic patterns of graduation and college attendance seen in the descriptive comparisons. The estimated impact of attending a charter high school on the probability of obtaining a high school diploma is positive in both Florida and Chicago. In Chicago, students who attended a charter high school were 7 percentage points more likely to earn a regular high school diploma than their counterparts with similar characteristics who attended a traditional public high school. The graduation differential for Florida charter schools was even larger, at 15 percentage points. The findings for college attendance are remarkably similar in Florida and Chicago. Among the study population of charter 8th graders, students who attended a charter high school in 9th grade are 8 to 10 percentage points more likely to attend college than similar students who attended a traditional public high school (see Figure 1).

As discussed above, there remains the possibility that unobserved changes occur between 8th and 9th grade that influence both high school choice and subsequent educational attainment. For example, dissatisfaction with performance in a charter middle school that is not captured by test scores (such as discipline issues or a poor fit between the student’s interests or ability and the curriculum being offered) could lead parents to choose to send their child to a traditional public high school. When we correct for this potential bias by examining students who attended charter or traditional public school based on proximity, we continue to find highly significant positive effects of attending a charter high school on both receipt of a high school diploma and college enrollment. The magnitude of the effects is large, roughly double the size of our main results.

This pattern suggests that, among students enrolled in charter schools as 8th graders, it is those who are less likely to graduate who are choosing to attend charter high schools. We can only speculate as to why this is so. It is possible that parents whose children are at risk of dropping out are more likely to choose charter high schools in a belief that the traditional public school environment would make it more likely that their child leaves school early. Alternatively, although we control for free or reduced-price lunch eligibility, it may be the case that low-income families have a stronger preference for charter schools. If so, families with children in charter high schools would be less likely to be able to afford to send their children to college.

Possible Mechanisms

The analyses reported above cannot explain how or why charter high schools appear to produce positive effects on their students’ educational attainment. Our study lacks data on operations and instruction in the charter schools, so we have little opportunity to explore the mechanisms contributing to their success. Nonetheless, we have a few pieces of information that permit exploratory analyses of factors that might play a role.

First, it is worth considering that charter high schools may raise rates of high school graduation and college enrollment directly, or indirectly through improved academic achievement. We attempt to distinguish between these explanations by controlling in the analysis for math and reading achievement as measured in the 10th grade. Controlling for 10th-grade test scores explains about half the graduation differential for charter high schools in Florida but less than 20 percent of the difference in Chicago. And it has an even smaller effect on the results for college enrollment, reducing the estimated effect of charter school attendance by only about 10 percent in both locations. These patterns suggest that the positive effects of charter school attendance on educational attainment are not due solely to measured differences in the achievement of students in charter and traditional public high schools. This result is similar to those found in some studies of Catholic high schools, which suggest larger benefits for attainment than for test scores.

Second, given that charter high schools tend to be much smaller than traditional public high schools, charter school effects might simply be attributable to their smaller size. In order to assess this possibility, we ran the analyses for high school graduation and college attendance again with an additional control for the total number of students attending the school. The results are comparable to those reported above, indicating that the estimated effects of charter high schools are not due to differences in school size.

Third, we consider the possibility that the charters’ success might be related to grade configurations that often differ from those of traditional public schools. In the traditional public school sector in both Chicago and Florida, high schools are almost always separate from middle schools. This is not the case for charter schools. In 2001–02, about 22 percent of charter schools in Florida offering middle-school grades also offered some or all high-school grades. As a result, about 30 percent of Florida charter 8th-grade students attended schools that also offered at least some high-school grades. In Chicago, 40 percent of charter middle schools offered both middle- and high-school grades, and nearly half of the 8th-grade charter students could attend at least some high-school grades without changing schools. This raises the possibility that the measured effects of attending a charter high school on educational attainment could simply reflect advantages of grouping middle and high school grades together, thereby creating greater continuity for students and eliminating the disruption often associated with changing schools.

In order to examine whether charter-school effects might be attributable to eliminating the transition between middle and high school, we restricted the Florida analysis to those students whose 8th-grade charter school did not offer 9th grade and ran our analyses again. For high school graduation, restricting the sample produces estimates that are nearly identical to the original estimates from our main method. Using the restricted sample and our alternative method, the estimates are about 30 percent smaller than when the full sample is employed, but still large. Meanwhile, estimates of the effect of attending a charter high school on college enrollment are even larger using the restricted sample than with the original sample that includes schools offering both 8th and 9th grade. In Florida, grade configuration is not a primary driver of the estimated positive effects of charter high schools on attainment. In Chicago, however, we could not run similar analyses because grade configuration is too strongly correlated with charter status; we therefore cannot rule out the possibility that positive results in Chicago could be partly attributable to eliminating the transition from middle school to high school.

Finally, we examined an interpretive concern arising from the fact that some charter schools in Florida are former traditional public schools that converted to charter status. If conversion schools were better-than-average traditional public schools to begin with, they may be distorting the estimated impact of charters on educational attainment. We calculated separate effects for Florida conversion and non-conversion (“de novo”) charters in Florida. (In Chicago, virtually all of the charter high schools in our sample were de novo charters). We found that although Florida’s conversion charters have significantly greater effects on high school graduation than do de novo charters, the impact of non-conversion charters is still sizable (nearly equal to the estimate in Chicago). For college attendance, the estimated positive impacts of Florida’s de novo charters are statistically indistinguishable from the estimated positive impacts of Florida’s conversion charters.

Conclusions

Although a number of recent studies analyze the relationship between charter school attendance and student achievement, this is the first analysis of the impacts of charter school attendance on educational attainment. We find that charter schools are associated with an increased likelihood of successful high-school completion and an increased likelihood of enrollment at a two- or four-year college in two disparate jurisdictions, Florida and Chicago. The reasons for these large charter-school effects are not clear. There is certainly room for future work to explore how differences in curricula, expectations, peer characteristics, and other factors may cause charter schools to diminish the high-school dropout rate and ease the transition to postsecondary schooling.

Our findings are consistent with some research on the efficacy of Catholic schools, which finds substantial positive effects of attending a Catholic high school on educational attainment. While just a first step, the results presented here and in the Catholic-school literature suggest that school-choice programs that include alternatives to traditional public high schools may reduce high-school dropout rates and promote college attendance.

Kevin Booker is researcher at Mathematica Policy Research, Inc. Tim R. Sass is professor of economics at Florida State University. Brian Gill is senior social scientist at Mathematica Policy Research, Inc. Ron Zimmer is associate professor at Michigan State University. This article is adapted from research reported in Charter Schools in Eight States (RAND Corporation, 2009).

Researchers have recently begun to learn more about the effects of time spent on learning from natural experiments around the country. This new body of evidence, to which we have separately contributed, suggests that extending time in school would in fact likely raise student achievement. Below we review past research on this issue and then describe the new evidence and the additional insights it provides into the wisdom of increasing instructional time for American students.

We also discuss the importance of recognizing the role of instructional time, explicitly, in accountability systems. Whether or not policymakers change the length of the school year for the average American student, differences in instructional time can and do affect school performance as measured by No Child Left Behind. Ignoring this fact results in less-informative accountability systems and lost opportunities for improving learning outcomes.

Emerging Evidence

More than a century ago, William T. Harris in his 1894 Report of the Commissioner [of the U.S. Bureau of Education] lamented,

The boy of today must attend school 11.1 years in order to receive as much instruction, quantitatively, as the boy of fifty years ago received in 8 years…. It is scarcely necessary to look further than this for the explanation for the greater amount of work accomplished…in the German and French than in the American schools.

The National Education Commission on Time and Learning would echo his complaint one hundred years later. But the research summary issued by that same commission in 1994 included not one study on the impact of additional instruction on learning. Researchers at that time simply had little direct evidence to offer.

The general problem researchers confront here is that length of the school year is a choice variable. Because longer school years require greater resources, comparing a district with a long school year to one with a shorter year historically often amounted to comparing a rich school district to a poor one, thereby introducing many confounding factors. A further problem in the American context is that there is little recent variation in the length of school year. Nationwide, districts generally adhere to (and seldom exceed) a school calendar of 180 instructional days. And while there was some variation in the first half of the 20th century, other policies and practices changed simultaneously, making it difficult to uncover the separate effect of changes in instructional time.

Among the first researchers to try to identify the impact of variation in instructional time were economists studying the effect of schooling on labor market outcomes such as earnings. Robert Margo in 1994 found evidence suggesting that historical differences in school-year length accounted for a large fraction of differences in earnings between black workers and white workers.

Using differences in the length of the school year across countries, researchers Jong-Wha Lee and Robert Barro reported in 2001 that more time in school improves math and science test scores. Oddly, though, their results also suggested that it lowers reading scores. In 2007, Ozkan Eren and Daniel Millimet examined the limited variation that does exist across American states and found weak evidence that longer school years improve math and reading test scores.

Work we conducted separately in 2007 and 2008 provides much stronger evidence of effects on test scores from year-to-year changes in the length of the school year due to bad weather. In a nutshell, we compared how specific Maryland and Colorado schools fared on state assessments in years when there were frequent cancellations due to snowfall to the performance of the very same schools in relatively mild winters. Because the severity of winter weather is inarguably outside the control of schools, this research design addresses the concern that schools with longer school years differ from those with shorter years (see research design sidebar).

While our studies use data from different states and years, and employ somewhat different statistical methods, they yield very similar results on the value of additional instructional days for student performance. We estimate that an additional 10 days of instruction results in an increase in student performance on state math assessments of just under 0.2 standard deviations. To put that in perspective, the percentage of students passing math assessments falls by about one-third to one-half a percentage point for each day school is closed.

Other researchers have examined impacts of instructional time on learning outcomes in other states, with similar results. For example, University of Virginia researcher Sarah Hastedt has shown that closures that eliminated 10 school days reduced math and reading performance on the Virginia Standards of Learning exams by 0.2 standard deviations, the same magnitude we estimate for the neighboring state of Maryland. Economist David Sims of Brigham Young University in 2008 took advantage of a 2001 law change in Wisconsin that required all school districts in that state to start after September 1. Because some districts were affected while others were not, he was also able to provide unusually convincing evidence on the effect of changes in the number of instructional days. He found additional instruction days to be associated with increased scores in math for 4th-grade students, though not in reading.

Collectively, this emerging body of research suggests that expanding instructional time is as effective as other commonly discussed educational interventions intended to boost learning. Figure 1 compares the magnitude of the effect of instructional days on standardized math scores to estimates drawn from other high-quality studies of the impact of changing class size, teacher quality, and retaining students in grade. The effect of additional instructional days is quite similar to that of increasing teacher quality and reducing class size. The impact of grade retention is comparable, too, though that intervention is pertinent only for low-achieving students.

Although the evidence is mounting that expanding instructional time will result in real learning gains, evidence on the costs of extending the school year is much scarcer and involves a good deal of conjecture. Perhaps the best evidence comes from a recent study in Minnesota, which estimated that increasing the number of instructional days from 175 to 200 would cost close to $1,000 per student, in a state where the median per-pupil expenditure is about $9,000. The total annual cost was estimated at $750 million, an expense that proved politically and financially infeasible when the proposal was recently considered in that state. Comparing costs of expanding instructional days with the costs of other policy interventions will be an analytic and policy exercise of real importance if the call for expanded instructional time is to result in real change.

Complicating this analytic task are differences in costs that exist across schools and states. Utilities, transportation, and teacher summer-labor markets vary widely across geographic areas, and all affect the cost of extending the school year. So, while the benefits of extending the school year may exceed the costs in some states or school districts, they may not in others. A further complication is the possibility of diminishing returns to additional instructional time. Our research has studied the effect of additional instructional days prior to testing, typically after approximately 120 school days. The effect of extending instructional time into the summer is unknown. Also, our research has focused on the variation in instructional days prior to exams, or accountable days. The effect of adding days after exams could be quite different.

Costs of extending school years are as much political as economic. Teachers have come to expect time off in the summer and have been among the most vocal opponents of extending school years in several locations. Additional compensation could likely overcome this obstacle, but how much is an unresolved and difficult question.

Teachers are not the only ones who have grown accustomed to a summer lasting from June through August. Students and families have camps, vacations, and work schedules set up around summer vacation. “Save Our Summers” movements have for years decried the benefits of additional instructional days and proclaimed the benefits of summer vacation, and the movements have grown as states have considered extending the school year and individual school districts have moved up their start dates. Longer school years might reduce tourism and its accompanying tax revenue. These additional costs likely vary by state and district, but are clearly part of the analytic and political calculus.

Time and Accountability

As education policymakers consider lengthening the school year and face trade-offs and uncertainties, it is important to recognize that expanding instructional time offers both opportunities and hazards for another reform that is well established, the accountability movement. Educators, policymakers, parents, and economists are sure to agree that if students in one school learn content in half the time it takes comparable students at another school to learn the same content, the first school is doing a better job. How students would rank these schools is equally obvious. Yet state and federal accountability systems do not account for the time students actually spent in school when measuring gains, and so far have no way of determining how efficiently schools educate their students.

One implication of this oversight is that accountability systems are ignoring information relevant to understanding schools’ performance. Year-to-year improvements in the share of students performing well on state assessments can be accomplished by changes in school practices, or by increases in students’ exposure to school. Depending on the financial or political costs of extending school years, those with a stake in education might think differently about gains attributable to the quality of instruction provided and gains attributable to the quantity.

To see how the contributions of these inputs might be separated, consider data from Minnesota. Between 2002 and 2005, 3rd graders in that state exhibited substantial improvements in performance on math assessments, a fact clearly reflected by Minnesota’s accountability system. But during that period, there was substantial year-to-year variation in the number of instructional days students had prior to the test date. In Figure 2, we plot both the reported test scores for Minnesota 3rd graders (the solid line) and the number of days of instruction those students received (the bars). Useful, and readily calculated, is the time series of test scores, adjusting for differences in the number of instructional days (the dotted line).

Comparing the reported and adjusted scores is useful for at least two reasons. First, it illustrates the role of time as a component of test gains. Overall, scale scores increased by 0.4 standard deviations from 2001–02 to 2004–05. Of this increase, a large portion was attributable to expansion in instructional time prior to the test date. Adjusting for the effect of instructional days, we estimate that scores increased by roughly 0.25 standard deviations, nearly 40 percent less than the reported gains.

Second, the comparatively steady gain in adjusted scores over the period provides evidence of improvements in instructional quality, independent of changes in the amount of time students were in class. The fast year-to-year increases in the first and last periods result in large part from increases in the amount of time in school, while the negligible change in overall scores between 2003 and 2004 does not pick up real gains made despite a shortened school year. Adjusted scores pick up increases in learning gains attributable to how schools used instructional time, such as through changing personnel, curricula, or leadership. The point here is that time-adjusted scores provide information that is just as important as the overall reported scores for understanding school improvements. A robust accountability system would recognize that more instructional time can be used to meet goals, but that more time is neither a perfect substitute for, nor the same thing as, better use of time.

The Hazards of Ignoring Time

Failing to account for the role of time in student learning not only means missed opportunity, it also creates potential problems. First, it can allow districts to game accountability systems by rearranging school calendars so that students have more time in school prior to the exam, even as the overall length of the school year remains constant. Beginning in the 1990s, districts in a number of states began moving start dates earlier, with many starting just after the first of August. The question arose whether these changes might be linked to pressures on districts to improve performance on state assessments. David Sims showed that Wisconsin schools with low test scores in one year acted strategically by starting the next school year a bit earlier to raise scores. Evidence of gaming soon emerged in other states as well. Wisconsin passed its 2001 law requiring schools to begin after September 1 to prevent such gaming; similar laws were recently passed in Texas and Florida.

The motives driving earlier start dates could spill over into other instructional policies. Minnesota moved its testing regimen from February to April in the wake of accountability standards, while Colorado legislators have proposed moving their testing window from March into April, with advocates suggesting that the increased time for instruction would make meeting performance requirements under No Child Left Behind more feasible for struggling schools. While administering the test later in the year has potential benefits in measured performance, grading the tests over a shorter time frame costs more, estimated at some $3.9 million annually in Colorado. Schools thus sacrifice educational inputs (such as smaller classes or higher teacher salaries) to pay for the later test date.

A second hazard involves fairness to schools at risk of being sanctioned for poor performance: these schools can face longer odds if weather or other schedule disruptions limit school days. The impact of instructional time on learning means that one factor determining the ability of schools to meet performance goals is not under the control of administrators and teachers. We illustrate the effects of time on making adequate yearly progress (AYP) as defined by No Child Left Behind by comparing the performance of Maryland schools the law identified as underperforming to estimates of what the performance would have been had the schools been given a few more days for instruction.

We begin with data from all elementary schools in Maryland that did not make AYP in math and reading during the 2002–03 to 2004–05 school years. We adjust actual performance by the number of days lost in a given year multiplied by the marginal effect of an additional day on test performance as reported in Marcotte and Hemelt’s study of Maryland schools. This allows us to estimate what the proficiency rates in each subject would have been had those schools been open for all scheduled instructional days prior to the assessment. We then compare the predicted proficiency rate to the AYP threshold.

We summarize the results of this exercise in Figure 3. The light bars represent the number of schools failing to make AYP in math and reading in various years. The dark bars are the number of those schools that we predict would have failed to make AYP if the schools had been able to meet on all scheduled days. We make these estimates assuming that low-performing schools would have made average gains with each additional day of instruction.

The average number of days lost to unscheduled school closings varied substantially over the period, from more than 10 to fewer than four and a half. Many schools that did not make AYP likely would have had they not lost so many school days. For example, we estimate that 35 of the 56 elementary schools that did not make AYP in math in 2002–03 would have met the AYP criterion if they had been open during all scheduled school days. Even if these schools were only half as productive as the typical school, 24 of the 56 flagged schools would likely have made AYP if they had been open for all scheduled days.

There is, however, a way to reduce risks like these for schools and to limit incentives for administrators to move start or test dates at the same time: that is to recognize and report time as an input in education. A simple and transparent way to do this is for state report cards, which inform parents about school outcomes and summarize the information on AYP status, to include information about the number of instructional days at test date as well as the total number of instructional days for the year. This information is readily available and already monitored by schools, districts, and states. Local and state education authorities could use it when assessing performance, for example, in hearing an appeal from a school that failed to meet its AYP goals. Further, this information could be used to estimate test scores adjusted for instructional days, to be used alongside unadjusted changes in performance. Distinguishing between gains due to expanded instruction time and better use of that time can enrich accountability systems and provide more and better information to analysts and the public alike.

Looking Ahead

There can be no doubt that expanding the amount of time American students spend in school is an idea popular with many education policymakers and has long been so. What makes the present different is that we now have solid evidence that anticipated improvements in learning will materialize.

Practical obstacles to the extension of the school year include substantial expense and stakeholder attachment to the current school year and summer schedule. The benefits of additional instructional days could diminish as school years are lengthened. Further, it is unknown how teachers would use additional instructional days if they are provided after annual testing is already finished. Simply extending the year well after assessments are given might mean that students and teachers spend more days filling (or killing) time before the end of the year. This would make improvements in learning unlikely, and presumably make students unhappy for no good reason.

Though the issue has seen little movement in the past and faces real opposition going forward, the policy climate appears likely to be favorable once the fiscal challenges now facing public school systems recede. It is our hope that policymakers and administrators who try to take advantage of this window of opportunity don’t harm reforms that have succeeded in improving learning outcomes and don’t implement reforms in a manner that would fail to do the same. Advocates for extended school years have so far said virtually nothing about whether or how accountability systems should accommodate longer school years.

Across the country, a small number of schools and districts are modifying or extending the academic year. The Massachusetts 2020 initiative has provided resources for several dozen schools to increase the number of instructional days they offer from 180 to about 200. Other examples include low-performing schools that have lengthened their school day in an effort to improve, and the longer school days, weeks, and years in some charter schools. However, such initiatives remain rare, with no systemic change in the instructional time provided to American students. Our work confirms that increasing instructional time could have large positive effects on learning gains. Encouraging schools and districts to view the school calendar as a tool in the effort to improve learning outcomes should be encouraged in both word and policy.

Dave E. Marcotte is professor of public policy at the University of Maryland, Baltimore County. Benjamin Hansen is a research associate at IMPAQ International, LLC.

Debates over tracking reached their high point in the United States in the 1990s. An influential report published in 1998 by the Thomas B. Fordham Foundation argued that the available research did not support the contention that tracking doomed impoverished students to inferior schooling, nor did it support universal adoption of the practice. Over the last decade, patterns in grouping students have changed markedly in the U.S.; high school students are no longer placed in rigidly defined general-education or noncollege tracks but have the flexibility to move between course levels for different subjects. These changes may have assuaged some critics, but the broader debate over tracking remains unsettled.

The central challenge in measuring the effect of tracking on performance is that schools that track students may be different in many respects from schools that do not. For example, they may attract a different pool of students and possibly a different pool of teachers. The ideal situation to assess the impact of tracking on test scores of different groups of students would be one in which students were assigned to tracking or nontracking schools randomly, and the performance of students could be compared across school types.

We shed light on these issues using data from Kenya. In 2005, each of 140 primary schools in western Kenya received funds from the nongovernmental organization International Child Support (ICS) Africa to hire an extra teacher. One hundred twenty-one of these schools had a single 1st-grade class and used the new teacher to split the students into two classes. In 61 randomly selected schools, students were assigned to classes based on prior achievement as measured by test scores. In the remaining 60 schools, students were randomly assigned to one of the two classes, without regard to their prior academic performance.

The results showed that all students benefited from tracking, including those who started out with low, average, and high achievement. At the tracking schools, the test scores of students who started out in the middle of their class do not seem to be affected by which section (top or bottom) the students were later assigned to. In other words, any negative effects of being with lower-achieving peers were more than offset in tracked settings by the benefit of the teacher being able to better tailor instruction to students’ needs.

Primary Education in Kenya

The Kenyan education system includes eight years of primary school and four years of secondary school. Like many other developing countries, Kenya has recently made rapid progress toward the goal of universal primary education. After the elimination of school fees in 2003, primary school enrollment rose nearly 30 percent, from 5.9 million in 2002 to 7.6 million in 2005. This is typical of what is happening in sub-Saharan Africa overall, where the number of new entrants to primary school increased by more than 30 percent between 1999 and 2004.

This progress creates its own new challenges, however. Pupil-teacher ratios have grown dramatically, particularly in lower grades. In our sample of schools in western Kenya, the median 1st-grade class in 2005 (after the introduction of free primary education, but before the class-size-reduction program we study here) had 74 students and the average class size was 83. These classes are heterogeneous in a number of ways: Students differ vastly in age, school readiness, and support at home. Many of the new students are first-generation learners and have not attended preschools, which are neither free nor compulsory in Kenya. These challenges are not unique to Kenya; they confront many developing countries where school enrollment has risen sharply in recent years. Understanding the roles of tracking and peer effects in this type of environment is thus critically important.

Our results are most likely to be directly applicable to settings where classes are large, the student population is heterogeneous, and few additional resources are available to teachers. It is unclear whether similar results would be obtained in different contexts, such as developed countries, where smaller class sizes may allow more tailored instruction even without tracking, and extra resources, such as remedial education, computer-assisted learning, and special education programs, may already provide tools to help teachers deal with different types of students.

Design of the Experiment

This study takes advantage of a class-size-reduction program and evaluation that involved primary schools in Bungoma and Butere-Mumias in Western Province, Kenya. Of 210 primary schools in these districts, 140 schools were randomly selected to participate in the Extra-Teacher Program. With funding from the World Bank, ICS Africa provided each of the 140 selected schools with funds to hire an additional 1st-grade teacher on a contractual basis starting in May 2005, the beginning of the second term of that school year. Most of the schools (121) had only one 1st-grade class, which was split into two classes when the new teacher was hired. The 19 schools that already had two or more 1st-grade classes added another class.

It is important to note that the incentives facing the newly hired teachers differed from those facing civil-service teachers already working in program schools. The new teachers had clear incentives to work hard to increase their chances of having their short-term contracts renewed and of eventually being hired as civil-service teachers—a desirable outcome in a society where government jobs are highly valued. In contrast, the difficulty of firing civil-service teachers implies that they had weak extrinsic incentives and may be more sensitive to factors affecting their intrinsic motivation.

Average class size was reduced from 84 to 46 students in the 140 schools that received funds for a new teacher. The program continued for 18 months, which included the last two terms of 2005 and the entire 2006 school year, and the same cohort of students remained enrolled in the program.

From the 121 schools that had originally only one 1st-grade class, 60 schools were randomly selected to assign students to one of the two classes by chance. We call these schools the “nontracking schools.” In the remaining 61 schools (the “tracking schools”), the children were divided into two sections according to their scores on exams administered by the school during the first term of the 2005 school year. The 50 percent of the class with the lowest exam scores were assigned to one section (the “bottom class”) and the rest were assigned to the other (the “top class”).

After students were assigned to classes, the contract teacher and the civil-service teacher were also randomly assigned to classes. In the second year of the program, all children not repeating the grade remained assigned to the same group of peers and the same teacher.

Data

Our initial sample consists of approximately 10,000 students enrolled in 1st grade in March 2005 in one of the 121 primary schools participating in the study. The outcome of interest is student academic achievement, as measured by scores on a standardized math and language test first administered in all schools 18 months after the start of the program. Trained proctors administered the test, which was then graded blindly by data processors. In each school, 60 students (30 per class) were drawn from the initial sample to participate in the tests. If a class had more than 30 students, students were randomly sampled.

The test was designed by a cognitive psychologist to measure a range of skills students may master by the end of 2nd grade. One part of the test was written and the other part oral, administered one-on-one. Students answered math and literacy questions ranging from counting and identifying letters to subtracting three-digit numbers and reading and understanding sentences.

To limit attrition from the experiment, proctors were instructed to go to the homes of sampled students who had dropped out or were absent on the day of the test and to bring them to school for the test. It was not always possible to find the child, however, and the resulting attrition rate on the test was 18 percent. However, there was no difference between tracking and nontracking schools in overall attrition rates. In total, we have postintervention test-score data for 5,796 students.

In addition, each school received unannounced visits several times during the course of the study. During these visits enumerators checked, upon arrival, whether teachers were present in school and whether they were in class and teaching, and then took a roll call of the students.

To measure whether the effects of the program persisted, the children who had been sampled for the first postintervention test were tested again in November 2007, one year after the program ended. During the 2007 school year, these students were overwhelmingly enrolled in grades for which their school had a single class, so tracking was no longer an option. Most of these students had reached 3rd grade by that time, but those repeating an earlier grade were also tested. The attrition rate for this portion of the experiment was 22 percent. Neither the proportion nor the characteristics of children who could not be tested differed between the tracking and nontracking schools.

The Impact of Tracking

We estimate the impact of tracking on student achievement by comparing the postintervention (18 months after the experiment began) test scores of students in the tracking and nontracking schools. Taking the average of students’ scores on math and literacy exams, we find that students in tracking schools scored 0.14 standard deviations higher than students in nontracking schools overall. When we adjust the comparison to take into account minor differences in student characteristics across the two groups of schools, the effect increases to 0.18 standard deviations. There was no significant difference between the impact of the program on math and literacy scores when we examined the subjects separately.

How large were these effects? A typical student with a literacy score one standard deviation above that of the average student could correctly spell 5.5 of 10 words included on the exam, while the average student could spell only two. Similarly, students with a math score one standard deviation above the average were able to perform single-digit multiplications, whereas those at the mean could not. The average effect of tracking was roughly one-fifth the size of these performance differences.

These gains persisted beyond the duration of the program (see Figure 1). When the program ended, most students had reached 3rd grade, and all but five schools had only one 3rd-grade class. The remaining students had repeated and were in 2nd grade where, once again, most schools had only one large class, since after the program ended they did not have funds for additional teachers. Even so, the test scores of students in tracking schools remained 0.16 standard deviations higher than those of students in nontracking schools overall (and 0.18 standard deviations higher with control variables). The persistence of the benefits of tracking is striking, as many evaluations find that the test-score effects of successful interventions fade over time. It seems that tracking helped students master core skills in 1st and 2nd grade that in turn helped improve their learning later on.

We also examine whether the effect of tracking differs between initially high-scoring students (who are grouped with other strong students in tracking schools) and initially low-scoring students (who are grouped with other low-scoring students in tracking schools). We find that both groups of students benefited from tracking, and by approximately the same amount. A year after the intervention ended, the effect persisted for both the top and bottom classes.

Tracking increases test scores for students taught by contract teachers. In fact, students initially scoring low who were assigned to contract teachers benefited even more from tracking than students who initially scored high. But students who initially scored low showed only a small and statistically insignificant benefit if assigned to a civil-service teacher. In contrast, tracking substantially increased scores for students who initially scored high and were assigned to a civil-service teacher. Below we discuss other evidence that tracking led civil-service teachers to increase effort when they were assigned to high-scoring students but not when assigned to low-scoring students.

Changes in Peer Achievement

Data from the tracking schools allow us to estimate the effect of being taught with a higher-achieving vs. lower-achieving peer group by comparing students with baseline test scores in the middle of the distribution. Because of the way tracking was done (splitting the grade into two classes at the median baseline test score), the two students closest to the median within each school were assigned to classes where the average prior achievement of their classmates was very different.

By comparing pairs of students right around the cutoff, we can estimate the effect of being the lowest-achieving child in the class compared to being the highest-achieving student in the class. We find that, despite the large gap in average peer achievement (1.6 standard deviations in baseline test scores) between the top and bottom classes, the students just below the cutoff have postintervention test scores similar to students just above the cutoff. Moreover, when we compare students around the cutoff at the tracking schools with students of similar ability at the nontracking schools, we find that students at the tracking schools score higher at the end of the intervention than the comparable students in the nontracking schools. These results imply that being the best student in a class of relatively weak students and being the worst student in a class of relatively strong students are both better than being the middle student in a heterogeneous class. This evidence suggests that students benefit from homogeneity because the teacher does not need to spend time addressing the needs of students performing at widely varying levels.

Learning from Peers vs. Learning from Teachers

We took a separate look at students in schools where students were not tracked but instead assigned to classes randomly. The random assignment of students and teachers within these schools made it possible to see whether and how peer achievement affected the performance of individual students when education took place in an untracked setting. We found that it did. If peer achievement was higher—0.10 standard deviations higher, to be exact—students learned 0.04 standard deviations more than they would have otherwise.

These results, taken together with those reported earlier, indicate that peer influence depends on whether or not classes are tracked. In untracked classes, where there is considerable heterogeneity of performance, students learn less if their peers are lower performing. At least in this particular setting, however, the homogeneous classes that are created by tracking seem to allow the teacher to deliver instruction at a level that reaches all students, thus offsetting the effect of having lower-performing peers. Interestingly, combining the direct effect of peer achievement with the fact that the median children in each school did not suffer from being assigned to the bottom track suggests that teachers focus their attention not on the median student in the class, but at students considerably above the median.

Why Did Tracking Work?

Two additional pieces of evidence shed light on the question of why tracking had such clear benefits. First, we look at teacher presence and effort. Do they spend more time in class and teaching? Then, we examine whether the test-score gains in tracking schools were concentrated among simpler or more complex tasks and whether this varied by students’ initial achievement levels. Our results confirm that students in tracked classes seem to have benefited from more-focused teaching and perhaps also from greater teacher effort.

Teacher absence is a major problem in Kenya, as in many developing countries. Only 59 percent of teachers were in class and teaching during unannounced visits to a comparable sample of schools that did not receive an additional teacher. Overall, teachers in tracking schools were 9.6 percentage points more likely to be found in school and teaching during random spot checks than their counterparts in nontracking schools, who were present and teaching only about half of the time. There were, however, large differences across teachers. The contract teachers were much more likely to be found in school and teaching (74 percent versus 45 percent for the civil-service teachers), and their absence rate was unaffected by tracking (see Figure 2). The civil-service teachers were 10 percentage points more likely to be in schools and teaching in tracking schools than in nontracking schools when they were assigned to the top class. This difference is statistically significant and amounts to a 25 percent increase in teaching time. However, the difference between tracking and nontracking school types was smaller and statistically insignificant for civil-service teachers assigned to the bottom classes.

These results suggest that teachers may be more motivated to teach a group of students with high initial scores than a group with low initial scores or a heterogeneous group. Recall that students assigned to the top class with a civil-service teacher benefited more from tracking than those assigned to the bottom class with a civil-service teacher. Increased teacher effort may help explain this pattern.

Another hypothesis consistent with both the tracking results and the effects from random peer assignment is that tracking by initial achievement improves student learning because it allows teachers to focus instruction. Teaching a more homogeneous group of students might allow teachers to adjust the material covered and the pace of instruction to students’ needs. For example, a teacher might begin with more basic material and instruct at a slower pace, providing more repetition and reinforcement, when students are initially less prepared. With a group of initially higher-achieving students, the teacher can increase the complexity of the tasks and pupils can learn at a faster pace. With a heterogeneous group, they may be compelled to cover both simple and advanced material, spending less time on each, which would hurt all students.

One way to examine this is to see whether children with different initial achievement levels gained from tracking differentially in terms of the difficulty of the material that they learned. While the results for language are mixed, the estimates for math suggest that, although the total effect of tracking on children in the bottom class is significantly positive for all levels of difficulty, these children gained from tracking more than other students on the easier questions and less on the more-difficult questions. Conversely, students assigned to the top class benefited less on the easier questions, and more on the more-difficult questions. In fact, they did not significantly benefit from tracking for the easier questions, but they did significantly benefit from it for the more-difficult questions. These results suggest that tracking helped by giving teachers the opportunity to focus on the competencies that children were not mastering.

Conclusion

A central challenge of education systems in developing countries—the context for which our results are most relevant—is that students in the same grades and classrooms are extremely diverse. Our results show that grouping students by preparedness or prior achievement and focusing the teaching material at the most appropriate level could potentially have large positive effects with little or no additional resource cost. One could also target more resources to the weaker group, further helping them to catch up with their more-advanced counterparts. It is often suggested that there is a trade-off between the value of targeting resources to weaker students, and the costs imposed on them by separating them from stronger students. We find no evidence for such a trade-off in this context.

Our results may also have implications for debates over school choice and voucher systems. A common criticism of such programs is that they may hurt some students if they lead to increased sorting of students by initial achievement and if all students benefit from having peers with higher initial achievement. If tracking is indeed beneficial, this is less of a concern.

Esther Duflo is professor of economics at the Massachusetts Institute of Technology. Pascaline Dupas is assistant professor of economics at University of California, Los Angeles. Michael Kremer is professor of economics at Harvard University.

An interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future is available here.

School choice supporters, including hundreds of private school students in crisp uniforms, filled Washington, D.C.’s Freedom Plaza last May to protest a congressional decision to eliminate the city’s federally funded school voucher program after the next school year (to see additional images of this event please click here). That afternoon, President Obama announced a compromise proposal to grandfather the more than 1,700 students currently in the District of Columbia Opportunity Scholarship Program, funding their vouchers through high school graduation, but denying entry to additional children. Both program supporters and opponents cite evidence from an ongoing congressionally mandated Institute of Education Sciences (IES) evaluation of the program, for which I am principal investigator, to buttress their positions, rendering the evaluation a Rorschach test for one’s ideological position on this fiercely debated issue.

School vouchers provide funds to parents to enable them to enroll their children in private schools and, as a result, are one of the most controversial education reforms in the United States (to see an interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future please click here). Among the many points of contention is whether voucher programs in fact improve student achievement. Most evaluations of such programs have found at least some positive achievement effects, but not always for all types of participants and not always in both reading and math. This pattern of results has so far failed to generate a scholarly consensus regarding the beneficial effects of school vouchers on student achievement. The policy and academic communities seek more definitive guidance.

The IES released the third-year impact evaluation of the Opportunity Scholarship Program (OSP) in April 2009. The results showed that students who participated in the program performed at significantly higher levels in reading than the students in an experimental control group. Here are the study findings and my own interpretation of what they mean.

Opportunity Scholarships
Currently, 13 directly funded voucher programs operate in four U.S. cities and six states, serving approximately 65,000 students. Another seven programs indirectly fund private K—12 scholarship organizations through government tax credits to individuals or corporations. About 100,000 students receive school vouchers funded through tax credits. All of the directly funded voucher programs are targeted to students with some educational disadvantage, such as low family income, disability, or status as a foster child.

Nineteen of the 20 school voucher programs in the U.S. are funded by state and local governments. The OSP is the only federal voucher initiative. Established in 2004 as part of compromise legislation that also included new spending on charter and traditional public schools in the District of Columbia, the OSP is a means-tested program. Initial eligibility is limited to K—12 students in D.C. with family incomes at or below 185 percent of the poverty line. Congress has appropriated $14 million annually to the program, enough to support about 1,700 students at the maximum voucher amount of $7,500. The voucher covers most or all of the costs of tuition, transportation, and educational fees at any of the 66 D.C. private schools that have participated in the program. By the spring of 2008, a total of 5,331 eligible students had applied for the limited number of Opportunity Scholarships. Recipients are selected by lottery, with priority given to students applying to the program from public schools deemed in need of improvement (SINI) under No Child Left Behind. Scholars and policymakers have since questioned the extent to which SINI designations accurately signal school quality because they are based on levels of achievement instead of the more informative measure of achievement gains over time.

The third-year impact evaluation tracked the experiences of two cohorts of students. All of the students were attending public schools or were rising kindergartners at the time of application to the program. Cohort 1 consisted of 492 students entering grades 6—12 in 2004. Cohort 2 consisted of 1,816 students entering grades K—12 in 2005. The 2,308 students in the study make it the largest school voucher evaluation in the U.S. to employ the “gold standard” method of random assignment.

Voucher Effects
Researchers over the past decade have focused on evaluating voucher programs using experimental research designs called randomized control trials (RCTs). Such experimental designs are widely used to evaluate the efficacy of medical drugs prior to making such treatments available to the public. With an RCT design, a group of students who all qualify for a voucher program and whose parents are equally motivated to exercise private school choice, participate in a lottery. The students who win the lottery become the “treatment” group. The students who lose the lottery become the “control” group. Since only a voucher offer and mere chance distinguish the treatment students from their control group counterparts, any significant difference in student outcomes for the treatment students can be attributed to the program. Although not all students offered a voucher will use it to enroll in a private school, the data from an RCT can also be used to generate a separate estimate of the effect of voucher use (see sidebar, page 50).

Using an RCT research design, the ongoing IES evaluation found no impacts on student math performance but a statistically significant positive impact of the scholarship program on student reading performance, as measured by the Stanford Achievement Test (SAT 9). The estimated impact of using a scholarship to attend a private school for any length of time during the three-year evaluation period was a gain of 5.3 scale points in reading. That estimate provides the impact on all those who ever attended a private school, whether for one month, three years, or any length of time in between (see Figure 1). Consequently, the estimate should be interpreted as a lower-bound estimate of the three-year impact of attending a private school, because many students who used a scholarship during the three-year period did not remain in private school throughout the entire period. The data indicate that members of the treatment group who were attending private schools in the third year of the evaluation gained an average of 7.1 scale score points in reading from the program.

What do these gains mean for students? They mean that the students in the control group would need to remain in school an extra 3.7 months on average to catch up to the level of reading achievement attained by those who used the scholarship opportunity to attend a private school for any period of time. The catch-up time would have been around 5 months for those in the control group as compared to those who were attending a private school in the third year of the evaluation.

Over time, in my opinion, the effects of the program show a trend toward larger reading gains cumulating for students. Especially when one considers that students who used their scholarship in year 1 needed to adjust to a new and different school environment, the reading impacts of using a scholarship of 1.4 scale score points (not significant) in year 1, 4.0 scale score points (not significant) in year 2, and 5.3 scale score points (significant) in year 3 suggest that students are steadily gaining in reading performance relative to their peers in the control group the longer they make use of the scholarship. No trend in program impacts is evident in math.

What explains the fact that positive impacts have been observed as a result of the OSP in reading but not in math? Paul Peterson and Elena Llaudet of Harvard University, in a nonexperimental evaluation of the effects of school sector on student achievement, suggest that private schools may boost reading scores more than math scores for a number of reasons, including a greater content emphasis on reading, the use of phonics instead of whole-language instruction, and the greater availability of well-trained education content specialists in reading than in math. Any or all of these explanations for a voucher advantage in reading but not in math are plausible and could be behind the pattern of results observed for the D.C. Opportunity Scholarships. The experimental design of the D.C. evaluation, while a methodological strength in many ways, makes it difficult to connect the context of students’ educational experiences with specific outcomes in any reliable way. As a result, one can only speculate as to why voucher gains are clear in reading but not observed in math.

Student Characteristics
The OSP serves a highly disadvantaged group of D.C. students. Descriptive information from the first two annual reports indicates that more than 90 percent of students are African American and 9 percent are Hispanic. Their family incomes averaged less than $20,000 in the year in which they applied for the scholarship.

Overall, participating students were performing well below national norms in reading and math when they applied to the program. For example, the Cohort 1 students had initial reading scores on the SAT-9 that averaged below the 24th National Percentile Rank, meaning that 75 percent of students in their respective grades nationally were performing higher than Chart 1 in reading. In my view, these descriptive data show how means tests and other provisions to target school voucher programs to disadvantaged students serve to minimize the threat of cream-skimming. The OSP reached a population of highly disadvantaged students because it was designed by policymakers to do so.

Did Only Some Students Benefit?
Several commentators have sought to minimize the positive findings of the OSP evaluation by suggesting that only certain subgroups of participants benefited from the program. Martin Carnoy states that “the treated students in Cohort 1 were concentrated in middle schools and the effect on their reading score was significantly higher than for treated students in Cohort 2.” Henry Levin likewise asserts that “the evaluators found that receiving a voucher resulted in no advantage in math or reading test scores for either [low achievers or students from SINI schools].”

The actual results of the evaluation provide no scientific basis for claims that some subgroups of students benefited more in reading from the voucher program than other subgroups. The impact of the program on the reading achievement of Cohort 1 students did not differ by a statistically significant amount from the impact of the program on the reading achievement of Cohort 2 students, Carnoy’s claim notwithstanding. Nor did students with low initial levels of achievement and applicants from SINI schools experience significantly different reading gains from the program than high achievers and non-SINI applicants. The mere fact that statistically significant impacts were observed for a particular subgroup does not mean that impacts for that group are significantly different from those not in the subgroup. For example, Group A and Group B may have experienced roughly similar impacts, but the impact for Group A might have been just large enough for it to be significantly different from zero (or no impact at all), while Group B’s quite similar scores fell just below that threshold.

From a scientific standpoint, three conclusions are valid about the achievement results in reading from the year 3 impact evaluation of the OSP:

• The program improved the reading achievement of the treatment group students overall.
• Overall reading gains from the program were not significantly different across the various subgroups examined.
• Three distinct subgroups of students—those who were not from SINI schools, students scheduled to enter grades K-8 in the fall after application to the program, and students in the higher two-thirds of the performance distribution (whose average reading test scores at baseline were at the 37th percentile nationally)—experienced statistically significant reading impacts from the program when their performance was examined separately. Female students and students in Cohort 1 saw reading gains that were statistically significant with reservations due to the possibility of obtaining false positive results when making comparisons across numerous subgroups.
  Why examine and report achievement impacts at the subgroup level, if the evidence indicates only an overall reading gain for the entire sample? The reasons are that Congress mandated an analysis of subgroup impacts, at least for SINI and non-SINI students, and because analyses at the subgroup level might have yielded more conclusive information about disproportionate impacts for certain types of students.

Expanding Choice
The OSP facilitates the enrollment of low-income D.C. students in private schools of their parents’ choosing. It does not guarantee enrollment in a private school, but the $7,500 voucher should make such enrollments relatively common among the students who won the scholarship lottery. The eligible students who lost the scholarship lottery and were assigned to the control group still might attend a private school but they would have to do so by drawing on resources outside of the OSP. At the same time, students in both groups have access to a large number of public charter schools.

The implication is that, for this evaluation of the OSP, winning the lottery does not necessarily mean private schooling, and losing the lottery does not necessarily mean education in a traditional public school. Members of both groups attended all three types of schools—private, public charter, and traditional public—in year 3 of the voucher experiment, although the proportions that attended each type differed markedly based on whether or not they won the scholarship lottery (see Figure 2). In total, about 81 percent of parents placed their child in a private or public school of choice three years after winning the scholarship lottery, as did 46 percent of those who lost the lottery. The desire for an alternative to a neighborhood public school was strong for the families who applied to the OSP in 2004 and 2005.

These enrollment patterns highlight the fact that the effects of voucher use reported above do not amount to a comparison between “school choice” and “no school choice.” Rather, voucher users are exercising private school choice, while control group members are exercising a small amount of private school choice and a substantial amount of public school choice. The positive impacts on reading achievement observed for voucher users therefore reflect the incremental effect of adding private school choice through the OSP to the existing schooling options for low-income D.C. families.

Parent Satisfaction
Another key measure of school reform initiatives is the perception among parents, who see firsthand the effects of changes in their child’s educational environment. Whenever school choice researchers have asked parents about their satisfaction with schools, those who have been given the chance to select their child’s school have reported much higher levels of satisfaction. The OSP study findings fit this pattern. The proportion of parents who assigned a high grade of A or B to their child’s school was 11 percentile points higher if they were offered a voucher, 12 percentile points higher if their child actually used a scholarship, and 21 points higher if their child was attending a private school in year 3, regardless of whether they were in the treatment group. Parents whose children used an Opportunity Scholarship also expressed greater confidence in their children’s safety in school than parents in the control group.

Additional evidence of parental satisfaction with the OSP comes from the series of focus groups conducted independently of the congressionally mandated evaluation. One parent emphasized the expanded freedom inherent in school choice:

“[The OSP] gives me the choice to, freedom to attend other schools than D.C. public schools….I just didn’t feel that I wanted to put him in D.C. public school and I had the opportunity to take one of the scholarships, so, therefore, I can afford it and I’m glad that I did do that.” (Cohort 1 Elementary School Parent, Spring 2008)

Another parent with two children in the OSP may have hinted at a reason achievement impacts were observed specifically in reading:

“They really excel at this program, `cause I know for a fact they would never have received this kind of education at a public school….I listen to them when they talk, and what they are saying, and they articulate better than I do, and I know it’s because of the school, and I like that about them, and I’m proud of them.” (Cohort 1 Elementary School Parent, Spring 2008)

These parents of OSP students clearly see their families as having benefited from this program.

Previous Voucher Research
The IES evaluation of the DC OSP adds to a growing body of research on means-tested school voucher programs in urban districts across the nation. Experimental evaluations of the achievement impacts of publicly funded voucher and privately funded K—12 scholarship programs have been conducted in Milwaukee, New York City, the District of Columbia, Charlotte, North Carolina, and Dayton, Ohio. Different research teams analyzed the data from New York City (three different teams), Milwaukee (two teams), and Charlotte (two teams). The four studies of Milwaukee’s and Charlotte’s programs reported statistically significant achievement gains overall for the members of the treatment group. The individual studies of the privately funded K—12 scholarship programs in the District of Columbia and Dayton reported overall achievement gains only for the large subgroup of African American students in the program. The three different evaluators of the New York City privately funded scholarship program were split in their assessment of achievement impacts, as two research teams reported no overall test-score effects, but did report achievement gains for African Americans; the third team claimed there were no statistically significant test-score impacts overall or for any subgroup of participants.

The specific patterns of achievement impacts vary across these studies, with some gains emerging quickly, but others, like those in the OSP evaluation, taking at least three years to reach a standard level of statistical significance. Earlier experimental evaluations of voucher programs were somewhat more likely to report achievement gains from the programs in math than in reading—the opposite of what was observed for the OSP. Despite these differences, the bulk of the available, high-quality evidence on school voucher programs suggests that they do yield positive achievement effects for participating students.

Conclusions
School voucher initiatives such as the District of Columbia Opportunity Scholarship Program will remain politically controversial in spite of rigorous evaluations such as this one, showing that parents and students benefited in some ways from the program. Critics will continue to point to the fact that no impacts of the program have been observed in math, or that applicants from SINI schools, who were a service priority, have not demonstrated statistically significant achievement gains at the subgroup level, as reasons to characterize these findings as disappointing. Certainly the results would have been even more encouraging if the high-priority SINI students had shown significant reading gains as a distinct subgroup. Still, in my opinion, the bottom line is that the OSP lottery paid off for those students who won it. On average, participating low-income students are performing better in reading because the federal government decided to launch an experimental school choice program in our nation’s capital.

The achievement results from the D.C. voucher evaluation are also striking when compared to the results from other experimental evaluations of education policies. The National Center for Education Evaluation and Regional Assistance (NCEE) at the IES has sponsored and overseen 11 studies that are RCTs, including the OSP evaluation. Only 3 of the 11 education interventions tested, when subjected to such a rigorous evaluation, have demonstrated statistically significant achievement impacts overall in either reading or math. The reading impact of the D.C. voucher program is the largest achievement impact yet reported in an RCT evaluation overseen by the NCEE. A second program was found to increase reading outcomes by about 40 percent less than the reading gain from the DC OSP. The third intervention was reported to have boosted math achievement by less than half the amount of the reading gain from the D.C. voucher program. Of the remaining eight NCEE-sponsored RCTs, six of them found no statistically significant achievement impacts overall and the other two showed a mix of no impacts and actual achievement losses from their programs. Many of these studies are in their early stages and might report more impressive achievement results in the future. Still, the D.C. voucher program has proven to be the most effective education policy evaluated by the federal government’s official education research arm so far.

The experimental evaluation of the District of Columbia Opportunity Scholarship Program is continuing into its fourth and final year of studying the impacts on students and parents. The final evidence collected from the participants may confirm the accumulation of achievement gains in reading and higher levels of parental satisfaction from the program that were evident after three years, or show that those gains have faded. Uncertainty also surrounds the program itself, as the students who gathered on Freedom Plaza in May currently are only guaranteed one final year in their chosen private schools. What will policymakers see as they continue to consider the results of this evaluation? The educational futures of a group of low-income D.C. schoolchildren hinge on the answer.

Patrick J. Wolf is professor of education reform at the University of Arkansas and principal investigator of the D.C. Opportunity Scholarship Program Impact Evaluation. The opinions expressed in this article are his own.

An unabridged version of this article is available here.

The ongoing global financial crisis is forcing many employers, from General Motors to local general stores, to take a hard look at the costs of the compensation packages they offer employees. For public school systems, this will entail a consideration of fringe benefit costs, which in recent years have become an increasingly important component of teacher compensation. During the 2005–06 school year, the most recent year for which U.S. Department of Education data are available, the nation’s public schools spent $187 billion in salaries and $59 billion in benefits for instructional personnel. Total benefits added about 32 percent to salaries, up from 25 percent in 1999–2000. The increase reflects the well-known rise in health insurance costs, but it also appears to include growing costs of retirement benefits, which have received much less attention.

Conventional wisdom holds that teacher pensions (along with other public pensions) are more costly than private retirement benefits, for reasons dating to an earlier era of low teacher salaries over lifelong careers. In spite of dissent from this view by some researchers (see sidebar), in this case we find that conventional wisdom is right: the cost of retirement benefits for teachers is higher than for private-sector professionals.

To track changes in retirement costs and compare employer contributions to retirement for public school teachers with those for private-sector professionals, we draw on recent data from a major employer survey conducted by the U.S. Department of Labor. These data show that the rate of employer contributions to retirement benefits for public school teachers in 2008 is substantially higher than for private professionals: 14.6 percent of earnings for teachers vs. 10.4 percent for private professionals. Moreover, the gap has widened over the four years the data have been available. Between March 2004 and September 2008, the difference more than doubled, rising from 1.9 to 4.2 percentage points (see Figure 1).

There are several reasons one might expect employer contributions to retirement to be higher for teachers. First, nearly all teachers are covered by traditional defined benefit (DB) pension plans, in which employees receive a regular retirement check based on a legislatively determined formula. These plans have, over the years, come to offer retirement at relatively young ages, at a rate that replaces a substantial portion of final salary. U.S. Department of Education data show a median retirement age for public school teachers of 58 years, compared to about 62 for the labor force as a whole. A teacher in her mid-50s who has worked for 30 years under a typical teacher pension plan will be entitled to an annuity at retirement of between 60 and 75 percent of her final salary. In nearly all plans this annuity has some sort of cost-of-living adjustment. One does not generally observe comparable retirement plans for professionals and lower-tier managers in the private sector, since most employers have replaced traditional DB plans with defined contribution (DC) or similar 401(k)-type plans, in which the employer and employee contribute to a retirement account that belongs to the employee. Nor do those traditional DB plans that remain typically reward retirement at such early ages; they more nearly resemble Social Security, where eligibility is age 62 for early retirement, and 66 and rising for normal retirement.

The Survey Data

Our analysis draws on data from the National Compensation Survey (NCS), an employer survey developed by the Bureau of Labor Statistics (BLS). The NCS survey is designed to measure employer costs for wages and salaries and fringe benefits across a wide range of occupations and industries in the public and private sectors. Although the BLS has been reporting quarterly fringe-benefit cost data for various public and private employee groups for more than a decade, only since March 2004 has the bureau broken out these fringe-benefit cost data for public school K–12 teachers. In this article we use those data to compare retirement benefit costs for public K–12 teachers with costs for private-sector professionals. We use the most detailed available private-sector comparison group, “management, professional, and related,” a category that includes business and financial managers, operations specialists, accountants and auditors, computer programmers and analysts, engineers, lawyers, physicians, and nurses.

We measure the cost of retirement benefits as a percentage of earnings. Virtually all states specify in law that the employer will contribute a certain percentage of teacher salaries to a DB pension fund (employee contributions are similarly specified), and it is commonplace to compare such contribution rates among the states. Similarly, private-sector employers offering DC plans will typically specify their contribution as a percentage of salary (often as a match to employee contributions). Unlike some other benefits (e.g., health insurance), if salaries change, the dollar costs for retirement benefits move proportionally. On the benefit side, the DB formula ties one’s starting annuity to final average salary, while the adequacy of a DC plan is commonly thought of in terms of the salary replacement rate. Thus it is natural to specify retirement costs as a percentage of salary, both for teachers and for private-sector professionals.

In making this comparison, we must account for the fact that, while all of the private-sector professionals are covered by Social Security, a number of public school teachers are not. Some of the higher cost of employer retirement plans for teachers is offset by lower employer contributions for Social Security benefits. Thus, we should compare the contribution rates for employer-provided retirement benefits and Social Security for both groups of workers. While the BLS reports the Social Security contribution rate for private professionals, it does not report a similar rate for teachers. However, we are able to make such an adjustment by multiplying the share of teachers covered by Social Security, which the BLS estimates to be 73 percent, times the employer contribution rate (6.2 percent). This assumes that the vast majority of teachers are below the Social Security earnings cap (currently $102,000) and that the share of teachers in Social Security has been steady over the four years for which we make the comparison.

A time series with quarterly data for these benefit percentages is reported in Figure 1. Two patterns are visible. First, the contribution rate is considerably higher for public school teachers than for private professionals. In the most recent quarter for which data are reported, ending September 2008, the employer contribution rate for public K–12 teachers (14.6 percent) was 4.2 points higher than that for private-sector professionals (10.4 percent). Second, the gap is widening. While the private sector contribution rate has been relatively flat over the four years, the rate for public school teachers has markedly increased, doubling the gap between them from one-fifth to two-fifths.

In one important respect, it is likely that the BLS data underestimate the cost of retirement benefits for public school teachers. Many public school districts (and some states) provide health insurance benefits for retired public school teachers. In the course of this research we were surprised to learn that retiree health insurance benefits are not included in the BLS employment cost estimates. Since private employers have largely eliminated this benefit, this means that our estimate of the gap in retirement benefits favoring public school teachers is low, although we cannot be sure of the extent of the underestimate.

Social Security and Teachers

While the overall employer contribution rate for public school teachers is higher than for private-sector professionals, the group average may mask differences between teachers who are and are not covered by Social Security. In order to assess this point empirically, we examined directly the data on employer contributions for teacher pension funds. We find that total employer contributions for both groups of public school teachers are higher than for private-sector professionals.

Most teachers are in statewide pension funds, with a relatively small number in district funds (e.g., New York City, Denver, St. Louis). Data on employer contributions for these plans are available in annual financial reports for each fund, which are surveyed by the National Association of State Retirement Administrators (NASRA).

Using data on contributions from NASRA and pension fund annual reports where necessary, and using weights based on the number of teachers employed in each state or district as reported in the NCES Common Core of Data, it is possible to compute average employer contribution rates for teachers. First we consider teachers who are in states and districts covered by Social Security. For these teachers, we calculate the weighted average employer contribution to be 9 percent of earnings. This can be compared to the estimate of employer contributions to retirement for private-sector professionals and managers, calculated from the BLS data as 4.7 percent for the comparable period (FY07). This is a 4.3 percent difference favoring public school teachers, almost double, in those states and districts where teachers are enrolled in Social Security, so the comparison is on an equal footing.

For states and districts where teachers are not in Social Security, we calculate the average employer contribution at 11.1 percent of earnings. Of course, this is considerably higher than the 4.7 percent retirement contributions for private-sector professionals, but, perhaps surprisingly, it even exceeds their employers’ combined contributions to retirement and Social Security, which averaged 10.3 percent for FY07. Thus, as Figure 2 shows, comparing teachers with professionals in private-sector employment, total employer contributions are higher for teachers whether or not they are also covered by Social Security.

Our analysis of evidence from the BLS National Compensation Survey and the NASRA Public Fund Survey shows that the employer contribution rates for public school teachers are a larger percentage of earnings than for private-sector professionals and managers, whether or not we take account of teacher coverage under Social Security. In addition, the BLS data show that the contribution rate for teachers is clearly trending upward.

Looking Ahead

What are the likely trends going forward for the cost of teacher retirement benefits? No one knows for sure, but we can identify the two key factors that will drive these costs: future developments in the benefits themselves and in their funding. The trend through much of the postwar period was to enhance the retirement formulas in various ways, including reducing the age or service requirement for full benefits. For example, just last year New York City agreed to enhance its pension formula for younger teachers. But there is evidence that benefit enhancement has generally abated in recent years. There are even a few states, including Texas, that have moved to reduce benefits for newly hired teachers. However, this is unlikely to reduce costs in the near future, since benefits for incumbent teachers are protected by law in most states.

The other factor to consider is the funding status of teacher pension plans. The vast majority of teacher pension plans are not fully funded. This means that contributions include both the “normal cost” of pension liabilities accruing to current employees and the legacy costs of amortizing unfunded liabilities accrued previously (due to a variety of reasons, including the original pay-as-you go nature of most plans, as well as unfunded benefit enhancements over the years). In theory, if the actuarial assumptions hold true going forward and no new benefits are enacted, the amortization costs will eventually disappear (after 30 years, under a typical funding schedule), in much the same way that a homeowner’s monthly expenses decline when the mortgage gets paid off.

However, the near-term prospects may be very different. For one thing, public pension funds face the possibility of important accounting changes. Unlike private pension funds, public fund actuaries have been allowed to discount future liabilities at a rate of about 8 percent, the assumed long-run market return on fund assets. Finance economists have argued that such a high discount rate is imprudent, however, and there have been signs that public accounting standards might move toward the private-sector rules, based on corporate bond and Treasury rates, which could reduce the discount rate to about 5 percent. This would dramatically raise the required amortization payments.

Finally, it bears noting that the market value of pension funds has fallen precipitously as of this writing (December 2008). Barring a major market recovery, pension funds across the country will have new, large unfunded liabilities. Under actuarial smoothing methods, these losses will be phased in, raising required amortization payments over the next few years. If the accounting rules for public funds also change, reducing the discount rate on liabilities, the employers of public school teachers, along with other public employers, will face a double hit, requiring sharp increases in contributions. By contrast, those private employers who have switched over to defined contribution plans in recent decades will be unaffected. In short, there are good reasons to believe that the contribution gap we have documented will continue to widen in coming years.

Robert M. Costrell is professor of education reform and economics at the University of Arkansas. Michael Podgursky is professor of economics at the University of Missouri–Columbia.

Refusing to consent to the underlying ideology, Swan failed repeatedly. The college threatened to expel him from the teacher training program unless he signed a contract agreeing to undergo diversity training and accept extra scrutiny of his student teaching. After a national civil-liberties group intervened on his behalf, Swan was allowed to continue in the program, and WSU has since revised its evaluation form. The new version requires professors to evaluate students’ “willingness to consider multiple perspectives on social and institutional factors that can impede or enhance students’ learning.” Dean of Education Judy Mitchell explained, “We’ve changed the format and clarified the words, but we haven’t changed the standards.”

Advocates of dispositions assessments of the kind in place at WSU defend the screening of pre-service teachers, whether at program entry or later on in the certification process, as standard practice and argue that “dispositions” are merely those attitudes and behaviors necessary to successful teaching. Critics see the combination of program accreditation standards, revised by the National Council for Accreditation of Teacher Education (NCATE) in 2000; a growing curricular emphasis on “social justice” issues; and a left-leaning education professoriate as yielding a one-sided approach to teacher education and the certification of teachers based on ideology, rather than teaching skills or mastery of content knowledge.

As a historian, I am most struck by the parallels between the dispositions assessments of today’s aspiring teachers and the evaluations of teachers’ mental hygiene and personality that began in the 1940s and continued for two decades. As is the case today, from 1940 to 1960 teacher educators sought to protect the interests of schoolchildren by socially engineering “desirable” characteristics in their teachers. What have changed are the personal qualities deemed most important for success in the classroom.

Assessing Teacher Dispositions

What is the purpose of dispositions assessment? What entity or body is in the best position to make this assessment? If the purpose is to ensure that access to children is denied to those who are truly deviant (sexual predators) or those who could harm children (drug dealers, felony offenders, child abusers), then it seems the assessment is best made by the government, which has the resources and responsibility to identify these people. If the purpose is to ensure that potential teachers have basic characteristics like honesty or fairness, existing standards such as university honor codes in higher education should suffice. If the purpose is to see how a teacher acts in a certain environment (be it an urban, suburban, or rural school, with a diverse or homogeneous student body), then perhaps those in that environment can best perform that assessment, taking into account the standards, mores, and preferences of the community. The ultimate employers of teachers, local school districts, can and do screen for the characteristics they want in their employees. Why, then, is it also necessary for teacher educators to assess the personal and political beliefs of aspiring teachers? Perhaps the policing of teacher personality and dispositions is just a way for teacher educators to extend their control even further into the public school classroom.

The harshest critics of dispositions assessment accuse education schools of acting as ideological gatekeepers to employment in public schools. Indeed, web site after web site shows schools of education that list among their teacher-education program goals the inculcation of political views alongside intellectual curiosity and such work habits as punctuality. The University of Alabama’s College of Education is “committed to preparing individuals to promote social justice, to be change agents, and to recognize individual and institutional racism, sexism, homophobia, and classism.…” In the teacher education program at the Harvard Graduate School of Education, students are asked to  “act as leaders and agents for organizational change in their classrooms, schools, and society…continually examine their own identities, biases, and social locations, seeking knowledge of students’ cultures and communities, and pursuing a complex understanding of societal inequities as mediated through classism, heterosexism, racism, and other systems of advantage.” Some program descriptions explain that requiring awareness of these issues and a commitment to addressing them ensures teachers will teach all children. In an October 2006 letter defending the conceptual framework of Teachers College, Columbia University, against accusations of political screening, President Susan H. Furhman wrote, “We believe that responsiveness to the diversity of students’ backgrounds and previous experiences are [sic] essential for effective teaching” (see Figure 1).

Not all universities make the leap from classroom behavior to ideology: The “Teacher Education Professional Dispositions and Skills Criteria” at Winthrop University in South Carolina are only basic indicators of professional commitment, communication skills, interpersonal skills (among them, “Shows sensitivity to all students and is committed to teaching all students”), emotional maturity, and academic integrity; acknowledging social inequities is not mentioned. The difficulty, however, in assessing dispositions, whether they espouse social justice or are seemingly harmless as at Winthrop, arises when the assessors make value judgments rather than encourage academic freedom and respect freedom of conscience. As the Swan case at Washington State University shows, some teacher education programs clearly demand allegiance to a particular perspective on the politics of education.

If schools encourage students to respond honestly to teacher education assignments, and then use any responses that differ from accepted beliefs as grounds for dismissal, that is political screening and a clear denial of academic freedom. A student accused Le Moyne College, a private, Jesuit-run school, of doing just that. In 2004, administrators dismissed the politically conservative graduate student after he wrote a paper on classroom management that questioned the value of multicultural education and expressed limited support for the use of corporal punishment in the classroom. At the Brooklyn College School of Education, some students complained after a teacher showed the Michael Moore film Fahrenheit 9/11 on the day before the 2004 presidential election. The university asked one student to leave, accused two others of plagiarism, and then denied the two students the right to bring a witness or an attorney to their hearing. K. C. Johnson, a faculty member who questioned the accusation of plagiarism and defended the students in Inside Higher Ed, then faced possible investigation by the university. The hallmarks of a professional program of teacher preparation within a university should be the free exploration of ideas. Yet it seems some teacher preparation programs substitute professional socialization, and the political conformity it requires, for a commitment to academic freedom.

The controversy over political screening of prospective teachers by teacher educators came to a head at the June 2006 reauthorization hearing for the National Council for Accreditation of Teacher Education (NCATE) with the U.S. Department of Education. Within the list of dispositions aspiring teachers might be required to possess, the agency had included “social justice,” a phrase that, to many, signals a value-laden ideology. Under pressure from a number of groups, NCATE president Arthur Wise announced that the agency would drop “social justice” from its accreditation standards; he maintains that social justice was never a required disposition.

NCATE’s definition of “dispositions” and its inclusion of social justice as part of that definition had caused considerable consternation. Among the groups represented at the hearing were the National Association of Scholars, which had filed the complaint, and the Foundation for Individual Rights in Education (FIRE), founded and headed by civil libertarians Alan Charles Kors, professor of history at the University of Pennsylvania, and Harvey Silverglate, a criminal defense attorney. FIRE, an organization dedicated to the preservation of free speech, has accused a number of universities, including Washington State University on behalf of Edward Swan, of evaluating students on the basis of their political views and thereby violating their First Amendment rights.

Arthur Wise has staked out NCATE’s position that dispositions are only “commonsense expectations” for teacher behavior and insists that the accrediting agency does not condone the evaluation of attitudes. Whether or not that is the case, most teacher education programs in this country receive accreditation from NCATE and follow its lead. Even though NCATE has now dropped “social justice” as a disposition, the agency stands behind dispositions assessment and institutions’ use of “social justice” as a curricular theme. The phrase appears in countless teacher-preparation program and course descriptions. Critics are not hopeful that NCATE’s action will curb abuses. In her testimony at the NCATE hearing, American Council of Trustees and Alumni president Anne D. Neal asked that the agency’s reauthorization be denied “until it affirmatively makes clear that teacher preparation programs are not expected to judge the values and political beliefs of teacher candidates and asks that its members review and revise their standards accordingly.”

Judging Fitness Is Nothing New

Society has long been concerned with the behavior, both inside and outside of the classroom, and the character of public school teachers. A century ago, local school boards carefully selected school teachers they deemed “fit to teach,” whose behavior comported with community values. They could not smoke or drink. Female teachers could not socialize with men while unchaperoned. They could not marry. They were not to display or engage in behaviors considered deviant, such as lesbianism. They were to dress conservatively and attend church. Violation could cost a teacher her job.

School officials and boards also scrutinized teachers’ political views. During World War I, the superintendent of the Cleveland public schools suggested firing those teachers sympathetic to Germany, and anti-war teachers did lose their jobs in New York City. In the 1920s and 1930s, more than a dozen states, typically those in which there were anti-communist crusades, required teachers to take loyalty oaths.

In public-school classrooms, as educational progressivism steadily gained influence during the first half of the 20th century, the focus in classrooms gradually shifted from rigorous academic study and discipline to children’s personality development and mental health. Education historian Sol Cohen describes the “medicalization” of education as the “infiltration of psychiatric norms, concepts and categories of discourse” into American education. Cohen reports that by 1950, there was “a national consensus on the role of personality development in American education” and that this included the view that “the school is basically an institution to develop children’s personality and that personality development of children should take priority over any other school objective.”

Attention turned as well toward the “mental hygiene” of the teacher, whose actions and attitudes would no doubt influence the children in her charge. As Douglas Spencer, instructor of psychological counseling at Teachers College, Columbia University, wrote in 1938, the teacher was to “demonstrate in her own personality adjustment sound mental health and emotional maturity.” As the 1940s began, a growing chorus of educators called for teacher qualification and selection to be based on mental health, first and foremost, and many expected this to be achieved through the teacher education process. However, market pressures on teacher education institutions made this problematic. Government policies provided tax funds for training teachers through the publicly supported teachers colleges, which did not have selective admissions requirements. Meanwhile, the number of both school-age children and college attendees grew steadily, with more than one-quarter of college degrees being granted in the field of education.

The rapid expansion of the teaching workforce hindered efforts to select teachers on mental hygienic grounds, even before the teacher shortage that developed in the 1950s. Reports of teachers with mental disturbances and even mental illnesses made professional and public headlines throughout the 1940s and 1950s. Public concern grew about maladjusted or neurotic teachers and their inability to ensure the proper psychological development of the children under their tutelage. Some feared that, as with contagious disease, psychological disorders would spread from teacher to child. Various personality traits of the maladjusted teacher emerged in the literature of the time. Shy, nervous, timid, easily excitable, disorganized, irresponsible, introverted, sexually repressed, or hot-tempered teachers were considered unfit for the classroom. A 1961 text, The Mentally Disturbed Teacher, documented purportedly true incidents about such teachers, suggesting that teachers who used corporal punishment could be mentally ill or that irritability in a teacher may be a sign of alcoholism, to take two examples. One suggestion for improving the mental health of the teaching body was for schools to keep a record of the teacher’s “attainments and attitudes,” including her cultural background and her community leadership.

As early as the 1940s, teacher education institutions began to use rating scales, placement tests, and personal interviews as screening devices for measuring mental hygiene and teacher personality. For some assessments, candidates filled out questionnaires; for others, faculty, administrators, or psychologists observed the teacher and made judgments. The University of Utah required teacher candidates to take the Minnesota Multiphasic Personality Inventory and the Strong Vocational Interest Blank. The College of Education at the State College of Washington used the Minnesota Teacher Attitude Inventory. Still other institutions employed a variety of assessment measures, such as the Rorschach test, James Cattell’s 16 Factor Personality test, the Guilford-Zimmerman Temperament Survey, the Thurstone Temperament Schedule, and a host of other batteries designed to explore the teacher’s behavior, personality, and attitude.

In 1953, Ruth A. Stout, director of field programs at Kansas State Teachers Association and later professor of education at Teachers College, Columbia University, completed a comprehensive study of admission practices in teacher education institutions. Stout surveyed 785 of 865 accredited teacher-training schools and found that a majority identified emotional stability as being of primary importance and that approximately 45 percent actually assessed students’ emotional stability, identifying it as the second most important criterion for determining fitness for teaching, behind academic credentials. Assessment of emotional stability became more important, Stout reported, as students progressed through their teaching preparation, with more institutions using it to determine admission to student teaching than to the teacher education program.

Research on Teacher Personality

Experimental and statistical research on personality development exploded onto the 1940s education scene, replacing earlier anecdotal surveys. The Journal of Experimental Education and the Journal of Educational Research published much of this research, which used psychological or personality indexes to “scientifically” determine the relationship between personality and “good teaching.” The ultimate goal was to connect personal traits with teaching effectiveness, thus enabling better selection of teacher candidates. Sometimes, researchers measured teacher success based on the observation of classroom supervisors. At other times, they used data on students’ class rank, college grades, or other measures of student performance.

The results of the research were as diverse as the assessment instruments used. Some found good teachers were more gregarious, adventurous, frivolous, artistic, polished, cheerful, kind, and interested in the opposite sex than teachers rated poorer in performance. Others found good teachers to be those whose attitudes were positive toward children and administrators. A few studies that tried to correlate teacher factors (both intelligence and personality) with effectiveness found teaching too complex to be influenced by any one or two factors. Nonetheless, institutions pushed forward with the use of personality tests to select among teacher candidates, often using multiple indexes, even as critics warned that some instruments had low predictive validity, that there was inconsistency in results, or that the lack of replication warranted cautious use.

In a 1956 review of the research on “School Personnel and Mental Health,” J. T. Hunt, a professor at the University of North Carolina, noted that “efforts to identify personality differences between superior and inferior school personnel, to isolate a ‘teacher personality,’ or to predict either competence or effectiveness of student teachers by means of psychometric or projective instruments, led to limited results.” Unlike most of the research he reviewed, Hunt recognized that personality was not a monolithic attribute, as there were many kinds and types of teacher personalities and roles. More presciently, Hunt called for research that would consider the “varying value standards of judges.” “Very little attention seems to have been paid,” he concluded, “to the actual attitudes and expectations of persons” who assess teachers. He called for research that placed university administrators under the personality microscope.

University of Chicago professors Jacob Warren Getzels and P. W. Jackson in 1960 followed Berkeley professor Fred Tyler’s lead in arguing that no consensus existed among researchers, and presumably educators generally, as to what was considered good teaching. Getzels and Jackson pointed out that the authoritarian teaching style considered “good” at the end of the 19th century had given way to the personal style brought into vogue with progressive education, which would in time give way to another. Without definitive criteria for good teaching, the personality indexes used in teacher personality research had no validity. The tests, they claimed, were chosen for “irrelevant reason” or for “no apparent reason at all.” Thus, the entire design of research on teacher personality was flawed.

Mental Hygiene as Curriculum

The mental hygiene perspective nonetheless held sway throughout the 1950s, and the concept of personality became as important in teacher education as academic and technical preparation for the classroom—as important as content knowledge and skills. The separation of the teacher into “technician” and “personality,” a distinction noted by mental hygienist Harry Rivlin in 1955, required that teacher education prepare students along both lines. Teacher educators often prioritized personality over other aspects of a teacher’s abilities.

In 1955, Percival Symonds, professor at Teachers College, Columbia University, where he taught mental hygiene, headed a study publicized by the Council for Research in the Social Sciences of Columbia University. His study recommended “a change in emphasis in teacher-training from intellectual courses to experiences for the better personal adjustment of teachers.” For a decade, Symonds had promoted the inclusion of psychotherapeutic principles and methods in the mental hygienic treatment of teacher maladjustment. He used psychotherapy in his classes; his students wrote autobiographies and he analyzed them. As he noted in a Newsweek article, he “first gained the pupil’s confidence to a point where they would feel free enough to drag all the family skeletons out of the closet”; of course, he found maladjustment everywhere. According to a 1955 Education article by Leon Mones, then an assistant superintendent in Newark, New Jersey, and a former principal, Symonds and others were openly advocating that the emotional life of the teacher become the focus of teacher preparation, since “it is the teacher’s personality that is the tool with which he works rather than the content in which he gives instruction.”

Educational psychology courses aimed at understanding children were standard fare for teacher preparation in the 1920s. But even by the mid-1940s, the goal of psychology coursework had become the teacher’s own mental health. Bank Street College of Education in New York, San Francisco State College, the University of Texas, and the University of Wisconsin incorporated lectures on mental health with “psychiatrically supervised individual guidance” of pre-service teachers. The experimental use of psychoanalysis in teacher education even received funding from the National Institute of Mental Health.

At Bank Street College, teacher educator and director of research Barbara Biber extolled the virtues of a program that applied “the concept of the unified nature of cognitive and affective development…on the teacher-training level” and was based on “a process of integrating new knowledge with an old self.” Bank Street faculty members looked for certain dispositions in their candidates: relatedness to children, an orientation to the psychology of growth, their relation to authority, their emotional strength, and their motivation. The Institute for Child Study at the University of Maryland emphasized the ideal of the “self-actualized individual” in its graduate-level instruction. A human relations seminar at the Merrill-Palmer School aimed “to help the individual teacher express and explore the values, meanings, and dynamics of personal and professional experiences, to achieve self-awareness, and to develop sensitive, understanding, responsive attitudes.”

Still, psychiatrists reported that teachers, especially novices, did not know how to handle their negative feelings. I. N. Berlin, a professor of psychiatry and psychiatric consultant to school districts in San Francisco, San Joaquin County, and Stockton, California, argued that some mental pathologies that were causal factors in teacher maladjustment and ineffectiveness in the classroom were, unfortunately, exacerbated rather than alleviated by teacher education. Berlin’s criticism of teacher training reflected the belief of some psychiatrists that there were limits to teacher education’s ability to ensure mentally healthy teachers.

Learning from History

The screening of prospective teachers for maladjustment 50 years ago and the dispositions assessments going on today have remarkable similarities. As William Damon of Stanford has noted, dispositions assessment “opens virtually all of a candidate’s thoughts and actions to scrutiny…[and] brings under the examiner’s purview a key element of the candidate’s very personality.” The same underlying assumption—that scientific means of selection and training could guarantee good teachers—held sway at mid-century with respect to mental hygiene. Teacher educators who guarded entry to the profession used the techniques of science to study, measure, and evaluate the teacher candidate as do those who guard entry today. Only the specific values and attitudes they appraise have changed. Advocates of dispositions assessment claim that their methods are “standards-based” and provide “accountability” —scientific-sounding catchwords that hold considerable weight in the current political climate. Both sets of desirable characteristics—summed up in the terms mental hygiene and social justice—are tied to progressivism and appear as core components of the teacher preparation curriculum, with the effect of deemphasizing academic knowledge, or at least requiring subject-matter learning and even pedagogy to make room for them. And hard evidence was and still is lacking. Researchers could never link with any certainty particular personality traits with effective teaching. Nor, as Frederick Hess explains, is there any scientific evidence that requiring teachers to have certain views about “sexuality or social class” ensures that they teach all students: “Screening on ‘dispositions’ serves primarily to cloak academia’s biases in the garb of professional necessity.”

The history of teacher screening reveals how deeply rooted such practices are in American teacher education. Whether the standard is mental hygiene or possessing the proper political and ideological disposition, the elimination of candidates who do not pass muster gives teacher educators the power to determine who gains access to a classroom based on the values the teacher educators prefer. While the courts have permitted certifying agencies to require “good moral character” of teacher applicants, as legal scholars Martha McCarthy and Nelda Cambron-McCabe note, they “will intervene…if statutory or constitutional rights are abridged.” Thus, while pledging loyalty to federal and state constitutions is a permissible condition for obtaining a teacher license, swearing an oath to progressivism is not. Given the evidence and the history, there should be real concern, as teacher educator Gary Galluzzo has said, that “students’ views and personalities are being used against them” whenever dispositions are assessed. Those committed to academic freedom within higher education should be concerned when professional socialization trumps freedom of conscience in teacher education programs.

Laurie Moses Hines is assistant professor at Kent State University Trumbull campus, where she teaches in the Cultural Foundations of Education program and in the history department.

In 2000, public elementary and secondary schools spent roughly $180 billion on teachers’ salaries and benefits, about half of their total expenditures; most of it was distributed according to fixed salary schedules that considered only a teacher’s education and years of experience. This system has its origins in the first half of the 20th century and was partly a response to the racial and gender discrimination that existed under more discretionary systems at that time.

However, over the past 20 years more educators have wondered whether such pay packages can attract, motivate, and retain high-quality teachers in a highly competitive professional world (see Forum). In response to such concerns, there was a flurry of merit pay activity in the early 1980s. Twenty-nine states had initiated some sort of merit pay program for teachers by 1986. Since then, however, almost all of them have been diluted or discontinued. A 1997 study by economists Dale Ballou and Michael Podgursky reported that 12 percent of school districts were using merit pay in some way, but the amount of incentive in these districts averaged only two percent of base pay.

Critics of merit pay argue that the falloff in such programs was due to the fundamental technical difficulties of accurately identifying effective teachers and rewarding good teaching practices. Proponents of performance-based pay insist that these experiments were too limited in scope and were destined to fail in the face of stiff opposition from teachers and unions.

Despite widespread pessimism among educators about whether merit pay systems can effectively reward good teachers, most of the limited empirical evidence has been surprisingly positive. For example, two studies (in 1992 and 1997) found that the math and reading test scores of students in South Carolina improved significantly when the students were taught by teachers receiving merit pay. Similarly, related and more recent literature suggests that mathematics students learn more when their teachers have certification in mathematics.

Policymakers should be cautious in interpreting this sort of evidence, however, as the apparent benefits of certification could merely reflect differences in the students placed in their classrooms. For example, teachers who receive merit pay may tend to select schools and classes whose students are high achievers for other reasons. Likewise, parents especially engaged in their children’s education may work to ensure their assignment to teachers with strong credentials. Such subtle differences may not be visible in the data typically available to researchers.

New Evidence

The effectiveness of these short-lived merit pay programs is exceptionally difficult to measure because of these selection effects. However, the fortuitous overlap of two Tennessee programs from the mid-1980s and 1990s provides an unusual opportunity to circumvent this problem. Project STAR (Student Teacher Achievement Ratio) was a large-scale class-size experiment that began with kindergarten students in the fall of 1985. At roughly the same time, the Volunteer State began directing pay increases to teachers deemed meritorious under a Career Ladder Evaluation System.

The fact that both teachers and students in schools participating in Project STAR were assigned randomly to classrooms allows for an especially rigorous test of whether a merit pay system can effectively reward good teachers.

Before describing the Tennessee programs in detail, however, we need to take a closer look at some of the objections to merit pay for teachers. One concerns the problem of designing valid evaluation procedures for measuring teacher performance. Under an efficient merit pay plan in any industry, employers should be able to explain clearly why an employee did not receive merit pay and what he or she would need to do to get it. Whether these conditions can be met in the teaching profession, where there is no single blueprint for effective practice, has been the most contentious issue surrounding merit pay. This evaluation problem is further complicated by the fact that schools have goals other than cognitive achievement (for instance, promoting citizenship, fostering individual development, and reducing drug use and violence) that are difficult to measure and are often achieved only with teachers’ cooperation.

These concerns raise the possibility that attempts to reward meritorious teachers could even have perverse consequences. For example, merit pay systems may discourage cooperation among teachers or otherwise foster a demoralizing and unproductive work environment.

While these problems may explain why merit pay plans have often been dismantled, some researchers suggest that they are excuses, not reasons. Dale Ballou, an economist at Vanderbilt University, has argued that merit pay is widely and successfully used in private schools, which suggests that there is nothing unique about education that makes merit pay infeasible or unattractive. Ballou notes that the amount of merit pay in private schools is quite large and that the teachers who report receiving it have earnings that are nearly 10 percent higher than their nonmerit counterparts. In contrast, the earnings of merit pay teachers in public schools are only 2 percent higher than their nonmerit colleagues. Ballou attributes the frequent dismantling of alternative compensation for public school teachers to union opposition.

The Career Ladder

Can we devise a merit pay system that overcomes the challenges of definitional clarity and valid measurement? Can we do so without directly incorporating measures of students’ progress on standardized tests?

The Tennessee programs initiated by Governor Lamar Alexander in 1984 offer some reason to believe that we can. At the same time, they underscore the considerable difficulty of doing so in a fair, equitable, and effective manner.

Part of Tennessee’s Comprehensive Educational Reform Act, the Career Ladder Evaluation System was both well funded and sophisticated in its approach to teacher evaluation. As Richard M. Brandt, a professor at the University of Virginia, wrote in 1995, the program was “perhaps the country’s most comprehensive experiment in summative evaluation.”

Governor Alexander, who would go on to become secretary of education under President George H. W. Bush and is now a U.S. senator, was more colloquial in his description, calling the program “an old-fashioned horse trade with teachers. Taxpayers said to teachers, ‘The state will pay you up to 70 percent more based on your performance if you’ll promise to be evaluated every five years.’”

Rung by Rung

While it lasted–for 13 years–the now-defunct Career Ladder had many of the elements that merit pay backers believed a good program should have, including multidimensional evaluations and a hierarchy of professional development (in other words, a career ladder) that was coordinated with significant financial and professional rewards.

The ladder had five distinct stages, ranging from probationary to master. Fast-track options allowed those who had been teaching before 1984 to advance immediately, subject to successful evaluations, to a career level matching their experience.

For new teachers, however, the first rung of the career ladder was a one-year probation supervised by two tenured teachers from their school. Subject to a favorable review by the school district, using state-approved criteria, these teachers were then placed on apprentice status for three years. At the end of those three years, the school district could recommend that the teacher be granted a five-year certification for professional, or Career Level I, status, which included a $1,000 salary supplement from the state.

Then, at the end of the five-year Level I stage, a teacher could either apply for another five-year Level I certification or seek a five-year certification as a Level II teacher. Advancement required evidence of superior performance, as defined by a state commission and the state board of education, but it also came with a $2,000 state supplement for those who chose a 10-month contract and $4,000 for those choosing an 11-month contract, a significant bonus to teachers’ salaries at the time.

At the end of the Level II certification period, the same kind of option was available: a teacher could seek recertification at Level II or pass more rigorous evaluations to receive a Level III certification and a salary supplement of as much as $7,000.

The evaluations that occurred at each stage of the career ladder assessed teachers on multiple “domains of competence” using several distinct data sources (such as student and principal questionnaires, peer evaluations, a teacher’s portfolio, and a written test). On the first three rungs of the ladder (probation, apprentice, Level I), the local school districts were responsible for evaluating and certifying performance. The key evaluator at these stages–typically the principal–received three to five days of state training on evaluation instruments and procedures. In contrast, the evaluations for certifications at Levels II and III were conducted largely by a three-member team of peers from outside the teacher’s district. These evaluators received three to four weeks of training and were often Level III teachers from other districts who had been borrowed for a year by the state certification commission. The extensive training provided to the Level II and Level III evaluators was considered appropriate since they fielded more complex evaluation instruments intended to discriminate among “good, superior, and outstanding” teachers.

Under the original formulation of the career ladder, participation was optional for veteran teachers and mandatory for new teachers. It was initially expected that new teachers who failed to advance to Level I status after their apprenticeship would be fired, since they would no longer be eligible for the state portion of their salary. However, in 1987, the career ladder was revised to make it optional for all teachers. The major consequence of failing to advance to Level I status was essentially the lost opportunity for the salary supplement.

Interestingly, it appears that relatively few teachers faced this cost. Nearly all of the state’s teachers (94 percent of them, according to one report) chose to enter the career-ladder program. A state audit in 1991 revealed that 95 percent of eligible teachers had achieved Level I certification, prompting criticism that the standards for this designation had been severely diluted. However, among teachers applying for certification at Levels II and III, the success rate was only 79 percent.

Though most teachers chose to participate, and the success rates for certification were quite high, some expressed criticisms that echoed issues often raised by merit pay critics: for example, that three classroom visits (some of them prearranged) were inadequate for evaluating teaching performance objectively and that separating the staff into levels strained relations among teachers and hurt morale. Even the application process was criticized for emphasizing, as the Christian Science Monitor reported, “cunning and endurance . . . rather than merit.” The criticisms suggest that, despite the relative sophistication of the career ladder, its efficacy in rewarding high-quality teachers remains an open question.

Project STAR

Coincidentally, a compelling way to evaluate the success of the career ladder system comes via data from Governor Alexander’s Student Teacher Achievement Ratio program. Project STAR was an experimental study of class-size reduction that also began in the fall of 1985. That year, it included 6,325 kindergarten students from 79 participating schools. The experiment lasted for three more years, following students through the 3rd grade. Overall, roughly 11,600 students participated, with additional students entering the participating schools in the 1st, 2nd, and 3rd grades. Participating schools were drawn from around the state and, by legislative mandate, included inner-city and suburban schools from larger metropolitan areas (Knoxville, Nashville, Memphis, and Chattanooga) as well as rural schools and those from smaller towns. All students in classrooms included in the experiment were given the Stanford Achievement Tests in math and reading in the spring of each year.

Pooling the information from the experiment’s four years yields a single data set with roughly 24,000 student observations for each subject. Roughly one-third of these observations are for black students, and nearly half were for students eligible for the free-lunch program. Fully 91 percent of the student observations in the dataset come from classrooms taught by teachers participating in the career ladder: 15 percent had teachers with probationary or apprentice status, 69 percent had teachers at Level I, while just seven percent had teachers who had reached Level II or III.

The key feature of the experimental design of Project STAR was that students and teachers within participating schools and grades were randomly assigned to one of three class types: small classes, regular-sized classes, or regular-sized classes with teacher aides. These random assignments allow us to use the STAR data to compare the performance of students assigned to career-ladder teachers with the performance of students in the same school and grade who were assigned to nonparticipating teachers.

Restricting the comparison to students attending the same school is essential because student-teacher pairings were random only within a given school. That is, the experiment did not move students and teachers to schools they would not otherwise have attended or staffed. This unfortunately means that some schools in the data set–those with classrooms taught by teachers with the same career-ladder status–do not offer useful information for looking at the effects of career-ladder status.

It should also be noted that student attrition from schools participating in the experiment was high, ranging from 20 to 30 percent each year, and that roughly 10 percent of students moved between small and regular classes. While most of the movement between classes was due to parental complaints or behavioral problems, the attrition figures could also reflect other factors unrelated to the study, such as students’ moving out of a school’s geographic zone or having to repeat a grade. However, if parents of students with unobserved propensities for high achievement sought out master teachers by class reassignment or by moving to another school altogether, our results would overstate the quality of career-ladder teachers.

Fortunately, we expect that these problems are less important for a study of the career ladder than for one about class size. Unlike a multiyear assignment to a particular class size, a one-year assignment to a particular teacher does not provide a strong incentive for attrition or reassignment. Students would be assigned a new teacher in the next academic year. By contrast, students placed in a large class were expected to remain in large classes through the 3rd grade.

Still, to evaluate whether the experiment successfully matched students and teachers randomly within schools, we examined the association between students’ traits and their assignment to a teacher of their own race. If the pairings of students and teachers were indeed random and remained so over time, we should find no within-school association between observed student traits and exposure to teachers in the career ladder. As expected, students’ race, gender, age, eligibility for the free-lunch program, and class-size assignment all exhibit small and statistically weak within-school relationships with assignment to a career-ladder participant.

Finally, because the student-teacher pairings were initially random, any statistically significant difference in performance between students with and without career-ladder teachers should be attributable to true differences in the quality of the teachers. The most conventional interpretation of such performance differences would be that the program provided effective incentives for teachers and that the evaluations carefully discriminated among teachers of high and low quality. However, the high pass rates on career-ladder evaluations suggest that these assessments were not particularly discriminating (at least through Level I). This raises the possibility that, if career-ladder teachers were more effective, it was simply because better teachers were more willing to negotiate the bureaucratic impediments to advancing on the career ladder. Nonetheless, even if the career ladder led only to self-sorting of teachers by quality, it would indicate that the program successfully directed its financial and professional rewards to meritorious teachers.

Results

To see what these Tennessee programs tell us about merit pay, let’s first look at the effects simply of having a teacher in the career-ladder program, ignoring for the moment the teacher’s specific level of accomplishment. To eliminate the effects of any chance differences in performance caused by other observable characteristics, our analysis takes into account students’ age, gender, race, and eligibility for the free lunch program; whether they had been assigned to a small class; and whether they were assigned to a teacher of the same race–which earlier research using these same data found to have a large positive effect on student performance (see “The Race Connection,” Spring 2004). We also include as control variables two conventional indicators of teacher quality: experience and possession of a graduate degree.

Our main results indicate that students with career-ladder teachers scored nearly 3 percentile points higher in mathematics than students with other teachers. They also suggest that reading scores were nearly 2 percentile points higher among these students, though the results for reading fall just short of conventional levels of statistical significance (see Figure 1).

The estimated effects on reading scores are statistically indistinguishable from zero primarily because they are less precise. If the effect on reading performance of having a career-ladder teacher were as precisely estimated as the effect of being in a smaller class, it would also be statistically significant. That it is not may reflect the fact that the experiment was designed to evaluate the effects of differences in class size, not the career-ladder program.

Regardless, our best guess is that having a career-ladder teacher in either subject had a quite large effect. The estimated gains associated with assignment to a career-ladder teacher equal 40 to 60 percent of the gains associated with assignment to a class with roughly 15 students rather than 22. Furthermore, the gains are approximately equivalent to a third of the black-white gap in test scores among students in the experiment.

When evaluating these results, it is important to keep in mind that 91 percent of the student observations in the data set came from classrooms with teachers certified by the career ladder. The benefits of having a career-ladder teacher are measured relative to a somewhat atypical base–namely, the small group of students whose teachers chose not to apply for the program or were unsuccessful in their application.

Our second analysis, therefore, considered not only the teacher’s participation in a career ladder, but also the teacher’s status within the program. That is, we looked separately at the effects of having a teacher at the probationary or apprentice level, at Level I, and at Level II or III.

In math, the career-ladder teachers at the probationary/ apprentice level and at Level I were the most successful at promoting achievement. In contrast, career-ladder teachers at the master level did not have a statistically significant effect on math scores (see Figure 2).

This surprising pattern could in theory reflect the success of the career ladder in attracting (and retaining) new, high-ability math teachers and in providing these new teachers with early mentoring and professional development. However, an alternative explanation is that novice teachers, many of whom quickly leave teaching, happen to be particularly adept at teaching math. The fact that we have already controlled for differences in teachers’ experience makes this explanation unlikely. Moreover, a similar pattern emerges when we look only at students with teachers having five or more years of experience, a good number of whom remained at the probationary/apprentice level (perhaps because fast-track options were not available in their area). In short, it appears that the career ladder simply was not very effective at distinguishing superior or outstanding math teachers from those who were merely competent.

In reading, by contrast, assignment to a Level II or Level III teacher was associated with a large and statistically significant increase in reading achievement, while estimates of the effects of having a teacher from both of the other two groups remained positive but statistically insignificant. This suggests the career ladder may have been modestly successful in identifying the most outstanding teachers in reading.

Conclusions

Overall, our results suggest that Tennessee’s Career Ladder Evaluation System was at least partially successful at rewarding teachers who were relatively effective at promoting student achievement. Though the program was voluntary for veteran teachers, the combination of large bonuses and relatively undemanding evaluations–at least at the lower levels–led the vast majority of teachers to enter. Nonetheless, assignment to a teacher who had been certified by the career-ladder evaluations led to large and statistically significant increases in mathematics scores and sizable, though statistically insignificant, increases in reading scores.

But our findings also suggest that the teachers who were on the highest rungs of the career ladder (and received the largest pay increases) were not consistently better at promoting student achievement. In reading, only students with a teacher at the highest levels of the career ladder made statistically significant gains. In contrast, the math-score gains associated with having a career-ladder teacher actually appear to have been concentrated among students with teachers on the lowest rungs of the career ladder. These mixed findings underscore the challenge of designing a system of teachers’ compensation that rewards quality in a fair and equitable manner–a political challenge as much as a technical one.

Despite some success in rewarding teachers for producing better student outcomes, the career ladder was a target of the same criticisms that challenge virtually all attempts to tinker with systems of teachers’ compensation. A few years of budgetary constraints helped kill the will to keep it all together. Thus, having made participation in the career ladder voluntary for teachers in 1987, it was perhaps inevitable that the Tennessee legislature in 1997 voted to prevent additional teachers from entering the program and becoming eligible for merit bonuses. Teachers already in the program, though no longer subject to regular evaluations, were allowed to keep their bonuses for the duration of their careers.

As Lamar Alexander lamented at the time, “Those who questioned the Model-T Ford didn’t try to kill it. They replaced it with something better.” Continuing debates over merit pay programs in districts in Tennessee and beyond indicate that efforts to find such a replacement are under way. But it may still be too early to tell whether the future for merit pay for teachers will resemble that of the Edsel or the Mustang.

-Thomas S. Dee is an assistant professor in the Department of Economics at Swarthmore College. Benjamin J. Keys is a graduate student in the Department of Economics at the University of Michigan.

Podcast: Robert Costrell and Michael Podgursky talk with Education Next.

An unabridged version of this article is available here.

Teacher pensions consume a substantial portion of school budgets. If relatively generous pensions help attract effective teachers, the expense might be justified. But new evidence suggests that current pension systems, by concentrating benefits on teachers who spend their entire careers in a single state and penalizing mobile teachers, may exacerbate the challenge of attracting to teaching young workers, who change jobs and move more often than did previous generations.

The design of teacher pension plans is a timely concern: like other public pension plans, those for teachers are becoming more costly. Employer contributions to pension funds tack on a larger percentage of earnings for public school teachers than for private-sector managers and professionals, and this gap is widening (see “Teacher Retirement Benefits,” research, Spring 2009, Figure 1). Those data do not yet reflect the impact of the stock market decline since 2007: the drop in the value of pension funds means further increases in employer contributions will be required to fund promised benefits. As fiscal concerns force states to reevaluate the costs of teacher pension plans, officials might also consider the plans’ consequences for teacher quality.

In earlier work we highlighted the peculiar incentives for retirement built into these plans (see “Peaks, Cliffs, and Valleys,” features, Winter 2008). Most plans create large spikes in pension wealth accumulation for teachers in their 50s. These spikes act as an incentive for teachers to stay in the classroom until their pension wealth reaches its peak and then push them into retirement shortly thereafter, as pension wealth accumulation turns negative.

We now extend this line of research by focusing on the distribution of pension benefits among teachers of varying career lengths and the penalties for those who switch systems. We examine pension formulas in six state plans and develop measures of the redistribution of pension wealth from teachers who separate early to those who separate later. We compare existing defined benefit (DB) teacher pension systems to fiscally equivalent systems that treat all teachers equally and find that the former often redistribute about half the pension wealth of an entering cohort of teachers to those who separate in their mid-50s from those who leave the system earlier. We then show that this back loading produces very large losses in pension wealth for mobile teachers. Compared to a teacher who has worked 30 years in a single state system, a teacher who has put in the same years but split them between two systems will often lose well over one-half of her pension wealth. It is difficult to justify such a system of rewards and penalties on grounds related to fairness or teacher quality.

Teacher Pensions 101

Public school teachers are almost universally covered by traditional defined benefit pension systems. In such a system, the employer has an obligation to provide a regular retirement check to employees upon their retirement. Typically, a DB teacher pension plan requires that both teachers and employers make a contribution each year to a pension trust fund. The salient characteristic of a traditional DB system is that for any individual, benefits are not tied to contributions.

More specifically, once a teacher is “vested” (usually after 5 or 10 years), she becomes eligible to receive a pension upon reaching a certain age or length of service. These eligibility rules vary across states, but they typically allow a teacher to draw a pension well before age 65, especially if she has been working since her mid-20s. Benefits at retirement are usually determined by a formula that takes into account years of service and the final average salary (FAS), which is an average of the last few years of salary (typically three). In Missouri, for example, teachers eligible for normal retirement earn 2.5 percent (the “multiplier”) for each year of teaching service. Thus, a teacher with 30 years of service would earn 75 percent of the final average salary. So if the FAS were $60,000, she would receive $45,000 every year for the rest of her life. If the teacher were to separate from service prior to being eligible to receive the pension, the first payment would be deferred and the amount of the pension would be frozen until that time. Once the pension payments begin, there is typically some form of inflation adjustment, although the specifics again vary from state to state.

We examined teacher pension plans in six states. While the states were not randomly chosen (we inhabit two of them), their plans are indicative of many teacher pension plans. Because the composite effect of each system is hard to discern by simply looking at the benefit formula, we examine patterns of pension wealth accumulation by age of separation.

Calculating Pension Wealth

We use the benefit formulas of pension plans to estimate the pension wealth of individual teachers. When an individual retires under a DB plan, she is entitled to a stream of payments that has a lump-sum value that we calculate using standard actuarial methods (which take into account expected mortality patterns and adjust the sum of payments to reflect the fact that they are received over many years rather than at a single point in time).

The heavy S-shaped curve in Figure 1 depicts pension wealth (net of employee contributions) for 25-year-old entrants to the Missouri teaching force who work continuously until they leave teaching at various ages. The salary schedule assumed is that of the state capital (Jefferson City), under which teachers receive experience-based salary increases and are also paid more if they have a master’s degree. The accumulation of pension wealth is not smooth and steady, but rises with fits and starts, due to rules of eligibility for early retirement and the like. In Missouri, after vesting at five years, a teacher is eligible for a pension at age 60. Her pension wealth—the current value of those deferred benefits—grows fairly steadily until age 45. The curve becomes steeper at age 46 because of a provision that allows teachers to begin collecting a pension when their age and years of service sum to 80, which brings her pension forward to age 59 and earlier. Then there is a big jump at age 50, because the 25th year of service makes a teacher eligible for an immediate pension (albeit with a reduced multiplier). Growth in pension wealth continues to be rapid in subsequent years as the multiplier is increased to its “normal” rate of 2.5 percent. Then, following a final bump in the benefit formula’s generosity at 31 years of service (age 56), net pension wealth starts shrinking. As is evident, complex pension rules lead to pension wealth curves that are irregularly shaped and bear no resemblance to the smoothly growing cumulative value of contributions.

(Pension) Wealth Redistribution

The result of these complex pension rules is that teachers who leave the profession in their 50s receive more pension wealth (as a percentage of cumulative earnings) than those who separate earlier. To develop a measure of the resulting redistribution, we compare existing DB systems to a fiscally equivalent plan where pension wealth is neutrally distributed: a cash balance (CB) system. CB systems calculate employee retirement benefits based on the cumulative contributions, with a guaranteed rate of return. Thus, pension wealth is a fixed percentage of cumulative earnings, regardless of retirement age.

In dollar terms, pension wealth grows smoothly under a CB system. Figure 1 compares the accrual of pension wealth under Missouri’s DB plan (the S-shaped curve) with the smooth accrual under a hypothetical CB plan. This diagram readily illustrates the redistribution of pension wealth toward those who retire in their 50s from those who leave teaching earlier. Teachers who retire before age 49 in Missouri receive less pension wealth than they would under a CB plan, while teachers who retire later receive considerably more.

We have developed a numerical measurement of this redistribution. Specifically, to compare net pension wealth across different ages of separation, we measure it at a fixed point in time, and we also estimate the frequency of separations at different ages. In this fashion, we can calculate weighted averages of net pension wealth for winners, losers, and the whole cohort of 25-year-old entrants. When we compare the Missouri plan to the fiscally equivalent CB plan, we find that 46 percent of pension wealth is redistributed from those leaving teaching at an average age of 36.6 to those separating at an average age of 54.2.

We made the same calculations of the distributional impact of the DB plans in the other states. In all states, the degree of redistribution is substantial. In Massachusetts, for example, average pension wealth is low, but 61 percent of it is redistributed. The degree of redistribution is also relatively high in Ohio (49 percent) and Texas (47 percent, for new hires), while it is somewhat lower in Arkansas (39 percent) and California (36 percent). As in Missouri, the redistributive gains are concentrated among those who retire in their 50s, while the losses are dispersed among all early leavers. This pattern holds particularly true for Massachusetts, where the gains are concentrated among just one-fifth of the cohort.

To summarize, there is significant variation among states in the magnitude of the gains and losses compared to a simple CB system, but all states redistribute net pension wealth to a substantial degree to those who retire in their 50s (after about 30 years of service) from those who leave a teaching position after shorter periods. In addition to the issue of equity, this has serious implications for teacher mobility, to which we now turn.

Moving Costs

It is well understood that DB pension plans penalize mobility, yet the sources of these costs are rarely delineated or quantified in a systematic way. There are several factors that reduce pension wealth when a teacher moves. First, teachers who leave a system before they are vested have no claim on a pension. Upon termination, or shortly thereafter, any teacher contributions are returned with interest (the rate varies, and can be well below market), but the teacher does not receive employer contributions. This is a major source of loss for many young teachers, since most teacher pension systems have a vesting period of five years or longer and the vast majority of early-career teacher turnover occurs in the first five years on the job. In fact, nine states have a 10-year vesting period for teachers. With such long vesting windows, many teachers will receive no employer contributions toward retirement as a result of their work in the classroom.

Although the effects of these vesting windows are large, they are at least fairly transparent for young teachers. This information is routinely provided to those newly hired. Even for teachers who are vested, however, there remain potentially large costs from mobility, and these are less obvious. One cost comes from the fact that teacher DB pensions are all based on final average salary. When a teacher leaves the profession before normal retirement age, the value of her annuity is tied to her salary at the time of her separation. No adjustment is made for ensuing salary growth or inflation.

Other costs to mobility arise from the service eligibility rules for normal and early retirement. Teachers who separate from a plan with, say, fewer than 20 years of service will often not be able to begin collecting their pensions until much later than teachers who remain in the plan until they meet eligibility requirements. At any given age, pension wealth is therefore lower for the mobile teacher—who has left one system early and entered another system late—simply because she can expect to collect fewer pension checks. Alternatively, she may be able to draw her pension at the same time as the teacher who stays in one system, but with a penalty. Either way, the costs are substantial.

Switching Systems

Pension wealth calculations similar to those above provide a comprehensive method for evaluating the costs of mobility. Specifically, let us continue to examine the pension wealth of a hypothetical teacher who enters at age 25 and works continuously. However, now, rather than working continuously in the same system, at age 40, after 15 years in state A, she moves to state B, which has the same pension formula and same pay grid, and ultimately retires. We assume that she collects two pensions, one in each of the states in which she worked. The pure mobility cost can be thought of as the loss from moving at age 40 to an identical state, but with zero creditable service.

The hypothetical wealth trajectory described above is depicted as the dotted curve in Figure 1 for Missouri. As discussed above, the heavy solid curve illustrates net pension wealth for continuous service under the DB plan, evaluated at date of separation. The dotted segment represents the wealth trajectory for a teacher who moves after 15 years, at age 40, diverging at that point from the solid curve for the teacher who stays. For the first five years, the dotted curve is flat since the teacher must get vested in the new system. After vesting, the teacher is entitled to two pensions, one from the old job and one from the new one. However, the loss from mobility continues to widen in the following years, as the teacher who stays becomes eligible for earlier and earlier retirement, while the teacher who moves does not earn enough service credit to advance the pension from age 60.

Under a continuous career, our hypothetical teacher would obtain 30 years of service by age 55, qualifying her for “normal” retirement benefits immediately at 75 percent of final average salary. This is worth $626,088 at age 55. The split career of the mobile teacher means that she receives two annuities, each of which is for 37.5 percent of final average salary, but the FAS for the first pension is of course much lower. In addition, neither the first nor the second pension would be drawn until “normal” retirement at age 60. This means that five years of pension payments are lost. These two factors together reduce the net pension wealth to $219,163, a loss from mobility of $406,925. This is the gap between the dotted and solid curves in Figure 1 at age 55. The cost of mobility is 65 percent of pension wealth.

By contrast, under the hypothetical cash balance system, also depicted in Figure 1, there is no loss from mobility. Net pension wealth, the cumulative value of employer contributions, is a constant percentage of cumulative earnings, regardless of whether they accrue in one job or two.

Table 1 provides summary calculations of these mobility losses for all six states. A glance down the first column shows substantial mobility costs in all six states, ranging from approximately $200,000 to more than $500,000. As the table also shows, these losses are large in relative terms as well, ranging from 41 percent to 74 percent of net pension wealth for teachers who stay.

Figure 2 depicts the sources of these losses, as well as the variation across states. For each state, the full bar gives the net pension wealth of a teacher who stays in the system to age 55, and the bottom portion, in black, is that of the mobile teacher. The middle portion gives the loss from mobility due to freezing FAS on her first job. The top portion gives the mobility cost imposed by service eligibility rules. Specifically, splitting 30 years of service credit between two jobs delays the first pension draw and can also affect the replacement rate (the annual pension as a percentage of FAS).

The costs from the split in service credit are generally large and vary across states. In Missouri, Arkansas, and Ohio, these rules lead to a delay of first pension draw from age 55 to 60, while in California, the first draw is delayed to age 57. In Texas, the mobile teacher delays first draw to 63, but she gains a higher replacement rate as a result. In Massachusetts, there is no delay for first draw, but the mobile teacher sacrifices a large increase in the replacement rate that is awarded to 30-year veterans. All in all, the service eligibility rules for early retirement, pension bumps, and the like—little known to the general public (and, we suspect, to many young teachers)—can impose large costs on teachers who move.

Final Considerations

Our work offers the first detailed analysis of the distribution of net pension benefits among teachers of varying ages of separation and the corresponding costs that teacher pension systems impose on mobile teachers. We find that in a typical DB system, compared to a neutral system, half an entering cohort’s pension wealth is redistributed to teachers who separate in their 50s, from those who separate earlier. One of the main reasons is that teachers who teach into their 50s can start collecting a pension immediately, while teachers who leave earlier often must defer their pension until age 60 or later, so they collect fewer payments over their retirement.

This inequality in benefits produces very large losses in pension wealth for mobile teachers. We estimate that teachers who split a 30-year career between two pension plans often retire with less than half the pension wealth accrued by teachers who complete a similar career in a single system. Again, one of the main reasons is that teachers who split their career often cannot begin collecting pension payments as early as those who stay in one system.

Our discussion has focused on teachers. However, the problems we have identified extend to other professional staff in public schools. School administrators are always included in teacher retirement systems. The market for administrators in urban school districts is increasingly becoming national in scope, yet for mobile administrators retirement benefit systems with 5- to 10-year vesting systems can have a devastating effect on retirement savings.

The impediments to mobility—for both teachers and administrators—may be particularly problematic for charter schools. Many charter schools are part of organizations (e.g., Knowledge Is Power Program [KIPP], Edison Learning, Imagine Schools) that operate in more than one state. Edison Learning, for example, operates schools in 16 states. As these schools attempt to replicate their school models, it is valuable to them to move staff from one location to another, particularly when they start new schools, in much the same way business firms relocate managers. As we have shown, current educator retirement benefit systems make such mobility very costly in those states where charter school employees are required to participate in the state’s teacher pension plan.

Such a system of rewards and penalties is hard to justify. To appreciate the importance of mobility, consider the large differences in the growth of public school enrollment between states. The National Center for Education Statistics projects that states such as Nevada and Arizona will see enrollment growth in excess of 40 percent between 2005 and 2017. Louisiana, Vermont, and Rhode Island can expect enrollment declines of 10 percent or more over this same period. Heavily populated states such as Michigan and New York can anticipate declines of between 5 and 6 percent. In a well-functioning labor market, one would see considerable movement of workers from areas of contracting demand to areas in which demand is increasing. In the case of teaching, however, the pension systems impose large costs on those who move.

The barriers to reform are primarily political. First, states have a coordination problem. It is in no state’s individual interest to facilitate mobility out of the state; to the contrary, states are inclined to keep average pension costs down by skimping on benefits for those who depart. In addition, the distribution of benefits within states between short-term and career teachers will be governed by the relative influence of junior versus senior educators in educator groups and state politics. Influence generally increases with seniority for a variety of reasons, and these are enhanced in the case of pension politics, because the benefits of pensions are far more immediate and tangible for senior educators than for junior ones. The opaque nature of final-average-salary DB systems, with their complicated eligibility rules, only reinforces this imbalance.

All that said, these barriers are not insurmountable. Similar issues arise in higher education, and yet the benefits of academic mobility have led many state and private universities to offer more portable retirement plans. As states grapple with the pension difficulties they now face, they should consider systems with smooth wealth accrual, such as the CB plan described in this article. Another alternative to consider might be a hybrid such as TIAA-CREF, which has features of both CB and defined-contribution plans and has proven popular in higher education. Such systems are more transparent, tie benefits more closely to contributions, and do not penalize mobility or job shopping among young teachers. At a minimum, education policymakers should consider experiments that provide actuarially fair alternatives to traditional DB plans for new teaching recruits, and evaluate their utility for recruiting and retaining high-quality teachers.

Robert M. Costrell is professor of education reform and economics at the University of Arkansas. Michael Podgursky is professor of economics at the University of Missouri–Columbia.

Are test scores the educational outcomes that parents value most? We tackle this question by examining the types of teachers that parents request for their elementary school children. We find that, on average, parents strongly prefer teachers whom principals describe as best able to promote student satisfaction, though parents also value teacher ability to improve student academics. These aggregate effects, however, mask striking differences across schools. Parents in high-poverty schools strongly value a teacher’s ability to raise student achievement and appear indifferent to student satisfaction. In wealthier schools the results are reversed: parents most value a teacher’s ability to keep students happy.

Data

This study combines data on teacher requests (by parents) and teacher evaluations (by principals) from 12 elementary schools in a midsized school district that asked to remain anonymous, in the western United States. The students in the district are predominantly white (73 percent), but there is a reasonable degree of diversity in terms of ethnicity and socioeconomic status. Roughly 35 percent of the white students are eligible for free or reduced-price lunch. Latino students, 84 percent of whom are eligible for free or reduced-price lunch, comprise 21 percent of the student population. Achievement levels in the district nearly match the average of the nation (49th percentile on the Stanford Achievement Test).

There is no formal procedure for parents to request specific teachers in the district. Principals report that they assign students to classes with an eye toward balancing race, gender, and ability across classrooms within the same grade. Parents submit requests during the spring or summer, and principals make assignments over the summer. During our analysis period, roughly 22 percent of parents requested a teacher each year and 79 percent of teachers received at least one parental request. Parents are also able to request that their child not be placed with a particular teacher (a “negative request”). Only about 9 percent of teachers received any negative requests, and 92 percent of teachers with negative requests had at least one positive request as well. Principals report that they are generally able to honor almost all requests, giving parents an incentive to truthfully reveal their first preference.

Parents in the district appear to have strong and varied preferences for teachers. Among those teachers receiving at least one request, the average number of requests was 6.2. Whereas the teacher at the 25th percentile received only 2 requests, the teacher at the 75th percentile received 8 requests. Moreover, there are often large differences between the most-requested and least-requested teacher within the same school, grade, and year: The average difference is 7.4, and in 10 percent of grades, the difference is larger than 17.

Our data include information on requests made for the 2005–06 school year (the “request year”) in the summer of 2005 for kindergarten through 6th-grade teachers in all 12 schools in our sample, as well as information from an earlier year for two of the schools. We exclude from our analysis those teachers parents could not have plausibly requested—mainly new teachers (unless parents specifically requested the “new” teacher), who comprised about 17 percent of those teaching in the request year. Note that we include teachers who did not receive any requests, as long as they taught in the same grade and school in the request year and the prior year. Our final sample consists of 256 individual teachers. Parents who made requests chose, on average, from among approximately three different teachers.

With the assistance of the district, we linked the parental request data to administrative data on teachers and students. Because the administrative files provide only a very coarse measure of family socioeconomic status—eligibility for the federal free or reduced-price lunch program—we constructed an additional proxy for family income by matching each student’s residential address to U.S. Census data on the median household income in the student’s neighborhood.

Finally, to supplement our information on teachers, we administered a survey to all elementary school principals in February 2003 and March 2006. In these surveys, we asked principals to evaluate their teachers along a variety of dimensions, including dedication and work ethic, organization, classroom management, parent satisfaction, positive relationship with administrators, student satisfaction, role model value for students, and ability to raise math and reading achievement. The average rating was roughly 8 on a scale of 1 to 10, indicating that principals were quite lenient in their assessments. On the basis of these survey results, we created three measures: (1) the principal’s overall assessment of the teacher’s effectiveness, which is a single item from the survey; (2) the teacher’s ability to improve student academic performance, which is a simple average of the organization, classroom management, reading achievement, and math achievement survey items; and (3) the teacher’s ability to increase student satisfaction, which is a simple average of the role model and student satisfaction survey items. If a teacher was rated by the principal on both the 2003 and 2006 surveys, we use the average of the two ratings.

In previous research using the 2003 principal survey data (see “When Principals Rate Teachers,” research, Spring 2006), we found that principals in the district are usually able to identify the most and least effective teachers in their schools, as measured by their students’ academic progress. However, principals appear to be less successful in differentiating between teachers near the middle of the distribution of teacher effectiveness.

What kinds of parents make requests?

We begin by examining the characteristics of families who make requests. This is important for two reasons. First, our analysis of parent preferences will reflect only the views of those parents who actually made requests, so it is important to understand this group. Second, whether different types of families are more or less likely to make a request has important implications. If high-income parents are more likely to make a request, and such requests are for better teachers on average, then the availability of requests could exacerbate the achievement gap between students from low- and high-income families, even if all families equally value academic achievement.

In this district, families that are not eligible for the federal lunch program are about twice as likely to make a request as those that are eligible: 30 percent of families who are not eligible for free or reduced-price lunch make a request compared with only 13 percent of eligible families. Interestingly, these fractions are nearly identical across schools with very different poverty levels. Thus the socioeconomic makeup of the school does not appear to affect whether parents make a request, although the socioeconomic status of the family does.

We also conducted a more sophisticated analysis that measures the relationship between a family’s demographic characteristics (such as eligibility for free- or reduced-price lunch, median household income of the student’s residential neighborhood, race, and student prior achievement level), a school’s poverty level, and the likelihood that the parent makes a request. These results confirm that, conditional on the characteristics of the family and student, parents in high- and low-poverty schools are about equally likely to make a request. However, parents of low-income students are about 6 percentage points less likely to make a request than parents of high-income students (9 percent vs. 15 percent). Additionally, parents from high-income neighborhoods are about 4 percentage points more likely to make a request than parents from low-income neighborhoods (17 percent vs. 13 percent). Finally, Hispanic parents are significantly less likely to request a particular teacher for their child than are other families in the district.

After taking into account differences in socioeconomic status, we found that parents of higher-achieving students are more likely to make a request, which perhaps reflects greater sophistication or interest on the part of these families. The parents of a student whose performance is 1 standard deviation above the mean are about 8 percentage points more likely to make a request than the parents of an otherwise similar student whose performance is 1 standard deviation below the mean (19 percent vs. 11 percent).

What kinds of teachers do parents request?

In general, parents who make a request exhibit a strong preference for teachers who have received higher overall ratings by the school principal. However, recall that the principals’ survey responses allowed us to construct separate measures of two distinct aspects of teacher quality: the ability to improve student achievement and the ability to provide an enjoyable classroom experience for students. While positively correlated, these two factors appear to reflect distinct characteristics that vary across teachers. Overall, we find that parents value the teacher’s performance on both the student satisfaction and achievement measures, but give more weight to the satisfaction measure.

Even more interesting, however, we find stark differences across schools in the type of teachers that parents tend to request. We find that parents making requests in high-poverty schools place less value on student satisfaction than those in lower-poverty schools. Conversely, parents in high-poverty schools value a teacher’s ability to improve student achievement considerably more than parents in lower-poverty schools.

On the other hand, within a school, a family’s own socioeconomic status is uncorrelated with the type of teacher a parent requests. That is, both more- and less-advantaged parents in low-income schools tend to request teachers that are rated highly in terms of their ability to improve student achievement. In contrast, parents from all backgrounds in higher-income schools tend to request teachers who are rated more highly in terms of their ability to improve student satisfaction. When we control for the socioeconomic status of both the student and school, our findings are the same: student characteristics are not related to the type of teachers that parents prefer, while school characteristics are strongly related to parental preferences for teachers.

To quantify these differences, we used our results to simulate parent choices (see Figure 1). For the sake of simplicity, we first consider a situation in which a parent can choose between two teachers: one teacher has an average rating for both achievement and satisfaction; the other teacher has an average rating for achievement, but a high rating on the satisfaction measure (i.e., a rating 1 standard deviation above the mean). We calculate the percentage of parents with average background characteristics who would choose the high-satisfaction teacher. Next, we change one characteristic of either the parent or school and calculate how this change would affect the percent of parents who would choose the high-satisfaction teacher.

In a school where 80 percent of the children are eligible for free or reduced-price lunch, the parents of the average child would have a 48 percent chance of selecting the teacher with a high-satisfaction and average achievement rating over the teacher with average ratings on both satisfaction and achievement. In other words, these parents are no more likely to choose the high-satisfaction teacher than if they had randomly chosen which teacher to request. In contrast, if the child attends a school where only 20 percent of the students are eligible for free or reduced-price lunch, there would be a 65 percent probability that their parents would select the high-satisfaction teacher. The 17 percentage point difference is large and statistically significant.

We then consider the scenario where the choice is between two teachers who have the same satisfaction rating but different achievement ratings, and see the opposite result. Parents in the lower-poverty school are no more likely than they would be by chance to select the teacher with a high achievement rating (51 percent), whereas parents in the higher-poverty school would choose the teacher with a higher achievement rating 62 percent of the time. Again, the difference of 11 percentage points is statistically significant.

As one might expect, parents of kindergarten children appear to value satisfaction more and academics less than other parents, though this difference is small and bordering on statistical insignificance. Grade level is otherwise unrelated to preferences for teacher attributes.

Parent requests and classroom effectiveness

It is important to emphasize that the results presented above reflect both what parents observe and what they value. To the extent that parents have less information on a particular teacher characteristic, our findings may underestimate parent preferences for this characteristic. In particular, one might be concerned that parents do not have accurate information on teachers’ ability to raise student achievement. For this reason, we focus primarily on information from the principal survey, which likely reflects teacher behaviors or qualities that parents might learn from observing the teacher’s classroom or speaking with friends and neighbors who have had experience with the teacher in the past.

To test the sensitivity of our results to this methodological decision, we constructed a value-added indicator that measures a teacher’s contribution to student achievement (accounting for a wide variety of student and classroom characteristics that could affect achievement independent of the teacher’s ability). We find that teachers who perform better on our value-added measure also receive more parent requests, even after controlling for the student satisfaction measure from the principal surveys. However, when we also control for the principal-reported academic measure, this relationship is no longer significant, although the relationships between parent requests and both principal-reported measures remain positive and significant. These results suggest either that the academic considerations parents value are better captured by principal ratings or that parents have difficulty observing how much value a teacher adds to reading and math test scores.

An explanation?

The results presented above suggest that parents in low-income schools strongly value student achievement and are essentially indifferent to a teacher’s ability to promote student satisfaction. The results are reversed for families in higher-income schools. At the same time, we find that parent preferences within schools are identical across several measures of family socioeconomic status. How should we interpret these results?

One possible explanation emphasizes the role of school context in the educational process, particularly the interaction between parents, schools, and students. In this view, high- and low-income parents have similar preferences for student outcomes, but face constraints that are correlated with school demographics. Because academic resources are relatively scarce in higher-poverty schools (e.g., there are more disruptive peers, lower academic expectations, fewer financial resources, and less-competent teachers), parents in these schools seek teachers skilled at improving achievement even if this comes at the cost of student satisfaction.

If this explanation were true, we would expect to find a positive association between school-level income and school-level academic inputs, and a negative association between school-level income and the differences in the value-added by teachers within the same school. The second prediction is simply a consequence of diminishing returns to academic inputs. More specifically, if the average quality of teachers in a school is already high, being assigned to one of the better teachers will have only a limited effect on student achievement.

To what extent are these predictions borne out in the data? A comparison of observable teacher characteristics across schools provides some support for the first prediction. As in most other school districts, the teachers in higher-poverty schools in our sample have fewer years of experience than their counterparts in lower-poverty schools (11.8 years vs. 14.0 years). In comparison to their counterparts, teachers in higher-poverty schools are less likely to have credits beyond a bachelor’s degree (66 percent vs. 78 percent) and are less likely to have attended the most prestigious local university (75 percent vs. 80 percent) for their undergraduate degree. In addition, the variance of our value-added measure is significantly higher within higher-poverty schools than in lower-poverty schools, even after we control for the experience level and other observable characteristics of teachers within each school, which supports the second prediction. Hence, while certainly not conclusive, the available evidence is consistent with the explanation offered above.

Conclusions

Our findings suggest that what parents want from school depends on the educational context in which they find themselves. In particular, in low-income schools where academic resources are scarce, motivated parents are more likely to choose teachers based on their perceived ability to improve academic achievement. On the other hand, in higher-income schools these parents seem to respond to the relative abundance of academic resources by seeking out teachers who also increase student satisfaction. This may reflect a parental preference for their children to enjoy school, or it might reflect parental preferences for teachers who emphasize academic facets that increase student satisfaction but are not captured by standardized test scores, such as critical thinking or curiosity.

In considering the policy implications of this research, it is important to recognize that our analysis reflects parent decisions conditional on school choice. In principle, students in this district can attend any school, although in practice the vast majority of students simply attend their neighborhood school. Because the school choice decision is quite different from the teacher choice decision, our findings do not map directly onto the school choice debate. However, the results represented here do inform other policy issues. For example, they suggest that the parents of low-income, minority, and low-achieving children are much less likely to take advantage of informal opportunities to exercise choice from among teachers. This highlights the potential adverse impacts of honoring parental requests on the equitable distribution of education resources. Our results also suggest that different socioeconomic groups are likely to react quite differently to accountability policies, such as those embodied in No Child Left Behind. In more affluent schools, parents are likely to oppose measures that increase the focus on standardized test scores at the cost of student satisfaction. More generally, programs that increase the focus on basic skills or classroom management at the expense of student enjoyment or other academic facets not measured on standardized tests are likely to be unpopular in more affluent schools.

Brian Jacob is professor of education policy and economics at theGerald R. Ford School of Public Policy at the University of Michigan.Lars Lefgren is assistant professor of economics, Brigham YoungUniversity. This article summarizes research that will be publishedin a forthcoming article in the Quarterly Journal of Economics.

We address two main questions about charter schools in the city. First, who enrolls in New York City’s charter schools? And, second, how well are the schools educating students? What we found is that, compared with other students in the traditional public schools, charter school applicants are more likely to be black and poor but are otherwise fairly similar. We also found that charter school students benefit academically from their charter school education. Charter school students in grades 3 through 8 perform better than we would expect, based on the performance of comparable students in traditional public schools, on both the math and reading portions of New York’s statewide achievement tests. There is not yet a sufficient number of charter school students in grades 9 through 12 for us to report achievement effects for this group.

Data Collection

Forty-seven charter schools were operating in New York City in the 2005–06 school year, the most recent for which we have test-score results, and all but five are included in the analysis presented here. Two schools, Manhattan Charter School and South Bronx Charter School for International Cultures and the Arts, are participating in our ongoing study but are not included in the analysis because they do not yet have any students in test-taking grades. One school, ReadNet Bronx Charter School, was in the process of closing in 2005–06. The absence of ReadNet Bronx from our evaluation is likely to have only a small impact on our assessment of student achievement because the school had only two years of test-taking students before it closed. The New York Center for Autism Charter School is not included in the study because it serves a very special population and is not compatible with many elements of the study. The United Federation of Teachers Elementary Charter School has declined to participate in the study so far, but it does not yet have any students in test-taking grades.

Charter schools must advertise their availability to all students eligible to attend public schools and are not allowed to select their students from among applicants. Instead, if a charter school in New York receives more applicants than it has places, it must enroll students based on a random lottery. Each spring, charter schools that are oversubscribed hold admissions lotteries.

Our study data are collected as follows: First, the information from each charter school application is sent to the New York City Department of Education for inclusion in its administrative database. This database contains entries for all students who attend New York City’s traditional public schools and for all students who attend New York City’s charter schools. A contractor for the department uses the maximum amount of information possible—for example, the student’s name, birth date, and Social Security number, if available—to match each applicant to a corresponding existing entry in the department’s database. The contractor then extracts information on each student’s demographic characteristics, enrollment, test scores, and certification for and participation in various programs such as free and reduced-price lunch, special education, and English-language services. This information is gathered from both the years before and the years after the application to a charter school and sent to us with an encrypted student identification number.

We first obtained application data on the lottery conducted in the spring of 2005 for the 2005–06 school year, and we requested application data from earlier years as well. Not all schools had archived this information or had requested all of the elements that would prove helpful in matching up their applicants. The 2005–06 application data therefore have the most complete coverage of schools and the most information on which to match. In order to be as representative as possible, the analysis of the characteristics of charter school applicants described below is based on the data from that year. In our achievement analysis, however, we use data from all lotteries for which we have application data.

The Applicants

Who applies to New York City’s charter schools? In answering this question, it is important to recognize that the charter schools are located in neighborhoods that are substantially poorer than and almost twice as black as the average New York City neighborhood. Charter school neighborhoods contain only one-third as many whites and Asians as the average New York City neighborhood. In fact, it is no exaggeration to say that if the charter schools draw from their neighborhoods, they will draw students who are 90 to 95 percent black or Hispanic. The charter schools are thus in a situation that people sometimes find confusing. Normally, if we say that a traditional public school is “more black” or “more Hispanic,” we mean to imply that the school has fewer white students. However, for New York City’s charter schools, “more black” or “more Hispanic” cannot imply “less white” because there are hardly any whites (or Asians) to be displaced. Instead, when we say a New York City charter school is “more black” than surrounding schools, it is automatically “less Hispanic” (and vice versa). Any school that disproportionately serves black students will disproportionately not serve Hispanic students. These are not two independent comments: they are the same comment!

As one might predict based on their neighborhoods, applicants to New York City’s charter schools are twice as likely to be black (64 percent versus 32 percent) and much less likely to be white or Asian (7 percent versus 28 percent) than the average public school student in New York City. Because charter school students are disproportionately likely to be black, they are somewhat less likely to be Hispanic (27 percent versus 39 percent). About half of charter school applicants are female, just like students in the traditional public schools (see Figure 2).

There is no simple explanation for the disproportionate appeal of charter schools to blacks. While a couple of charter schools—Harriet Tubman and Sisulu-Walker—are named after a black person, most of the charter schools, not a few, disproportionately draw black students. Nor does the explanation seem to involve strong language barriers for Hispanics. Traditional public schools and charter schools located in areas with significant Hispanic populations provide the same level of Spanish-language translation for school materials. In both sets of schools, key materials, such as applications, school calendars, and school descriptions are usually available in Spanish. A more complex story is needed. For instance, black parents may feel more comfortable “disagreeing” with their regular school assignment than Hispanic parents do, particularly if the parents in question are recent immigrants.

A common proxy for poverty is a student being certified to receive a free or reduced-price lunch. (To get certified, a student’s household income must be less than 185 percent of the federal poverty line.) Using this proxy, we find that the applicants to charter schools are much more likely to be poor than is the average New York City student (93 percent versus 74 percent).

Unfortunately, charter schools and regular public schools have some information recorded differently in the New York City database, and these differences cause charter schools’ numbers of special education and English language learner students to be understated. Nevertheless, the data that we have suggest that, at the time they applied, 11.1 percent of charter school applicants were participating in special education. This is about the same percentage as in the New York City schools overall (12.5 percent). The data we have also suggest that, at the time they applied, 4.2 percent of charter school applicants were classified as English language learners, while 13.6 percent of New York City’s students were classified as such. Because of our concerns about the differences in the recording of English proficiency status, we cannot draw the conclusion that charter schools appeal disproportionately to students who are proficient in English. But the fact that charter schools appeal disproportionately to black students is probably reflected in applicants being more likely to be English speakers.

We do not have good data that would help answer the question of whether charter schools disproportionately draw high or low achievers. Because most students enter charter schools before the 3rd grade when state-mandated testing begins, only 36 percent of applicants in our study have prior test scores on record and this group is not representative of all applicants.

Student Achievement

The basic strategy we use to evaluate the effect of charter schools on student achievement is to compare students who are awarded a seat in a charter school through a lottery with students who enter the lottery but are not awarded a seat. About 91 percent of all charter school applicants participated in lotteries. The random assignment to the two separate groups of students who are otherwise similar—in their measured characteristics and the fact that they expressed a desire to attend a charter school—enables us to isolate the impact of attending a charter school.

We first wanted to confirm that the two groups contained similar students. As expected, when we compared students who were awarded a seat in a charter school to those who were not, we found no statistically significant differences on any of the demographic or predetermined program eligibility characteristics we could measure.

We use common statistical procedures to estimate the effect on math and reading test scores of each additional year of actual attendance at a charter school. Our results therefore reflect the performance of students who, if offered a seat in a charter school, choose to enroll—that is, those who comply with the experimental treatment. In some applications, having an estimate of a program effect that is valid only for compliers is problematic, because it would be useful to know what would happen if the program were expanded to other populations. In the case of charter schools, however, an estimate of their effect on students who enroll is exactly what we want, as the basic idea behind charter school reform is that only students who want to should attend them. Our present approach also assumes that each year of charter schooling has the same effect on student achievement. When we investigated whether each year of attendance at a charter school had a different effect, we found no evidence to support the idea of different effects in different years. However, we plan to return to the question in subsequent analyses when we will have more variation in the number of years students attend charter schools.

We use test-score data from the years 2000–01 to 2005–06 from the 36 charter schools that enroll students in grades 3 through 12. However, because the number of students in grades 9 through 12 is too small to produce statistically significant results at this time, our discussion will focus on the results for the 32 schools that enrolled 3rd through 8th graders in the relevant years. For them, the number of test-score observations included in the analysis ranges from almost 7,800 in grade 5 to 3,000 in grade 8.

We first present our results in the way most often used by researchers: standard scores. These scores, which are generated by dividing a scale score by its standard deviation, are helpful because they allow researchers to compare the effects of charter schools to the effects of other interventions, like class-size reductions. Our results indicate that, on average, New York City’s charter schools raise their 3rd through 8th graders’ math achievement by 0.09 of a standard score and reading achievement by 0.04 of a standard score, compared with what would have happened had they remained in traditional public schools (see Figure 3). We find no evidence that the improvement in achievement differs between boys and girls or between blacks and Hispanics.

To put these results in context, consider the Tennessee STAR Experiment, which produced some of the literature’s highest estimated effects for class-size reduction. The Tennessee experiment suggested that a 10 percent reduction in class size in grades K–3 raised students’ standard scores by 0.06. Furthermore, this was a one-time effect: even if students stayed in smaller classes for multiple years, their achievement rose only once, by 0.06. In contrast, the average charter school student improved by 0.09 in math and 0.04 in reading for each year of charter school attendance.

Another way to present the results is in terms of New York State’s performance levels. In 2005–06, depending on the grade, a student’s math scale score had to rise by an average of 32 points to go from the top of the Performance Level 1 range (“failing” or not meeting learning standards) to the bottom of the Performance Level 3 range (“proficient” or meeting learning standards). The equivalent required rise in a student’s reading score was 44 points.

We estimate that, depending on the grade, students’ math scale scores rise by 3.75 to 3.98 points and their reading scale scores rise by 1.53 to 1.61 points for every year they spend in charter schools. Again, these improvements are measured relative to what would have happened to the same students in traditional public schools. Another way to think about these gains is to understand that, for every year they spend in a charter school, students make up 12 percent of the distance from failing to proficient in math. They make up 3.5 percent of the distance from failing to proficient in reading.

There are several possible explanations for the effects of charter schools being larger in math than in reading. The most likely explanation, we believe, is that schools largely control math education, but that both families and schools exert strong influence over reading skills. If, for instance, the families of students who were and were not awarded a seat through a lottery had the same effect on reading and families controlled half the gains in reading, then the difference between the estimated math and reading effects would be fully explained.

Keep in mind, these annual gains are relative to whatever gains the students would have been expected to make in the traditional public schools had they not been awarded a seat through the lottery. Because most of the students in our study have been attending a charter school for between one and three years and no student has attended for more than six years, we are uncomfortable extrapolating our finding beyond four years of enrollment in a charter school.

We also estimated a separate effect on achievement for each of the 32 charter schools with students in grades 3 through 8. The results for about one-third of these schools are very imprecise, usually because they had very few students in test-taking grades during the analysis years. Based on the remaining schools for which we have reasonably precise estimates, however, we found a good deal of variation in achievement effects. About 19 percent of charter school students attend a school that is estimated to have a positive effect on math that is very large: greater than 0.3 of a standard score per year. Another 56 percent attend a school that is estimated to have a positive effect that is large: between 0.1 and 0.3 of a standard score. 18 percent attend a school with a positive but small to moderate effect. Only 6 percent attend a school that is estimated to have a negative effect on math, and these estimated effects are all small. The effects on reading are similarly distributed across a range, with 80 percent being positive and only 8 percent being negative.

School Policies

The variation in achievement effects among charter schools raises the question of whether one can identify specific policies that are associated with charter school success (see sidebar, “New York City Charter Basics”). To provide hints at possible answers, we conducted some preliminary analysis on the question using the math and reading results from the 32 schools that enrolled elementary and middle school students.

We want to be clear that our analysis cannot establish definitively whether the policies of charter schools cause changes in student achievement. We can describe only associations between policies and achievement effects, and the distinction between association and causation is very important in practice in the charter school context. Charter schools may adopt policies for reasons that we do not observe and it may be that it is these unobserved reasons that actually affect achievement. For instance, suppose that charismatic school leaders were a key cause of positive achievement effects, and suppose that charismatic leaders just happened to like long school years. We cannot measure charisma, but we can measure the length of the school year. Therefore, we might find an association between a long school year and positive achievement effects, even if the charisma, and not the long school year, caused higher achievement. A school that lengthened its school year would be disappointed in the results, not realizing that what it had really needed to do was to hire a charismatic leader.

That caution given, there are a few clear and interesting associations to be noted. We find no relationship between how long a charter school has been in operation and student achievement after controlling for school policies. However, if we do not control for school policies and look at the simple correlation between a charter school’s years in operation and student achievement, we find that older schools have more positive achievement effects. The fact that this correlation disappears when we include such policies in our analysis suggests that the reason older schools have more positive achievement effects is that they adopt more effective policies.

A long school year is associated with positive achievement effects, and we estimate that schools with years that are 10 days longer are associated with average student achievement that is 0.2 standard deviations greater. This is a large effect, and a 10-day difference among school calendars is quite common. In fact, 12 days is the standard deviation in the length of the school year among charter schools. We should note, however, that a long school year tends to go part and parcel with several other policies, such as a longer school day and Saturday school, and this should make us cautious about assigning too much importance to a longer school year in and of itself. A more conservative conclusion would be to think of the package of the three policies having a positive association with student achievement.

We also find that class size, optional afterschool programs, and most math and reading curricula seem to have no relationship to student achievement. Everyday Math and Open Court reading curricula did have negative and statistically significant associations with achievement effects. We discourage readers from interpreting these as causal effects, however, since an equally plausible interpretation is that these are curricula that schools adopt when their students are struggling.

New York City Charter Basics New York City has three charter school authorizers. Of the schools covered in this report, the State University of New York authorized 20 , the chancellor of the New York City schools authorized 19, and the New York State Board of Regents authorized 3. Three types of organizations operate charter schools in New York City: nonprofit community-grown organizations (CGOs), nonprofit charter management organizations (CMOs), and for-profit education management organizations (EMOs). CMOs and EMOs are formal organizations that exist to manage charter schools, and they function somewhat like firms that have a strong brand and that establish fairly independent branches or franchises (see “Brand-Name Charters,” features). CMOs and EMOs typically make overarching curricular and policy decisions, conduct back-office activities, and provide something of a career ladder for teachers and administrators within their network of schools. The CMO with the most schools in New York City in 2005–06 was the KIPP Foundation, and the EMO with the most schools was Victory Schools. CGO schools may be founded by a group of parents, a group of teachers, a community organization that provides local social services, one or more philanthropists, or the teachers union. More often than not, the founding group combines people from a few of the groups listed above. Fifty-six percent of the charter school students covered by this report attend 23 schools operated by CGOs; 19 percent attend 12 schools that are affiliated with CMOs; and 25 percent attend 7 schools run by EMOs. As these percentages suggest, the average school operated by an EMO has considerably larger enrollment than the average school operated by a CGO or a CMO. Missions and Policies Every charter school describes itself in a carefully crafted mission statement that sets out its vision, educational philosophy, and focus. Based on these statements, we can categorize the schools roughly into five groups: those that have a child-centered or progressive educational philosophy and typically seek to develop students’ love of learning, respect for others, and creativity (29 percent of students); those with a general or traditional educational mission and a focus on students’ core skills (28 percent of students); those with a rigorous academic emphasis, which have mission statements that focus almost exclusively on academic goals such as excelling in school and going to college (25 percent of students); those that target a particular population of students, such as low-income students, special needs students, likely dropouts, male students, and female students (11 percent of students); and those in which a certain aspect of the curriculum, such as science or the arts, is paramount (7 percent of students). There are a number of reasons to expect that charter schools will choose different policies and practices: They are independent and fairly autonomous. Their operating agencies have a variety of histories and priorities. All are young schools and more likely to experiment with new policies than are established schools. At the same time, there are reasons to think that New York City’s charter schools will share certain policies. They commonly serve disadvantaged students; they are all under pressure to attract parents and to satisfy a small number of authorizers; one school may deliberately imitate another by adopting a policy that seems to be working in the other school; schools may also imitate one another unconsciously (as when teachers who have worked at one school are hired by another and bring their knowledge with them). The common characteristics of charter schools reveal which innovations seem most promising to urban school leaders empowered to set their own policies (see Figure 4). About 64 percent of students attend a charter school with a school year of 190 days or longer, and 20 percent attend a school with a school year of 200 days or longer. By way of comparison, the modal school year in the United States is 180 days or 36 weeks. About 55 percent of students attend a charter school with a day that lasts eight hours or longer, 67 percent attend one with an optional afterschool program, and about 57 percent attend one with Saturday school that is mandatory for all or at least some students (for instance, students who are struggling academically). About 49 percent of students attend a charter school that has a system of bonuses for successful teachers, and 17 percent of students attend a charter school whose teachers are unionized. Most of the students in charter schools whose teachers are unionized attend one of the five charter schools that were formerly traditional public schools but converted to charter status. In addition, about 91 percent of charter school students attend schools that require uniforms, and about 95 percent attend schools that voluntarily administer standardized exams on a regular basis for diagnostic purposes. The advisory system is used by nearly all the charter schools that serve middle or high school grades. In an advisory system, a teacher or pair of teachers is assigned to a group of students for an entire school year. Teachers meet frequently (often daily) with their students and are responsible for tracking their progress and preventing them from “falling through the cracks.” Because students in elementary grades are assigned to one teacher for most of the school day, advisory systems would be duplicative and are therefore not used by elementary schools. About 52 percent of students attend charter schools that ask their parents to sign “contracts.” Because these contracts are not enforceable, it is best to think of them as a method of trying to ensure that parents know about the school’s policies and expectations. Some parents may also feel morally bound to abide by the contract. Just over half the students attend a charter school that reserves one or more seats on its board for parents. About 21 percent attend one with a disciplinary policy that fits the “no broken windows” school of thinking, which holds that encouraging small courtesies and punishing small infractions (usually at the classroom level) are important. This is in contrast to disciplinary strategies that focus more on preventing or punishing large infractions (often at an administrative level above the classroom). The charter schools employ a variety of math and reading curricula, with no curriculum being dominant. The most popular are Saxon Math (41 percent of students) and Core Knowledge (38 percent of students.) Fifty-four percent of students have an extended English or language arts period of 90 minutes or more, and the same percentage have an extended math period. While the Children First initiative in New York City mandates a daily “literacy block” of 90 minutes for elementary school grades, the city requires that traditional public elementary schools have between 60 and 75 minutes of math instruction daily, depending on the grade.

Conclusion

In sum, in the largest lottery-based evaluation of charter schools to date, we find that charter schools in New York City are having positive effects on the academic progress of the students who attend them. These effects are largest in charter schools that have extended the length of the school year, though we cannot establish definitively that this is the reason for their exceptional performance. We also find that the students applying to charter schools in New York City are more likely to be black and eligible for a free or reduced-price lunch program than students in the public schools in the district.

While it is reasonable to extrapolate the findings to other urban students who are similar to New York City applicants, we would argue against these results being applied to students who differ substantially from applicants to the charter schools. In particular, the results should not be applied to students who are substantially more advantaged or to students who would not be interested in applying to the types of charter schools available in New York City, even if they were conveniently located in the students’ area.

That said, our results provide a strong basis for recommending the continued expansion of charter schooling in the Big Apple and in other large cities with similar student populations.

Caroline M. Hoxby is professor of economics at Stanford University and director of the Economics of Education program at the National Bureau of Economic Research. Sonali Murarka is a project manager at the National Bureau of Economic Research. They are, respectively, principal investigator and project manager of the New York City Charter Schools Evaluation Project.

Not surprisingly, school board members agree that one of their most important goals is to help students learn. According to a 2002 national survey, student achievement ranks second only to financial concerns as school board members’ highest priority. We wondered, though, do voters hold school board members accountable for the academic performance of the schools they oversee? Do they support sitting board members when published student test scores rise? Do they vote against members when schools and students struggle under their watch?

Existing accountability policies assume that they do: states shine light on school performance by providing the public with achievement data. Voters and parents are expected to make use of these data in choosing school districts or schools, and to hold administrators and school board members accountable for the schools’ performance at each election. The idea is that voters will replace incumbents with new members when performance is poor and support incumbents over challengers when performance is strong. Indeed, there are very few other ways in which district officials can be held accountable for school performance. Neither the federal No Child Left Behind Act (NCLB) nor the states impose direct sanctions on members of school boards that oversee large numbers of underperforming schools.

Our questions led us to undertake the first large-scale study of how voters and candidates respond to student learning trends in school board elections. We analyzed test-score data and election results from 499 races over three election cycles in South Carolina to study whether voters punish and reward incumbent school board members on the basis of changes in student learning, as measured by standardized tests, in district schools. In addition, we assessed the impact of school performance on incumbents’ decisions to seek reelection and potential challengers’ decisions to join the race.

We found that in the 2000 elections, South Carolina voters did appear to evaluate school board members on the basis of student learning. Yet in the 2002 and 2004 elections, published test scores did not influence incumbents’ electoral fortunes. As we’ll see, the possible reasons our results differed so dramatically from one time period to the next hold important implications for the design of school accountability policies. But let’s first take a closer look at our methods and findings.

South Carolina

Once we set out to study local school board races, we encountered tall hurdles to obtaining election results. Only one state, South Carolina, centrally collects precinct-level election data for school board races. In all other states, obtaining precinct-level election results requires gathering and organizing election returns from hundreds of individual counties and election districts.

So we took a close look at South Carolina. In most respects, South Carolina elections and school boards are similar to those across the rest of the country. All but 4 of the state’s 46 counties hold nonpartisan school board elections. Approximately 80 percent of school board members receive some compensation, either a salary, per diem payments, or reimbursement for their expenses. Over 90 percent of South Carolina’s 85 school boards have between 5 and 9 members, while the largest board has 11. And, as is common practice in other states, nearly 9 out of 10 South Carolina school districts hold board elections during the general election in November.

Perhaps the most important difference between South Carolina and most other states when it comes to local school politics is the role played by the state’s teachers unions, which are among the weakest in the country. In other states strong teachers unions may mobilize high turnout among members, their families, and friends, and punish and reward board members for their treatment of teachers rather than hold them accountable for student test scores. South Carolina school boards are unlikely to be beholden to the unions, which should make the boards more responsive to the broader public.

Roughly half of the state’s 85 districts hold school board elections in any two-year election cycle. We collected precinct-level election returns for all school board races in three election cycles, 2000, 2002, and 2004. We also obtained school-level student achievement data from the South Carolina Department of Education. We began our analysis with 2000 because it was the first cycle of elections after South Carolina started administering the Palmetto Achievement Challenge Test (PACT) to students in grades 3 to 8 in 1999. These tests, based on the South Carolina Curriculum and Standards, are given in both reading and math. We averaged the reading and math percentile scores to produce a composite score for each school. Because we wanted to examine whether voters are more concerned with student performance districtwide or in their local neighborhood, we computed two measures of average school performance to include in our analysis. The first is the average test score for each district. The second is the average test score for the public school that is located closest to an election precinct.

Searching for Accountability

We began our analysis by comparing the vote shares of incumbent school board members who ran and faced an opponent with the test-score performance of the schools and districts they represented. We were careful to separate the effect of school performance from the effects of other factors that could reasonably influence an incumbent school board member’s vote share. For example, we considered whether voters evaluate student outcomes relative to spending by measuring the effect of changes in the district’s property tax rate. We also took into account features of the election, including whether it was held as part of the November general election or on another date, when turnout is likely to be lower. Additionally, we accounted for the partisanship of the electorate, measured by the Democratic candidate’s share of the presidential vote, and demographic characteristics, such as race, age, and gender. We also adjusted for potential differences in how voters from precincts with higher and lower average test scores respond to changes in test scores. For example, voters from precincts with lower test scores might respond more strongly when test scores improve than do voters from precincts with test scores that already were very high.

In 2000, 67 incumbents from 37 school boards ran for reelection in contested races in South Carolina. Of these 67 incumbents, 50 were reelected, and the median vote share for all incumbents in competitive races was 58 percent.

We found that incumbent school board members won a larger share of the total vote in a precinct when test scores in that precinct improved. We estimate that improvement from the 25th to the 75th percentile of test-score change—that is, moving from a loss of 4 percentile points to a gain of 3.8 percentile points between 1999 and 2000—produced on average an increase of 3 percentage points in an incumbent’s vote share. If precinct test scores dropped from the 75th to the 25th percentile of test-score change, the associated 3-percentage-point decrease in an incumbent’s vote share could substantially erode an incumbent’s margin of victory. In districts where percentile scores had increased in the year preceding the election, incumbents won 81 percent of the time in competitive elections; in districts where scores had declined, incumbents won only 69 percent of the time.

Citizens therefore did seem to base their assessment of incumbents on changes in test-score performance during a board member’s tenure, exactly the type of accountability many supporters of NCLB had hoped for.

We were interested to find that the average school test score for the precinct, rather than the district, had a significant effect on an incumbent’s vote share. The significant relationship with precinct test scores and the absence of a relationship with district scores suggests that voters were more concerned with school performance within their immediate neighborhood than across the district.

The Later Elections

With the evidence from 2000 in hand, we were initially surprised that all indications of a relationship between school performance and an incumbent school board member’s vote share vanished after the passage of NCLB in 2002.

We reanalyzed the data in a number of different ways, but were unable to find any indication that voters cast their ballots based on changes in test scores. We included administrative data from teacher, parent, and student ratings of local schools; we considered the potential relationship between vote share and test-score changes over the previous two or three years; we examined the deviation of precinct test scores from district means; we looked at changes in the percentage of students who received failing scores on the PACT; we evaluated the relationship between vote share and the percentage change in the percentile scores rather than the raw percentile point changes; and we turned to alternative measures of student achievement, such as SAT scores, exit exams, and graduation rates. None of these approaches yielded clear evidence of a link between school performance and voter behavior in school board elections.

Even when we estimated the probability that an incumbent won a majority of the votes in each precinct, or accounted for test-score changes and levels as a function of dollars spent on students, or measured the relationship between an incumbent’s vote share in one election and the previous election, the overwhelming weight of the evidence indicated that school board members were not being judged on improvement or weakening in school test scores.

Strategic Politicians

So far, we’ve discussed the experience of incumbents who ran against an opponent. Many incumbents, however, either did not run for reelection or ran unopposed. For example, in 2000, 42 of the 157 sitting board members in 39 school districts who were up for reelection did not run for office. Among the remaining 112 who sought to retain their seats, more than one-third, 45, did not face a challenger. The 67 incumbents who ran opposed in 2000 represented less than half of the sitting board members whose seats were in play that election.

School performance as measured by test scores may have helped determine which candidates sought reelection and which faced a challenger. If board members and potential challengers anticipate that voters will punish incumbents for poor school performance, declining test scores may lead board members to retire rather than endure defeat. A drop in test scores may also encourage opponents to run for office, either because they believe that incumbents are now vulnerable to defeat or because disgruntled citizens feel compelled to run for office when schools perform poorly.

Although exact election filing dates vary by school district, most candidates for seats on South Carolina’s school boards must decide whether to run by mid-September for a November election. PACT scores, however, are typically released to the public in late September or early October. Incumbents and potential challengers may not know the exact size of precinct or district test-score changes, but they could very well have impressions of the direction and rate of student learning trends. School board members and some challengers have observed the schools firsthand and have listened to accounts from principals and teachers. By monitoring the coverage of education issues on local television and in the print media, candidates may also have a sense of the extent to which voters are likely to use student test-score performance to evaluate candidates. And although we do not know this with any certainty, it is possible that school board members have access to test-score results before they are released to the public.

We decided to assess the relationship between test-score trends and incumbents’ decisions to run for reelection, and then to estimate the effect of test-score trends on the probability that an incumbent who runs faces an opponent. Our basic approach in this analysis was to compare the probability of running (or running and facing a challenger) between incumbents who oversaw districts with stronger and weaker year-over-year test scores. Because candidates either run for election in every precinct or do not run at all, we focused only on district test scores. As with our analysis of the relationship between test scores and vote share, we accounted for a number of factors that could reasonably influence a candidate’s decision to run for office. These included the incumbent’s vote share in the previous election, which might serve as a signal of the likelihood of victory to both the incumbent and potential challengers, and whether board members received compensation for their service, under the assumption that paid positions would be more attractive.

Our results indicate that incumbents may bow out in anticipation of being held accountable for poor test-score performance by schools in their district. During the 2000 election, incumbents were less likely to seek reelection when their district’s test scores declined over the preceding school year. If a district experienced a drop from the 75th to the 25th percentile of test-score change, our results lead us to expect that incumbents will be 13 percentage points less likely to run for reelection. In fact, 76 percent of incumbents sought reelection in districts with improving test scores; in districts with falling scores, only 66 percent did. The results did not hold for the later elections. Just as we found no evidence in the 2002 and 2004 elections that a large block of voters held incumbents accountable for poor test scores, we failed to find any indication that incumbents in 2002 and 2004 based their decisions about running for reelection on student learning trends.

When we looked at the behavior of the challengers, we once again saw evidence of their responding to test scores during the 2000 election, but no indication in 2002 or 2004 (see Figure 1). In 2000, a drop in test scores within the district significantly increased the likelihood an incumbent would face a challenger. If a district’s test-score change fell in the 25th rather than the 75th percentile, we estimate that an incumbent experienced an 18-percentage-point increase in the probability of facing a challenger. On the ground, the data show that 74 percent of incumbents who ran for reelection in districts with declining scores faced a challenger; in districts with improving scores, only 49 percent of incumbents faced a challenger.

What Happened in 2000?

Why did voters, incumbents, and potential challengers care about test scores in 2000 but not in 2002, or in 2004? The most likely explanation involves changes in media coverage of education issues. The amount and content of media coverage of student test scores differed substantially between 2000 and the latter two election years.

The 2000 elections were the first to follow the passage of the state’s accountability system. Journalists devoted ample space to issues that either directly or indirectly concerned student learning trends. Charleston’s Post and Courier, the Herald in Rock Hill, Columbia’s The State, and the Associated Press State & Local Wire, which serves numerous other South Carolina papers, regularly carried stories about the state of South Carolina’s schools. Both incumbents and challengers frequently identified student achievement generally, and test scores in particular, as the single most important issue in the 2000 school board election. Newspaper editorials that endorsed candidates in the 2000 election regularly underscored ways in which individual incumbents and challengers did, or said they would, improve student achievement. And 45 percent of the newspaper articles about school board races in the two months prior to the election mentioned student test scores.

In the 2002 and 2004 elections, however, media coverage shifted to other issues, such as the closing of schools, the racial composition of schools and boards, disciplinary problems, and sports programs. In these years, only 30 and 34 percent of articles, respectively, touched on test scores. The decline in media attention leads us to suspect that concerns about student learning trends probably did not stand at the forefront of voters’ or candidates’ thinking in the 2002 and 2004 elections.

The tone of articles about the state’s accountability system also shifted drastically during the 2002 and 2004 election cycles. From 1998 to 2000, most stories adopted a fairly neutral tone, introducing the public to the new accountability system and offering tepid praise and criticism of the testing regimen. After the 2000 election, journalists portrayed considerably more skepticism in their coverage of student achievement trends. Reporters devoted stories to errors in PACT’s scoring, security breaches in school testing, flaws in the science and social studies portions of PACT, district efforts to get ahead by changing their test dates, confusion regarding the comparability of test scores over time, missing PACT scores, and conflicts between school evaluations under the state and national accountability systems.

At the same time that administrative irregularities and mishaps attracted public scrutiny, teachers, district officials, and various other interest groups began to challenge the value of standardized tests more generally. One 3rd-grade teacher was quoted as saying, “These tests cannot and never will truly measure what a child actually knows, how a child sees the world, what a child genuinely understands and grasps, and what kind of life that child lives outside the school walls.” A school district associate superintendent claimed, “The problem with PACT is it doesn’t tell you what your child knows and doesn’t know.” The Palmetto State Teachers Association questioned the value of the state’s testing regimen, noting on its web site, “The current statewide tests do not provide immediate diagnostic information needed to improve student achievement or provide information to help teachers plan to meet the needs of each student. The testing process is time consuming, and spending weeks on high-stake testing is NOT in the best interest of children.” And as Andrew HaLevi, the Charlestown County School District 2000 Teacher of the Year, wrote in a 2001 op-ed for the Post and Courier, “The PACT needs to be seen for what it is: a vehicle for politicians to say that they are tough on education (and educators). This may make for good politics, but it makes for bad educational policy.” Reacting to the rising criticisms directed toward PACT, voters may have grown disenchanted with the state’s accountability system and removed test-score performance from among the criteria on which they evaluated school board candidates.

There are, of course, several other plausible explanations for why South Carolinians voted based on test score performance in 2000 but not in 2002 and 2004. The timing of the public release of the test scores is one. The 2000 scores were released in late October, whereas scores in 2002 and 2004 were released in early October and early September, respectively. In 2000, the release of scores so close to the election date and the media coverage that followed may have primed voters to evaluate candidates on student test scores. In the other two election years, the gap of a month or two between the release of scores and election day may have allowed the issue of test scores to fade from voters’ minds.

Another possibility is a major change in the reporting of test information. NCLB requires schools to notify parents directly about the performance of their schools. In 1999 and 2000, the first two years of PACT testing, scores were reported in their raw form in the materials that parents received. Beginning in 2001, official PACT reports to parents used a simpler rating scale that classified each school into one of five performance categories ranging from unsatisfactory to excellent. Under this scheme, almost every school received a rating of at least average. Indeed, a Department of Education news release in 2002 ran with the headline, “Schools receive higher Absolute ratings on report cards; 80% average or better.” Although the raw scores were contained deeper in the reports, if most schools appeared to be average or better, parents may not have been prompted to hold incumbents accountable for poor school performance. Incumbents and potential challengers may also have become less responsive to scores when the testing regimen began to give nearly every school a passing mark.

Implications for Policy

The evidence from South Carolina shows that voters do at least sometimes evaluate school board members on the basis of student learning trends as measured by average school test scores. Changes in average school test scores from year to year can affect the number of votes incumbents receive, the probabilities that they run for reelection, and the likelihood that they face competition when they do.

But the absence of a relationship between average school test scores and incumbents’ electoral fortunes in the 2002 and 2004 school board elections raises important questions about the assumptions underlying accountability systems. School board elections give the public the leverage to improve their schools. If voters do not cast out incumbents when local school performance is poor, they forfeit that opportunity. As debate continues over components of NCLB, policymakers should consider whether it is realistic to assume voters will in fact use the polls to drive school improvement.

Christopher R. Berry is assistant professor at the Harris School of Public Policy Studies at the University of Chicago, where William G. Howell is associate professor.

Despite the old habits and the old research, however, school districts across the nation have been slowly but steadily bucking convention. Several large systems, including Chicago (beginning in 1996), New York (2004), and Philadelphia (2005), now require students in particular grades to demonstrate a benchmark level of mastery in basic skills on a standardized test before they can be promoted. Florida (2002) and Texas (2002) have taken the lead among states in forbidding social promotions. In 2000, the most recent year for which national enrollment data are available, these five school systems alone enrolled nearly 20 percent of the nation’s 3rd-grade students. (For more on Chicago’s policy, see Alexander Russo, “Retaining Retention,” features, Winter 2005; and Robin Tepper Jacob and Susan Stone, “Teachers and Students Speak,” features, Winter 2005.)

But is this new approach to grade promotion effective? And what about those studies that say retention doesn’t work? Proponents of the new programs believe that schools do students no favor by
promoting them if they don’t have the skills to succeed at a higher level. But because these arguments, however plausible, have little research to support them, we set out to determine if they have scientific merit. Our findings from Florida suggest that the use of standardized testing policies to end social promotion can help low-performing students make modest improvements in reading and substantial improvements in math.

Florida’s Program to End Social Promotion

Over the past several years Florida has attempted substantial reforms of its struggling public school system, the fourth-largest in the country and one that consistently ranks close to the bottom on academic indicators, including high-school graduation rates and scores on the National Assessment of Educational Progress (NAEP). The Sunshine State had instituted school voucher programs, increased the number of charter schools, and devised a sophisticated accountability system that evaluates schools on the basis of their progress as measured by the Florida Comprehensive Assessment Test (FCAT). But in May 2002, the state legislature made one of its boldest moves, revising the School Code, the state’s education law, to require 3rd-grade students to score at the Level-2 benchmark or above on the reading portion of the FCAT in order to be promoted to 4th grade.

The hurdle created for students was not terribly high. The state’s department of education describes a student who scores at Level 2 (of five levels) as having “limited success” against the state standards; only students who score at Level 3 or above are considered to be proficient for the purposes of evaluating schools under No Child Left Behind. Even so, roughly 24 percent of 3rd graders tested in Florida in 2001–02, the year before the retention policy was introduced, performed below Level 2. This number fell slightly, to 22 percent, in the 2002–03 academic year.

Not all these students were retained, however, even after the policy change. The law allowed for exceptions to the retention policy if a student had limited English proficiency or a severe disability, scored above the 51st percentile on the Stanford-9 standardized test, had demonstrated proficiency through a performance portfolio, or had already been held back for two years. Altogether, roughly 40 percent of the 3rd-grade students who scored below the Level-2 threshold in 2002–03 were promoted.

The Problem with Earlier Studies

Traditionally, the retention of a student, uncommon as it was, resulted from an individual teacher’s assessment of the student’s ability to succeed at the next level. But such teacher discretion, while arguably desirable as a matter of policy, is the primary reason earlier studies of social promotion are flawed. We must assume from studying those retention programs, which are still the predominant practice in schools throughout the United States, that students who were held back were fundamentally different from students who were promoted. Because teachers were considering intangible factors, even when race, gender, family income, and academic achievement are the same, there was no way to isolate the effect of being held back, much less to make reasonable conclusions about the effects of retention on a student’s academic achievement or the probability of his dropping out of high school. Are students who were retained less likely to graduate because they were retained? Or were they retained because of characteristics that also predisposed them to drop out? Because the retention policies were subjective, we will simply never know.

There are also reasons to believe that subjective retention policies affect students differently than policies that use promotion criteria like performance on standardized tests. If promotion depends on an individual teacher’s assessment of a child, then that child is not likely to know what he or she must do to avoid being held back. Also, if few students were being held back, then those students might perform worse because they felt excluded and inferior. A policy that holds back thousands of students might dilute this sense of being singled out. Finally, subjective assessments of students are vulnerable to inappropriate influences, including teachers’ prejudices and pressure brought by parents, in ways that objective criteria of performance might inhibit.

Implementing objective standards, even if they were accompanied by subjective exemptions, might significantly change the effects of retention in ways that previous research could not anticipate or measure. For research purposes, objective retention policies also create a useful comparison group of students not subject to retention. In the case of Florida’s program to end social promotion, for example, we can compare students who were subject to the threat of retention with students who would have been had they been born a year later.

What a Difference a Year Makes

To determine the impact of ending social promotion for 3rd graders in Florida, we compared low-scoring 3rd graders in 2002, the first students to be subject to the program, with low-scoring 3rd graders from the previous year. Of the 43,996 3rd graders in 2002 for whom we have valid test scores on both FCAT math and reading assessments, 60 percent were actually retained. By contrast, of the 45,401 3rd graders in 2001 for whom we have valid test scores, only 9 percent were retained. Our analysis assumes that the students from the two school years should be similar in all respects except for the year in which they happened to have been born. We analyzed the test-score improvements made between each student’s first 3rd-grade year and the following year on both the state’s own accountability exam and the Stanford-9, a nationally normed exam administered at the same time as the FCAT but not used for accountability purposes.

We measure FCAT performance using developmental-scale scores, which allow us to compare the test-score gains of all the students in our study, even though they took tests designed for different grade levels. Developmental-scale scores are designed to measure academic proficiency on a single scale for students of any grade and in any year. For example, a 3rd grader with a developmental-scale score of 1,000 and a 4th grader with a developmental-scale score of 1,000 have the same level of academic achievement; if a student gets a developmental-scale score of 1,000 in 2001 and then gets the same score of 1,000 in 2002, this indicates that the student has not made any academic progress in the intervening year. The developmental-scale scores required to reach Level 2 on the FCAT reading test were consistent for each year’s cohort.

We began by measuring the effect on all low-scoring 3rd graders of simply having been subject to the new policy. That is, we did not distinguish in our initial analysis between students who were actually retained and those who received an exemption and were promoted to the next grade. This analysis provides an estimate of the average impact of the policy change on all students in the state performing below the Level-2 benchmark. It also allows for the possibility that exempted students enjoyed spillover benefits from the retention policy, since they were now being instructed in a system in which fewer students in 4th grade were unprepared to do grade-level work.

To identify the policy’s average impact, we compared the gains in developmental-scale scores made by students who first entered 3rd grade in 2002 and scored below the FCAT benchmark with gains made by students who first entered 3rd grade in 2001 and scored below the FCAT benchmark. In making this comparison, we took into account other factors that could affect achievement gains, such as the student’s race, whether the student received a free or reduced-price school lunch, whether the student was deemed Limited English Proficient, and the student’s precise test score during his first 3rd-grade year. With these differences accounted for, the only distinction between the two groups of students was assumed to be that the former group entered the school system a year later and was therefore subject to the new policy in 3rd grade.

As discussed above, however, many low-scoring 3rd graders were granted exemptions and promoted to the 4th grade even under the new policy. We therefore also evaluated the effect of actually being retained, again controlling for race, eligibility for free or reduced-price lunch, English proficiency, and baseline test scores. In conducting this analysis, we also needed to account for the fact that the students who were held back were a select group of students who could differ in important ways from the promoted students. Presumably, teachers and other decisionmakers expected these students, unlike promoted students, to benefit from an additional year as 3rd graders. Fortunately, the fact that simply having entered school a year later increased the probability of retention for all low-scoring students again provides a way around this obvious selection problem. In essence, the statistical method we use compares those retained students that our data suggest would not have been retained the previous year with a comparable group of students who were not retained. Our results therefore indicate the effect of retention on those students who were held back as a result of the new policy.

During this time, Florida was engaged in other education reforms as well: instituting several school-voucher programs, increasing the number of charter schools in the state, and improving the system used to assign grades to schools based on the FCAT. However, it is reasonable to assume that whatever effect these other policies have on our analyses is minor. In order for the existence of another policy to affect our results significantly, we would have to believe that the program substantially improved the education of the 3rd graders in 2002–03 without having a similar effect on the previous year’s cohort. Moreover, while a sudden policy change could conceivably explain the overall improvements between the two cohorts, it is difficult to see how such a change could cause substantially larger gains among those students actually retained.

Retention Works

Our fundamental findings from an analysis of the 3rd-and 4th-grade data for these two years indicate that the performance of students identified for retention, regardless of whether they were retained or exempted and promoted, exceeded the performance of low-performing students from the previous year who were not subject to the retention policy; and students who were actually retained made the larger relative gains.

Students identified for retention by the Florida policy gained 0.06 of a standard deviation in reading on both the FCAT and Stanford-9 over equally low-performing 3rd graders from the previous school year (see Figure 1). In math, students identified for retention surpassed low performers who were not subject to the policy by 0.15 standard deviations (4.8 percentiles) on the FCAT and 0.14 standard deviations (4.4 percentiles) on the Stanford-9.

Students who were actually retained experienced even larger relative improvements (see Figure 2). Retained students performed better than low-scoring students who were promoted by 0.13 standard deviations (4.10 percentiles) on the FCAT and 0.11 standard deviations (3.45 percentiles) on the Stanford-9 in reading. In math retained students improved 0.30 standard deviations (10.0 percentiles) on the FCAT and 0.28 standard deviations (9.3 percentiles) on the Stanford-9 over promoted students.

Some critics of the new retention policies argued that teachers and schools would respond to them by manipulating test scores, either directly by cheating or indirectly by teaching students skills that would help them to improve their test scores but would not provide real academic proficiency. This argument would have merit only if we found strong gains on the high-stakes FCAT and no similar gains on the low-stakes Stanford-9, for which there is no incentive to manipulate scores. But our results are consistent between the FCAT and the Stanford-9, indicating that there have been no serious manipulations of the high-stakes testing system. If teachers are in fact changing their curricula with the intent to “teach to” the FCAT, they are doing so in ways that also contribute to gains on the highly respected Stanford-9. This would indicate that teachers have made changes resulting in real increases in students’ proficiency.

An unexpected benefit of the retention policy is the improvement in math scores. This might seem odd, given that it is the reading portion of the FCAT that students must pass to earn promotion and that the rhetoric supporting Florida’s retention program emphasizes that it will improve student literacy. Of course, the math gains could simply reflect the fact that math skills are learned primarily in schools, while reading is practiced both in and outside of school. For this reason, evaluations of school reforms frequently find stronger effects in math than in reading. Alternatively, it may be that students who were retained specifically because of their poor reading skills are particularly poor in that subject and that this limits their room for improvement.

We also explored the possibility that the objective retention program could have different effects on students of different races. Our results show gains of similar sizes by the three racial groups for which we have an adequate sample size to have reasonable confidence in our findings: white, black, and Hispanic. The exception is for whites’ performance on the FCAT reading test. It is difficult for us to interpret why white students would fail to benefit from the retention policy as measured by the FCAT reading test but would be shown to benefit as measured by the Stanford-9 reading test.

Our results also suggest that low-scoring Florida 3rd graders who were given an exemption and promoted might have bene­fited from another year in the 3rd grade. This does not mean that it would be wise to eliminate all exemptions to the testing requirement. There are certainly students for whom testing is either inappropriate or whose performance on other academic measures could reasonably indicate that they would be better served by moving on to the next grade. However, our findings do indicate that teachers and school systems should be cautious when granting exemptions.

What It Means

At first glance our findings seem inconsistent with evaluations of Chicago’s program ending social promotion, to our knowledge the only similarly designed retention policy to be evaluated using comparable methods. In Chicago, students in the 3rd, 6th, and 8th grades must exceed benchmarks on the Iowa Test of Basic Skills (ITBS), a respected standardized test, in order to be promoted to the next grade. In a study conducted in 2004 by scholars at the Consortium on Chicago School Research, the performance of 3rd- and 6th-grade students who scored just below the benchmark on the ITBS, most of whom were retained because of the mandate, was compared with the performance of students who scored just above the benchmark, most of whom were promoted. The Chicago researchers were able to measure test-score performance for two years after implementation of the program. They found benefits from the program after one year, similar to what we found in Florida, but discovered that those benefits went away after the second year. Third-grade students were not affected, and 6th-grade students were negatively affected by the policy in their performance on the ITBS reading test. The findings on the Chicago retention program emphasize the importance of following the progress of retained students in Florida over time.

Still, the Chicago policy differs from Florida’s in some respects. In 1999 the Chicago policy stopped allowing students to be retained twice, which Florida’s policy does allow. This difference might reduce teachers’ motivation to work with already retained students, whom they now can expect to be promoted the next year regardless of their performance. Other programs with different and more stable retention policies might show different results.

Finally, while our study provides valuable information about the effectiveness of Florida’s policy to end social promotion, it does not offer a full catalog of the policy’s benefits or of its potential costs. It will be some time before we can examine whether retention increased or reduced the probability of dropping out of school later on. Most important, it does not provide any information about the program’s effects on students’ academic progress the first time they were in 3rd grade. The policy’s greatest benefits could result not from retention itself, but rather from increased efforts on the part of teachers and even students to avoid being retained in the first place.

-Jay P. Greene is professor and head of the Department of Education Reform, the University of Arkansas; he is also a senior fellow at the Manhattan Institute. Marcus A. Winters is a doctoral fellow, the University of Arkansas and a senior research associate at the Manhattan Institute.

Students will test the limits of acceptable behavior in myriad ways better known to school teachers than to judges; school officials need a degree of flexible authority to respond to disciplinary challenges; and the law has always considered the relationship between teachers and students special.
— Supreme Court Justice Stephen Breyer

In Morse v. Frederick, a 2007 First Amendment student free speech case, the Supreme Court held that a school official may restrict student speech at a school-supervised event when that speech is viewed as promoting illegal drug use. Filing a separate opinion, Justice Stephen Breyer echoed concerns expressed by his conservative colleagues that school authority was being undermined by legal challenges. Since the 1960s, courts have become increasingly involved in regulating U.S. schooling in general, but especially in the area of school discipline. Justice Breyer noted in his opinion, “Under these circumstances, the more detailed the Court’s supervision becomes, the more likely its law will engender further disputes among teachers and students.”

School discipline is a critical area for research, as student interaction with school institutional authority is one of the primary mechanisms whereby young people come into contact with and internalize societal norms, values, and rules. It is thus significant that the number of cases reaching state and federal appellate courts has surged back up to levels attained during the early 1970s when civil rights cases had a central place on the national political agenda (see Figure 1). Our research indicates that both educators and students understand the former’s authority to be more limited and the latter’s rights more expansive than has actually been established by case law.

School Discipline in Court
Until the late 1960s, parents and students rarely challenged the disciplinary actions of school authorities, viewing common schools as providing instruction, instilling virtue, and fostering the ideals of our nation. Then, as conceptions of youth rights began to shift, and as institutions that provided support for the expansion of these rights emerged, students and parents, with the support of public-interest lawyers, began to question and challenge school disciplinary practices in court.

Table 1 summarizes key school-related rulings from the Supreme Court over the last 40 years. From 1969 to 1975, amid increasing legal challenges to the regulation of student expression in school, the Court’s rulings largely confirmed students’ rights to various free expression and due process protections. The most important decision affecting how schools approach student discipline was Goss v. Lopez, decided by the Supreme Court in 1975. During a patriotic assembly at Central High School in Columbus, Ohio, in 1971, expressions of student unrest over the lack of African American curricula turned into a week of demonstrations and disturbances. Dozens of students were suspended for up to 10 days without formal hearings or notification of the specific charges against them. The Supreme Court case hinged on whether the disciplinary actions improperly denied students their rights to a public education. In ruling for the students, the Court granted “rudimentary” due process rights to those suspended from school for fewer than 10 days, as well as “more formal protections” for students facing longer exclusions.

In recent years, courts at all levels have dealt with cases challenging the enforcement of “zero-tolerance” policies that establish severe and nondiscretionary punishments for violations involving weapons, violence, drugs, or alcohol. At the same time, an increasing number of cases have appeared in lower courts that involve students and families suing schools for failing to provide adequate discipline within school facilities. These cases have alleged climates that permit bullying, sexual harassment, or other forms of school violence (including school shootings). Thus, in recent years, schools have been sued for both disciplining students and not disciplining them.

Since 1975, the Supreme Court has generally been less favorable toward students than it was during the early years of the civil rights movement.. This shift in orientation occurred for diverse reasons, including growing public concern about the level of violence and disorder in public schools, the changed political climate following the end of the Vietnam era, and a pattern of increasingly conservative judicial appointments during the Nixon, Reagan, and Bush administrations. The Supreme Court’s 2007 decision in Morse v. Frederick continued the post-1975 pattern of sympathy with schools that are facing challenges to their disciplinary authority, but did not, as some of the media coverage implied, alter the general contours of student rights as previously established. Its June 2009 decision in Safford United School District v. Redding, in which eight justices agreed that a near strip-search of an 8th-grade girl suspected of concealing prescription-strength ibuprofen was unconstitutional, at first glance appears to be an exception—a sign that the courts will continue to watch over the shoulders of school officials to ensure that reasonableness and proportionality prevail. Yet a majority on the court ruled that the administrators who conducted the search could not be held personally liable because of the uncertainty of the law in this area.

Appellate Case Patterns
While Supreme Court decisions are important because every school in the nation must adhere in principle to its rulings, these few landmark cases do not encompass the universe of legal challenges regarding school discipline and related policies. To discern the larger contours of the legal climate facing schools, we analyzed all appellate-level federal and state court cases in which school efforts to discipline and control students have been challenged. As a whole, decisions in these cases are often complex and contradictory in providing practical guidance to schools regarding specific disciplinary matters. We included cases involving the use of state agents (such as the police) acting on behalf of school authorities to deal with students in the vicinity of school grounds. We excluded instances of conflicts between schools and teachers (such as teacher dismissal cases) and between schools and nonstudent outsiders (such as drug- and weapon-free-zone cases that did not involve students), as well as student rights cases focused exclusively on free speech issues (that is, those not combined with the school’s use of suspension, expulsion, corporal punishment, and transfer). We also excluded cases in which students allege that school authorities have breached their duty to maintain safety in the school and to protect students from harm.

Of course, we did not include the vast majority of litigation, which was either settled before hearing or never reached state and federal appellate courts. Still, our methods provide a way to gauge the general character and broad trends in legal challenges that contemporary educators face. Appellate-level court cases define case law, generate media coverage, influence public perceptions, and can be tracked over time as an empirical indicator of the broad parameters of court climate toward school discipline. We found that not only has the frequency of legal challenges greatly varied over time, but the content and direction of outcomes has shifted as well.
The newfound willingness to challenge school authority became evident in the surge of litigation during the late 1960s. In part because of increased institutional support from public-interest legal advocacy groups and the legal services program of the Office of Economic Opportunity, from 1968 to 1975 an average of 39.1 public school K—12 cases per year reached the appellate level. After important legal precedents were set and institutional support waned, the average number of cases declined but then took a sharp upturn from 1993 on, with a peak of 76 cases in 2000 and a total of 65 in 2007. We present here the overall number of cases rather than a relative measure accounting for public school enrollment, given that media coverage and individual understandings reflect the former indicator. Nevertheless, a measure of state and federal court cases calculated per enrolled student would demonstrate similar upward trends, more than doubling from the years 1976—1992 to the 2003—2007 period.

The substance of the cases brought before the courts has also varied over time, with protest and free expression cases decreasing markedly through 1992 (see Figure 2a). Recently, courts have witnessed a reemergence of these issues. Cases involving alcohol and drugs rose during the intermediate time periods that coincided with national attention to the “War on Drugs” and then diminished. Those involving weapons and violence have increased to nearly 40 percent of all K—12 public school discipline cases since 1993. In addition, school discipline court cases increasingly have involved student disability. From 2003 to 2007, 18 percent of cases included discussion of student disability status. Since the 1970s, legal entitlements and protections have grown for students classified as disabled because of learning, physical, or behavioral handicaps (including psychological disorders that are associated with the manifestation of student misbehavior). Special education students thus gained additional protections related to school discipline, particularly in cases in which infractions could be attributed to the individual’s disability.

Over time, we found that courts in general have become less favorable to student claims across these areas of litigation (see Figure 2b). However, since the number of court challenges has increased in recent decades, the likelihood of a school facing a legal environment in which a student has recently been successful in a court challenge over school discipline has not significantly diminished.

Socioeconomic Disparities
Many of the early school discipline cases were brought to ensure that the rights of less-advantaged students were protected. New evidence suggests, however, that litigation is increasingly used strategically and instrumentally by families from relatively privileged origins to promote the interests of their children. Research (by Irenee Beattie, Josipa Roksa, and Richard Arum) that examined appellate court cases from 2000 to 2002 found that, on average, those cases emerged from secondary schools with 29 percent nonwhite students compared to 37 percent nonwhite students in the national population of secondary schools (the latter weighted for enrollment size to be comparable to the court case data); appellate cases also emanated from schools with more educational resources per student (student/teacher ratios of 16.3 compared to 17.5 nationally).

National surveys of teachers and administrators reveal a similar middle-class bias in legal challenges. A reanalysis of a Harris survey of teachers and administrators conducted by Melissa Velez and Richard Arum for Common Good in 2003 examined the proportion of public school educators (a combined sample of teachers and administrators) who reported that either they or someone they knew personally had been sued by a student or parent. Educators in suburban schools with less than 70 percent nonwhite students had a 47 percent probability of having experienced contact with an adversarial legal challenge compared to a 40 percent chance for educators in all other schools. Although much of the development of student rights originally emerged from concern about nonwhite students in urban areas, educators in those settings had only a 41 percent probability of contact with a legal challenge.

In collaboration with colleagues working on the School Rights Project (Lauren Edelman, Calvin Morrill, and Karolyn Tyson), we conducted a national telephone survey of 600 high school teachers and administrators and site-based surveys of 5,490 students and 368 educators on perceptions and experiences of the law in schools. In our site-based work, which included in-depth interviews and ethnographic fieldwork, we examined 24 high schools with varying legal environments situated across three states (New York, North Carolina, and California), stratified by school type (traditional public, charter, and Catholic) as well as by student socioeconomic composition. We found that 15 percent of public school teachers and 55 percent of public school administrators have been threatened with a legal suit over school-related matters. For administrators with more than 15 years of experience in the position, the figure rose to 73 percent. Administrators’ actual experience with being sued for school-related matters occurs at a lower rate (14 percent), but is still the source of considerable professional anxiety, given that these cases—following Wood v. Strickland (1975)—include vulnerability to personal liability claims. We again found that legal challenges are concentrated in schools with more-privileged students. When we looked solely at administrators working in urban public schools with more than half of students eligible for free lunch, we found—albeit with a sample of only 16 cases—not a single report of administrators being sued for a school-related matter.

That legal mobilization is dependent on economic resources needed to pursue such challenges is in general not surprising. We documented evidence of this association, however, to illustrate that regardless of the institutional and political origins of student rights, today legal mobilization in schools largely reflects patterns of socioeconomic inequalities. In the School Rights Project, we found that white students were nearly twice as likely as nonwhite students to report having pursued a formal legal remedy for a perceived rights violation.

Legal Understandings and School Practices
Legal mobilization is a relatively rare occurrence, a small tip of a much larger legal-dispute pyramid. School discipline today is profoundly shaped by legal understandings that are only partially and indirectly related to formal regulation and case law. We highlight here the extent to which both students and educators have developed an expansive definition of legal rights of students, the relationship between this sense of legal entitlement and school disciplinary practices, and perceptions of the fairness and legitimacy of various school disciplinary practices.

The institutionalization of student due process protections goes well beyond appellate case law, having been enshrined in extensive state statutes and administrative regulations. The accompanying sidebar (page 65) provides a sense of the extent to which law has come to permeate school practices by highlighting codified disciplinary procedures in New York City. While discipline policies vary across schools, districts, and states—and as the nation’s largest school district the New York City public schools are likely more bureaucratized and formalized in matters of school discipline than smaller districts—the scale, scope, and level of complexity of the legal regulations affecting day-to-day school practices appear quite formidable.

Generally speaking, educators and students have developed a set of legal understandings that assumes a broad and expansive definition of student legal entitlements. Following the Goss decision, students have been granted rudimentary due process protections when facing minor discipline and more formal due process protections when facing more serious forms of discipline (such as long-term expulsion or suspension). The Goss decision delineated procedural safeguards, stating that “the student be given oral or written notice of the charges against him and, if he denies them, an explanation of the evidence the authorities have and an opportunity to present his side of the story.” More formal due process protections may include the right of students to “summon the accuser, permit cross-examination, and allow the student to present his own witnesses. In more difficult cases, he (the disciplinarian) may permit counsel.”

We are interested in individuals’ perceptions of such protections, since students’ and educators’ beliefs about rights likely have real consequences for school authority and disciplinary procedures. In the School Rights Project, we specifically asked students and teachers which due process protections were required when students faced various disciplinary sanctions. We found that while expectations of formal due process protections were broadly diffused for students when facing major disciplinary actions, many of them had also come to expect these legal entitlements when facing minor day-to-day discipline. For example, 62 percent of public school students in our sample believed that, if faced with long-term suspension or expulsion, they were legally entitled to at least one of the following: a formal disciplinary hearing, opportunity to be represented by legal counsel, opportunity to confront and cross-examine witnesses bringing the charges, or opportunity to call witnesses to provide alternative versions of the incident. Approximately one-third of students also believed that they were legally entitled to some form of formal due process protection when they had their grades lowered for disciplinary reasons (33 percent), were suspended from extracurricular activities (36 percent), or faced in-school suspension (35 percent).

We found that students’ sense of legal entitlement was expansive, and that teacher and administrator expectations of required student due process protections were even more so. For example, when asked about lowering student grades for disciplinary reasons, approximately half of public school teachers and administrators responded that this action was prohibited; among the educators who did think such disciplinary actions were permissible, 32 percent reported that students subject to such disciplinary sanctions were entitled to formal due process protections.

In the School Rights Project, we found that increased perceptions of student legal entitlements correlate with decreased reports of the fairness of school discipline. This conclusion mirrors James Coleman’s finding that Catholic school students in the 1980s were significantly more likely to perceive school discipline to be fair than public school students, who possessed far greater formal legal protections. Educators and students have developed a generalized sense of legal entitlements, while school practices have, in many settings, become increasingly authoritarian, with student misbehavior often subject to criminalization and formal legal sanction. These internal contradictions enhance students’ sense of the unfairness of school discipline. Longitudinal research has demonstrated that students who perceive school discipline as unfair are more likely to disobey teachers, disrupt classroom instruction, and in general fail to develop behaviors conducive to educational success and related positive outcomes.

Also, in recent decades schools have moved away from disciplinary practices that rely on the judgment, discretion, and action of professional educators and have turned instead to reliance on school security guards, uniformed police, technical surveillance, security apparatus, and zero-tolerance policies. The latter techniques are ill suited to the pedagogical task of enhancing the moral authority of educators to support the socialization of youth, that is, the internalization of norms, values, and rules.

Conclusion
Citizens, legislators, judges, and policymakers have begun to recognize and question legal interventions in situations involving school discipline and authority. We add to this discussion our findings that the legal understandings underlying school discipline policies depart in significant ways from the case law on which they are assumed to be based, according expansive rights and protections to students, even as the courts have tended to side with school authorities. We also document that although public-interest lawyers were initially motivated to expand student legal rights as part of a larger strategy to reduce social inequality, legal challenges to school disciplinary actions are disproportionately the province of white and higher-income students and their families.

The expansion of student legal entitlements has been accompanied by the increasing formalization and institutionalization of school discipline. As educators’ discretionary authority over school discipline has been challenged and undermined, counterproductive authoritarian measures such as zero-tolerance policies have been implemented in its place. But to be educationally effective, school discipline requires that educators have moral authority and students perceive their actions as legitimate and fair. Ironically, the expansion of student legal rights, rather than enhancing youth outcomes, has increased the extent to which schools have relied on authoritarian measures, decreased the moral authority of educators, and diminished the capacity of schools to socialize young people effectively.

As various social and political actors consider legal regulatory reforms, it is important to recognize that the expansion of students’ legal entitlements has also increased the potential for student dissent in U.S. schools, whether of a political, religious, or ideological character. At the same time, individual students and families with sufficient resources are able to contest what they perceive as unfair disciplinary sanctions or rights violations. These gains have come at a pedagogical and societal cost, as the resolution of school disciplinary matters has increasingly moved—as Justice Breyer feared—from the schoolhouse to the courthouse.

Richard Arum is professor of sociology and education at New York University, where Doreet Preiss is a research fellow and doctoral candidate. This essay is adapted from “Still Judging School Discipline,” in Joshua M. Dunn and Martin R. West, eds., From Schoolhouse to Courthouse: The Judiciary’s Role in American Education (Brookings, 2009).

Understanding whether troubled children in fact generate spillover effects in school is important for two reasons. First, the existence of substantial spillovers caused by family problems such as domestic violence would provide an additional compelling reason for policymakers to find ways to help troubled families. Second, because many education policies change the composition of school and classroom peer groups, it is important to understand how such changes may affect student achievement. For example, a common concern regarding the ongoing push to “mainstream” emotionally disturbed students in regular classroom settings is that doing so may undermine the performance of other students. Similarly, the tracking of students into classrooms based on ability or academic performance may group disadvantaged children with the most disruptive students. The validity of these concerns hinges on whether and how classroom exposure to troubled peers affects student achievement and behavior.

Credibly measuring negative spillovers caused by troubled children has been difficult. Most data sets do not allow researchers to identify troubled children. Even when such students are identified in the data, it is difficult to determine if a disruptive child causes his classmates to misbehave or if his classmates cause him to be disruptive, what scholars of peer effects call the “reflection problem.” In addition, troubled children are likely to attend the same schools as other disadvantaged children. One must rule out the possibility that the disruptive student and his classmates misbehave due to some common unobserved factor.

We overcome these problems in this study by utilizing a unique data set in which information on students’ academic achievement and behavior is linked to domestic violence cases filed by their parents. This data set allows us to identify troubled children more precisely than we could by using conventional demographic measures. Moreover, we can identify children who are troubled for specific family reasons and not because of their peer group. This allows us to measure peer effects free from the reflection problem, providing a rare opportunity to test the notion that even one “bad apple” impedes the learning of all other students.

Our results confirm, first, that children from troubled families, as measured by family domestic violence, perform considerably worse on standardized reading and mathematics tests and are much more likely to commit disciplinary infractions and be suspended than other students. We find also that an increase in the number of children from troubled families reduces peer student math and reading test scores and increases peer disciplinary infractions and suspensions. The effects on academic achievement are greatest for students from higher income families, while the effects on behavior are more pronounced on students who are less well-off. The results of our analysis provide evidence that, in many cases, a single disruptive student can indeed influence the academic progress made by an entire classroom of students.

Data

In our study, we use a confidential student-level data set provided by the school board of Alachua County in Florida. This data set consists of observations of students in the 3 rd through 5th grades from 22 public elementary schools for the academic years 1995–96 through 2002–03. The Alachua County school district is large relative to school districts nationwide, with roughly 30,000 students; in the 1999–00 school year, it was the 192nd largest among the nearly 15,000 districts nationwide. The student population in our sample is approximately 55 percent white, 38 percent black, 3.5 percent Hispanic, 2.5 percent Asian, and 1 percent mixed race. Fifty-three percent of students were eligible for the federal free or reduced-price lunch program.

The test-score data consist of reading and mathematics scores from the Iowa Test of Basic Skills and the Stanford 9, both nationally normed exams. Reported scores indicate the percentile ranking on the national test relative to all test-takers nationwide. Because the reading and math results are so similar, we use a composite score calculated by taking the average of the math and reading scores. The average student in our data scored at the 53rd percentile, or just above the national norm.

Yearly disciplinary records, which include incident type and date, are available for every student in our sample. Incidents are reported in the system if they are serious enough to require intervention by the principal or another administrator. We focus on three behavioral outcomes from these records: the probability the student was involved in a disciplinary incident, the total number of disciplinary incidents per student, and the probability the student was suspended. In a typical year, 18 percent of the students in our data set were involved in a disciplinary incident, the average student was involved in 0.56 incidents, and 9 percent of students were suspended.

We gathered domestic violence data from public records information at the Alachua County courthouse, which included the date filed and the names and addresses of individuals involved in domestic violence cases filed in civil court in Alachua County between January 1, 1993, and March 12, 2003.Cases are initiated when one family member (typically the mother) petitions the court for a temporary injunction for protection against another member of the family (most often the father or boyfriend). Students were linked to cases in which the petitioner’s first and last name and the first three digits of the residential address matched the parent name and student’s residential address in the annual school record. In that way, we were able to identify the set of students who could be matched to a domestic violence case from1993 to 2003. In total, 4.6 percent of the children in our data set were linked to a domestic violence case filed by a parent, split equally between boys and girls. Sixty-one percent of these children were black, while 85 percent were eligible for subsidized school lunches.

Students linked to a domestic violence case performed at lower levels academically and were more likely to have been involved in a disciplinary incident than other students in the district. Boys exposed to domestic violence, for example, performed at the 37th percentile academically, as compared with the 52nd percentile for boys who were not exposed. Forty-three percent of boys exposed to domestic violence were involved in a disciplinary incident, as compared with 25 percent of boys who were not exposed. Girls exposed to domestic violence performed at the 41st percentile academically and 19 percent of them were involved in a disciplinary incident, as compared with the 55th percentile and 11 percent for girls who were not exposed to domestic violence (see Figure 1).

Measuring Peer Effects

Our main analysis examines the impact of troubled children on their peers. We assume there is no feedback loop in which a student’s peers cause the domestic violence in the household. This assumption appears reasonable, as none of the most likely determinants of domestic violence can plausibly be caused by an elementary school child or her peers.

To overcome the bias that results from self-selection into peer groups, our main analysis compares cohorts of students in the same grade at the same school in different years. For example, we compare the 3rd graders in a given school this year with the 3rd graders in the same school last year to see whether the cohort with more students exposed to domestic violence had higher or lower student achievement. Restricting the comparisons to students attending the same school ensures that any effects we observe reflect the impact of troubled students and not the fact that schools with more such students differ in unobserved ways from other schools. We measure peer domestic violence at the cohort level (that is, across all students in a grade at a school) as opposed to the classroom level due to the possible sorting of students into classrooms according to their achievement and behavior. We also adjust for differences among students in a large set of individual characteristics—most importantly whether particular students had been directly exposed to domestic violence—but also race, gender, subsidized lunch status, and median zip code income.

Results

Our results indicate that troubled students have a statistically significant negative effect on their peers’ reading and math test scores. Adding one troubled student to a classroom of 20 students results in a decrease in student reading and math test scores of more than two-thirds of a percentile point (2 to 3 percent of a standard deviation). The addition of a troubled peer also significantly increases misbehavior of other students in the classroom, in effect causing them to commit 0.09 more infractions than they otherwise would, a 16 percent increase. These are effects that could accumulate over time if the same students are repeatedly exposed to troubled peers.

These average effects also mask a few interesting differences across student groups. We find that troubled peers have a large and statistically significant negative effect on higher income children’s math and reading achievement, but only a small and statistically insignificant effect on the achievement of low-income children. However, we find the opposite pattern for disciplinary outcomes. The presence of troubled peers significantly increases the misbehavior of low-income children, but does not increase the disciplinary problems of higher-income children (see Figure 2).

Results of examining the differential effects of peers from troubled families by race and gender show relatively large negative and statistically significant test-score effects on white boys and statistically insignificant effects on black boys, black girls, and white girls. Adding one troubled peer to a classroom of 20 students reduces white boys’ reading and math scores by 1.6 percentile points and black boys’ reading and math scores by 0.9 percentile points (the effects on girls are negligible). Troubled peers increase disciplinary problems for all subgroups except for white girls. The effects are largest for black girls. One troubled peer added to a classroom of 20 students increases the probability that a black girl commits a disciplinary infraction by 2.2 percentage points (an increase of 10 percent over what would otherwise be the case).

Finally, we examined whether troubled boys affect their peers differently than do troubled girls. Across all outcome variables, both academic and behavioral, the negative peer effects appear to be driven primarily by the troubled boys, and these effects are largest on other boys in the classroom. The results indicate that adding one troubled boy to a classroom of 20 students decreases boys’ test scores by nearly 2 percentile points (7 percent of a standard deviation) and increases the probability that a boy will commit a disciplinary infraction by 4.4 percentile points (17 percent). Apparently, troubled boys generate the strongest adverse peer effects, and other boys are most sensitive to their influence.

Testing Key Assumptions

Of critical importance to our method is the assumption that students are not systematically placed into or pulled out of a particular grade cohort within a school depending on the domestic violence status of the student or his peers. For example, if parents who really value education were more likely to pull their children out of a cohort with a particularly high proportion of peers from troubled families, such nonrandom selection would cause us to erroneously attribute lower performance to the presence of the troubled peers.

We performed several additional analyses to probe the robustness of our results to this critical assumption. As a first test for nonrandom selection of students into or out of particular schools and cohorts of students, we examined whether peer family violence appears to have an effect on cohort size or student characteristics such as race, gender, and income. In the absence of nonrandom selection, we expect to find no correlation between these characteristics and the peer family violence variables. This is indeed what we find.

Next, we noted that some parents may be more likely than others to put their children in private schools or move to a different school zone because of a particularly bad cohort, but that parents may be less likely to pull one child out of the school due to a particularly bad cohort when that child has a sibling in the same school. When we calculated peer effects only on children with siblings in the school, the results were essentially the same as those for the full sample.

One might also be concerned that some families are, for some reason, unable to remove their children from cohorts with a large number of troubled peers. To check this potential cause of nonrandom selection, we calculated results based only on comparing students to their siblings. We found that the sibling in the cohort with more children from troubled families has lower test scores and more disciplinary problems. These within-family results are roughly two-thirds the size of the estimates for the full sample, but the differences between the two sets of results are not statistically significant.

For a final check, we added controls for a full set of cohort-level variables, including race, gender, participation in the federal subsidized lunch program, and median zip code income. These variables control for any potential changes in cohort characteristics not captured by our full set of individual controls in the main analysis. In addition, this allows us to examine whether the presence of children exposed to domestic violence is merely a proxy for other peer characteristics, such as family income. The results indicate that the negative peer effects are not likely driven by observable factors, such as family income, that are correlated with domestic violence.

Collectively, these tests provide strong evidence that our findings are not the result of families changing schools in response to the number of children from troubled families in their child’s grade at an assigned school.

Discussion

In addition to knowing how children from troubled homes affect their peers through interaction with their cohort at school, one may also wish to know the precise way in which the troubled families cause the peer effects. This is a particularly challenging task given that researchers have consistently found, as we have, that domestic violence is correlated with other negative family characteristics, such as poverty, unemployment, low levels of education, and substance abuse. While we cannot conclusively attribute the effects found to the causal effect of domestic violence per se, we can exploit the timing of the domestic violence filings to provide suggestive evidence of whether the negative spillovers are due to domestic violence or some other factor correlated with it.

Specifically, we examine whether the negative spillovers associated with children from troubled families are smaller after the parent files the case than before the case is filed. Survey research shows that on average, violence had occurred in the family for more than four years prior to the reporting of the incident. However, 87 percent of the respondents indicated that the reporting of the incident “helped stop physical abuse.” Consequently, if domestic violence itself is causing the negative spillovers on the child’s classmates, then we would expect the spillovers to be smaller when the parent of the peer had already filed for the injunction against domestic violence.

To investigate whether exposure to domestic violence is the potential mechanism through which the spillovers occur, we constructed two peer domestic violence variables: reported and as yet unreported violence. By definition, reported domestic violence means that the petition for the injunction was filed before the student test was taken and unreported domestic violence signals that the filing occurred after the test date.

We find substantially larger effects for the proportion of peers with unreported domestic violence (that is, those whose parents had not yet filed for the injunction) than for those with past domestic violence. For example, the test-score effects for troubled boy peers on boys are statistically insignificant for reported violence, while they are large and highly significant for unreported violence. The larger peer effects for unreported domestic violence suggest that the violence in the home may itself be playing a role in driving the effects. However, we remain cautious with this interpretation, as we have no direct information regarding the details of the family environments for students in our sample.

Conclusion

Our findings have important implications for both education and social policy. First, they provide strong evidence of the validity of the “bad apple” peer effects model, which hypothesizes that a single disruptive student can negatively affect the outcomes for all other students in the classroom. Second, our results suggest that policies that change a child’s exposure to classmates from troubled families will have important consequences for his educational outcomes. Finally, our results provide a more complete accounting of the social cost of family conflict. Any policies or interventions that help improve the family environment of the most troubled students may have larger benefits than previously anticipated.

Scott Carrell is assistant professor of economics at the University of California–Davis. Mark Hoekstra is assistant professor of economics at the University of Pittsburgh.

The federal law No Child Left Behind (NCLB) requires states to “restructure” any school that fails for six years running to make Adequate Yearly Progress (AYP) toward full proficiency on the part of all students by the year 2014. The law provides a number of restructuring options, including turning over the school’s management to a private for-profit or nonprofit entity. Only a few school districts nationwide have sought help from either type of organization in the management of low-performing schools. But in 2002 the School District of Philadelphia, at the request of the state of Pennsylvania, asked entities of both types to participate in a substantial restructuring of many of its lowest performing schools. The restructuring initiative was directed by the Philadelphia School Reform Commission (SRC), which contracted with for-profit organizations to manage 30 elementary and middle schools and with nonprofit organizations to manage 16 schools.

The policy intervention in Philadelphia raises questions of general interest: Do students at schools assigned to for-profit or nonprofit managers learn more than would be expected had those schools remained under school district management? Is for-profit management more or less effective at raising achievement than nonprofit management? Told most simply, the Philadelphia story provides a threefold answer to these questions: 1) for-profits outperform district-managed schools in math but not in reading; 2) nonprofits probably fall short of district schools in both reading and math instruction; and 3) for profits outperform nonprofits in both subjects. However, the answers require both explication and qualification.

The Theoretical Debate

The distinction between for-profit and nonprofit management has been a topic of continuing discussion in the scholarly literature on school reform. Nobel Prize–winning economist Milton Friedman theorized that for-profit firms are more effective because they have clear economic incentives to lift student performance. The firm can build its reputation (and in the long run generate a profit) only if it becomes known for running effective schools. Others have suggested, however, that for-profit firms are likely to cut costs and thereby shortchange students in order to benefit the firm’s owners and shareholders. The debate over nonprofit organizations takes a different form. Some have argued that nonprofit managers are likely to be effective because they have close ties to the community in which they are embedded and can enlist the energies of committed entrepreneurs, who devote all available resources to enhance student performance. But others caution that nonprofit managers may not have the experience, resources, or economic incentives necessary for building quality educational institutions.

The Intervention

Only after an intense political struggle did the Philadelphia school district ask for-profit and nonprofit managers to assume responsibility for a number of its schools. In 2001 Pennsylvania governor Tom Ridge, a Republican and school voucher supporter, indicated he would not support any increase in funding for the Philadelphia school district until an independent entity had assessed its financial practices and educational effectiveness. Philadelphia’s mayor at the time, Democrat John Street, knowing the district was facing a $215 million deficit, agreed to the study, and the state department of education asked Edison Schools to carry it out. Edison, a for-profit firm that manages charters and other schools under contracts with school districts, reported that the Philadelphia school district had spent $10 billion over a decade without being held accountable for the results. Governor Ridge refused to distribute any more state aid beyond current levels unless the school district agreed to a new partnership with the state.

When the local press reported that Edison was expected to assume management of many of the city’s schools, the local teachers union mobilized in opposition, and groups of parent and student activists held rallies throughout the city. As the turmoil was reaching its climax, Governor Ridge resigned from office to become the nation’s secretary of homeland security, and Pennsylvania’s lieutenant governor, Mark Schweiker, became governor. Upon assuming office, Schweiker said he would not serve beyond the current term, ending in 2002, a decision that weakened his leverage vis-à-vis the Philadelphia school district.

The stage was set for a compromise that would save face for all the parties involved. It was agreed that the school board should be replaced by a School Reform Commission, three members of which would be appointed by the governor and two by the mayor. The SRC decided that only a limited number of the lowest-performing schools in the district would be turned over to private management. Edison Schools was not to be the only private provider. Instead, seven entities—three for-profit and four nonprofit—were chosen. The SRC explained its decision by saying that multiple providers would yield information on the kind of management that was most effective. The SRC asked Edison Schools to manage 20 of the schools; another 5 each would become the responsibility of two other for-profit companies, Victory Schools and Chancellor Beacon Academies. Sixteen of the low-performing schools would be managed by nonprofit entities—the University of Pennsylvania (3 schools), Temple University (5 schools), Foundations (5 schools), and Universal (3 schools).The SRC also appointed a reform-minded superintendent, the energetic and outspoken Paul Vallas, who had instituted a series of reforms in Chicago at the behest of Mayor Richard Daley.

The for-profit firms were more experienced in running schools but they had fewer local political connections than did the nonprofit entities, as none had operated programs within the city itself. Though Edison was held in high regard by the Republican state secretary of education, it faced strong opposition within Philadelphia, especially from the local teachers union. Edison Schools could claim considerable experience at running schools, however, as it was the manager of 100 district and charter schools nationwide. Victory Schools, a company that offered single-sex education within classrooms, was the manager of schools in New York state and Baltimore, Maryland. To strengthen its local connections, Victory hired a former district employee to head up its Philadelphia effort. Chancellor Beacon operated some 80 private and charter schools, but it had not previously managed schools under contract with a school district. Shortly after the intervention began, Superintendent Vallas canceled the district’s contract with Chancellor Beacon, and its five schools were either brought back under district control or assigned to other providers.

By contrast, the nonprofit entities were—and have remained—politically well-connected institutions. The University of Pennsylvania is a Philadelphia icon, a highly prestigious Ivy League institution with a history dating back to Benjamin Franklin. Temple University’s status is less exalted, but it is nonetheless an established Philadelphia institution of higher learning. Foundations was created by one of the school district’s former associate superintendents and is staffed by many former district employees. Foundations also had close ties to a politically influential state representative active in community development programs. Universal, a community development corporation founded by Kenny Gamble, an immensely successful writer of soul music, has strong ties to Islamic leaders within Philadelphia’s black community.

Temple University and the University of Pennsylvania drew on the resources of their schools of education. Rather than taking on a general reorganization of the schools, they focused mainly on providing professional development to teachers, formal and informal assessment feedback to teachers, and within-classroom coaching services to students. Foundations operated afterschool programs and favored a teaching approach that relied on computer-based learning in which students progress at their own pace. Universal was known for boosting economic development and providing social services, but it had only limited experience managing schools.

The school district restricted the prerogatives of both for-profit and nonprofit managers in a number of ways. Management had to operate within the framework of the district’s collective bargaining agreements with its union employees, and teachers were allowed to transfer to other schools within the district if they wished. Many teachers chose to transfer, as could be expected given the fact that the schools in which they had been teaching were considered most in need of new management. The district also retained control over many aspects of school management, including the school calendar and the code of conduct for teachers and students.

All of that happened in 2002. Then over the following six years, several leadership changes occurred. Sandra Dungee Glenn was appointed by Democratic governor Edward Rendell (who had succeeded Schweiker) as the SRC’s new chair. Glenn was a former community organizer and active in Democratic politics. Superintendent Paul Vallas left for New Orleans, where he took on the daunting task of rebuilding the city’s post-Katrina school system. And Arlene Ackerman, who had previously served as superintendent in San Francisco and Washington, D.C., was hired by the SRC as the new superintendent.

Under its new leadership, the SRC removed five schools from the management of the for-profit firms (four from Edison, one from Victory) and one school from the management of a nonprofit entity (Temple). Acting upon the superintendent’s recommendation, the SRC decided that the six schools should come under direct district control once again because they had not made Adequate Yearly Progress (AYP), as required by No Child Left Behind. “It’s been six years—it is time to sort it out,” said SRC chair Glenn. Ackerman indicated that many more schools could come back under district control once a full-scale review had been undertaken.

Factors other than educational considerations may have influenced the selection of schools the district brought back under its control. As noted above, the nonprofit schools were well connected politically, while the for-profit firms were not. But it is also possible that the nonprofits were the more effective educational institutions. To see whether that was the case, we use a rigorous research design to estimate the impact of for-profit and nonprofit management in Philadelphia.

The Data

The Philadelphia school district supplied the information on which we base our analysis. Test-score, demographic, and school enrollment information on Philadelphia students in grades 2 to 8 from2001 through 2008 are available for each student. The test-score data come from three different tests. The Pennsylvania System of School Assessment (PSSA) is the test currently used for holding schools accountable for improved student learning in Philadelphia. But when the private management intervention began in the fall of 2002, that system of measuring school performance was still a work in progress. However, two other tests were given to some Philadelphia students between 2001 and 2006: the Stanford 9 and the TerraNova, both of which are nationally normed. In order to place the information from these tests on a common scale, we followed the standard practice of standardizing all scores by test, grade, and year to have a mean of zero and standard deviation of one.

We next classified schools as under for-profit management, under nonprofit management, or under regular district management. The average combined reading and math test scores one year prior to the management change at schools assigned to for-profit and nonprofit entities were 0.39 and 0.13 standard deviations below the Philadelphia average, respectively, while the pre-intervention scores of the full set of 142 regular public schools were 0.19 standard deviations above the district average. Because of that disparity, we limited the schools included in the comparison group to the lower half of all regular district schools. Those 71 schools had prior test scores that were 0.15 standard deviations below the district average, a level of performance much closer to those at the schools placed under new management. Restricting the comparison group in this way allows us to make a cleaner, if not an exact, comparison while maintaining a sufficient number of schools to be able to detect sizable management impacts at conventional levels of statistical significance.

Method

The Philadelphia intervention does not provide the opportunity for a random assignment study of the impact of for-profit and nonprofit management. Schools assigned to intervention status were not chosen randomly, but selected on the grounds that they were in greatest need of intervention. We therefore employed a “difference-in-differences” analysis to estimate the impact of attending a for-profit or nonprofit privately managed school (relative to attending that school had it remained under district management). The treatment groups consist of schools managed by each type of private provider, and the comparison group includes the regular public schools with test scores below the median for all regular district schools, as discussed above. To identify the effect of treatment, we calculate the difference between average annual test-score gains made by students at treated and comparison schools before and after the intervention began. So, for example, if test-score gains at the schools treated by for-profit management were 20 percent of a standard deviation higher than gains before treatment, while comparably measured gains at the comparison schools were only 15 percent of a standard deviation higher, the estimated effect of for-profit management would be the difference between them, or 5 percent of a standard deviation.

Our use of annual gain scores provides an estimate of treatment effects based on the extent to which students at each school do better or worse than would be expected, given their initial test scores. We also include student fixed effects, which account for changes in the composition of the schools’ student populations over time that cannot be explained by the limited set of student characteristics for which information was available in the district’s database. Finally, we control for the demographic characteristics of students’ peers and whether students have recently moved from one school to another.

The inclusion of student fixed effects means that students are compared only to themselves over time when estimating the effect of each kind of management. Estimates that include student fixed effects require at least three years of scores, which provide two changes in scores from one period to another, typically called gain scores (even though in some cases they are losses, not gains). In our analysis, there are 68,677 students for whom the data allow us to compute at least two gain scores in math (and a similar number for the reading estimation). For another 46,875 students one gain score is available. With one gain score, it is possible to use models that control for observable background characteristics, but those models cannot adjust for unobserved student characteristics (such as parental education or student commitment). We nonetheless use such models to estimate informally the impacts on all students for whom at least one gain score can be calculated. Those informal estimates require assumptions that are quite restrictive, however, so the best estimates available to us are only for the approximately 68,000 students that could be included in the model that employed a fixed-effects analysis.

Despite the tens of thousands of student observations, those in schools under private management are clustered within only 30 schools managed by for-profit organizations and 16 managed by nonprofit ones. As a result, the annual impacts of the intervention must be as much as 20 to 30 percent of a standard deviation in order to be detected at conventional levels of statistical significance. That substantial an annual impact is seldom detected for large-scale structural interventions in education. For that reason, we also discuss large impacts that fall short of statistical significance when the pattern of results is consistent over the six year time period.

Results

The impact of nonprofit management appears to have been negative. At schools under nonprofit management, students learned, on average for the six years, 21 percent of a standard deviation less in math each year than they would have had their school remained under district management. In terms of years of schooling, the negative impacts on math performance were, on average, approximately half a year’s worth of learning each year, a large effect (see Figure 1). However, the negative impact was statistically significant in only the first year after the intervention began. In reading, the average adverse impact of nonprofit management was roughly 10 percent of a standard deviation annually, about 32 percent of a year’s worth of learning. The adverse effect on reading performance was statistically significant in only the first year after the intervention began.

As mentioned earlier, impacts may have been somewhat different for the students with fewer test scores, who could not be included in our preferred model. Adjusting for that possibility yields an adverse impact of nonprofit management of 18 percent of a standard deviation on math performance and 14 percent on reading scores.

The effect of for-profit management was generally positive, although only the math impacts are statistically significant. At schools under for-profit management, students learned on average 25 percent of a standard deviation more in math each year of the six years of the intervention than they would have had the school been under district management. The estimated impact each year was roughly 60 percent of a year’s worth of learning, a large, statistically significant impact. Our adjustment using results from an alternative model that includes the larger number of students yields a positive annual impact of for-profit management on math performance of 12 percent of a standard deviation, 29 percent of a year’s worth of learning.

The estimated average annual impact on reading performance of for-profit management relative to district management is a positive 10 percent of a standard deviation, approximately 36 percent of a year’s worth of reading. However, that impact is not statistically significant. The adjusted impact was just 4 percent of a standard deviation, about 14 percent of a year’s worth of learning.

The differential impact of for-profit and nonprofit management is especially sizable. Using the estimates given above, students in schools under for-profit management gained between 70 percent and greater than a full year’s worth of learning in math more each year than they would have had the schools been under nonprofit management. All of these differences are statistically significant. In reading, students learned approximately two-thirds of a year more in a for-profit school than they would have had the school been under nonprofit management. All but one of the differences are statistically significant.

Taking Back Five Schools

Our analysis provides compelling evidence that schools do much better under for-profit than under nonprofit management. Year after year, students learned substantially more in reading and math if they attended a school under for-profit rather than one under nonprofit management. Yet in 2008 the district reassumed control of only one school under nonprofit management while not renewing the contracts for five schools under for-profit management. To ascertain whether that decision had a strong educational basis in the district’s own test-score database, we used the same methodology to estimate the impact on student learning of the five schools for which the for-profit management contract had been terminated.

The results are mixed but provide little support for the district’s decisions. The reading performance of students at the five schools was, on average, 18 percent of a standard deviation below what could have been expected had the schools been under district management, a difference that is statistically significant in three of the six years. Also, nonprofit schools whose contracts were not revoked had an impact on reading performance that was 10 percent of a standard deviation more positive than that of the for-profit schools whose contracts were terminated. However, that difference is not statistically significant in any year.

Math performance of students at the for-profit schools was 35 percent of a standard deviation higher than would have been the case had the schools been under district management. Also, it was 56 percent of a standard deviation higher than would have been the case had the schools been under nonprofit management. Those large differences are statistically significant in most years.

Our results indicate that nonprofits outperformed the five for-profits in reading in five out of six years, although the difference was only 3 percent of a standard deviation in 2008, and in no year were the differences statistically significant. In math, the five for-profits had strongly positive impacts in all years, while the nonprofits had decidedly negative ones, leading to very large, statistically significant differences between the two groups of schools in all years. The large differences in math clearly offset the statistically insignificant differences in reading.

If the Philadelphia school district cares only about reading results, and places no weight on math results, our data could be used to support the policy choice that was made, provided no attention is paid to the statistical insignificance of the reading finding. But if the two subjects are given equal weight in evaluating a school, our results provide no support for the decisions made by the school district with respect to renewing for-profit and nonprofit management contracts.

Discussion

Care should be taken before generalizing from the Philadelphia experience concerning the relative advantage of for-profit and nonprofit management. It is possible that for-profit entities have a greater vested interest in enhancing student achievement, because only in that way are they likely to survive over the longer run. But other factors in Philadelphia could easily account for the same result. The two main for-profit providers had much more experience with school management than did any of the nonprofit organizations. The nonprofits seem to have been selected more for their strong political ties than for any history of effectiveness at delivering educational services. Others have reported that newly formed charter schools under both for-profit and nonprofit management appear to become more effective as they gain in experience. That could easily account for the pattern of results reported here. Still, it is disconcerting to discover that impacts of non-profits compared unfavorably with those of the for-profits six years after the intervention began, presumably a long enough period for a new school manager to learn from experience.

Paul E. Peterson, director of the Harvard Program on Education Policy and Governance (PEPG), is editor-in-chief of Education Next.
Matthew M. Chingos is a PEPG research fellow.

Scholars have attempted to discern the effects of competition between the public and private sectors within the United States and in other countries, but no study, to our knowledge, has attempted to measure systematically the causal impact of competition by looking at variation across countries. Until now, research has been stymied by the fact that any simplistic statistical correlations between the extent of competition and student achievement that might be found are suspect. Countries where more people choose to invest in private schools may have other attributes, such as more income or a greater commitment to education, that lead to higher levels of achievement. If this is the case, any positive correlation between private schooling and student achievement could reflect a country’s income or educational commitment rather than any beneficial effects of competition. Or it may be the case that low-quality public schools increase the demand for private schooling. If so, then it could appear that competition lowered the quality of public schooling when in fact the causal connection was in the opposite direction.

In this study, we solve this conundrum by taking advantage of the historical fact that the amount of competition in education today varies from one country to another for reasons that have little to do with contemporary school quality, or national income, or commitments to education. The extent of private schooling stems in large part from the Catholic Church’s decision in the 19th century to build an alternative system of education wherever they were unable to control the state-run system.

Nineteenth-century Catholic doctrine strongly opposed Catholic attendance at state-run schools that were not controlled by the Church. In the United States, for example, Catholics perceived government-operated “common schools” to be Protestant-dominated institutions that were only ostensibly nonsectarian. Local parishes responded by establishing separate schools in which children received Catholic-infused instruction. The United States was not the only country where this happened. Catholic school systems developed in many other countries, but their size depended on the percentage of Catholics living in that country during this critical period (see sidebar). (In countries where Catholicism was the state religion, there was no perceived need for private schools, however.) As a result, even today the size of the private education sector—and thus the amount of competition between public and private schools—is related to the size of the Catholic population in 1900.

To connect the historical past to competition’s effect on achievement today requires two analytic steps. We first estimate the statistical relationship between the size of the Catholic population in 1900 and the extent of private schooling today in order to capture only that share of the private sector’s size that can be attributed to 19th-century Catholic policies—policies we assume to be otherwise unrelated to contemporary student achievement. Having estimated this relationship between Catholicity in the past and competition in the present, we then use that estimate to isolate the causal effect of private school competition on the achievement of individual students across 29 countries.

Our results confirm that countries with larger shares of Catholics but without an official Catholic state religion in 1900 have significantly larger shares of privately operated schools in 2003. More important, private school competition attributable to past Catholic policies generates higher student achievement in mathematics, reading, and science today. We also show that competition between the public and private sector positively affects the achievement of students attending public schools. Spending on education is also reduced, suggesting that school systems are more productive if they are more competitive.

PISA 2003

For the information on contemporary student achievement we rely on the well-regarded data sets compiled by the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) in 2003. Working closely with official government agencies, PISA gathered information on the mathematical, scientific, and reading literacy of nationally representative student populations in all 30 OECD countries. The term “literacy” signifies that the PISA measured not only the students’ knowledge of math, reading, and science, but also their ability to use that knowledge to meet real-life challenges. In 2003, PISA made a special effort to measure math literacy, allocating 70 percent of testing time to questions in this subject. PISA assessed the achievement of 15-year-old students in each country, regardless of the grade they attended. This means that, in most participating countries, PISA tested students nearing the end of compulsory schooling.

For purposes of this analysis, we constructed a data set that contained pupil-level test scores for about 220,000 students. We also were able to obtain from PISA student reports of their background characteristics and administrator reports on the characteristics of each student’s school, including such things as school resources and whether the school was public or private. All that information was available from 29 of the 30 OECD countries. (France had to be dropped from the analysis because it did not supply any information on the characteristics of its participating schools.)

We defined a school as private if the principal reported that it was managed directly or indirectly by a nongovernment organization (e.g., a church, trade union, business, or other private entity). A public school was defined as one being managed directly or indirectly by a public education authority, government agency, or governing board appointed by government officials or elected by public franchise. We used these definitions to calculate the share of private schools in a country. Throughout our study, this figure serves as our measure of the extent of contemporary private school competition in each country.

The size of the private sector so defined ranges widely across countries. In the Netherlands, more than three-quarters of 15-year-old students attend privately operated schools. Private school shares in Belgium, Ireland, and Korea are also well above one-half. By contrast, the share of students attending privately operated schools in Greece, Iceland, Italy , New Zealand, Norway, Poland, Sweden, and Turkey is below 5 percent. Just over 6 percent of the American 15-yearolds sampled by PISA attended private schools, a figure that corresponds closely to official estimates of private enrollment at the secondary level from the U.S. Department of Education (see Figure 1).

ESTIMATING COMPETITIVE EFFECTS

Recall that our analysis involves two steps. First, we estimate the amount of contemporary private school competition across our 29 countries that can be accounted for by the share of each country’s population that was Catholic in 1900. Where Catholicism was the official state religion, we assign a value of zero for this variable (even though the size of the Catholic population was quite large). That decision is not as odd as it sounds, as we are interested in Catholicism only insofar as it was a factor contributing to the creation of a private sector, something that clearly was not the case in those countries where Catholicism was the state religion and Catholics had no reason to object to the education provided in state-run schools.

The second step uses the connection between past Catholicism and the contemporary size of the private sector to estimate the impact of competition on student achievement. Specifically, we measure the relationship between Catholic-induced private school competition in a country and the PISA test scores of individual students in math, reading, and science.

In taking this approach, we assume that the density of Catholics in 1900 is not directly related to student achievement today, independent of effects that may occur via school competition. While this assumption cannot be proven, there are good reasons to believe it is well founded. Protestant Christians have historically placed a greater emphasis than have Catholics on the value of education, because Protestants thought individual

Bible reading helped one along the road to salvation. Catholics placed greater emphasis on remaining connected to the traditions and practices of the Church. Interestingly enough, in those 22 majority-Christian countries for which data on literacy in 1900 are available, one finds a strong negative association between Catholic population shares and literacy rates. This strong negative correlation exists even after accounting for the lower gross domestic product (GDP) per capita, which is associated with lower literacy rates, in countries with larger Catholic population shares. So to the extent that we find any beneficial effect of Catholic-induced private school competition, its size is probably depressed by cultural values related to Catholicism. In other words, our approach is more likely to yield underestimates than overestimates of competitive effects.

Of course, the historical prevalence of Catholicism could also have had other consequences, apart from a greater reliance on private schooling, that indirectly affect student achievement. For example, the share of Catholics in a country could have an effect on current GDP per capita or education spending per student. We therefore account for the effect of both of these factors in all of our analyses.

In estimating the effect of private school competition on student achievement, we also adjust for the effects of a host of other factors that can affect individual student performance. In addition to the country-level factors of per capita GDP and education spending per student, we include in our analysis information on the presence or absence of external exit exams (which research suggests are associated with higher achievement) and information on whether the country had a Communist government in 1970 (which may have affected both the size of the private sector and achievement). Student and family background characteristics used in the analysis include a student’s gender, immigration status, exposure to early childhood education, the number of books in the home, and parental occupation and work status. Finally, we account for school resources such as class size, availability of materials, teacher certification and preparation, and amount of time for instruction.

PRIVATE SCHOOL COMPETITION AND STUDENT ACHIEVEMENT

The first step of our analysis confirmed the existence of a statistically strong relationship between the extent of private school competition in 2003 and a country’s Catholic population in 1900, much as the historical record would suggest. A 10-point increase in the percentage of Catholics in 1900 is associated with a 4.7-percentage-point increase in the share of students enrolled in privately operated schools in 2003 (see Figure 2). These results support our basic reasoning that as long as Catholics could not be sure that the emerging public school systems of the 19th century would provide education in line with their church’s demands, they tended to resist state schooling and establish their own private schools alongside the state sector. The consequences of historical differences in denominational shares across countries persist to this very day.

The results from the second step of our analysis are equally striking. Let us begin with the results related to student achievement in mathematics, the subject most extensively assessed in PISA 2003. A 10-percentagepoint increase in the share of national student enrollment in private schools attributable to a historically larger share of Catholics induces an improvement in PISA math scores of 9.1 percent of a standard deviation (see Figure 3). As a benchmark for interpreting the magnitude of this effect, note that the difference in average mathematics test scores between the two grades with the largest share of 15-year-olds (9th grade and 10th grade) in the PISA study was 22.1 percent of a standard deviation. This “grade-level equivalent” provides a rough idea of how much a typical student learns during one school year. By this metric, our estimate of the effect of a 10-percentagepoint increase in private school enrollment is equivalent to 41 percent of a year’s worth of learning in high school.

Because we are able to draw on evidence from a relatively small sample of only 29 countries, the statistical precision of our estimate is not very high. That is, we can say with 95 percent confidence that the effect of a 10-percentage-point increase in the private school share is between 3.9 and 14.2 percent of a standard deviation in test scores. Still, this means we have a very high degree of confidence that the real effect is larger than zero. The bottom line is that students in countries whose larger shares of Catholic population in 1900 induced them to have larger shares of privately operated schools today performed significantly better on the PISA 2003 math test.

As an additional step to address any lingering concerns about Catholicism’s direct influence on student achievement, we conducted both stages of our analysis again, this time accounting for the relationship between contemporary differences in the share of Catholic adherents in a country and student achievement. We found that historical Catholic shares continue to be a strong predictor of the extent of private school competition in a country. In addition, the estimated effect of Catholic-induced private school shares on student achievement increases relative to our first version of the analysis, which did not account for contemporary Catholic adherence. There is now a 12.2 percent of a standard deviation increase in test scores for each 10-percentage-point increase in the private school share in a country. This larger estimate suggests that the true effect may be closer to the upper bound of the interval we identified above.

Why the stronger relationship between private school competition and student achievement? We reason that this may be because, in the latter approach, the Catholic-induced school share was reflecting the slightly negative direct effect of contemporary Catholic adherence on student achievement, a relationship that reveals itself in this version of the analysis, although the estimated effect is just shy of statistical significance. Considered together, these results increase our confidence that we are describing a real, causal relationship between private competition and student performance, rather than effects of cultural differences related to religious adherence.

The estimated effects of the private school share on student achievement are somewhat smaller in science and reading than in math, but they remain substantial, positive, and statistically significant (see Figure 2). A change in the historical Catholic population share that produces a 10-percentage-point increase in the extent of contemporary private school competition generates an improvement of about 5.5 percent of a standard deviation in both science and reading—or more than one-fifth of a grade-level equivalent in these subjects.

To gain additional insights, we also re-ran both stages of our analysis while accounting for the average share of funding that private schools receive from the government. The inclusion of this variable hardly affects our results, suggesting that our findings reflect competitive effects stemming from the private operation of schools and not from differences in funding policies.

EFFECT ON PUBLIC SCHOOL STUDENTS

The previous portions of our study investigated the impact of private competition on student achievement in the educational system as a whole. But what about the effect of private school competition on public schools? To answer this question, we removed all students attending a privately operated school from the sample in each country and analyzed only the academic achievement of students in the public sector.

These results are somewhat more difficult to interpret than our findings above, as they combine the effects from competition with the consequences of student sorting. In other words, some of what we find may be due to high-ability students (and their parents) being more likely to choose private schools, leaving the weaker students in the public sector.

Nonetheless, the results suggest that public school students profit nearly as much from increased private school competition as do a nation’s students as a whole. While our estimates of the effects are somewhat smaller than the estimates for students in both the private and public sectors, the results are not statistically distinguishable. It therefore appears that much of the increased performance of education systems with higher levels of private school competition accrues to students who attend public schools.

EDUCATION SPENDING

The analysis so far has been limited to educational outcomes, estimating the effect of private school competition on students’ achievement. In doing so, we have controlled for possible effects of differences in educational inputs such as class sizes, availability of materials, and aggregate expenditure per student in the country. We wondered, though, whether private school competition also affects the input side of the educational process, specifically educational spending per student.

We again used a two-stage process, with the first stage using historical Catholicism to predict the Catholic-induced share of current private school competition in each country. Then, in a second stage, we measured the relationship across countries between the Catholic-induced share of competition and the cumulative educational expenditure per student up to age 15—a measure that includes both public and private spending. We continued to account for a range of country- and student-level characteristics when making these comparisons, but we now excluded measures of school resources that are likely to be affected by spending levels.

Our results show that private school competition, in addition to raising student achievement, substantially reduced the average spending level of the educational system. Changes in historical shares of Catholics in the population that are associated with a 10-percentage-point increase in the private school share today lead to a $3,209 reduction in cumulative spending per student, or 5.6 percent of the average OECD spending level of $56,947 (see Figure 3).

CONCLUSION

Our findings from an international study of 29 countries speak quite clearly. Competition from private schools improves student achievement, and appears to do so for public school as well as private school students. And it produces these benefits while decreasing the total resources devoted to education, as measured by cumulative educational spending per pupil. Under competitive pressures from private schools, the productivity of the school system measured as the ratio between output and input increases by even more than is suggested by looking at educational outcomes alone. Ironically, although Catholics historically placed less emphasis on education than did adherents of many other religions, their resistance to state-run schooling in many countries helped create institutional configurations that continue to spur student achievement.

Martin R. West is assistant professor of education at Brown University and an executive editor of Education Next.
Ludger Woessmann is professor of economics at the University of Munich and heads the Department of Human Capital and Innovation of the Ifo Institute for Economic Research.

The explanation for this odd fact: since 1981 Chile has had a more comprehensive school choice system than any other country in the world, as well as a system of publicly available information on student test performance. Scholars have thus seen Chile as a place to test theories of school choice. Do students with vouchers learn more in private schools or in those run by municipalities? What is the impact of a voucher system on equality of educational opportunity? The answers to these and related questions have been just about as varied as the number of scholars who have inquired into the matter. On balance, the bulk of the research shows a small educational advantage for students who attend privately operated voucher schools rather than municipal ones. But hardly any study looks at differences among the voucher schools, and none has examined differences between private schools in networks and those that operate on a stand-alone basis. Yet interest in school networks has escalated since many operators of charter schools in the United States have begun to expand their operations beyond a single school. Some have argued that this is the ideal way for connecting school choice to school improvement. If effective schools can expand either by setting up or by “franchising” other schools, school quality can gradually improve. But others say that the formation of networks of schools will lead to a standardization that will undermine the vitality of individual school communities.

Chile is an ideal place for exploring these questions. In 2002, only 53 percent of students were still being educated in municipally run schools, which nonetheless received a good deal of their funding from the vouchers paid for by the national government. Another 9 percent of students attended fee-based private schools that were independently operated and received no government assistance whatsoever. For the most part, these were schools with well-established reputations that served the country’s upper class. The remaining students attended what might be called voucher schools, because the schools, while private, had been since 1981 heavily dependent on the subsidy that the schools received from the national government for each student they enrolled. This sector is the fastest growing segment of the Chilean educational system.

Like American charter schools (see “Brand-Name Charters,” features), Chile’s privately run voucher schools may be part of a larger organization or school network, or operate on their own. Most schools are of the stand-alone or “mom-and-pop” variety: 25 percent of all students in Chile attend such schools. But another 13 percent of students attend schools that are part of a network of two or more schools.

The schools, inside and outside of networks, vary from one another in many ways. Some are operated by teachers who once worked in municipal schools. Others are run by business entrepreneurs. Fifty-nine percent of the network schools are run by nonprofit entities, either religious or secular. For-profit organizations operate the remainder. Some schools as well as some networks are religious. Most networks, and especially those in rural areas, consist of just two or three schools. Only about 20 percent of primary (K–8) private voucher school students attend schools that belong to networks that have more than three schools (see sidebar).

Proponents of school networks say that the networks facilitate the flow of information (such as research on best practices) among schools and provide political benefits, credibility, and legitimacy in the eyes of the community. They argue that larger schooling operations have more access to private investments and loans to expand than smaller operations do. Supporting this view, research on public charter schools in the United States indicates that well-established charter school networks can build credibility for fund-raising more easily than stand-alone charter schools can. The underlying hypothesis is that, all else bring equal, the more a schooling organization facilitates transactions between members of a school’s community, the better the school’s performance. The research conducted so far shows that stand-alone charter and brand-name schools, like their district counterparts, vary widely in quality.

Critics of school networks fear unintended negative consequences. They argue that large centralized operations create hard-to-manage bureaucracies and make it difficult to maintain order and create a sense of community among students, parents, teachers, and administrators. Opponents also claim that large schooling operations grant too much authority to administrators and other professionals far removed from the classroom. Some critics are concerned that consolidation encourages standardization. For instance, they maintain that school networks must establish a brand to be successful, which necessitates relatively uniform operations and services from site to site. They argue that this branded approach to education stifles innovation.

Very little factual information is available to sort out the credibility of these claims and counterclaims. It is thus of interest to examine the Chilean experience, where both network and stand-alone voucher-subsidized schools have been operating for several decades. Information on more than one-quarter million students who were 4th graders in 2002 allows us to compare Spanish language and mathematics achievement in network and stand-alone voucher-subsidized schools. Our findings suggest that network schools in Chile are more effective than stand-alone schools, and that larger networks tend to outperform smaller networks. While we cannot be certain whether the higher performance of network schools is because good schools were the ones to expand or whether networking, by itself, had a positive impact, our results nonetheless add considerably to the sparse information currently available on a question of substantial policy interest.

School Reform in Chile

During the 1980s, the Chilean government decentralized the administration of schools, transferring responsibility for public school management from the Ministry of Education to municipalities (recognized neighborhoods in Chile around which municipal services are organized). The government also changed the education financing scheme. Municipalities began to receive funding from the central government according to the number of students who chose to attend the municipal schools. Any enrollment loss had a direct effect on their education budgets. Equally important, privately run schools that had not charged tuition began receiving the same per-student voucher as the public schools. Tuition-charging (elite) private schools mostly continued to operate without public funding.

Despite the parity in funding, significant differences remained between municipally run schools and privately run voucher schools. First, starting in 1994 municipal elementary schools were not allowed to charge parents fees, while all privately run voucher schools could. Second, municipal schools were required by law to accept all who applied. Private voucher schools, in contrast, were allowed to consider results from admissions tests and parent interviews when making admission decisions. Third, municipal schools had to comply with labor laws that made it virtually impossible to fire a low-performing teacher. Privately run voucher schools had greater freedom to terminate employment. In addition, municipal school teachers received salary increases incrementally, based on years of experience. There were no rules with regard to incremental salary increases in the private school sector.

The policies sparked a movement of students from municipal to previously existing private schools as well as the establishment of new institutions. In 1981, 15 percent of the nearly 2.9 million Chilean K–12 students had been attending private schools that received some public subsidy, and another 7 percent attended elite, unsubsidized private schools. By 1990, 34 percent of students attended privately run voucher schools; by 2002, enrollment in such schools reached 38 percent of the roughly 3.4 million in total enrollment (see Figure 2). The trend continued after 2002, the year in which the data for this study were collected. By 2005, 43 percent of students were enrolled in privately run voucher schools. As indicated above, about one-third of the voucher schools belong to networks, while the remaining two-thirds operate independently.

Beginning in 2003, after our data were collected, the Chilean government sought to alter several features of the system, although not all of the changes have been fully implemented: Rather than providing vouchers at a flat rate, voucher amounts are to be tied to family income. Private voucher schools can no longer select students in primary school; for secondary school admission, they can administer tests, but they cannot conduct parent interviews. In addition, Congress recently passed legislation that will hold schools accountable for student achievement and improvement over time.

The political debate continues. In 2006, widespread student protests of inequalities in the education system prompted debate over whether entrepreneurs should be able to own and run private voucher schools for profit. Proposed legislation, which initially prohibited for-profit education organizations, now would require that such entities make available to the public information on their profitability as well as their use of voucher funds.

Data and Methodology

Our study is based on student-level data from Chile’s national standardized test, Sistema de Medición de la Calidad de la Educación (Educational Quality Measurement System—SIMCE), which assesses students in grades 4, 8, and 10 in language, mathematics, history and geography, and natural sciences. In 2002, SIMCE evaluated 274,863 4th graders. Complementing student test scores are parent and teacher questionnaires, which include socioeconomic and environmental information on the students, their families, their peers, and their schools.

Because we lacked complete data for some schools, our study includes 252,202 students. Fifty-eight percent of the students were attending municipal schools, 24 percent were attending stand-alone schools, and 18 percent were attending network schools. In addition to separating out municipal and stand-alone private schools, our analysis subdivides the network schools into five groups—those that are in networks of two, three, four, five, and more than five schools—for a total of seven categories. (We have excluded information from the elite, independent schools that receive no government subsidy.) We compare the test scores of students in each of the seven categories, taking into account differences in the students’ socioeconomic characteristics, including parent schooling, self-reported household income, the number of non-school books in the home, and the quality of the peer groups (calculated by averaging family background and home resources for all students in the classroom). We also included some school-level control variables—whether or not the school was located in a rural area, the total number of students per school, and the average monthly tuition a school charges.

If these variables fully account for differences in student and peer demographics across the various categories of schools, then this strategy will provide unbiased evidence on the relative effectiveness of municipal, stand-alone, and network schools. We cannot account for other factors that could be significant. For example, the average student attending a privately run voucher school, whether network or stand-alone, may have parents who place a higher value on education than those of the average student attending a municipal school. Because we do not have a measure of parent commitment to education, we may confuse the effect of having a committed parent with that of attending a private school. Similarly, the “brand name” value attached to network schools may enable them to select more-qualified students, on average, than their independent counterparts.

As a result, simple comparisons of student outcomes in municipal, stand-alone, and network schools might give misleading estimates of the impact of schools on student achievement, even after adjusting for the measured characteristics of the students who attend each type of school. In order to correct for this selection bias, we restrict our analysis to differences across students in the type of school they attend that result from the types of schools available to them. More specifically, we assume that an individual’s probability of choosing a given school type is affected by the school density (that is, the number of schools per square kilometer) of each type in her municipality. All else being equal, students are more likely to choose schooling alternatives that are more densely concentrated in their municipalities. The crucial assumption made by our method is that school choice is influenced by local school supply, but school densities at the community level do not directly influence student achievement.

Though every precaution has been taken to make the comparison exact, it is still possible that our results overestimate the benefits of privately subsidized schools over municipal ones. For example, it is possible that voucher schools are to be found in greater density in higher income areas, where more parents are willing to pay additional fees for their children to attend higher-quality schools. To the extent that is happening, our results could be biased toward finding greater voucher benefits than is actually the case. For that reason, our comparisons between private and municipal schools should be interpreted cautiously. However, that potential source of bias is unlikely to affect comparisons between stand-alone private schools and network ones, the main focus of this analysis.

Results

We report our results in terms of standard deviations of student test scores. The difference in performance between American 4th and 8th graders on the National Assessment of Educational Progress is about one full standard deviation, suggesting that students improve by one quarter of a standard deviation each year. Although comparable measures of the rate of student learning are not available for Chile, researchers studying the Chilean school system typically consider a difference in student achievement of 10 percent of one standard deviation to be a small to moderate effect. Without accounting for any differences in students’ socioeconomic status, the Spanish language and mathematics test scores of students who attend network schools are considerably higher than the scores of those attending stand-alone schools. After controlling for student and peer attributes and for selection bias, we still find a substantial positive and statistically significant effect of attending a network school on student achievement. Students at network schools score 19 percent and 25 percent of a standard deviation higher than students at stand-alone schools in Spanish language and math, respectively. We also find that students at municipal schools do significantly worse than students at stand-alone schools on achievement tests (19 percent and 16 percent of a standard deviation in Spanish language and math, respectively), although, as discussed above, we are less confident in these results because of the difficulties of accounting for the selection of students into and by private schools.

Although these results provide some evidence of the effectiveness of school networks, a more precise analysis is needed to understand the optimal size of a network. We examined whether larger networks are more effective than smaller ones and found that, both with and without correcting for student and peer socioeconomic characteristics and selection bias, students at schools that are part of networks of three or more schools consistently outperform students at schools in networks of only two schools.

Figure 3 shows the results from our estimations. Students in schools in larger networks generally learned more than students in stand-alone schools. The results for Spanish language achievement show students in schools in networks with three schools learn 24 percent of a standard deviation more, those in networks of five schools learned 50 percent of a standard deviation more, and those in networks of more than five schools learned 23 percent of a standard deviation more. The effects on mathematics achievement are similar. Students who attend schools in networks with three schools learn 37 percent of a standard deviation more than students in stand-alone schools. The percentages for those in networks of five and more than five schools are 36 and 34, respectively.

Prior research in Chile and in the United States has demonstrated that, all else being equal, Catholic schools outperform public schools and other private schools. Since some of the network schools were affiliated with Catholic churches, that fact could be the explanation for the apparent positive benefits that come from networking. To determine whether that was the case, we checked whether the school owners were Catholic. Only 13 percent of the students attended such schools, however. And after adjusting for Catholic affiliation, the differences between network and stand-alone schools remained large and significant. In other words, the superior performance of network schools is not driven by the number of them that are Catholic.

Policy Implications

This paper compares the academic achievement of 4th graders in municipal schools, stand-alone schools, and network schools. Controlling for individual and peer characteristics as well as selection bias, we find that students in network schools outperformed those in stand-alone schools in both Spanish language and math. The stand-alone schools outperformed municipal schools but not by as large a margin. It also is of interest that students generally performed better in networks of large size. Most clearly, those in networks that contained three or more schools generally outperformed those in networks with only two schools.

Possible explanations for the positive school network effect include the substantial benefits of scale for employing education professionals and administrators, the bulk purchases of supplies and equipment, and the costs of implementing innovations in the curriculum. School networks may also benefit from greater access to credit and private investment than that extended to small individual schools in Chile. In addition, it may be that operating within a larger communal organization reduces agency problems; encourages interactions between parents, teachers, administrators, and students; and influences the development of professional school communities.

Of course, it is also possible that good schools are invited to join networks, while weaker schools are left on their own. In a competitive schooling environment, low-quality schools may be unable to attract students and additional resources needed to expand operations.

The results of this paper add evidence to the debate in the United States over the desirability of creating networks of charter and voucher schools. The findings provide some ground for optimism about the effects of networking on student achievement. Policies that provide incentives for schools to establish a network or to be managed by an organization that runs a network of schools may have the potential to increase educational outcomes.

Gregory Elacqua is professor at the Universidad Diego Portales in Santiago, Chile, and former policy advisor to the Minister of Education of Chile. Dante Contreras is senior researcher at the United Nations Development Program (UNDP) Chile and associate professor, Universidad de Chile. Felipe Salazar is researcher at the Universidad Diego Portales.

According to the latest international math and science assessment conducted by the Organisation of Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) (see Figure 1), the United States remains a long distance from that target. Rather than worrying about the consequences, some have begun to question what all the fuss was about. Education researcher Gerald Bracey, for example, has argued that no one has “provided any data on the relationship between the economy’s health and the performance of schools. Our long economic boom suggests there isn’t one—or that our schools are better than the critics claim.”

Truth be told, the Bracey critique is not entirely misplaced. Most commentators rely more on the commonsense understanding that countries must have good schools to succeed economically rather than presenting conclusive empirical evidence that connects what students learn in school to what subsequently happens in a nation’s economy. Even economists, the people who think the most systematically about the way in which “human capital” affects a nation’s economic future, have skirted the heart of the question by looking only at “school attainment,” namely the average number of years students remain in school.

Using average years of schooling as an indicator of a country’s human capital has at least two major drawbacks. First and foremost, the approach assumes that students in diverse school systems around the world receive the same educational benefits from a year of schooling. A year of schooling in Papua New Guinea and a year of schooling in Japan are treated as equally productive. Second, this measure does not account for learning that takes place outside the classroom—within families, among peers, or via the Internet, for example.

A more direct measure of a country’s human capital is the performance of students on tests in math and science, something that might be called the average level of “cognitive skills” among those entering a country’s work force. At one time, internationally comparable information on student performance was not available for a sufficient number of countries over a long enough period of time to allow for systematic study, which is why economists relied upon the less informative measures of school attainment. Now that test-score data for many countries over an extended period of time are readily available, it is possible to supplement measures of educational attainment with these more direct measures of cognitive skills.

In a series of studies conducted over several years, the four of us have explored the role of both school attainment and cognitive skills in economic growth. Beginning in the mid-1960s, international agencies started conducting tests of students’ performance in mathematics and science at various grade levels. We used performance on 12 of these standardized tests as rough measures of the average level of cognitive skill in a given country. With this information, we could assess how human capital relates to differences in economic growth for 50 countries from 1960 to 2000, more countries over a longer period of time than any previous study. We were also able to pay close attention to institutional factors that influence economic growth, such as openness of the economy and protection of property rights.

What we discovered gives credence to the concerns expressed in A Nation at Risk. The level of cognitive skills of a nation’s students has a large effect on its subsequent economic growth rate. Increasing the average number of years of schooling attained by the labor force boosts the economy only when increased levels of school attainment also boost cognitive skills. In other words, it is not enough simply to spend more time in school; something has to be learned there.

We also discovered that the size of the impact of cognitive skills depends on whether a nation’s economy is open to outside trade and other external influences. The more open the economy, the more important it is that a country’s students are acquiring high levels of cognitive skills. As the world becomes increasingly interdependent or “flat,” to use New York Times columnist Thomas Friedman’s familiar terminology, enhancing human capital will become increasingly critical. As the world continues to change, the United States can ill afford to rest easily on its past accomplishments.

Measuring Cognitive Skills

Reaching these conclusions required a multistep analysis. The first step was to use the 12 PISA and other international math and science assessments, dating back to 1964, to construct an index of cognitive skill levels for a large sample of countries at various points in time. Because the number of countries participating in the 12 test administrations changed from one administration to the next, and because testing agencies have made no attempt to link their results to one another, we needed to develop comparable scores for each test. This required a norm against which each test could be calibrated. Fortunately, we could construct that norm by using information from tests in the United States, the country that has had the earliest, most sophisticated, and most comprehensive system of testing. The United States has participated in all of the international tests since 1964, and it has also maintained a separate longitudinal testing system of its own, the National Assessment of Educational Progress (NAEP). With that information in hand, it was possible to calibrate scores on each of the separate international tests to one another via the connection of those tests to the NAEP. To obtain further precision, we used the variation in scores across a subset of the more-advanced developed countries to obtain an estimate of the spread in scores across countries. By following these two steps, we were able to aggregate all available scores for each country into measures of average cognitive skill levels for each country.

The 50 countries for which we were able to develop a comparable measure of cognitive skill levels include the 30 democracies that have market economies and have been accepted as members of the OECD, most of which are at a relatively high level of economic development. The other 20 countries are at lower levels of economic development. In Figure 2, you can identify top performers like Finland and Japan, average performers such as the United States and Germany, and low performers that include Albania, the Philippines, and South Africa.

Impact on Economic Growth

We wanted to use this new information to compare the economic benefits of higher levels of just school attainment with the benefits of higher levels of cognitive skills. We therefore took measures of average educational attainment and average cognitive skill levels for as many countries as possible and examined their relationship to the average annual growth rate in the country’s gross domestic product (GDP) per capita from 1960 through 2000.

First, we looked just at the impact of average school attainment on the economic growth rate. (An adjustment was made for the initial level of GDP because it is “easier” to grow if you are starting out at a lower level; that is, it is easier to copy more productive technologies than to initiate progress on your own.) When we performed this analysis, we found, as other economists before us, that when the average number of years of schooling in a country was higher, the economy grew at a higher annual rate over subsequent decades. Specifically, we found that, across the 50 countries, each additional year of average schooling in a country increased the average 40-year growth rate in GDP by about 0.37 percentage points.

That may not seem like much, but consider the fact that since World War II, the world economic growth rate has been around 2 to 3 percent of GDP annually. Lifting it by 0.37 percentage points is a boost to annual growth rates of more than 10 percent of what would otherwise have occurred, a significant amount.

But the impact of improved cognitive skills, as measured by the performance of students on math and science tests, is considerably larger. When we performed the analysis again, this time also including the average test-score performance of a country in our model, we found that countries with higher test scores experienced far higher growth rates. If one country’s test-score performance was 0.5 standard deviations higher than another country during the 1960s—a little less than the current difference in the scores between such top-performing countries as Finland and Hong Kong and the United States—the first country’s growth rate was, on average, one full percentage point higher annually over the following 40-year period than the second country’s growth rate. Further, once the impact of higher levels of cognitive skills are taken into account, the significance for economic growth of school attainment, i.e., additional years of schooling, dwindles to nothing (see Figure 3). A country benefits from asking its students to remain in school for a longer period of time only if the students are learning something as a consequence.

Another indication of the importance of education quality to economic growth lies in our ability to explain global variation in GDP growth. When we tried to account for economic growth with information only about school attainment levels and the level of a country’s GDP in 1960, we were able to explain only one-quarter of the differences we saw among countries. But when we also included cognitive skills in our statistical models of economic growth, we were able to attribute nearly three-quarters of the differences among countries to these three factors. In other words, higher levels of cognitive skill appear to play a major role in explaining international differences in economic growth.

Of course, the initial level of economic development, schooling attainment, and cognitive skills are not the only factors that affect economic growth. Could it be that some other factor we have overlooked is responsible for the close connection between test scores and economic growth?

Other economic research has identified two additional factors that affect a country’s economic growth rate: the security of its property rights and its openness to international trade. When those two factors are taken into account, the positive effect of cognitive skills on annual economic growth becomes somewhat smaller, but is still 0.63 percentage points per half of a standard deviation of test scores. This is the best available estimate of the size of the impact of cognitive skills on economic growth.

Other commonly discussed determinants of economic growth are fertility and geography. However, when we took into account the total fertility rate and common geographical proxies, such as latitude or the fraction of the land area of a country that is located in the tropics, neither of these additional variables was significantly associated with economic growth. Once again, the strong effect of cognitive skills remained clear.

We performed a variety of additional tests to assess the validity of these basic results. For example, we estimated the relationships over shorter periods of time, used different subsets of international tests, and compared smaller groups of the 50 countries.

One of our tests was particularly interesting. We thought it possible that the effect of cognitive skills could be the result of the presence in our sample of East Asian countries, most of which have both high levels of cognitive skill and rapidly growing economies. To see whether the inclusion of those countries in our study influenced our results, we excluded them from one of our models. The impact of cognitive skill remained very powerful, albeit diminished.

We also looked at cognitive skills as measured in the 1960s through the mid-1980s to see what their impact was on growth between 1980 and 2000, ensuring that the cognitive skills themselves were not caused by the economic growth. Again, our basic findings remained intact. Finally, we looked at whether a country’s estimated cognitive skills affected the earnings of immigrants working in the United States. Higher home country cognitive skills translated into higher earnings if the immigrants were educated in their homeland but not if educated in the United States.

Our commonsense understanding of the importance of good schools can thus be documented quite precisely. A highly skilled work force can raise economic growth by about two-thirds of a percentage point every year.

More Rocket Scientists or Basic Skills for All?

To gain additional insight into the relationship between cognitive skills and economic growth, we examined the separate impact of improvements at different levels of a nation’s distribution of skills. Loosely speaking, is it a few “rocket scientists” at the very top of the distribution who spur economic growth, or is it “education for all” that is needed?

To address this question, we measured the share of students in each country who reach a threshold of basic competency in mathematics and science, as well as the share of students who perform at very high levels. To estimate the importance of basic competency, we identified the share of students performing at least at a very basic level, that is, no more than one standard deviation below the international average of all OECD countries. In the average OECD country in our study, 89 percent of the students achieved at least at this very basic level. The share of students with at least basic skills ranged widely among countries, from as low as 18 percent in Peru to 97 percent in the Netherlands and Japan. To show a country’s ability to develop a large cadre of high-performing students, we identified the share of students performing at very high levels—at or above one standard deviation over the OECD average. On average across all countries, 6 percent of students performed at that high level. Once again, countries varied enormously in this respect, the percentage ranging from as low as 0.1 percent in Colombia and Morocco to 18 percent in Singapore and Korea and 22 percent in Taiwan.

Which is more important for growth—having a substantial cadre of high performers or bringing everyone up to a basic level of performance? The answer, it seems, is not one or the other but both! When we estimated the importance of each within the same model, we found each of them to be separately important to economic growth. That is, both the performance of countries in ensuring that almost all students achieve at basic levels and their performance in producing high-achieving students seem to matter.

The reasons that a substantial cadre of highly skilled citizens and near-universal basic skills matter are not difficult to imagine. Even if a country is simply making use of new technologies developed elsewhere, as is often the case in developing parts of the world, the more workers that have at least basic skills, the easier it will be for them to make use of those new technologies. Some workers need a high level of skill so they can help adapt the new technologies to their countries’ particular situation. In countries on the technological frontier, substantial numbers of scientists, engineers, and other innovators are obviously needed. But so is a labor force that has the basic skills needed to survive in a technologically driven economy.

But even if the results seem intuitively correct, they should be taken as suggestive rather than definitive, because the two measures of cognitive skills are closely related to one another and our models have difficulty in separating out the precise impact of each individually. Most countries that have a high percentage of students with very high cognitive skills also are ones in which basic skills are near universal. Conversely, countries with a substantial percentage of students lacking even basic skills tend to be those that have only a small percentage of highly skilled students. Still, that pattern is not a perfect one, so we are able, at least tentatively, to identify the impact of each type of human capital, and we are quite confident that we can recommend that countries both concentrate resources on their “best and brightest” and make sure that “no child is left behind.”

The Impact of Becoming a World Leader

What would it mean for economic growth, then, if a country like the United States, currently performing somewhat below the average of OECD countries, managed to increase its performance by 50 points (or 0.5 standard deviations) so that it would score alongside the world leaders? (On average on the PISA 2006 math and science exams, countries such as Canada and Korea scored about 50 points higher than the U.S., Hong Kong and Taiwan about 60 points higher, and Finland as many as 74 points higher.) That increase of 50 test points is exactly what George H. W. Bush and the nation’s governors together promised in 1989 the United States would achieve by the year 2000.

Unfortunately, no such gains were realized. But had the promise been fulfilled by 2000, our results suggest that GDP would by 2015 be 4.5 percent greater than in the absence of any such gains (see Figure 4). That 4.5 percent increment in GDP is equal to the total the U.S. currently spends on K–12 education. In other words, had that money effectively raised cognitive skills by the 50 test points that would have brought the United States close to world leadership, the economic returns to the country would probably have been enough to cover the entire cost of education in 2015 and after.

Figure 4 shows that the benefits of successful reform grow even more vivid when we look farther out. Over 75 years, even a reform that takes effect in 20 years (instead of the governors’ 10 years) yields a real GDP that is 36 percent higher than it would be if there was no change in the level of cognitive skills.

None of this is meant to suggest that schooling is the only factor contributing to a society’s cognitive skill development. Family, individual ability, and health combine with school quality to determine a student’s level of achievement. Yet there is every reason to believe that the single best route to higher levels of cognitive skill is strengthening a country’s education system. After all, most people think that is the system’s primary purpose.

An American Exception?

The United States has never done well on international assessments of student achievement. Instead, its level of cognitive skills is only about average among the developed countries. Yet the country’s GDP growth rate has been higher than average over the past century. If cognitive skills are so important to economic growth, how can we explain the puzzling case of the U.S.?

Part of the answer is that the United States may be resting on its historic record as the world’s leader in educational attainment. In addition, the United States has other advantages, some of which are entirely separate and apart from the quality of its schooling. The U.S. maintains generally freer labor and product markets than most countries in the world. There is less government regulation of firms, and trade unions are less powerful than in many other countries. Put more broadly, the U.S. has generally less intrusion of government in the operation of the economy, including lower tax rates and minimal government production through nationalized industries. Taken together, these characteristics of the U.S. economy encourage investment, permit the rapid development of new products and activities by firms, and allow U.S. workers to adjust to new opportunities.

Those economic institutions seem to matter on their own and in conjunction with cognitive skills. Our analyses suggest that the value of a high-quality education system is substantially diminished in closed economies. We estimate that the effect of a one-standard-deviation improvement in cognitive skills on annual economic growth is 0.9 percentage points per year in closed economies, identified by heavy restrictions on international trade, but 2.5 percentage points in open economies. It may be that rich human capital combines with a laissez-faire economy to foster robust economic growth.

It is also the case that, over the 20th century, the expansion of the U.S. education system outpaced the rest of the world. The U.S. pushed to open secondary schools to all citizens. Higher education expanded with the development of land grant universities, the GI Bill, and direct grants and loans to students. The extraordinary U.S. higher-education system is a powerful engine of technological progress and economic growth in the U.S. not accounted for in our analysis. By most evaluations, U.S. colleges and universities rank at the very top in the world.

Although the strengths of the U.S. economy and its higher-education system offer some hope for the future, the situation at the K–12 level should spark concerns about the long-term outlook for the U.S. economy, which could eventually have an impact on the higher-education system as well. The U.S. higher-education system may also be challenged by improvements in higher education across the world. Other countries are doing more to secure property rights and open their economies, which will enable them to make better use of their human capital. Most obviously, the historic advantage of the U.S. in school attainment has come to an end, as half of the OECD countries now exceed the U.S. in the average number of years of education their citizens receive. Those trends could easily accelerate in the coming decades.

Not Just a Matter of Money

Our evidence of a clear, strong relationship between cognitive skills and economic growth should encourage continued reform efforts. Improvements in mathematics performance called for by No Child Left Behind would matter, contrary to what critics sometimes suggest. Yet reformers should bear in mind that money alone will not yield the necessary improvements. Many expensive attempts around the world to improve schooling have failed to yield actual improvements in student achievement.

Economic growth flows only from reforms that bring actual improvements in cognitive skills. Identifying what works and how to implement it on a society-wide scale remains a challenge, not only for the U.S. but also for many nations across the globe. But, if we are to remain economically competitive, we need to solve the puzzle of our schools and meet the governors’ challenge. We should not, simply because we have failed to meet them in the past, decide that the goals were not legitimate or important.

Eric A. Hanushek is a senior fellow at the Hoover Institution of Stanford University. Dean T. Jamison is professor of health economics in the School of Medicine at the University of California, San Francisco. Eliot A. Jamison is an investment professional at Babcock & Brown. Ludger Woessmann is professor of economics at the University of Munich and heads the Department of Human Capital and Innovation of the Ifo Institute for Economic Research. The opinions expressed in this article are those of the authors and do not necessarily reflect those of their employers.

But the success of the charter school movement has been as uneven as it has been widespread (see Figure 1). There are remarkable differences in the number of charter schools and enrollment between states, and even between school districts within the same state. Take Arizona and Minnesota. The two states were early leaders in the charter school movement, both passing legislation highly favorable to the establishment and support of charter schools. Yet, in the 2005–06 school year, more than 10 percent of Arizona’s enrollment was in charter schools, while only 3 percent of Minnesota students attended a charter school.

The patchwork pattern of success for the charter school movement in the United States raised two big questions in our minds. What factors led some states to grant charter schools a great deal of latitude and provide solid financial support, while others adopted less permissive legislation? And, why, even among states with similar enabling legislation, do charter schools flourish in some places but not in others?

Several well-regarded researchers have tried to explain the differences in charter school legislation. We decided to build on their work, in an effort to produce a more complete account of the politics of the charter school movement. Like those conducting the previous studies, we considered the role of state demographics and party politics. We also used new data to see whether the academic performance of students in traditional public schools and the influence of teachers unions affect the strength of charter school legislation in a state. But this was only one part of our larger project. Marshalling demographic, financial, political, and school performance data from 1990 to 2004, we took the novel step of assessing patterns in the presence of charter schools and in their enrollments at both the state and local levels.

Our Approach

The first thing we needed to do was identify U.S. charter schools and their locations and determine their enrollments. The most recent comprehensive catalog of charter schools and school enrollments is the National Center for Education Statistics (NCES) Common Core of Data. We combined the NCES file with the Center for Education Reform (CER) directory of charter schools to create a master database of 3,066 schools for the 2003–04 school year. Next, we calculated the total number of charter schools and the total enrollment in charters and traditional public schools in each school district. In all, at least 1,000 of the 14,000 districts in the U.S. contained at least one charter school.

With this information in hand, we set out to answer three questions about the charter school movement at the state level. One, why did some states pass charter laws earlier than others? To answer this question, we first studied how D.C. and the 37 states that passed charter laws before 1999 differed from the remaining states that had not adopted charter laws by 1999. For the 40 states that passed a charter law by the 2003–04 school year, we also investigated how earlier and later adopters, grouped by year of the law’s enactment, differ from one another. We wanted to know, for instance, how Minnesota, the state that passed the nation’s first charter school law in 1991, is different from Maryland, which passed the most recent enabling legislation in 2003.

Two, we wondered why some states enacted laws highly favorable to charter schools while others passed more-restrictive statutes. In this part of our study, we compared states based on the rating of their laws by CER, which is an advocacy organization for charter schools. CER rates the “strength,” or permissiveness, of the laws’ provisions. CER judges each law against 10 criteria, each scored on a 1–5 scale, with a total possible score of 50 for laws most favorable to charter schools. These criteria include whether the state grants charter schools an exemption from collective bargaining, the number of chartering authorities beyond local school boards, the number of new charter schools permitted, and whether charters are granted waivers from certain state and local laws. Arizona, Michigan, and Minnesota have enacted relatively “strong” legislation, that is, legislation that provides considerable latitude to charter schools. Other states, such as Kansas, Tennessee, and Virginia, have adopted charter legislation with much more restrictive provisions. In our study, we assigned states without charter school laws a CER law strength score of 0. Although an advocacy organization may have an incentive to understate the strength of these laws, it is unlikely that any overall downward bias would create problems for our assessment of states relative to one another (and, in fact, these scores have been used in a number of other scholarly studies).

Finally, we asked why some states have a greater percentage of public school students enrolled in charter schools than other states. In all our analyses of state laws and statewide enrollment, we wanted to know how characteristics of the state affect the level of support for charters. We were primarily interested in assessing the role of demographics, student achievement, and the extent of school choice currently available in shaping how states participate in the charter school movement. Because the presence of charter schools in an area might affect both student achievement and the decisions of families to move to a district, we measured state demographics and student achievement during the 1989–90 school year, several years before the first charter laws took effect. When studying the pattern of charter school enrollment across the country, we took into account how each of three factors contributes to or retards charter school growth: per pupil expenditures (also measured during the 1989–90 school year), length of time a charter law was on the books, and degree of permissiveness of each state’s charter school law, as measured by the CER index.

Demographics and Politics

Before we proceed, let’s consider why we would expect demographics to shape charter school politics and participation. It is possible that different racial, ethnic, and economic groups demand different curricular approaches: for example, bilingual education, arts, vocational instruction, or programs for gifted or at-risk students. In areas with greater racial and ethnic diversity and economic inequality, families and policymakers may support charter schools as a means of satisfying diverse educational preferences. Indeed, many charter school founders explicitly state that satisfying the educational needs of a target student group is central to their mission. Or it may be that demographics matter because changes in the demographic composition of local school districts increase the desire among families to sort students into similar peer groups. Other researchers have found that white students in charter schools transferred from schools that, on average, had a higher proportion of nonwhite students than their new charter school. In any case, we expected states and communities with more heterogeneous populations to be more supportive of the charter school movement.

A word about our measure of student achievement. It was not clear to us whether parents are most concerned with the absolute level of student performance or with performance relative to what they expect given resources in the school, household, and community. For example, expectations about high-school dropout rates could vary with K–12 expenditures and local poverty. Whether parents, and policymakers, are ultimately “satisfied” with public education may hinge on how well schools are performing relative to local expectations. We conducted our analysis alternately using absolute student achievement, measured with statewide mean SAT scores for the 1989–90 school year and the mean high-school dropout rate calculated from 1990 census data, and with a second measure that represents the deviation of actual achievement from expected student performance.

The two achievement measures, SAT scores and dropout rates, have the benefit of reflecting student achievement at both ends of the ability distribution. A disadvantage, however, is that SAT scores and dropout rates are much more closely aligned to secondary-school performance than to elementary-school performance. And only about 25 percent of charter school students were enrolled in a secondary grade during the 2003–04 school year. We did, however, have a reasonable amount of confidence in the use of these achievement measures because mean SAT scores and high-school graduation rates are very public indicators of educational outcomes, and parents and policymakers are likely to consider these measures when they take positions on charter schools.

To estimate expected student performance, we used a statistical procedure that predicts student achievement (i.e., SAT scores and dropout rates) in the state based on school characteristics and household demographics. The school characteristics we used include per pupil expenditure in the state. Our household demographic measures included the share of blacks and Hispanics in the state population, the fraction of adults who are college educated, and median household income. For instance, if a state has a higher dropout rate than one would predict given its characteristics, we considered that state’s school system to be underperforming. Compared with parents and policymakers in states where expectations are being met or exceeded, those in states with underperforming schools may be more supportive of education reforms such as charter schools.

Of course, whether educational preferences based on demographics or dissatisfaction with existing school performance manifest themselves in support for charter schools depends on other circumstances as well: notably, the political power of opponents to charter schools, the most prominent opponents being teachers unions; and the degree of school choice already available to parents. We estimated the power of unions with the fraction of all teachers in a state who are union members. We based our assessment of the degree of school choice in a state on several indicators, including the fraction of students in the state enrolled in private schools and an index that measures the extent of choice parents have between school districts in the state as of the 1989–90 school year. The landscape of school choice options may be important in that demand for public charters could be greater where parents have fewer schooling options.

The demographic and political characteristics of a state and character of the state law authorizing charter schools undoubtedly matter in some way for the fate of charter schools in a state, but most decisions about charter school formation and attendance are made within school districts—by founders who decide to start a new school, by authorizers who empower them to do so, and, ultimately, by parents who decide to enroll their students. Therefore, we also gave our attention to two indicators of the spread of charter schooling. We measured expansion of charters across districts: first, by whether the district had at least one operating charter school in the 2003–04 school year and, second, by the fraction of public school students enrolled in charter schools during the 2003–04 school year.

We compared districts with at least one charter to districts with no charters and compared districts with higher and lower enrollments in charter schools to search for differences among districts that could explain the variation. As with our analysis of differences among states, we estimated the roles played by demographics, student achievement, and degree of school choice currently available in explaining variation in support for charter schools. We also accounted for whether the district is urban, suburban, or rural; whether the district is elementary, secondary, or unified; per pupil expenditure in the district during the 1989–90 school year; and differences attributable to the state. State-level differences included the strength of charter laws, statewide demographics, existing school choice policies, number of school districts, and the presence of charter support or opposition groups that operate throughout the state.

Explaining Support from State to State

State demographics, student academic performance (measured relative to expectations), and teachers union strength all play important roles in shaping state charter legislation and student participation. Let’s begin with demographics. States with larger Hispanic populations tended to pass laws supporting charter schools earlier and were likely to pass more-permissive legislation. For example, states with an Hispanic-population of 14 percentage points (two standard deviations) higher than the average were about 10 percent more likely to pass a charter law. These states also passed laws that ranked 12 to 14 points higher on the CER strength index (out of a total of 50). There is not, however, strong evidence that these states also had a greater proportion of their students enrolled in charter schools in 2003–04.

Considering the effects of Hispanic population on charter laws, we were surprised that the fraction of a state’s population that is black did not affect the likelihood that a state would pass a charter law, nor did we find any relationship to the timing of passage or the strength of the law. We were very interested to discover that the size of a state’s black population does, however, have a strong relationship with a state’s charter enrollment. For example, a 12.1-percentage-point increase (one standard deviation) in the fraction of a state’s population that is black is associated with roughly a 2-percentage-point increase in charter school enrollment in the state. This is effectively double the charter school enrollment in the average state. One potential explanation for this finding is that black voters, being heavily concentrated in the Democratic Party, are not traditionally swing voters who could influence state legislators’ positions on charter legislation. But once charters are established, African American families are active supporters of charter schools.

Strong charter laws appeared earlier in states where the fraction of adults with at least a college education was higher. For example, states in which the college-educated share of the population was 2.4 percentage points higher passed laws on average six months earlier. States with more-educated populations also had a greater fraction of students enrolled in charter schools. The size of this estimated effect is similar to the effect of a larger black population, about 2 percent greater enrollment in charter schools accompanying a one-standard-deviation increase in the fraction of college-educated adults in the state. Our data did not allow us to explain what lies behind this relationship, but we do have two conjectures. Highly educated citizens may have a greater willingness to experiment with education reforms, or there may be a “supply side” phenomenon: more educated adults translates into a larger pool of charter suppliers.

When we looked at changes in state demographics over time, we found that states with growing Hispanic and college-educated populations were more likely to pass early charter laws. We also found that states with growing income inequality during the 1980s were more likely to pass laws and to pass stronger charter laws during the 1990s.

States with higher-than-expected SAT scores were less likely to pass charter school legislation; tended to adopt such legislation later, if at all; and passed weaker laws. There is no evidence, however, of a statistically significant relationship between state SAT performance and enrollment in charter schools.

A higher- or lower-than-expected high-school dropout rate has no clear relationship to the passage or strength of charter legislation, but does, interestingly, have a strong relationship with charter school participation. States with higher-than-predicted dropout rates had significantly higher enrollment in charters. If the dropout rate is an additional 2 percentage points, or roughly one standard deviation, higher than expected, a state experiences a 1-percentage-point increase in charter school enrollment.

What could account for the differences in the estimated effects of SAT performance and the high-school dropout rates? It may be that SAT scores, as a very public measure of school performance, lead to agitation for charter laws, but that charters themselves are more likely to target students at risk of dropping out, and therefore participation is more closely associated with dropout rates.

Finally, membership in teachers unions has a substantial impact on the legal status of charter schools. States where a greater fraction of teachers were covered by a union contract in 1987 were much less likely to pass a charter law in the 1990s, more likely to pass a law later (if at all), and more likely to pass a weaker law. A one-standard-deviation increase in the fraction of teachers who are unionized, that is, an additional 20 percent of instructional employees covered by union contracts, means a state is 20 percent less likely to pass a charter law. We were initially puzzled to find that, conditional on the successful passage of a charter law and controlling for law strength, the fraction of students enrolled in charter schools appears to increase with the fraction of teachers in the state who are unionized. We share some further thoughts on these contradictory findings in our analysis of school districts.

We found little evidence that the extent of choice among districts in a state is related to either charter school legislation or participation, nor did we find a relationship between income or income inequality and legislative support for charter schools. We did find a positive relationship between the fraction of students enrolled in private schools prior to the passage of charter laws and law passage and strength. This may be due to private school parents supporting public charter schools as a substitute for private schools, or it may be related to broad dissatisfaction with public schools and a generally higher demand for alternatives.

Some have argued that the political composition of the governor’s office and the state legislature may influence the success of charter school laws. We reconducted all our statistical analysis, taking into account party control of the governor’s office and state legislature from 1990 to 2004. We never found party control of the state government to have an effect on the probability that a law passed, the year a law passed, the strength of the law, or participation in charter schools once we controlled for other state characteristics. In fact, of the 40 states passing a law, 24 passed it when a Republican was governor and 16 when a Democrat was governor, and about two-thirds of the states passing a charter law did so when there was no single party controlling both the legislature and the governor’s office.

School District Support for Charter Schools

We began to review the findings of our district analysis by first checking that the results from our statistical procedures were consistent with well-known patterns of enrollment in charter schools. As we expected, school districts with only secondary schools or both secondary and elementary schools were more likely to have a charter school in 2003–04 than districts with only elementary schools. School districts primarily based in a large or mid-sized city or large town were more likely to have a charter school than suburban or rural school districts.

We found that the fraction of a school district’s population that is Hispanic has no clear relationship with either the presence of charter schools or enrollment in the district. As in our state analysis, an increase in the fraction of a school district’s population that is black makes a district more likely to have a charter school in operation and to have a greater share of its students enrolled in charter schools. We estimated that an increase of 11 percentage points in the black population in a district increases the share of students enrolled in charter schools by about 6 percentage points. Considering that the average share of students enrolled in charter schools in school districts with a charter school is only about 10 percent, this is a substantial effect.

We also examined changes in demographic characteristics between 1980 and 1990, and found that districts with a rising fraction of black or college-educated individuals saw greater participation in charter schools. In addition, we found that districts where income inequality was rising saw higher participation in charter schools. This finding may reflect a divergence in preferences for education programs, for example, or an increasing diversity of needs within school districts that raise the demand for charter school options.

School districts that already had higher fractions of students enrolled in private schools, even accounting for the urban or rural location of the district, had a greater likelihood of having a charter school open in their district by 2003–04 and a greater share of their students enrolled in charters. Whether this pattern is indicative of general receptiveness on the part of these districts toward alternatives to public schools or a long-standing dissatisfaction with traditional public schools, it certainly suggests that private schools do not serve as a hindrance to the start-up of public charter schools. Our measure of the current level of choice in the public school system has no statistically significant relationship with charter support within school districts.

Districts with higher-than-predicted high-school dropout rates were, like states with high dropout rates, more likely to have charter schools and a greater share of students enrolled in charters. An 8-percentage-point increase in the adjusted dropout rate (that is, the average deviation from the predicted dropout rate) was associated with about a 2-percentage-point increase in the fraction of students enrolled in charters.

We were very interested to see the relationship between the strength of teachers unions and charter school enrollment, which we first glimpsed in our state analysis, reemerge in our study of districts. Contrary to what one might expect given the opposition—or at least hearty skepticism—of teachers unions to the charter school movement, districts with a greater union presence were more likely to have a charter school and to have a greater share of public school students enrolled in charter schools in 2003–04. A 39-percentage-point increase in the fraction of teachers unionized is associated with about a 2-percentage-point increase in the fraction of students enrolled in charters.

What lies behind this relationship between teachers’ unionization and support for charter schools? We cannot say for sure with our data, but it may be that parents are more likely to support charter schools in heavily unionized states, perhaps in a desire for more local control and less bureaucracy, or a desire for curricular and personnel policies that are less influenced by the union. Strong unions are more successful than weaker ones in opposing liberal charter legislation, but once a charter law is adopted, it seems that parents see charters as an avenue for reform in districts where unions have a strong hold on traditional public schools.

Conclusions

Understanding why states pass laws favorable to charters and why charter school participation is higher in some areas than others is important if we want to identify locations where school choice reforms are likely to present meaningful alternatives to traditional public schools and where they are likely to remain reforms on the margins. Our findings suggest that there are several forces propelling the expansion of the charter school movement. One of the most powerful is growing diversity in state and district populations. States and school districts with more blacks and college-educated adults have a substantially larger share of their students in charter schools than other districts. Failure to meet expectations for student academic performance, as measured by SAT scores and high-school dropout rates, also fuels the passage of charter laws, leads to the passage of stronger charter laws, contributes to the creation of charter schools, and boosts charter school enrollment. Unexpectedly, we discovered that a leading opponent of the charter school movement, teachers unions, appears to contribute indirectly to the expansion of charter schools. In states where teachers are unionized at the highest rates, charter laws were less likely to be enacted, were passed later, and were less favorable to charter schools in states that did adopt charter laws. But in states and school districts with strong unions and charter laws of similar strength, more families have sought out alternatives for their children in charter schools. All else being equal, a highly unionized teaching labor force goes hand in hand with the creation of more charter schools and more students learning in charter school classrooms.

-Christiana Stoddard is assistant professor in the department of agricultural economics and economics at Montana State University. Sean Corcoran is assistant professor of educational economics at the Steinhardt School of Education, New York University.

Increasingly frequent journalistic accounts report that schools are responding to No Child Left Behind (NCLB) by engaging in what has come to be known as “educational triage.” Although these accounts rely almost entirely on anecdotal evidence, the prospect is of real concern. The NCLB accountability system divides schools into those in which a sufficient number of students score at the proficient level or above on state tests to meet Adequate Yearly Progress (AYP) benchmarks (“make AYP”) and those that fail to make AYP. The system gives no credit to schools for moving students closer to proficiency or for advancing already-proficient students. If schools intent on meeting minimum competency benchmarks practice educational triage, they dedicate a disproportionate amount of their limited resources to “bubble kids,” students who might otherwise perform just below the proficiency threshold. While these marginally performing students are likely to benefit from increased attention, reallocation of instructional attention leads to a tradeoff whereby the achievement gains of the marginally performing students come at the expense of both the lowest- and highest-performing students.

With congressional proceedings on NCLB’s reauthorization under way, the time is opportune to take a hard look at educational triage claims. If the current law’s minimum competency standard produces gains among students near the proficiency threshold but disadvantages others, the rules of the accountability system need to be modified, perhaps to reward improvements across the entire achievement distribution.

To search for evidence of educational triage, I analyzed three years of test-score and other data on 300,000 students in public schools in a western state. I found none. I concluded that these schools were not responding to NCLB by trading off achievement among students with different baseline levels. Rather, they were successful at raising the performance of students who were otherwise at risk of failing the state test without sacrificing the performance of lower- and higher-performing students. Even in failing schools, students above the proficiency threshold made gains that were greater than one would expect if schools were concentrating resources on students near the threshold. When academic achievement is measured with test-score performance in this state, the much-politicized argument that NCLB compromises the educational needs and opportunities of high-performing, academically accelerated students holds no water.

THE STATE'S ACCOUNTABILITY PROGRAM

The U.S. Department of Education in 2003 approved the state’s accountability plan, which was designed to meet federal guidelines and regulations associated with NCLB. The plan requires all public schools in the state to meet proficiency standards in math and reading for all students and for each of 10 student subgroups, and to test a minimum of 95 percent of students in each subgroup to avoid sanctions. The accountability program measures students’ content knowledge and skills using an Internet-enabled testing system developed by the Northwest Evaluation Association (NWEA), a national nonprofit organization that provides assessment products and related services to school districts. NWEA compares spring assessment results to grade-specific benchmarks to gauge whether individual students, subgroups of students, and schools meet the state’s proficiency standards.

The particular demographic characteristics of this state limit the generalizabilty of my findings to other, more demographically diverse states. The state is disproportionately white and rural, with much smaller than typical schools and districts. Approximately 83 percent of students are white, 12 percent are Hispanic, and the remaining 5 percent are black, Asian, Pacific Islander, American Indian, or Native Alaskan. Roughly 40 percent of students were identified as economically disadvantaged based on their eligibility for free and reduced-price lunch. Schools that did not make AYP have a higher percentage of Hispanic students than schools that did (18 percent vs. 10 percent) and a higher percentage of students eligible for free and reduced-price lunch (48 percent vs. 39 percent). Despite the disadvantage of atypical demographics, this state offered the unique advantage of being able to measure achievement gains within the same school year. The state tests each student twice per year, permitting for measurement of individual students’ fall-to-spring test-score gains.

DATA

Data in this study are from the NWEA Growth Research Database. Starting with the 2002–03 school year, NWEA administered tests in mathematics, reading, and language arts to more than 90 percent of the state’s students. NWEA furnished fall and spring test scores for the first three years after enactment of the state’s accountability program (2002–03 through 2004–05 school years) for students in grades 3 through 8. My analysis focuses on math scores. The statewide percentage of students scoring in the proficient and advanced categories in math has ranged from a low of 53 percent for 8th graders in 2003 to a high of 90 percent for 4th graders in 2005.

The NWEA data set also provides demographic information about students, including the student’s grade in school, gender, race, ethnicity, and eligibility for free or reduced-price lunch. School-level characteristics include school type and school size. Although NWEA gathers data for students in traditional public schools, charter schools, and private schools, I limited the study to students enrolled in traditional public schools or public charter schools because private schools are not included in the state’s accountability program. I removed from the study sample very small schools, those with fewer than 34 students being tested, given their systematically different treatment under the state’s accountability system.

IDENTIFYING EDUCATIONAL TRIAGE

My objective was to detect shifts in how schools committed resources to different students, resources such as textbooks and teachers, but also such inputs as teacher attention and choice of curriculum and instructional strategies. Obviously, no formal accounting system tracks the distribution of resources directed at individual students. So I turned to an indirect measure of resource allocation. I infer the priorities of administrators and teachers from educational outcomes, as measured by student performance on the state’s math test. If there is a greater-than-expected increase in the achievement of students just below the state’s proficiency standard, and this occurs in tandem with a less-than-expected increase in the achievement of high- and low-performing students, then I can conclude that educational triage has transpired.

Is it reasonable to expect administrators and teachers to be able to identify students who are likely to be on the cusp of the proficiency threshold at the spring test administration? When speaking of states that use NWEA assessments, the answer is yes. NWEA furnishes classroom teachers and building principals with proficiency reports for each student within days of the fall test administration. The reports include a projection of each student’s performance on the spring test.

Consider a hypothetical example of the distribution of changes in test scores under educational triage. In Figure 1, the y-axis is the amount of growth in a student’s test score from the fall to the spring test administration. The x-axis identifies a student’s distance from the state-defined proficiency threshold. The vertical line in the middle of the graph is the threshold a student needs to cross to be considered proficient. The farther a student lies below the performance threshold in the fall, the more likely that student is to fail the spring assessment. The inverted “V” depicts the simplified pattern of gains one would expect to see if a school disproportionately targets resources, such as instructional time and teacher focus, to students particularly important to its accountability rating, that is, to students hovering around the state-defined proficiency threshold. If this practice were the case, the greatest fall-to-spring achievement gains would occur among students around the threshold, while other students would struggle to match expected test-score gains.

My basic strategy, then, was to compare fall-to-spring test-score changes among students who were expected to be either nearer or farther from the state-defined proficiency threshold following spring testing. A question of particular interest was whether schools that failed to make AYP in the previous school year responded strongly to the incentive to target instruction to students at risk of falling just short of the proficiency threshold.

STATISTICAL CONTROLS

The first step in my statistical analysis was to rank all the students within the same grade and year by their performance on the fall exam. I then divided the students into 20 groups of equal size and calculated a standardized test-score change for each student that measured their respective performance relative to students within the same performance group. Use of standardized test scores helps address what statisticians call reversion to the mean. When one takes repeated measures of some event or behavior, such as test-score performance among students, the measurements at the low and high ends of the resultant distribution tend over time to converge toward the average value for the population under study. With respect to students and test scores, reversion to the mean suggests that students with scores in the upper or lower tail of the test-score distribution are likely to perform closer to the average when tested more than once. This effect may mask a school’s actual response to the threat of failing AYP by producing the illusion that schools are helping low-performing students while neglecting high-performing students. By comparing each student’s gain to gains among students who performed at a similar level and would have experienced a similar, natural shift toward the average score, I can better separate legitimate test-score gains and losses from change associated with mean reversion.

I also made my best effort in the statistical analysis to isolate the change in test scores that could be attributed reasonably to the resources schools dedicate to teaching students. Specifically, I separated out the effects on test-score gains of a student’s race and ethnicity, as well as accounted for the influence of a student’s peers, by evaluating the influence of demographic characteristics of the student body, including average income level and percentage of minority students. Given a large data set, I also was able to account for characteristics of schools that I could not directly measure but that might influence student achievement over the school year. Finally, I took precautions against shaping the results according to changes in test difficulty from year to year for each grade and for students in a given school.

RESULTS

Despite many media claims of educational triage, I found no evidence of failing schools engaging in coordinated targeting of students near the state-defined proficiency threshold. In the state under study, public schools that had failed to make AYP focused instruction on the entire range of low-performing students in the subsequent school year, and did so without negative impact on high-performing students.

Figure 1b shows the changes in standardized test scores, across the full range of student performance, that can be attributed reasonably to teacher and school performance and to decisions about how the school allocates resources among students. In schools that failed to make AYP in the previous year, students who were expected to fall well below proficiency gained more than students nearest the proficiency threshold. The lowest-performing students gained about 0.20 standard deviations, roughly twice the improvement of those students whose expected gains were to leave them just below proficiency. Students expected to be proficient did not lose ground; the most advanced students performed comparably to other already-proficient students.

In schools that did make AYP, lower-performing students met expectations, with the largest of those gains coming from students expected to perform the weakest. It is interesting to note, in contrast, that higher-performing students in these schools lost ground from one year to the next. Remarkably, proficient students enrolled in failing schools experienced larger test-score gains than proficient students in non-failing schools. Remember that these patterns of gains and losses would look much different if schools did indeed engage in educational triage. Under educational triage, students near the proficiency threshold would attain the largest gains, while students dispersed away from this threshold and toward the tails of the achievement distribution would suffer diminished performance.

CONCLUSION

Although there is no evidence that schools in the study sample targeted resources to particular students, they may have allocated resources toward outcomes measured by the accountability system. For instance, schools may have taught to the tests in math and reading while neglecting science, social studies, the arts, and physical education. The apparent absence of educational triage in one state does not invalidate documented accounts of the practice in particular schools, nor encroach upon other arguments to modify NCLB’s proficiency-based school rating system. Nonetheless, as the reauthorization debate continues, policymakers should take note that educational triage was not evident in the first statewide analysis of the issue.

Matthew G. Springer is research assistant professor of public policy and education at Vanderbilt University’s Peabody College and director of the federally funded National Center on Performance Incentives.

It is now possible to provide some answers to these questions by exploring the relative effectiveness of recently hired New York City public school teachers who entered the profession through alternate routes. Using a large data set provided by the New York City Department of Education (NYC DOE), we analyzed student test scores as well as information about the students, their teachers, classrooms, and schools. With this rich array of data, we compared the effectiveness of recently hired alternatively certified (AC) and uncertified teachers to that of their traditionally certified counterparts in improving student learning in math and reading during grades 4 through 8.

The results of our study of New York City public school teachers confirm a simple truth: some teachers are considerably better than others at helping students learn. For example, elementary-school students who have a teacher who performs in the top quartile of all elementary-school teachers learn 33 percent of a standard deviation more (substantially more) in math in a year than students who have a teacher who performs in the bottom quartile. Yet as we embrace this piece of conventional wisdom, we must discard another: the widespread sentiment that there are large differences in effectiveness between traditionally certified teachers and uncertified or alternatively certified teachers. The greatest potential for school districts to improve student achievement seems to rest not in regulating minimum qualifications for new teachers but in selectively retaining those teachers who are most effective during their first years of teaching.

New Routes to the Classroom

State laws were originally designed to regulate teacher quality by specifying minimum pre-service course-taking and exam performance. Reflecting these traditions, the majority of prospective public-school teachers have completed one or two years of full-time study in a university education program and passed exams such as the widely used PRAXIS tests to satisfy those requirements.

In the 1990s, that traditional system for regulating the flow into teaching began to break down. Faced with difficulties recruiting enough certified teachers, many school districts hired large numbers of uncertified teachers. When No Child Left Behind and state laws mandated a certified teacher in every classroom, districts turned to the growing number of teachers entering the profession through alternative certification (AC) programs. According to the National Center for Education Information (NCEI), 48 states and the District of Columbia currently have AC programs, and approximately 50,000 teachers became certified through alternative routes in 2004–05 NCEI estimates about one-third of all new teachers enter the field with alternative certification. Participants in the AC programs are usually required to possess a bachelor’s degree and pass state licensing exams before entering the classroom and to enroll in a teacher-education program, taking coursework at night, during their first years in teaching.

We wanted to know whether staffing classrooms with uncertified and AC teachers shortchanges students. In other words, are students essentially paying the cost of training these teachers on the job, in the form of lost academic achievement?

Knowing the relative merits of traditionally certified, AC, and uncertified teachers is particularly important because AC and uncertified teachers are more likely to work in urban areas with low-income and low-achieving students, those most in need of a high-quality teacher who can foster their learning. This is one reason the New York City public schools are an ideal setting to explore how a new teacher’s certification status might affect how much students learn. Besides being the largest and one of the most diverse school districts in the country, New York City is a major employer of uncertified and AC teachers. More than 50,000 new teachers were hired in New York schools between the 1999–2000 and 2004–05 school years. Uncertified and AC teachers accounted for, respectively, 34 percent and 20 percent of these new hires.

Teacher Certification in New York City

Recruiting a sufficient number of certified teachers has been a long-standing challenge for the New York City Department of Education. During the 1999–2000 school year, approximately 60 percent of all new teachers hired were uncertified. Recruiting difficulties have been more severe in schools with the lowest average achievement levels.

Since that time, the NYC DOE has taken a number of steps to decrease its use of uncertified personnel, one of which has been to expand its recruitment of alternatively certified teachers. Between the 1999–2000 and 2004–05 school years, uncertified hires fell from 60 percent to 7 percent of all hires, while the fraction who were alternatively certified rose from 2 percent to 36 percent. It is highly unlikely that this shift was a matter of previously uncertified teachers entering AC programs. The two populations—uncertified and AC teachers—differ in a number of ways: AC teachers are less likely to be black or Hispanic, tend to be several years younger when hired, and attended colleges with substantially higher median SAT scores (see Figure 1).

The major source of new AC teachers has been the New York City Teaching Fellows (NYCTF) program. Created in the summer of 2000, NYCTF is a partnership between the New York City Department of Education and The New Teacher Project (TNTP). TNTP is a national nonprofit organization, founded in 1997, that helps school districts recruit AC teachers. TNTP has also worked with school districts in Miami, New Orleans, Oakland, Philadelphia, Washington, D.C., and a number of other urban and rural communities.

The NYCTF program was created as a response to pressure from the state government to hire only certified teachers in the city’s lowest-performing schools. After the NYC DOE failed to hire only certified teachers for these schools in 1999–2000, it was sued by state education commissioner Richard Mills. In 2000, additional motivation to produce and hire AC teachers was provided by a New York State law that required all teachers in the state to be certified in their subject of instruction by the 2003–04 school year. The number of teaching fellows hired grew from 350 in 2000–01 (less than 5 percent of new hires) to 2,500 in 2003–04 (more than 30 percent of new hires) and 2,000 in 2004–05 (more than 25 percent of new hires). Teaching fellows represent the vast majority of all AC teachers signed on in New York City since 1999 Between the 1999–2000 and 2004–05 school years, some 9,000 teaching fellows were hired.

New York recruits AC teachers through several other sources as well: the high-profile, nationwide Teach For America (TFA) program, the Peace Corps Fellows Program, and the Teaching Opportunity Program Scholars. (The latter two programs are relatively small.) New York City hired 1,544 new TFA teachers during the 1999–2000 to 2004–05 school years. Here, we limit our discussion to the findings on teaching fellows and the substantial number of teachers who come from the TFA program.

Measuring Teacher Effectiveness

It is a complicated task to determine how much difference a teacher makes in student achievement and whether or not that difference depends on how she entered the teaching profession. We need to compare the effectiveness of teachers in each certification group while separating out each of their students’ baseline level of achievement (measured with test scores from the prior school year) and other characteristics of the student, classroom, school, and teacher that could affect student achievement.

We used data from the New York City Department of Education, which cover the 1998–99 to 2004–05 school years and grades 3 through 8, the grades in which students in New York City take standardized math and reading examinations. Because we need a prior year test score for each student in each grade in order to estimate the contribution made by the student’s teacher, we can only study 4th- through 8th-grade teachers.

The student data include test scores, race and ethnicity, eligibility for the federal free and reduced-price lunch program, and status as an ESL or special-education student. We know about a student’s prior attendance record and any suspensions from school. It is important to account for these student characteristics when evaluating an uncertified or AC teacher’s contribution to student learning compared to that of a certified teacher. Uncertified teachers, teaching fellows, and TFA corps members all tend to teach in schools that, relative to those employing more certified teachers, have a higher percentage of minority students; more low-income, ESL, and special-education students; and students with lower achievement levels. We can place each student in a classroom and school so we know about the demographic and educational characteristics of students’ schoolmates as well as the size of their class.

The NYC DOE data also include identification numbers for students’ math and reading teachers, which were often the same for elementary-school students. This is a crucial piece of information, enabling us to match a student to his or her teacher. The NYC DOE payroll system was the source of information about each teacher’s certification program and years of teaching experience. We are interested in evaluating teachers based on their certification at the time they are hired. Thus, uncertified teachers who later gain regular or alternative certification are still, in our study, considered uncertified. The study sample includes some 10,000 teachers in total, drawn from four groups: regular certified, regular uncertified, teaching fellows, and Teach For America.

Certification and Effectiveness

Simply put, a teacher’s certification status matters little for student learning. We find no difference between teaching fellows and traditionally certified teachers or between uncertified and traditionally certified teachers in their impact on math achievement. Classrooms of students assigned to TFA teachers actually scored 2 percent of a standard deviation higher than students assigned to traditionally certified teachers. In reading, students assigned to teaching fellows did underperform students assigned to traditionally certified teachers by 1 percent of a standard deviation. These are the only instances in which we find that a teacher’s initial certification status has statistically significant implications for student achievement. The picture of teacher effectiveness looked the same when we separately examined teachers in elementary schools, middle schools, and schools with above- and below-median test scores.

A Close Look at Teacher Experience

We also measured how teaching effectiveness improves with experience in New York City public schools. New York’s teachers are no different from other teachers around the country. Teachers make long strides in their first three years, with very little experience-related improvement after that. The students of third-year teachers score 6 percent and 3 percent of a standard deviation higher in math and reading, respectively, than students of first-year teachers.

Since the first few years of experience are so important, we decided to take a closer look at how uncertified and AC teachers fare against traditionally certified teachers at different levels of teaching experience. We compared the effectiveness of teachers from among the certification groups in their first, second, and third years of teaching.

We found that teaching fellows, TFA corps members, and uncertified teachers may fare slightly worse as rookie teachers than certified teachers, but they quickly make up the lost ground (see Figure 2). For example, first-year teaching fellows underperform traditionally certified teachers in their first year by 1 percent of a standard deviation in math, but third-year teaching fellows outperform third-year traditionally certified teachers by 2 percent of a standard deviation. In reading, rookie teaching fellows underperform first-year teachers with traditional certification by 2 percent of a standard deviation. Yet by their third year of teaching, teaching fellows are eliciting student achievement as well as third-year traditionally certified teachers.

How do AC and uncertified teachers manage to catch up to traditionally certified teachers? This improvement need not reflect on-the-job learning. It could be that the propensity for ineffective teachers to leave could be higher among those uncertified and alternatively certified. This would improve the average performance of these groups relative to traditionally certified teachers, even though individual teachers did not improve at higher rates.

We can estimate the rate of improvement that is attributable to experience (as opposed to the departure of weak teachers) by following individual teachers and measuring the change in their performance from one year to the next. We can then see if these changes, say from year one to year three, tend to be greater or smaller for teachers from different certification groups.

There is some evidence that AC and uncertified teachers learn more from experience. Whether they teach math or reading, TFA teachers seem to learn at the same pace as traditionally certified teachers. The first two years of teaching bring substantially more improvement in both math and reading instruction for teaching fellows than for traditionally certified teachers. Given the same initial effectiveness as a traditionally certified teacher, our results indicate that, after two years on the job, a teaching fellow’s students would score 3 percent of a standard deviation higher on average in math and reading. Uncertified math teachers’ gains from experience also outpace those of traditionally certified teachers. Given the same initial effectiveness as a traditionally certified teacher, an uncertified third-year teacher’s students would score 3 percent of a standard deviation higher, on average, in math.

Attrition

In debates over certification standards, alternative certification programs have been criticized for high turnover. Alternatively certified teachers may be just as good as traditionally certified teachers, the critics say, but they are more likely to leave teaching just when they are learning the ropes. Teach For America programs have been the focus of much of this criticism, since they ask their corps members for only a two-year commitment.

The critics may have a point. When faced with the choice of two teachers of equal initial effectiveness but differing expected turnover rates, a principal or district recruiter ought to choose the teacher with lower expected turnover. However, the strength of this preference depends on two things: the actual difference in turnover rates and the difference in effectiveness between an experienced and a novice teacher. Fortunately, both of these differences can be estimated with the NYC DOE data.

First, we set out to determine the average rate of attrition, by year of teaching experience, among teachers hired since the 1999–2000 school year. We used a statistical procedure that allows us to adjust for factors that might affect attrition, such as teacher age and the characteristics of the school in which they taught. We also adjusted for the state-mandated departure of teachers who remained uncertified at the end of the 2002–03 school year, which artificially raised the attrition of uncertified teachers in that year. (Starting in the fall of 2003, school districts in New York State were no longer permitted to employ uncertified teachers, although in schools with severe teacher shortages some continued to teach with two-year Modified Temporary Licenses.)

Perhaps surprisingly, after adjusting for these factors, teaching fellows have attrition rates similar to those of traditionally certified teachers. In fact, teaching fellows have slightly lower attrition rates in the first two years than traditionally certified teachers. After five years, approximately 50 percent of both groups still teach in the New York City public schools. Teachers who were initially uncertified are a little less likely to stick around than teaching fellows or traditionally certified teachers. Given the estimated rates of attrition, 45 percent would remain with the district in their fifth year.

Teach For America corps members do have much higher exit rates. By the fifth year, only about 18 percent of corps members would remain with the district. Much of the attrition for TFA corps members comes after two years of teaching. Presumably, this reflects the fact that TFA corps members sign up for a two-year teaching commitment.

We used these attrition rates to estimate the proportion of each group of teachers who would be in their first, second, and third year of teaching and so on, up to 30 years. Roughly 45 percent of TFA teachers would be in their first or second year of teaching during any given school year. In stark contrast, traditionally certified teaching and teaching fellows would be less than half as likely to be so green, with only 20 percent in their first or second year.

Despite the higher attrition among TFA corps members, the implications are not nearly as dire as some critics of the program suggest. The impact on student outcomes is negative but rather modest. When we compare teacher effectiveness in our simulated teacher pool, we estimated that TFA teachers would need to produce about 2 percent of a standard deviation in additional math and reading achievement in their students to offset the impact of the higher turnover rates. At least in math, the size of the Teach For America advantage during the first years of teaching is just large enough to offset the cost of higher turnover. (Although TFA corps members’ effect on students’ reading scores is not as great, the payoff to experience is also considerably lower in reading achievement.) In other words, both the critics of Teach For America, who point to high attrition rates, and the supporters of Teach For America, who maintain that the corps members are more effective than other novice teachers, are right. But on net, it is a wash.

Teachers Matter Even If Certification Does Not

The above discussion should not be taken to imply that teachers do not matter. Although it may not matter much whether a child is assigned to a certified or uncertified teacher, it certainly does matter to which teacher a student is assigned. Interestingly, the range of effectiveness within each certification group is wide and the distribution of effectiveness is roughly the same.

There are a number of potential sources of error in measurements of teacher effectiveness based on student achievement data. For instance, because many classrooms have 20 to 25 students, a few particularly talented or particularly disruptive students can radically change the classroom average. Moreover, there may be other factors, unrelated to class size, that lead to swings in classroom performance, such as a dog barking in the parking lot on the day of the test or special classroom “chemistry” reflecting a good match between a particular classroom of students and a particular teacher. We wanted to measure the part of a teacher’s effectiveness that persists over time. We used a statistical method that separated our estimate of overall effectiveness into a fixed component, which is stable from year to year, and an idiosyncratic component that changes across classrooms and across years.

When we looked at the persistent component of teacher effectiveness, we found that the best teachers have a large positive impact on their students’ academic performance relative to that of a less effective teacher. For example, the top quarter of elementary-school teachers improve student achievement in math by 33 percent of a standard deviation more than the bottom quarter of teachers do. Among middle-school teachers, the difference is slightly less but still important, at 20 percent of a standard deviation. To put this in perspective, the advantage of being the student of a teacher in the top quarter of effectiveness rather than the bottom quarter is roughly three times the advantage of being taught by an experienced teacher rather than by a novice, and more than ten times any advantage created by teacher certification!

Knowing When to Be Choosy

Traditionally, states and districts have regulated teacher quality by focusing on initial qualifications. In writing the No Child Left Behind Act, Congress followed that same logic, requiring states to live up to the minimum hiring standards they have established. But is a highly qualified teacher more likely to be a highly effective teacher? Our results suggest not. States and districts could learn a lot about teacher impacts on student achievement during the first few years on the job, as long as they assemble their student and teacher data to compare end-of-year performance among classrooms of students with similar baseline characteristics. That is an obvious first step. But states and districts also need to find ways to directly observe and rate in-class performance, using some combination of principals, peers, and external observers.

We believe states need to develop the infrastructure for assessing the performance of novice teachers during their first few years on the job. There are some models being developed, such as Connecticut’s Beginning Educator Support and Training (BEST) program, Charlotte Danielson’s performance evaluation rubrics, and the National Board for Professional Teaching Standards. These approaches are only beginning to be validated against student performance on standardized tests. That is a process that will take time, and it is probably too early to say that there is one best approach to assessing teacher effectiveness.

Admittedly, not all private employers terminate their less effective employees. Employees who do not perform up to expectations may remain in entry-level positions, left in the proverbial “mailroom. ” Schools are very different organizations from most private firms. Notably, there is no equivalent to the corporate mailroom. Less effective teachers, when they earn tenure, are assigned classrooms of students just like more effective teachers.

Nevertheless, there are some labor markets that operate in the way we propose. The market for faculty at top universities is one such example. Arguably, colleges and universities have a lot more information at the time they hire faculty than school districts do. They have a copy of several papers someone has written; they have spoken with a candidate’s advisors, whom they often know personally; they invite the candidate to campus for a full day of interviews. Yet despite this intensive screening, the top departments remain at the top by selectively retaining only the most prolific researchers from among their junior faculty.

Ironically, current collective-bargaining agreements already give district leaders the flexibility to terminate ineffective teachers during their first two or three years on the job. For instance, in New York City, teachers’ contracts may not be renewed if they receive an unsatisfactory rating from their principal during their first three years of teaching. Our estimates suggest that just one year yields a substantial amount of information on teachers’ effectiveness. Three years would give a school district a lot of information.

Nevertheless, there are several potential impediments to districts’ implementing this kind of policy. Individuals who believe they may be dismissed will be less attracted to teaching in a district with selective retention. Districts may have to make other changes, such as increasing salaries for teachers clearing the tenure hurdle, in order to recruit enough teachers to fill available positions. The “highly qualified teacher” requirements in the No Child Left Behind Act already make it hard for districts to hire sufficient numbers of novice teachers. Our proposal would require districts to hire even more. Those provisions will have to be rethought during the law’s upcoming reauthorization.

By shifting the focus away from “qualifications, ” we are not proposing to open the floodgates into teaching. Nor are we intending to imply that teaching is based on innate talent rather than developed skill. Instead, we simply want to move the dam further downstream from the time of initial recruitment, and postpone assessments of teacher’s effectiveness for a year or two until districts have much more useful information about which teachers are performing well and which are performing poorly. Only then will we have hope of living up to the aspirations embodied in the No Child Left Behind Act.

Thomas J. Kane is professor of education and economics at the Harvard Graduate School of Education. Jonah E. Rockoff is assistant professor of economics and finance at Columbia Business School. Douglas O. Staiger is professor of economics at Dartmouth College.

But in recent years a new concern has surfaced: What if it’s not the district but rather the specific school a child attends within a district that matters most for accessing educational resources? Mounting evidence suggests that districts commonly distribute different amounts of funding, even when schools serve the same types of students. Our research and that of others indicate that schools with predominantly junior teachers receive fewer salary dollars than do schools staffed with veterans. Further, districts often compound these inequities by distributing a smaller share of unrestricted funds to the same schools that are shortchanged in salary dollars.What we don’t yet know about school funding inequalities is whether and how these discrepancies have changed in recent years. Nor is there much information available about how spending differences within districts compare to differences between districts in the same state.

In this study, we address these questions by taking an in-depth look at funding differences between and within Texas school districts over the course of a decade, from the 1993–94 to 2002–03 school years. Within Texas, we focus our attention on large school districts, those with more than 25,000 students. In 1994, the state had 29 districts with an enrollment greater than 25,000, and that number increased to 39 by 2003. These districts serve about half of all Texas public school students.

Texas’s large districts are useful cases for two reasons. First, other studies of school funding equity have suggested that funding discrepancies are greatest in the largest and most urban school districts. By focusing on large districts, we are more confident that we are identifying the full extent of inequality that exists between schools. At the same time, we recognize that we may not be able to generalize our findings beyond such districts.

Second, the state of Texas has in recent years aggressively addressed funding inequalities between districts. In 1993, following a state supreme court order to equalize public school spending, the state’s school finance system adopted a provision known as the “Robin Hood” law that requires property-rich districts to subsidize poorer districts within the state. Studying Texas districts and schools allows us to assess whether and how policies designed to reduce inequities between districts affected inequalities between schools within districts.

Our findings demonstrate that, at least in Texas, funding decisions within districts currently have a greater impact on a school’s resources than inequalities in access to revenues across school districts. Although reforms have been successful in reducing inequalities between Texas districts, variation in funding within districts remains high. As a result, examining only data aggregated to the district level, still the standard practice in equity studies, misses much of the inequality in funding across schools.

Data and Methodology

We obtained financial and descriptive information about Texas districts with more than 25,000 students and the schools within those districts from a large database published by the Texas Education Agency. The database reports financial allocations to schools by district. For each school, we know the nontargeted, or noncategorical, allocations made for each student who attends the school as well as how much the school received for five targeted groups of students: students eligible for free or reduced-price lunch, students eligible for bilingual education programs, students with disabilities, gifted students, and students in vocational education programs. We exclude charter schools from our analysis, as their funding levels are not determined by the same policies that affect traditional public schools.

We first examine the differences between schools in noncategorical resources by comparing each school’s per-pupil funding to the average per-pupil funding in the district. We make this comparison by calculating the ratio of each school’s per-pupil noncategorical expenditure to the district’s average per-pupil noncategorical expenditure. For example, if School A receives $4,000 per pupil in noncategorical funds but the district average per pupil is $5,000, then the ratio for School A is 0.8 (or 80 percent of the district average). Since the ratio compares individual school funding to the average for the district, we know that any funding differences we see are entirely the product of intradistrict rather than interdistrict variation.

Next, we compare total spending, including funds allocated specifically for students eligible for free or reduced-price lunch and bilingual, vocational, or gifted education. We exclude special education funds from this analysis because of large variations in funding depending on disability type.

For each district, we compute the district’s average expenditure for each student-need group. For example, one district may allocate an average of $500 on top of the nontargeted allocations for each gifted student, while another district might allocate only an average of $200. This average is effectively an implicit spending weight unique to each district, determined by dividing the sum of all allocations made on behalf of each student type by the number of students in that category.

We then calculate a ratio, called a Weighted Student Index (WSI), of the actual funding received by each school to the funding we would expect if schools received the district’s average allocation for its particular mix of students. The WSI allows us to compare per-pupil funding in schools while accounting for the types of students a school serves. A school with a WSI of 0.7 receives 70 percent of what we predict the school would be allocated, given its student population, if all the schools in the district received the same amount for each student of each type enrolled in the school.

We follow standard practice among school finance researchers who are interested in studying potential inequality at both ends of the spectrum, and calculate for each school year in our study the coefficient of variation for the differences in funding within districts. We define the coefficient of variation as the standard deviation of the population divided by its mean. Since the mean value of our two spending indexes is 1.0, the coefficient of variation is actually equivalent to the standard deviation in our analysis. The value of 0 indicates perfect equity, with larger values signaling greater disparities in the allocation of funds. Researchers studying spending differences between districts have established 0.1 as an acceptable level of equity, and we follow this convention in our analysis of between-school spending differences.

As an additional point of comparison, we also examine spending inequalities between districts. In this analysis, we adjust spending figures to reflect differences in district size and in the costs of providing education before calculating the coefficient of variation. For both the between-schools and between-districts analyses, the dollars analyzed include total operating funds from federal, state, and local governments, and use real-dollar teacher salaries.

The Funding Picture

Throughout the decade we study, the 1993–94 to 2002–03 school years, noncategorical funding between schools within Texas districts was considerably less equal than between districts. The coefficient of variation calculated in the between-school analysis was consistently higher than that calculated in the between-district analysis. We removed the state’s four largest urban districts from the sample and found between-school inequities were still much higher than inequities between districts.

There has been modest progress toward equity of noncategorical funds across districts and schools in Texas over the last decade (see Figure 1a). At the district level, the coefficient of variation in 1994 was 0.09, dropping to 0.07 in 2003. The coefficient of variation among schools for the 1993–94 school year was 0.17, dropping to 0.14 by the 2002–03 school year. The good news is that in 2003, the coefficient of variation across schools in 24 of Texas’s 39 largest school districts was less than 0.1. The average coefficient of variation across schools, however, exceeded this benchmark in each year.

When we examined noncategorical per-pupil funding in the state’s four largest school districts—Austin, Dallas, Fort Worth, and Houston—the levels of inequity were even higher and each district was remarkably different from the others. In Dallas, Fort Worth, and Houston, the coefficients of variation were nearly always more than 0.15, meaning that one-third of the schools in these districts had spending levels that deviated from their district’s average by 15 percent (or $225,000 for a school of 500 when average spending is $3,000 per pupil). In contrast, Austin had a coefficient of variation near 0.15 for most of the decade, but dipped to the 0.1 level for three years, from 1997–98 to 1999–2000. Houston ranged between 0.2 and 0.25, except for one year, while Dallas had the highest levels of inequality, hovering around 0.3 until the 2000–01 school year, when it experienced a dramatic drop in the level of inequality in the district, indicating that a greater percentage of schools were funded at or near the district’s average allocation per pupil.

During the decade we studied, Fort Worth made steady improvements toward equity in noncategorical funding across its schools, while Austin’s allocations became less evenly distributed over the last five years in our study. And while there appear to be some equity gains in these four districts over the last two years of this analysis, there is no clear long-term trend toward improvement.

Figure 1b shows the equity picture for total funding over the period. While inequities both between and within districts have decreased over the past 10 years, there is still greater variation across schools than across districts. Taking into account resources expended for particular student types, then, does not change the patterns in noncategorical spending described above in any meaningful way.

The Impact of School Characteristics

Of course, we should not assume that all inequalities in spending between schools are necessarily perverse. District officials in Texas might point out that there are reasons aside from special student needs that could legitimately prompt uneven funding among schools. School level, school size, and academic performance are often cited as factors that shape strategic funding allocations to schools. Districts might, for example, allocate a relatively larger share of resources to high schools because they are expected to provide a diverse curriculum. Similarly, a district could be spending more on its lowest-performing schools to support improvement efforts. As discussed above, though, previous research documents spending differences resulting from less intentional factors, primarily differences in teacher salary costs due to different levels of teacher experience.

In order to investigate the role of both the intentional and unintentional factors, we explore the extent to which various school characteristics explain variation in the allocation of resources within a school district. We look at level of school (high school, middle school, or elementary school), total enrollment, percentage of the student body that is white, average experience of teachers, and school performance, as measured by the school’s academic rank within the state. Using data from the 2001–02 school year for our sample of large Texas districts, we estimate the amount of variation in total per-pupil funding, measured by a school’s WSI, that we can attribute to such school organizational characteristics.

We find that, as expected, high schools tend to receive more funding. On average, a high school’s WSI is 0.18 higher than an elementary or middle school, indicating that high schools received 18 percent more than elementary or middle schools within the same district with similar student populations. There is some indication that the lowest-performing schools in a district have higher WSI scores, and additional analyses reveal that this pattern is concentrated among elementary schools. Unexpectedly, we do not find a clear or strong relationship between school size and WSI values. Nor do we find that schools with a larger share of white students have a meaningful increase in their WSI.

Schools with more experienced teachers and the lowest–performing schools receive slightly more funding from the district, with higher WSI by 0.01 and 0.04, respectively. In other words, these schools typically received 1 to 4 percent more than the district average, or $15,000 to $60,000 per school of 500 students in a district where the average school expenditure is $3,000 per pupil.

These findings aside, it turns out that relatively little of the differences in funding by schools is explained by these school and district characteristics. We can account for only slightly more than one-third of all the variation between schools in the same district. If spending is not strongly influenced by observable school characteristics, we have to question whether it is driven by a district strategy at all. What we haven' within and outside the district, organizational habits, and de facto policies at play in the system that can and likely do affect how districts distribute resources among schools.

Weighted Student Formulas

There have been calls for district policies that would transparently and equitably allocate district resources among schools, with the use of an explicit formula. Districts that adopt a weighted student formula (WSF) for funding schools allocate funds according to the specific student types enrolled. Spending increments, or weights, are deliberately determined for each student need. Were the districts in this analysis to allocate funds using a strict WSF system, we would not have found any inequities between schools. Each school would receive exactly the average allocation for its mix of students, and the coefficient of variation for each district would be 0.

To date, only a handful of districts have implemented WSF, and certainly not in the strict sense described here. In 1998, Houston implemented a modified version of WSF, known as student-based budgeting, in which a base amount is set per student and a percentage added for each special need, such as bilingual education. While we would not feel comfortable claiming, based on the analysis here, that student-based budgeting has been the cause of greater equity in Houston’s school funding system, our findings do show that despite an initial increase in the coefficient of variation, Houston schools have over the longer term made modest improvements in equity since the strategy was put into place.

There are reasonable concerns about the consequences of WSF funding. First, district leaders may not select appropriate weights. They may choose, for instance, to allocate too little for each student in bilingual education or too much for each student in gifted programs. Second, some inequality is likely to remain even with a WSF system. WSF models typically ignore the effect of differences in teacher experience levels and, therefore, teacher salary across schools by adjusting the allocations for real salaries after the weighted formula has been applied.

Regardless of whether WSF systems are the answer, there is a clear and simple policy lesson in the experience of large Texas school districts. We should not assume that school finance reforms directed at resolving resource inequalities between school districts will ensure those resources are equitably distributed among schools and their students.

Marguerite Roza is research assistant professor, Kacey Guin is research associate, Betheny Gross is senior research associate, and Scott DeBurgomaster is research assistant at the Center for Reinventing Public Education at the Evans School of Public Affairs at the University of Washington.

To assess the influence of neighborhoods on children’s education, we need to compare large numbers of children in similar families who reside in different communities. Unfortunately, the task is not as simple as finding families who already live in higher- and lower-poverty neighborhoods. Even if these families share characteristics that we can measure, such as income and parental education levels, there may be differences we cannot measure that both led these otherwise similar families to reside in their respective neighborhoods and influence the academic performance of their children. This possibility leaves us uncertain about how much neighborhoods, as compared to family influences, matter for children’s academic performance.

What we really want is to be able to identify low-income families in high-poverty neighborhoods who would all prefer to live in better areas. We need some of these families to relocate to lower-poverty communities, but which families move and which remain must be decided by a process outside their control. If we see differences in academic performance between the children in the relocated families and the children who remained in the higher-poverty neighborhood, we can then be reasonably certain that these changes are due to the new neighborhoods, and by extension the new schools, and not family characteristics.

In 1994, the U.S. Department of Housing and Urban Development (HUD) began a pilot program that provided researchers with just this type of experimental data. From 1994 to 1997, HUD carried out a lottery that awarded housing vouchers to families in five U.S. cities: Boston, Baltimore (see “All Over the Map,” features), Chicago, Los Angeles, and New York. The Moving to Opportunity (MTO) Fair Housing demonstration program was designed to move poor families with children out of public housing developments located in census tracts with 1990 poverty rates of at least 40 percent and into the private housing market of more affluent neighborhoods.

Families had to take the initiative to enter the housing voucher lottery, thus indicating their preference for moving. The lottery process was used to randomly assign each family to one of three groups. The first group received housing vouchers that could be used to rent in the private market in any neighborhood. These vouchers were part of the preexisting Section 8 (now called “Housing Choice Voucher”) program, which awards housing vouchers to low-income families. (Families residing in public housing or using vouchers to rent apartments in the private market are generally required to pay 30 percent of their income in rent.) The second group also received vouchers, but, new to the MTO program, the use of the vouchers was restricted in the first year to low-poverty neighborhoods, that is, a neighborhood in a census tract with a poverty rate of less than 10 percent, based on 1990 census data. The third group did not receive either voucher, creating a control group of families against which those in the other two groups could be compared. During the first four years of the program, a total of 4,248 families entered the lottery. Some 1,209 were offered the Section 8 unrestricted vouchers; 1,729 families were offered the restricted vouchers; 1,310 received no voucher. Forty-seven percent of the families who were offered restricted vouchers and 59 percent of the families who were offered unrestricted Section 8 vouchers “leased-up” new apartments under the terms of the program. In our analysis, families who were offered vouchers but did not use them remained part of the experiment in order to maintain the randomly assigned groups. However, our analysis takes account of the fact that not all families who were offered a voucher used it.

We set out to evaluate the impact of the MTO program on neighborhoods, schools, and educational outcomes across all five cities and on the full age range of children affected by the MTO program. We expected that in 2002, the year of our study, children whose families were offered vouchers of either type would, on average, live in lower-poverty neighborhoods, attend higher-performing schools, and would, as a result, have improved educational outcomes. Our approach was to compare the educational outcomes of children whose families were offered housing vouchers to those of children in families who entered the lottery but were not offered vouchers. Using data on more than 5,000 children between the ages of 6 and 20, we examined mid-term outcomes four to seven years after a family entered the lottery.

We found that families offered housing vouchers in the MTO demonstration program on average moved to communities with substantially fewer poor and substantially more educated neighbors, but with only marginally fewer minority neighbors. Despite these neighborhood improvements, the families sent their children to schools where students were performing at only modestly higher levels on state exams than in the schools attended by children in the comparison group. Nor did we find evidence of any improvement in reading scores, math scores, behavior, or school engagement. At least for the children in the Moving to Opportunity experiment, the promise that better neighborhoods would bring greater academic achievement has thus far gone unfilled.

Data

In 2002, in collaboration with HUD and Abt Associates, we were able to achieve an effective response rate of 90 percent in obtaining information on families participating in the program four to seven years after they entered the lottery. We collected data on one adult and up to two children from each family in our study. We interviewed parents and we also interviewed children age 8 or older. In the interviews, parents were asked about their children’s academic performance and behavior. Children were asked about their neighborhoods, schools, academic performance, and behavior. Interviewers administered a battery of achievement tests to all children.

Our sample of over 5,000 children is divided roughly equally between girls and boys. The average child is just over 13 years old. The sample consists mainly of minority children; approximately two-thirds are non-Hispanic African Americans and about 30 percent are Hispanic (black or non-black). The majority of the children are from female-headed households.

We identified changes in neighborhoods and schools by constructing address and school histories for each family. We used several sources to compile the address histories, including contacts with the families, the National Change of Address system, and credit reporting bureaus. To determine a neighborhood’s level of poverty, we located the street address within the 1990 and 2000 census tracts and used the percentage of residents in the tract who live below the poverty line, as measured by the census data. We interpolated poverty levels for years between 1990 and 2000, and extrapolated post-2000. Our expectation was that neighborhoods have a cumulative impact on children. We incorporated this assumption in our analysis by averaging the characteristics of all the neighborhoods the children lived in between their families’ entrance in the lottery and our follow-up in 2002. The longer a child lived in a neighborhood, the more weight we gave that neighborhood’s statistics in our calculation.

Interviewers constructed school histories for each child by asking the adult in the family for the names and grades of all the schools the child had attended since the lottery. We were able to use the names and addresses of schools to retrieve information about school enrollment and type from the National Center for Education Statistics’ Common Core of Data and Private School surveys. State education departments and the National Longitudinal School-Level State Assessment Score Database provided us with average test scores at the schools.

Our primary measures of each student’s academic performance were the reading and math scores of MTO children from the Woodcock-Johnson-Revised (WJ-R) battery of tests, which the interviewers administered. The WJ-R can be used across a wide range of ages, produces scores that are highly correlated with other commonly used achievement tests, and has been standardized on a nationally representative sample.

We obtained additional information about schools and the students from interviews with the adults and children. We asked children about their school’s climate, including its safety and the level of disruption by other students. We assessed how engaged the children were with learning by asking them directly about how hard they worked at school, how often they were tardy, and how many hours they spent reading each day.

Interviewers asked the adults whether their children exhibited various behavior problems. Specifically, we measured a child’s degree of behavior problems as the fraction of eleven problems, including “is disobedient at school” and “has difficulty concentrating, cannot pay attention for long,” that the adult reported as “sometimes” or “often” true for the child. We also used the survey to gather information from the adults about other educational outcomes, such as whether the child had been held back a grade, been suspended, or taken any remedial or advanced classes.

Measuring the Impact of the MTO Program

In a randomized experiment, the baseline characteristics of the treatment and control groups should, on average, be the same. In the case of the MTO program, the random assignment of housing vouchers by lottery does appear to have created experimental and control groups that are similar on characteristics that we can observe, including age, gender, race, frequency of learning and behavior problems, the employment status and education of parents, and the reported reasons for wanting to move, including the desire for better schools. This experimental setup should allow us to attribute any subsequent differences between the groups as the product of the offer of a voucher. To increase the precision of our estimates of the effects of being offered a voucher, we controlled statistically for the influence of other characteristics of the child and parent on educational outcomes.

After estimating the average effect of being offered a voucher by comparing the entire treatment and control groups, we used information on how many families actually moved to gauge the impact on those who relocated to a new neighborhood. Assuming that the families who were offered a voucher were not affected if they did not use the voucher, the size of the impact is essentially the estimated impact of being offered a voucher divided by the fraction of families who moved. Recall that 47 percent of families who were offered restricted vouchers moved to lower-poverty neighborhoods. This means that the impact of actually using a restricted voucher is approximately twice as large as the impact of being offered a restricted voucher.

New Neighborhoods

The MTO program had huge success in moving families out of public housing and into lower-poverty neighborhoods. The offer of a restricted housing voucher, for use in a tract with a poverty rate less than 10 percent, led to children living in neighborhoods with poverty rates averaging 12.6 percentage points lower than those where families that had not been offered a voucher lived, an average poverty rate of 33 percent for the restricted voucher group compared to an average poverty rate of 45.6 percent for the control group. When we calculate the impact for families who actually used the restricted vouchers, we estimate that families moved into neighborhoods with an average poverty rate 28 percent lower than the neighborhood in which they would have lived if they had not used the vouchers (see opening Figures).

The impact of using a Section 8 unrestricted voucher was also sizable, though smaller than the impact for the restricted voucher group. Children whose families actually used unrestricted vouchers moved to neighborhoods where the average poverty rate was 17 percent lower than the average poverty rate of the neighborhood in which they would have lived.

Whether a family used a restricted or unrestricted housing voucher, children lived in neighborhoods with approximately 5 to 7 percentage points lower male unemployment as well as more college-educated adults and fewer minorities. Restricted voucher families resided in neighborhoods with 13 percentage points more adults with a college degree and 17 percentage points fewer minorities. The changes for families who used unrestricted vouchers were about 30 to 40 percent as large.

What we did not see, however, were families moving en masse to affluent, white suburbs. Even among the families who used restricted vouchers, only 18 percent lived in neighborhoods where the poverty rate was below the state median. Additionally, only 21 percent resided in neighborhoods where more than half of the residents were non-Hispanic whites.

New Schools

The changes in the schools children attended were more modest than the improvements in neighborhoods. Children in families who moved with restricted vouchers attended schools with slightly higher average scores on state exams. The average effect was 9.5 percentiles, bringing the average percentile rank for children whose families used restricted vouchers from the state’s 15th percentile to the 24th percentile. Students also attended schools with about 10 percentage points fewer minority students and almost 13 percentage points fewer classmates eligible for the federal lunch program. The use of unrestricted vouchers again produced changes about 30 to 50 percent as large.

The schools attended by students in families using vouchers had higher-performing students, but they also had more-advantaged student bodies. As a result, it is unclear whether these schools were actually more effective in raising student achievement. When we compared their performance to other schools in the same states, taking into account differences in the backgrounds of their students, the advantage for the schools attended by housing voucher students disappeared. We therefore cannot rule out the possibility that the increased school test scores simply reflect differences in the students served and are not indicative of more effective schools.

What did students think of their new schools? We found no differences between students from families offered vouchers and those from the control group, when the children themselves were asked to characterize the climate of their schools. We are unsure how to interpret these results. The lack of positive effects on self-reported measures of school climate could reflect a real lack of improvement, but it could also be due to changes in the frame of reference among students in their new neighborhoods. This certainly must be the case among the youngest children, who never attended schools in their original neighborhoods.

Student Test Scores

When it comes to improving student test scores, the MTO program does not appear to have had an overall impact (see Figure 1). Our statistical estimate puts the impact of moving with a restricted voucher at four-hundredths of a standard deviation increase in combined reading and math test scores, and the estimate is not statistically significant. We also failed to find evidence that suggests an advantage to using an unrestricted Section 8 voucher.

Nor were the impacts of the program more favorable for younger children, who had spent fewer years in high-poverty neighborhoods and, we hypothesized, were more amenable to change. We tested for varying impacts by age in two different ways. First, we divided our sample of children into three roughly equal groups, ages 6–10, 11–14, and 15–20. We then estimated the impacts separately for each age group. We also tested for the possibility that the effect of moving to a lower-poverty neighborhood decreases gradually with the child’s age. The estimates of the impact of a voucher on student test scores are statistically insignificant regardless of whether we look within age groups or for a year-by-year correlation.

We did find noteworthy impacts on combined reading and math test scores in two of the five MTO sites. In Baltimore, use of a restricted voucher appears to produce a statistically significant increase in test scores of 25 percent of a standard deviation, an effect that is concentrated among elementary school–aged children. Oddly, in New York, use of an unrestricted voucher is associated with a decline of one-half of a standard deviation. For the three other cities, none of the estimates were large or statistically significant.

We estimated the impact on test scores separately for girls and boys, students with different races and ethnicities, and children grouped by characteristics measured at the time of the lottery that are predictive of low, moderate, and high test scores. Again, we usually failed to find that students from voucher families performed differently than students from the control group. The one exception was the statistically significant positive impact of the use of a voucher on the achievement test scores of non-Hispanic African American children. Among these students, restricted-voucher use is correlated with notable improvements in reading scores, with average scores almost one-fifth of a standard deviation higher than for the control group. The estimated impact of using an unrestricted Section 8 voucher among African American children is not statistically significant and is approximately 0.1 standard deviations. When we analyzed this relationship by MTO site, we discovered that these improvements in reading for African Americans appear only in the Baltimore and Chicago sites. In these two sites, our sample includes virtually no other racial or ethnic groups. Considering the lack of evidence for an impact on African Americans in the other three sites, and the increased likelihood of incorrectly identifying a chance occurrence as a real impact when comparing multiple subgroups within a sample, we encourage readers to interpret these findings with caution.

Behavior and Attitudes toward School

As with test scores, we generally did not find large, statistically significant differences in behavior or attitudes toward school between children from families with and without vouchers. Students from both restricted and unrestricted voucher families and the control group had similar numbers of behavior problems and were equally likely to have repeated a grade, been suspended in the last two years, or had a parent or guardian called to the school to discuss a problem the child was having. Additionally, we could not find differences between the children in their engagement with school, which we measured with the children’s self-reported attention in class and tardiness. Nor were children from voucher families any more or less likely to attend remedial or advanced classes.

There are three exceptions to the similarity between the voucher and control groups. First, moving with a restricted voucher appears to be associated with increased behavior problems for 11- to 14-year-olds. Second, we find that moving with a restricted, though not an unrestricted, voucher has an adverse impact on the behavior of boys. Boys from families that used restricted vouchers scored, on average, one unit higher on the behavior problems index. The estimate for girls from the same group suggests that they may have fewer behavior problems than girls in the control group, but the estimate is not statistically significant. Third, for boys and girls with characteristics associated with higher test scores, the family’s use of a restricted voucher appears to be associated with more behavior problems.

Why Didn’t Student Performance Improve?

The Moving to Opportunity program moved families into lower-poverty neighborhoods and children into schools where students performed at modestly higher levels. So why were there virtually no improvements in children’s test scores, incidence of behavior problems, and school engagement, even among children who were preschoolers at the time their families moved?

One plausible explanation lies in the fact that families who used vouchers later took steps that undid some of the potential advantages of their initial moves to middle-class neighborhoods. Children in families that moved using a restricted voucher resided an average of 3.1 years at their new addresses. Subsequent moves, however, tended to be to tracts that were considerably less affluent than the neighborhoods in which they first used the voucher. Moreover, for many families who remained in their new tracts, the poverty rate in their neighborhood increased around them. The census tracts to which MTO voucher families initially moved had substantially higher poverty rates on average in 2000 than they did in 1990. Despite this trend, however, voucher families did reside in considerably more affluent neighborhoods, on average, than control families.

Second, although families with vouchers moved to less-impoverished neighborhoods, most did not move to racially or ethnically integrated neighborhoods. By 2002, three-fifths of families that moved with vouchers were living in neighborhoods with 80 percent or more minority populations. For a variety of reasons, these neighborhoods may not have high-quality schools or other public services usually found in more-affluent neighborhoods where whites make up a greater share of the residents. We can do little more than speculate about this here, however, because we do not have the data to test whether this was the case.

Third, today many urban school systems offer school choice. In fact, over 30 percent of the control-group children in Chicago and Los Angeles were attending magnet schools. School choice may have allowed families that moved with vouchers to continue sending their children to schools in their old neighborhoods.

The final plausible explanation is related to the relative affluence of MTO families’ new neighborhoods. Certainly, voucher families moved to neighborhoods that were substantially less poor, and the improvement was very likely greater than any that could be produced by community-based neighborhood improvement programs. Nevertheless, the neighborhood improvements did not involve moving to truly affluent neighborhoods. Previous nonexperimental studies have indicated that neighborhood effects on academic performance are primarily found in comparisons of children in affluent areas to those in middle-class neighborhoods. Lacking random assignment, these studies are not fully convincing. Nonetheless, it may be the case that children from low-income families who moved into high-income suburbs would experience notable improvements in academic achievement.

In the end, we were surprised and disappointed by the inability of the Moving to Opportunity experiment to help poor children succeed in school. After following more than 5,000 MTO children over five years, we can offer some tentative conclusions about policy interventions designed to improve student performance. Moving poor families to neighborhoods that, while less poor, have schools that are only marginally better than those in the original neighborhoods is unlikely to solve the children’s academic problems. Interventions might be more profitably focused on factors more directly related to the child, family, and school. The question remains whether these same conclusions will hold after a full decade of living in new neighborhoods, especially for the youngest children who, because of the MTO program, will have spent their entire lives outside of public housing projects. We will be collecting additional data next year and look forward to learning more.

Lisa Sanbonmatsu is a postdoctoral fellow at the National Bureau of Economic Research. Jeffrey R. Kling is senior fellow and deputy director of economic studies at the Brookings Institution. Greg J. Duncan is professor of education and social policy at Northwestern University. Jeanne Brooks-Gunn is professor of child development and education at Columbia University.

Details on these findings are available in “Neighborhoods and Academic Achievement: Results from the MTO Experiment,” Journal of Human Resources 41:4 (Fall 2006), 649-691. An earlier version of this paper is available at: http://www.nber.org/~kling/mto/mto_ed.pdf.

Many supporters of school choice argue that neighborhood assignment to public schools results not in what public school advocates celebrate but in just the opposite: schools that are less likely to contain a diverse mix of students and that are more internally segregated along racial lines than are schools of choice. In recent years, a number of empirical studies of the effects of school choice on civic values have been published. As the extent of school choice in American education continues to grow—the latest data from the Department of Education show that 26 percent of American students attended a school other than their closest neighborhood public school—it is time to take stock of the evidentiary record on whether assigned public schooling better prepares students for their responsibilities as citizens in a democracy.

Studying the Effects of Choice on Civic Values

For this review, I examine the results of 21 quantitative studies regarding the effects of school choice on seven civic values that relate to the capacity of individuals to perform as effective citizens in our representative democracy. The values, in order from the most studied to the least studied, are political tolerance, voluntarism, political knowledge, political participation, social capital, civic skills, and patriotism.

The studies are divided into two categories, based on the statistical rigor with which the investigation was conducted. To qualify for inclusion in this review, a study had to be a quantitative analysis that controlled for observed differences in the backgrounds of the students attending different schools. To be classified as rigorous, the study also had to attempt to correct for the tendency of students and families to sort themselves into different schools and school sectors based on unobserved factors, a research challenge commonly referred to as selection bias. Those studies classified as rigorous used experimental data or employed sophisticated statistical techniques that credibly adjust for the possibility of selection bias when analyzing nonexperimental data. These more-rigorous studies should be weighted more heavily in any assessment of school sector impacts.

Most of the studies included in the analysis focus on students in private schools. Only three studies present results for students in charter or magnet schools. Therefore, the results described below primarily map out the effects of private schooling on civic values.

Findings are divided into three categories. A finding is categorized as signaling a traditional public school advantage if the evidence suggests that such a schooling arrangement produced a statistically significant (at the 90 percent confidence level or better) increase in the realization of the particular civic value. A finding is classified as supporting a choice school advantage if attendance at a public or private school of choice generated a statistically significant positive effect on a civic value. Findings of no significant difference between traditional public and choice schools are classified as neutral.

As can be seen in Figures 1a and 1b, the 59 findings from existing studies suggest that the effect of private schooling or school choice on civic values is most often neutral or positive. Among the group of more-rigorous studies, 12 findings indicate statistically significant positive effects of school choice or private schooling on civic values and 10 suggest neutral results (see Figure 1). Only one finding from the rigorous evaluations indicates that traditional public schooling arrangements enhance a civic value.

The studies that employ only basic adjustments for likely self-selection paint an even rosier picture of the positive effects of school choice on civic values (see Figure 1). Of the 36 findings, 21 indicated a school choice advantage in promoting preparation for citizenship. Thirteen neutral results appear in this collection of analyses, and two findings show benefits from traditional public schooling. The reader is cautioned not to draw strong conclusions from these studies, however, since they employed only rudimentary methods for addressing the problem of selection bias. We now consider the specific civic values that appear to be affected by school choice arrangements.

Studies of Political Tolerance

Democratic citizenship requires that we respect the rights of others, even if we profoundly disagree with their opinions. The most commonly used method of measuring such political tolerance first asks respondents to either think of their least-liked political group or select one from a list that includes such groups as the Ku Klux Klan, American Nazis, the religious right, and gay activists. It then asks whether respondents would permit members of the disliked group to exercise constitutional rights such as making a public speech, running for political office, and teaching in the public schools. Other studies simply ask respondents whether they would permit various activities from a group with whom they disagree, without first asking them to choose their least-liked group. In either case, responses are aggregated into a tolerance scale.

With one exception, the findings regarding the effect of school choice on political tolerance are confined to the neutral-to-positive range. Eleven findings—five of them from the more-rigorous studies—indicate that school choice increases political tolerance. For example, one experimental voucher study in Washington, D.C., found that nearly one-half of the students who switched to a private school said they would permit a member of their disliked group to live in their neighborhood, compared with just over one-quarter of the students in the public school control group. Three studies that used sophisticated nonexperimental techniques to control for selection bias also found positive effects of choice arrangements on political tolerance. These studies were of 8th-grade students in Dallas–Fort Worth attending private secular and nonevangelical religious private schools, 8th-grade students in private secular schools in New York City, and Massachusetts students attending secular private schools.

The more-rigorous studies produced eight findings that school choice arrangements neither increase nor decrease political tolerance. For example, three experimental studies found neutral results of school vouchers on the political tolerance of middle-school students, in the Washington program after three years, a Dayton program after two years, and a San Francisco program after two years.

Six of eight findings from the less-rigorous studies of the effects of school choice on political tolerance indicate a school choice advantage. Three studies concluded that secular private schools have a positive effect on political tolerance. One analysis reported that Catholic schooling boosted tolerance. Another study found that religious schools in general increase the political tolerance of their students. A third report concluded that private schooling of any type improves political tolerance among Latinos.

An observational study by Jay Greene and his colleagues reported no effect of school type on the political tolerance of Texas adults, and David Campbell’s analysis of the National Household Education Survey (NHES) found that students in traditional public schools had higher levels of political tolerance than students in non-Catholic religious schools.

Studies of Voluntarism

The ideal citizen not only tolerates dissent but also actively serves the community. With one exception, studies regarding the extent to which private schooling or school choice affects the likelihood that students or parents will volunteer their time in community enterprises range from neutral to positive.

Four voluntarism findings emerged from rigorous studies, with three of them favoring school choice. The study of 8th graders in New York City and Dallas–Fort Worth found that private school students are 21 percent more likely to volunteer—and dedicate more hours to community service—than comparable public school students. A study using just the sample of 8th graders in New York City found that private schooling promotes volunteer activity if the students attend religious private schools. A third rigorous study looked at the effects of school choice on the likelihood of parents volunteering in New York City and the New Jersey suburbs and found that parents in school choice districts are about 6 percent more likely to volunteer than are comparable parents in nonchoice districts. Finally, students in private secular schools in New York City are nearly 17 percent less likely to volunteer than comparable public school students, a finding that favors traditional public schooling arrangements.

Studies of voluntarism employing less-rigorous statistical methods produced 10 findings; half favor school choice, whereas the other half report no effects of school type. The findings in support of school choice show higher voluntarism among students in religious schools, parents of students in religious schools, parents who home school, students in any type of private school, and students in public charter schools. Other studies reported that voluntarism rates were similar between the students in secular private, non-Catholic religious, and magnet schools and their peers in traditional public schools. Two studies reported similar rates of volunteering between the parents of students in secular private and traditional public schools.

Studies of Political Knowledge

Presumably, democratic citizens will be more active and effective in public forums to the extent they are knowledgeable about politics and current events. Researchers typically measure political knowledge by administering brief civics quizzes of 3 to 10 items pertaining to central features of the U.S. Constitution as well as contemporary public figures. Five studies produced nine findings regarding the effect of school choice on political knowledge. Only three of these findings are from rigorous studies. R. Kenneth Godwin and Frank Kemerer, in their analysis of students in schools in New York City and Dallas–Fort Worth, found that choice students scored higher than traditional public school students on political knowledge regardless of whether they attended private schools in general or evangelical private schools in particular. The only experimental study of school choice and political knowledge found no significant difference in average political knowledge levels between recipients of vouchers and comparable students in public schools.

Three studies that employed basic statistical methods report six findings regarding the effect of school choice on political knowledge, two showing a choice school advantage. Richard Niemi and his colleagues drew on the NHES data to conclude that private schooling increases political knowledge. David Campbell’s more fine-grained analysis of the same data reported that only Catholic private schools demonstrated a clear political knowledge advantage. Campbell found that students in non-Catholic religious, secular private, and public choice schools all evidenced political knowledge levels that were comparable to students in traditional public schools. In the earliest known empirical study of the effect of school choice on civic values, James Coleman and Thomas Hoffer concluded that students in Catholic and public schools were similar in their average levels of political knowledge.

Studies of Political Participation, Social Capital, Civic Skills, and Patriotism

Beyond being tolerant, community minded, and well informed, we also expect well-trained citizens to be politically active possessors of social capital with civic skills who are loyal to their country. Unfortunately, relatively few studies have queried the extent to which school choice arrangements foster such attributes. One rigorous study, by Thomas Dee, concluded that Catholic schooling increases voter turnout as adults. Jay Greene and his colleagues conducted less-sophisticated studies that found that Latinos who received all of their K–12 education in private schools were 16 percent more likely to say they voted in the last presidential election than comparable Latinos who were educated exclusively in public schools. They also reported that Texas adults who were educated at least partly in private schools were 9 percent more likely to have voted recently, all else being equal. An observational study by Christian Smith and David Sikkink found that parents who enroll their children in private religious schools or who home school them are more politically active than are otherwise comparable parents who enroll their children in public schools. Parents of students in private secular schools do not differ significantly from public school parents in political participation.

Two rigorous studies reported findings regarding the effects of school choice on social capital, typically defined as a close connection with one’s community via social networks, group norms, and cooperation for mutual benefit. Mark Schneider and his colleagues concluded that the responsibility to choose their child’s school increases the social capital of parents. Paul Peterson and David Campbell reported no difference in the levels of social capital between voucher users and control group members in their experimental analysis of the Children’s Scholarship Fund.

In the first study of education and social capital, Coleman and Hoffer employed basic statistical methods and found that Catholic schooling was associated with higher levels of social capital. Greene and his colleagues replicated those results on a national sample of Latino adults.

No experimental studies have been conducted on the effects of school choice on civic skills or patriotism. Two studies that applied basic statistical methods to the 1996 NHES data generated diverse findings regarding the effect of school choice on civic skills. The survey asked students, During this school year, have you done any of the following things in any class at your school: Written a letter to someone you did not know? Given a speech or an oral report? Taken part in a debate or discussion in which you had to persuade others about your point of view? Students in private high schools were more likely to have engaged in these three activities than comparable students in public high schools, according to one study. The second study found that students in Catholic schools scored slightly higher than comparable students in assigned public schools. No significant differences in civic skills were uncovered between students in assigned public schools and comparable students in non-Catholic religious or secular private schools.

An observational study of patriotism employed an index that includes five questions about students’ visceral attachment to their country and its symbols (such as the flag and the Pledge of Allegiance). New York City 8th graders in private schools scored somewhat lower on patriotism, on average, than comparable students in public schools. One weakness of this analysis is the patriotism scale employed, which could be interpreted as a measure of national chauvinism or jingoism.

The Catholic Schooling Effect

Several prominent scholars have claimed that Catholic schooling may be largely responsible for the generally positive school choice effects on civic values. Would the likely effects of choice on political tolerance, voluntarism, and other democratic values disappear or turn negative with Catholic schools out of the picture? Figure 2 excludes all results based on comparisons between public and Catholic school populations or that focus exclusively on the experiences of Latinos (who, if privately schooled, predominantly attend Catholic schools). Study groups identified simply as “private religious” are considered Catholic and excluded for purposes of this analysis, since most religious private schools in the U.S. are Catholic. Twenty-two results showing a school choice advantage remain, suggesting that secular private schooling enhances political tolerance, that charter schooling increases voluntarism, and that education at an evangelical private school increases political knowledge. Twenty findings indicate that school choice has no clear effect, positive or negative, when schools other than Catholic schools are chosen.

Three findings showing a traditional public school advantage remain, suggesting that evangelical Protestant schools reduce political tolerance, that secular private schools decrease voluntarism, and that private schooling of any sort may diminish a particularly passionate form of patriotism. As all the negative effects shown in Figure 1 also appear here, it seems non-Catholic schools of choice are responsible for the few negative effects of choice arrangements on civic values observed here. However, non-Catholic schools of choice also appear to generate many positive outcomes regarding democratic values. These results suggest that the expansion of school choice is more likely to enhance than diminish the civic values of our next generation of citizens, even if none of the new choosers end up in communitarian-infused Catholic schools.

Discussion

All of the studies reviewed draw on data either about the various school sectors as they existed in the 1980s and 1990s or from modestly sized school choice experiments. The demographic composition of the various school sectors and the independent effects of private schooling and school choice on the civic values reviewed here would likely change somewhat under a complete or even larger-scale school choice regime. One should therefore be cautious in drawing strong conclusions from the empirical record to date on school choice and civic values.

The empirical picture regarding the effects of school choice on civic values raises some concerns. The lone study on fostering patriotism indicates that public schools may hold an advantage over schools of choice. In one study of voluntarism, attending private secular schools apparently reduced the likelihood of volunteering. Attending an evangelical Protestant school was found to decrease political tolerance in one study and increase political knowledge in another, causing scholars such as Stephen Macedo to worry that such schools may produce young adults who are strongly equipped to act politically on their intolerance. The Madrassa schools of radical Islam remind us that private schools of choice can serve to undermine democratic values. It would seem reasonable to require some minimal oversight and regulatory constraints on private schools that accept public monies, such as prohibitions against teaching hate. As important as these concerns are, the record to date suggests that civic values tend to be enhanced, or at least not harmed, by the exercise of school choice.

What aspects of choice schools generate these modestly positive civic values outcomes? No direct evidence yet exists regarding the specific conditions or practices of choice schools relative to traditional public schools that would explain this pattern of results. One theory is that schools of choice foster strong education communities typified by regular parental involvement and a concern for the welfare of all members. Yet several other plausible explanations also deserve attention. Teachers in private schools may be freer to infuse instruction with moral values and discuss controversial issues than public school teachers. Students who regularly encounter value-based claims and perspectives may become more tolerant of people with value-based positions that differ from their own. They also may feel more motivated to volunteer for activities that seek to bring about social and political change.

The most intriguing explanation, in my opinion, for the apparent school choice advantage in promoting civic values is a generally higher level of order and discipline in schools of choice. Public charter schools and private schools tend to be more well-ordered education institutions than neighborhood public schools, especially in urban environments. A well-ordered and nonthreatening education environment likely contributes to students’ feelings of security and confidence. Such feelings might be a necessary precondition for young people to develop a willingness to tolerate potentially disruptive political ideas and political groups and to venture out into the community to promote social causes, an idea suggested by Alan Peshkin in his case study of a Christian fundamentalist school. There is a clear theoretical justification for linking a well-ordered education environment with stronger civic values, and I hope that future studies will explore this possibility.

Other aspects of schooling might also promote higher levels of civic values among students, be they in assigned public schools or schools of choice. Effective instruction itself likely promotes civic values, as better-educated citizens tend to be more knowledgeable about politics, more tolerant, and more active in their communities. Some preliminary studies suggest that students are more likely to embrace civic values as adults if they had the opportunity to participate in student governance or voluntary activities as students, or at least witnessed adults who modeled proper civic behaviors in their schools. There is less empirical support for curricular interventions aimed at boosting civic values. Civics classes appear to increase civic values such as tolerance only modestly, and only if they are customized to focus explicitly on that particular value. There is no evidence that taking a required civics course in junior high or senior high school, in and of itself, enhances civic values.

In summary, the empirical studies to date counter the claims of school choice opponents that private schooling inherently and inevitably undermines the fostering of civic values. The statistical record suggests that private schooling and school choice often enhance the realization of the civic values that are central to a well-functioning democracy. This seems to be the case particularly among ethnic minorities (such as Latinos) in places with great ethnic diversity (such as New York City and Texas), and when Catholic schools are the schools of choice. Choice programs targeted to such constituencies seem to hold the greatest promise of enhancing the civic values of the next generation of American citizens.

Patrick J. Wolf is professor of education reform and 21st century chair in school choice at the University of Arkansas College of Education and Health Professions.

A complete list of the studies used to generate the above findings is provided here.

Often the victories for adequacy are only the beginning of prolonged and inconclusive struggles within the ruling courts and between the courts and legislatures or governors. But sometimes the outcomes are radical. In a path-breaking suit in Kentucky, the state supreme court in 1989 found virtually everything about that state’s schools to be unconstitutional, and the legislature responded with major reforms. More recently, in March 2006, an appellate court in New York State ordered its elected officials to increase operating aid for New York City schools by between $4.7 and $5.63 billion a year and by $9.2 billion over five years for capital improvements. Adequacy lawsuits have proved a serious threat to the right of citizens to have their taxes determined by elected officials who are in a position to weigh competing claims for public support and to judge the relative efficacy of spending for particular purposes.

Adequacy as a Political Campaign

At first glance it appears ironic that plaintiffs have enjoyed a higher rate of success in adequacy cases than in those grounded in equity. Courts would seem to have greater legitimacy and competence in adjudicating the latter. The irony disappears, however, if school finance lawsuits are viewed as political rather than legal events. As political events, equity cases compelled the redistribution of spending for education, inciting a strong reaction from those property-rich school districts with the most to lose. Adequacy cases have the clear political advantage: they aim to enlarge the educational pie. Districts rich or poor and urban or rural, teachers and administrators, equipment suppliers, consultants, building contractors, pension funds—along with the advocacy organizations that everywhere push for more school spending—can detect such opportunities for gain and join forces, at least up to the point at which remedies are specified and the bigger pie begins to be sliced.

Adequacy lawsuits are political events: they allocate things of value, and they propel the courts into an institutional sphere normally reserved for the legislature and the governor. Head litigators in adequacy lawsuits know that judicial decisions depend on implementation by the political branches and are alert to ways in which this might be achieved. At a 2005 conference of the adequacy movement, winning lawyers from North Carolina, Montana, and Kansas constituted a panel devoted to the subject of converting court victories into solid remedies. Beyond speaking of standard litigating tactics, such as picking plaintiffs, witnesses, and exhibits, they spoke of success at spinning the media, hiring public relations firms, and engaging a lobbying firm to work with the legislature (in Kansas), all standard political tactics. One lawyer hinted at success in having a school board attorney “from one of our [plaintiff] districts” appointed to the state supreme court (again, Kansas). They spoke of the utility of lawsuits for agenda setting—of keeping school spending inescapably before the legislature.

Implementation of national statutes is always problematic in a federal system and with a national legislature that habitually underfunds its promises. The adequacy campaign, a national movement committed to litigating in state courts, conceives of itself as stepping into this breach. The standards-and-accountability movement—which spread nationwide through the 1990s and reached a climax with passage of the No Child Left Behind Act (NCLB) in 2002—has provided a political stepstool to adequacy suits. The keynote speaker at the adequacy movement conference was Representative George Miller, a California Democrat and one of the principal authors of NCLB. He told the conference, “You have to continue to litigate. Only through litigation will we capture attention.… You can help us realize the goals and live up to the promise of No Child Left Behind.” Michael Rebell, leader of the adequacy movement in New York City and more broadly, has chastised critics for failing “to grasp that the education adequacy lawsuits have become the driving force for achieving the aims of the standards-based-reform movement.”

The Expanding Reach of the Courts

Because the challenge to separation of powers from adequacy lawsuits is so plain, one might expect them to have given rise to constitutional debates within the states. Angered legislators have sometimes proposed constitutional amendments in defense of their prerogatives. Conservative Republican members of the Kansas legislature in 2005 tried to couple school spending that had been compelled by courts with a proposed amendment that would have prohibited courts from ordering the legislature to make appropriations. The proposal failed to get the two-thirds majority in the Kansas house that was needed for submission to the electorate. More school spending had support from Democrats and a few Republicans in the legislature despite the challenge to the institution.

Legislatures per se are not normally defendants in the lawsuits, and so cannot mount their own defense in court. State officials who are in charge of the defense do not necessarily have strong incentives to conduct it vigorously. No attorney general has yet won a large following by opposing more spending on schools or supporting the constitutional principle of separation of powers. State superintendents of instruction, who often have a great deal of influence in shaping the defense, have even less incentive to oppose increased spending on schools.

Given the absence of widespread constraints on the courts from legislators pushing back, it has been up to the courts to work their way through the issue of whether or not these cases are appropriate for court determination (“justiciability”). Most courts have elected to advance into the legislative terrain, all the while denying that they are doing any such thing.

Justiciability has forced adequacy advocates to overcome two arguments. One is that judicial action violates the principle of separation of powers because school spending is a political question. The other is that the language of state constitutions is unclear and, therefore, provides no justification for regulating the elected branches’ policies. For political reasons rather than constitutional ones, the political question doctrine is likely to remain a troubling issue while constitutional language will not.

The Political Question Doctrine

Based on evidence from state courts, where its application is wildly uneven in remarkably similar cases, the political question doctrine does not have much force of its own. Courts deploy it or ignore it as they wish and use it only if they are predisposed not to enter into a controversy.

Justice William F. Brennan gave the standard formulation of the political question doctrine in Baker v. Carr, describing it as “a function of separation of powers.” His detailed definition could justify dismissing education reform litigation for many reasons, among them “a lack of judicially discoverable and manageable standards for resolving it … or the impossibility of deciding without an initial policy determination of a kind clearly for nonjudicial discretion.” For advocates of judicially imposed reform, “judicially manageable standards” has been a long-standing obstacle, as it requires that there actually be a solution. If courts do not think that they have a manageable solution, institutional self-interest can restrain them. In San Antonio Independent School District v. Rodriguez, a Supreme Court case on equity of school finance, Justice Lewis F. Powell in 1973 cited the lack of judicially manageable standards as a reason for leaving the issue to elected bodies.

The lack of manageable standards has been a continuing source of frustration for education reform litigants. The standard of equal spending ultimately proved unattractive to plaintiffs since it provided powerful incentives to simply reduce spending for everyone. Justiciability here faced both legal and political obstacles: equal spending failed to promise more money for the poverty populations of central cities, where per-pupil expenditures were often relatively high (see “Educational Jujitsu,” features, Fall 2002). Arguments for anything more than equal spending seemed devoid of precise content or guidance. The solution to the dilemma came courtesy of the standards movement. According to Rebell, the standards movement “provided the courts with practical tools for developing judicially manageable approaches for implementing effective remedies.” All that remained was marrying standards to the idea of adequacy. Adequacy tied to standards solves the legal and political problems of justiciability.

Defining Adequacy

In Rose v. Council for Better Education the Kentucky Supreme Court established an “operative definition of adequacy,” which other state courts have since “adopted,” according to Rebell. The court concluded that an adequate education requires among other things “sufficient oral and written communication skills” for functioning “in a complex and rapidly changing civilization,” “sufficient knowledge of economic, social and political systems to enable the student to make informed choices,” and a “sufficient grounding in the arts to enable each student to appreciate his or her cultural and historical heritage.” Since the Kentucky court did not mandate a specific set of reforms, this broad definition is more political rhetoric than a reasonable judicially manageable standard. Leaving aside the inherent ambiguity of terms such as “sufficient,” “informed,” and “grounding,” the court’s definition in fact assumes that in a complex and rapidly changing society the skills needed, and therefore constitutionally required, will change as well.

On close inspection it becomes clear that there is no evidence of inadequacy without evidence of inequity. Two prominent and recent adequacy cases—from New York (Campaign for Fiscal Equity v. New York) and Kansas (Montoy v. State)—show that when courts attempt to overcome the problem of justiciability either they will founder trying to establish what an adequate education actually is or they will retreat to the legally safe but politically dangerous standard of equity.

In CFE v. New York, Judge Leland DeGrasse ruled that an adequate education included the “foundational skills that students need to become productive citizens capable of civic engagement and sustaining competitive employment,” the “intellectual tools to evaluate complex issues, such as campaign finance reform, tax policy, and global warming,” the ability to “determine questions of fact concerning DNA evidence, statistical analyses, and convoluted financial fraud.” These requirements are frustratingly vague, a fact DeGrasse inadvertently demonstrated when arguing that New York City’s public schools were inadequate.

When marshaling evidence for inadequacy, DeGrasse looked at what he called the “inputs” and “outputs” of the system. The inputs were “the resources available in public schools” and the outputs were the “measure of student achievement.” Evidence for the inadequacy of the inputs was based solely on equity. For example, New York City teachers were found on a variety of levels to be inferior to their statewide counterparts. DeGrasse was also unable to present any independent standards of inadequacy when discussing outputs. New York City public schools have lower graduation rates and test scores than other New York schools. This is at best a demonstration of inequity.

Socioeconomic factors initially seem to offer a way out of this dilemma. DeGrasse offers a very grim picture of the socioeconomic condition of New York City public school students. They suffer from poverty, homelessness, poor health, teen pregnancy, and frequent change of residence. Such obstacles raise the question of whether the lower “outputs” of the school system are the result of inadequate “inputs.” DeGrasse seems to make the case that the quality of the New York City schools is not to blame. The state’s highest court, the Court of Appeals, apparently recognized this even as it approved DeGrasse’s ruling, stating, “Decisions about spending priorities are indeed the Legislature’s province, but we have a duty to determine whether the State is providing students with the opportunity for a sound basic education. While it may be that a dollar spent on improving ‘dysfunctional homes’ would go further than one spent on a decent education, we have no constitutional mandate to weigh these alternatives.”

In Montoy v. State, the Kansas Supreme Court blurred the line between equity and adequacy even more. The Kansas legislature allowed a variety of different taxes based on local circumstances such as high cost of living, low enrollment, and extraordinarily declining enrollment. But the state supreme court struck all of these down because of their “disequalizing effects.” Normally such accommodations would be allowed under rational basis scrutiny, but the court objected because they could possibly lead to unequal amounts of spending. The supreme court did state that “once the legislature has provided suitable funding for the state school system, there may be nothing in the constitution that prevents the legislature from allowing school districts to raise additional funds for enhancements to the constitutionally adequate education already provided.” However, the court gave no indication at what point “suitable funding” would be reached such that some school districts could spend more than others. For the time being, the court is demanding more spending alone to equalize expenditures across school districts.

The adequacy advocates driving the litigation have searched along with the courts for conceptual foundations. Rebell says that a “core constitutional concept” has emerged from recent adequacy lawsuits. This concept, he says, “emphasizes that an adequate education must (1) prepare students to be citizens and economic participants in a democratic society; (2) relate to contemporary, not archaic educational needs; (3) be pegged to a ‘more than minimal’ level; and (4) focus on opportunity rather than outcome.”

These components are hopelessly unclear. For instance, to explain the meaning of “to be citizens and economic participants in a democratic society” he says that “there is widespread agreement that an adequate system of education is one that ‘ensures that a child is equipped to participate in political affairs and compete with his or her peers in the labor market.’” As evidence of this agreement, he quotes the Vermont Supreme Court’s opinion in its largely equity-,  rather than adequacy-based decision, which held that the state constitution guarantees “preparation ‘to live in today’s global marketplace.’” The idea that education should “relate to contemporary, not archaic educational needs” means that as “the level of skills necessary to participate as a citizen and as a wage-earner in society rise, expectations for an adequate education will also necessarily rise.” Defining a generality with more generalities does not make a generality more precise. Thus, adequacy advocates turn to money. The courts ensure “the availability of essential resources.” As CFE v. New York shows, the easiest way to gauge “essential resources” is by comparison with other school districts.

Constitutional Language

The state constitutions’ education clauses also raise questions about the appropriateness of judicial intervention based on separation of powers. However, the language is unlikely to undermine the movement as the political question doctrine potentially could. The notion that the constitution requires an “adequate” education is politically popular. But that does not mean that the interpretation is proper.

State education clauses are characterized by generality and often by their delegation of authority to the legislature. Some clauses simply require free public schools. Others imply a standard of quality such as “thorough and efficient” or of “high quality.” The strongest give education a special status, calling it “fundamental” or “primary.” While scholars and activists have made much of these differences, constitutional language has had little apparent influence on state courts. Adequacy suits have failed in states with stronger language such as Maine and Illinois, but won in states with weaker language such as North Carolina and New York. The reason is the distinctions between weak and strong education clauses have been too finely drawn. It is not unfair to call all of them, as Clayton Gillette has, “inherently nebulous.” What for instance does it mean to say that education is a “primary” obligation of a state? How does one know when the state has not made it a “primary” obligation?

The obvious question is whether it is appropriate for the judiciary to find the standards that it imposes on legislatures in these generalities. In states that have rejected adequacy suits, the courts’ analyses have hinged on the inherent arbitrariness of finding a specific standard and the unconstitutionality of applying a static interpretation on clauses whose meaning must evolve. The Illinois Constitution, with one of the most demanding education clauses, says that the state must “provide for an efficient system of high quality public educational institutions and services.” But twice the court held that “[i]t would be a transparent conceit to suggest that whatever standards of quality courts might develop would actually be derived from the constitution in any meaningful sense.”

Since education clauses provide little textual substance, it is unsurprising that their analysis by courts is occasionally nothing more than a bald assertion obscured by fallacious reasoning. In Abbeville v. State from South Carolina, the state supreme court simply asserted that the education clause, in spite of its lack of qualitative language, must have a qualitative component. In CFE v. New York, another state with a spare education clause, Judge De Grasse without apology explained that in education litigation courts “are called on to give content to Education Clauses that are composed of terse generalities,” which in New York’s case is “The legislature shall provide for the maintenance and support of a system of free common schools, wherein all the children of this state may be educated.” From that clause, De Grasse determined that the New York City schools were unconstitutional in everything from library expenditures to arts courses. The judge had become completely unmoored from the text and was sailing in purely policy waters.

Policymaking in the Courts

Despite the assurances of adequacy advocates that courts now have the tools necessary for implementing effective reforms, there are reasons for skepticism. For one, there is a well-developed body of literature documenting the institutional difficulties that courts have in creating social change, beginning with Donald Horowitz’s pioneering book of 1977, The Courts and Social Policy. This literature grew up around the study of federal courts, and to the extent that state courts are beginning to behave like federal courts, much of it applies. Horowitz said that litigation is a poor vehicle for making policy because among other things the adversarial format produces unreliable information and artificially isolates issues that are connected in the real world. Examples of these problems can be found in adequacy litigation.

An example of judicial action with inadequate information is to be found in Kansas, where a willfully blinkered court chose to rely on one consultant’s study, by the firm of Augenblick & Myers (A&M), in ordering how much the legislature should appropriate. In Montoy v. State, the Kansas Supreme Court said it would be guided by the A&M study because 1) it was “competent evidence presented at trial”; 2) the legislature “maintained the overall authority to shape the contours of the study”; 3) it was “the only analysis resembling a cost study” before the court; and 4) the state board of education and department of education had concurred with the results. The implication of this reasoning—other than that legislatures must follow the recommendations of studies that they commission—is that the court was unwilling to seek as much information as possible. The court assumed the reliability of the study and impugned the motives of members of the legislature who disputed its findings. It repeatedly said that it must make its decision “based solely on the record before us,” an artificial but convenient standard peculiar to litigation.

A second institutional defect is that courts must isolate problems that are connected and need a comprehensive approach if they are to have any chance of being solved. Education is a broad and complicated area of public policy, which is intertwined with other broad and complicated areas of social policy. As courts look at the problems of education through the narrow lens of the legal process, their approach is inherently piecemeal.

A Radical Transformation Is Underway

If active and continuing judicial supervision of school spending were to be institutionalized, the result would be a radical—and unnecessary—revision of the American system for appropriating public funds. Judgments of courts in combination with a new industry of costing-out consultants would be substituted for the bargaining and mutual adjustment—that is, the politics—of state legislatures. Indeed, this new day has already dawned, according to a presentation that the financial consultant John Myers made to the National Association of State Budget Officers in the summer of 2005. “Historically, adequacy was determined politically using input measures and available resources,” he said. “Now adequacy is technically determined and output orientated.”

If money—and money alone—were all that is required to educate the nation’s children, and if courts alone could provide the money, then perhaps one would be willing to entertain, if only for a fleeting moment, this constitutional departure. But then one would recall that other public functions exist, such as health, transportation, and higher education, that make large and urgent claims on the budgets of state governments; that problems other than a lack of money afflict the schools, such as students who arrive unprepared for learning or life in a classroom; and that evidence for the efficacy of money per se is at best mixed. One might then be less willing to have the core institutions of democratic government cast aside.

The adequacy movement would like to secure a foundation in federal law for its claims. This might be done by importing, via amendments to the No Child Left Behind Act, some of the rights language produced by state courts. “We want to see the issue of equity on the national agenda,” Arthur E. Levine, then president of Teachers College of Columbia University, told an interviewer in 2005. Rebell has moved to Teachers College to direct its equity campaign. A West Coast branch of the movement has set up operations as the Chief Justice Earl Warren Institute on Race, Ethnicity and Diversity at the school of law at the University of California, Berkeley. One of its initial projects in 2004–05 was to convene an interdisciplinary working group called “Rethinking Rodriguez: Education as a Fundamental Right.” The aim was to inquire into what would be required to make education a fundamental right—“that is, a right belonging to all children, protected by an enforceable guarantee of ‘adequacy’ or ‘equality’ or both.”

The successes of the adequacy movement in state courts thus are to be seen as stepping stones to the broader arena of national legislation and litigation. If the adequacy-cum-equity advocates succeed—wedding centralization and judicialization in a regime of a federally guaranteed right to education and federally prescribed school spending—transformation of the traditionally local and democratic governance of schools in the United States, already far advanced, will be complete.

Josh Dunn is assistant professor, the University of Colorado–Colorado Springs. Martha Derthick is professor emeritus, the University of Virginia. The unabridged version of this essay is available in Martin R. West and Paul E. Peterson, eds., School Money Trials: The Legal Pursuit of Educational Adequacy, forthcoming from the Brookings Institution Press.

Vouchers in New York, Dayton, and D.C.
by WILLIAM J. HOWELL, PATRICK J. WOLF, PAUL E. PETERSON, & DAVID E. CAMPBELL

Vouchers in Charlotte
by JAY P. GREENE

Significant, but Not Decisive
by DAN GOLDHABER

ILLUSTRATION BY DAN VASCONCELLOS.

What is strange about the debate over teacher policy, however, is that it has taken place largely within the confines of the existing public school system. Hardly anyone talks about how the growing movement toward parental choice and competition, in the form of vouchers and charter schools, will affect the teaching profession. Only the teachers unions seem to hold a clear, if unspoken, opinion on this question. Though they never say so, the unions obviously think that school choice would harm teachers. Both the National Education Association (NEA) and the AFT stand resolutely opposed to vouchers, and their attitudes toward charter schools have ranged from tepid support to outright hostility. While these unions don’t craft their agendas solely with the interests of teachers in mind, they seldom—if ever—oppose policies that would benefit their members.

Yet economic theory suggests that school choice would change the teaching profession in ways that would fulfill many of the reform movement’s goals. In short, theory predicts that schools that faced stronger competition would favor teachers who raised the schools’ ability to attract students. These schools presumably would strive to attract and retain teachers who were especially talented or hard-working or who possessed rare skills. In turn, you would expect their tolerance for less-effective teachers to wane. You would expect, in fact, that teaching would be transformed into a true profession, where workers are rewarded not only on the basis of seniority but also on the basis of their skills and performance.

Making a Profession a Profession

In the current system, teachers unions tend to compress wages within a school district so that teachers with the same seniority and the same degree are likely to receive similar (if not identical) salaries. Even nonunionized districts tend to adopt salary scales that resemble union scales. So teachers who excel at their jobs or teach hard-to-staff subjects, such as math and science, are paid the same as if they were mediocre or could be replaced easily.

This is one reason why teaching, as it stands, is not as attractive to candidates with high aptitude, a strong work ethic, or math and science skills. The evidence suggests that such people are less likely to start teaching than their lower-aptitude peers (even when those being compared are all certified to teach) and less likely to remain in teaching. This situation has worsened over time, as other professions such as management, law, and medicine have opened their doors to women. Women with high aptitude or math and science skills have chosen such professions over teaching, perhaps because these professions do reward workers based on their skills and performance. By making teaching a more market-oriented profession, school choice might also make it a more attractive career to people who would thrive in such an environment.

Why might teachers welcome such a shift? In a word, professionalism. Both the NEA and the AFT support so-called professionalization agendas that aim to make the route into teaching look more like the routes into the legal and medical professions. Much as lawyers and doctors ensure quality within their professions through bar examinations and board certification, the AFT and the NEA call for higher certification standards for new teachers (but rarely for current teachers), apprenticeships, peer review, and rewards for teachers who earn additional credentials, such as National Board certification. The movement also includes proposals to give teachers more control over the profession through independent, state-level standard-setting boards and to give the National Council for the Accreditation of Teacher Education, the body that accredits education schools, more control over the number and origin of undergraduate degrees granted in education each year.

In most professions, however, credentials such as board certification merely maintain minimum standards. The size of a professional’s client list or how much a professional earns usually depends on how the market values her services, since professionals typically compete in a market setting as employees of a competitive firm, in self-employment, or in partnership. The teachers unions, however, want to use credentials to restrict the supply of teachers while opposing competition, making their range of policies more characteristic of a craft union than of a profession. Professionalization in this sense is a misnomer for the current movement. Yet the name and rhetoric of the movement suggest that many teachers do want teaching to become more professional. Economic theory suggests that school choice would make it so, in the sense that it would raise demand for highly talented and skilled teachers who receive rewards closely linked to their performance.

Just a Theory?

Without evidence, of course, this is just a theory. And without full-fledged school choice, it is a tough theory to test, though not impossible.

Comparisons between public and private school teachers already give us some insight into how choice might affect the teaching profession. After all, the revenues of private schools depend on their ability to attract tuition-paying students. They thus have an added incentive to hire teachers who will help them do so. What do they look for? In a comprehensive comparison of private and public school teachers, economists Dale Ballou of the University of Massachusetts Amherst and Michael Podgursky of the University of Missouri–Columbia found that, in making hiring decisions, private schools value high aptitude more than public schools do. They also found that salaries in private schools correspond to aptitude and scarce skills (such as math and science skills) more than they do in public schools.

Private schools, however, are not a perfect guide to what teachers will experience under a system of school choice. Parents who pay private-school tuition must also continue to pay taxes that support public schools. Private schools thus face financial constraints that a true school of choice would not. As a result, they typically pay teacher salaries that are about 60 percent of local public school salaries.

To get a broader picture of how choice affects teachers, I used data both from traditional forms of school choice (choice among public schools through choice of residence and choice among private schools) and from charter schools. My analytical strategy relied on a simple economic argument: If schools that face stronger competition prefer certain characteristics in their teachers, then they should hire more teachers who possess those characteristics and pay them a higher wage. In economic terms, they would show greater demand for teachers with those characteristics; they would be willing to pay a higher price in order to buy more of them.

My findings suggest that school choice would change the teaching profession in significant ways. Schools that face tougher competition have greater demand for teachers who attended well-regarded colleges, majored in subject areas (especially in math and science), and who put in more effort and show more independence. These schools are more likely to hire such teachers and to pay them higher wages than they would earn in schools that face less competition. In general, they also have less demand for certification and master’s degrees. They pay teachers who hold such credentials less than similarly educated teachers earn in schools that are less choice-driven.

Making Choices

Before I discuss the results of this study, let me briefly explain the measures of choice I used and where some of the data came from.

Within the present K–12 education system, parents and students can make three types of choices. First, they can choose their public schools by choosing where to live (this is called Tiebout choice, after the economist Charles Tiebout, who called attention to its importance). Some metropolitan areas, however, contain more school districts than others, allowing for more choices among public schools. Compare the cities of Miami and Boston, for instance. In Miami, one school district encompasses the entire metropolitan area. Parents have almost no choice among public schools. Boston, by contrast, has 70 districts within a 30-minute commute of downtown and even more in the surrounding metropolitan area. In an area with so many choices, a town’s reputation and real-estate values depend heavily on the quality of its local school district, giving residents added incentive to ensure that their schools compare favorably with those in other towns.

Various factors besides the degree of choice might affect a district’s demand for certain qualities in its teachers. For instance, a school district in an area with high employment in the technology field might show greater demand for teachers who possess skills in math and science. But it also might be a school district that happens to face strong competition. Without controlling for such a coincidence, any comparison would seem to show that competition affected the demand for math and science skills in this district, when, in fact, another factor contributed to the district’s demand as well. Likewise, teachers in an area where the population is more educated tend to have attended more selective colleges. You must adjust the data to take account of this, or they may give you the false impression that competition raises the demand for high-aptitude teachers. Other factors, such as household income, ethnic makeup, and union membership among teachers, can also affect a district’s demand for certain qualities. After controlling for all these factors, I compared teachers in areas where parents have more choices among public schools with teachers in areas where they have fewer.

Second, parents can choose to send their kids to private school. But again, some metropolitan areas have more low-tuition private-school options than others, mainly due to historical variations in religious populations. Private schools in areas with large, long-standing religious populations tend to have higher endowments and more established donor bases. This allows the schools to charge a tuition that is lower than the actual cost of educating a child, thereby making it easier for parents to choose a private school. In some metropolitan areas, up to 15 percent of the elementary school population is enrolled in private schools where the tuition is about two-thirds of the per-pupil expenditure. (Religiously affiliated schools typically charge $1,600 in tuition while spending about $2,300 per student.) Other metropolitan areas have less than 1 percent of their elementary school populations enrolled in such schools. They may have seats available in schools where the tuition is higher and there are no tuition subsidies, but this hinders a parent’s ability to choose a private school. After again controlling for factors other than competition that might affect a private school’s demand for certain teachers, I compared teachers in areas where parents have more choices among private schools with teachers in areas where they have fewer.

Finally, and most recently, parents can choose to send their kids to a charter school. I compared teachers in charter schools with those in private and public schools in the same regions. Here you must control for the fact that the existence of a charter school may reflect factors in an area that also determine the area’s demand for certain teacher characteristics. I also tested for composition effects, meaning that a charter school might enter an area and favor the same teacher characteristics that the public schools want. After the charter entered the area, you might see teachers with that characteristic slowly draining away from the public schools. This is unlikely, given that a charter school has little incentive to move into an area where schools are already doing what it plans to do, but I tested for it anyway. For the comparison among charter, public, and private school teachers, I assumed that charter and private schools face more competition than public schools, since a greater share of charter and private schools get funding only if they attract students.

For both public and private schools, I used data from the Schools and Staffing Survey (SASS), a comprehensive survey of public and private school teachers and administrators. I examined variables such as salary, years of teaching experience, whether teachers planned to stay in teaching for the next few years, the number of hours per week (on top of required hours) that teachers spent on activities related to their students’ academic progress, and the number of hours that they spent on their students’ extracurricular activities. I also looked at where they earned their baccalaureate degrees, whether they were math or science majors, the number of undergraduate and graduate courses they took in math and science, and whether they earned their degrees in a field of the arts and sciences (rather than in education).

The SASS does not cover charter schools, so I had to conduct a survey of my own. I distributed surveys to one administrator and two randomly selected teachers in every charter school that was open as of October 1998. The survey questions were taken from the SASS. Data from the survey will be described in a forthcoming monograph.

Results

Remember that my test for whether school choice raises demand for certain teacher characteristics is two-fold: 1) whether a school that faces stronger competition hires teachers with more of a certain characteristic; and 2) whether that characteristic earns a premium in an environment of greater school choice. Remember also that I performed two different comparisons: 1) a within-sector comparison of teachers in both the public and private sectors; and 2) an across-sector comparison of teachers in public, private, and charter schools.

The first question I asked was whether competition raises demand for teachers who possess high aptitude. This is important to know because research has shown that teachers’ aptitude, as measured by scores on standardized tests, significantly affects student achievement. The SASS, unfortunately, does not have a measure of the aptitude of individual teachers, since only some states require teachers to take proficiency tests. Even if they did, teachers would not have taken comparable tests anyway. The SASS does, however, list which college teachers attended. This is a good measure of the quality of their education and, to some extent, of their aptitude. Colleges take account of students’ aptitude, high-school grades, extracurricular activities, and character in making acceptance decisions, so the selectivity of a teacher’s college summarizes a complex set of information on aptitude, work ethic, leadership, creativity, and so on.

The most obvious way to use a teacher’s college as a proxy for aptitude would be to assign that teacher the college’s average SAT score. But most teachers come from colleges that range from nonselective to somewhat selective, where SAT scores are poorly defined because so many students do not take admissions tests. Their average SAT scores would not be an accurate proxy for an individual teacher’s aptitude. So, instead of SAT scores, I used the rating each college received in the 1982 Barron’s Profiles of American Colleges. I chose the 1982 version because it represented a middle ground: Older teachers would have graduated before 1982, younger teachers after. In any case, the rankings are fairly stable from year to year, so the choice of year doesn’t make much difference. I assigned the teachers from each school a number that corresponds to that school’s Barron’s rating, from 9 for Barron’s top rating, “most competitive,” to 1 for its lowest, “nonselective,” which is reserved for schools that will accept nearly anyone with a high school diploma or GED.

My first finding was not so surprising. Several scholars have shown that teachers are drawn mainly from the bottom half of the college-going population’s aptitude distribution. Likewise, most of the teachers in the SASS sample came from schools rated between 4 (“competitive plus”) and 1 (nonselective).

My findings also suggest, however, that enhanced choice and competition could help to remedy this situation and thereby bolster the skills of the nation’s teaching corps. I found that teachers who work in areas with higher Tiebout and private school choice attended more selective colleges. Teachers in areas of maximal Tiebout choice attended colleges that were ranked 0.4 levels higher than in areas of minimal Tiebout choice. Areas with a high degree of private school choice have teachers who attended colleges that were ranked 0.1 levels higher than areas with minimal private school choice.

Because most teachers attended schools that rated in the 1–4 range, I was also interested in whether competition raised demand for teachers who went to schools rated 4 (competitive plus) or higher. It appears to. For instance, about 20 percent of public school teachers went to such schools, compared with 36 percent of charter school teachers and 36 percent of private school teachers (see Figure 1). In public schools, teachers from such colleges earn 3 percent more than their peers. In charter schools, they are paid 6 percent more, suggesting that charter schools have greater demand for teachers who have graduated from colleges that rate at least “competitive plus.” Their need to attract students in order to receive any funding seems to drive their hiring of higher-aptitude teachers.

Similarly, a teacher in an area with many public school choices (maximal Tiebout choice) is 8 percent more likely than a teacher in an area of minimal Tiebout choice to have attended a college that rates at least “competitive plus.” Teachers who attended such colleges also earn 8 percent more than their peers. A high degree of private school choice boosts the likelihood that a teacher attended a college rated at least “competitive plus” by 5 percent. The degree of private school choice does not have a meaningful effect on the wage for this characteristic.

Skills in Short Supply

My second question was motivated by the evidence that schools face a shortage of teachers who have math and science skills (only 7 percent of teachers in the United States were math or science majors). Indeed, schools need teachers who have baccalaureate degrees in any field of the arts and sciences (as opposed to education) because they have subject knowledge that is useful in grades 7 through 12.

Again, my findings suggest that enhanced competition and choice may help to ameliorate these shortages. Stronger competitive pressures appear to raise the demand for teachers who have math and science skills. In an area of maximal Tiebout choice, a teacher is 15 percent more likely to have majored in math or science than a teacher in an area of minimal Tiebout choice. A teacher in an area with a high degree of private school choice is 10 percent more likely to have majored in math or science than a teacher in an area with minimal private school choice. Teachers who majored in math or science earn 16 percent more if they work in an area with maximal Tiebout choice than they would if they worked in an area with minimal Tiebout choice. In private schools, math or science majors earn 14 percent more if they work in an area with a high degree of private-school choice.

In charter schools, 10 percent of teachers are math and science majors, versus 8 percent of public school teachers and 7 percent of private school teachers. In the public sector, math and science majors earn about 4 percent more; in charters, 8 percent.

The results also suggest that schools facing more competition are better at retaining such teachers. In general, math and science teachers are more likely to leave teaching.They have 1.2 fewer years of experience and are 6 percent less likely to say that they plan to keep teaching than other teachers. But in an area of maximal Tiebout choice and a high degree of private school choice, math and science teachers have only .4 fewer years of experience and are no less likely to say that they plan to continue teaching. This may be a result of such teachers’ being paid more in districts that face more choice-based incentives.

Schools that face stronger competition also appear to favor teachers who majored in subject areas. Of teachers in charter schools, 56 percent majored in a field of the arts and sciences (as opposed to education), compared to 37 percent of public school teachers and 42 percent of private school teachers (see Figure 1).

Effort and Independence

I also wanted to know whether school choice increased demand for teachers who put in more effort and show more independence. I used data that measure the number of hours teachers spend on instructional tasks (such as tutoring) and on noninstructional tasks (such as directing a school play) beyond the hours required in their contracts. The measure of independence is a teacher’s assessment of the control that she exercises over her teaching (particularly teaching methods and the organization of material).

In general, schools that face more Tiebout choice demand more effort and independence from their teachers, while the effect of private school choice is unclear. Teachers in an area with maximal Tiebout choice spend two extra hours per week on instructional work and one extra hour per week on noninstructional work than teachers in an area of minimal Tiebout choice. A high degree of Tiebout choice also raises the wage paid for every extra instructional hour per week by .7 percent and for every extra noninstructional hour per week by .3 percent.

Charter school teachers work 13 extra instructional hours per week, versus 9 extra hours for public school teachers and 9 extra hours for private school teachers. The difference between the charter and public school teachers is highly statistically significant. In public schools, teachers are not compensated for the extra instructional hours they work; in charter schools, they are paid 5 percent more for an extra 13 hours of work. In public schools, teachers claim their level of control over their teaching is 4.8 on a 6-point scale, while charter and private school teachers rate their level of control at 5.6 on a 6-point scale.

The professionalization movement among teachers includes a sharp focus on certification and on the importance of holding a master’s degree, so I was interested in how choice affects schools’ demand for such credentials. It is important to note that many teachers-union contracts specify salary increases for master’s degrees and for being certified. So examining the data shows whether choice motivates schools to make such salary increases larger than they would be in the absence of choice.

Overall, about 44 percent of public school teachers, 41 percent of charter school teachers, and 28 percent of private school teachers have master’s degrees (see Figure 2). The public sector pays teachers who hold master’s degrees about 25 percent more; in charter schools, they are paid about 20 percent more. Furthermore, nearly all public school teachers are certified, while only 87 percent of charter school and 65 percent of private school teachers are. In private schools, teachers who hold certification are paid 1.4 percent less than uncertified teachers; charter schools give a similarly negative premium, but this finding is not statistically significant. This suggests that private schools may slightly prefer uncertified teachers and that charter schools probably have less demand for them than public schools do.

Another way to look at this question is whether teachers actually teach in the areas for which they are certified. Most states put few restrictions on teachers—once certified—taking on assignments that are relatively remote from the fields in which they were certified. About 97 percent of public school teachers claim to be certified in their teaching area, while only 83 percent of charter school and 54 percent of private school teachers do (see Figure 2). Public school teachers who teach in their areas of certification earn a substantial wage premium, 9 percent, compared with a premium that is not meaningfully different from zero for charter teachers and a 2 percent premium for private school teachers. All in all, it appears that public schools have a greater demand for degrees and certification than private or charter schools.

Broadly speaking, my findings suggest that enhanced competition and choice raise the demand for high aptitude, skills in math and science, subject-area expertise, effort, and perhaps independence among teachers. Choice also seems to lower schools’ demand for certification and master’s degrees. These findings further suggest that school choice has the potential to create a professional environment for teachers in which more motivated and skilled teachers earn higher pay for such qualities. For new teachers, this could quickly change the profession in areas where a growing number of choice schools offer a large share of the new teaching positions. It would take longer for veteran teachers to feel the pinch. They would notice a change only when their schools began to feel competitive pressures and, as a result, began to demand teachers whose characteristics attract parents. Some teachers, obviously, would dislike such changes. Less able or less motivated incumbent teachers might find themselves earning smaller salary increases than some of their peers. Such teachers might be more likely to leave the teaching profession. That would be a welcome reversal of the current pattern, where the most able teachers are also the most likely to exit early.

–Caroline M. Hoxby is an associate professor of economics at Harvard University and a visiting fellow at the Hoover Institution, Stanford University.

According to voucher advocates, parents are mainly concerned about school quality. In the absence of vouchers, only parents with enough money are able to seek out good schools by going private; but under a voucher system, they argue, with the cost of private education much reduced (or zero), many more parents would be able to—and would want to. This is particularly true for parents who are disadvantaged or trapped in failing public schools, for they have the most to gain from new opportunities. The upshot, advocates say, is that a voucher system would have very positive effects on society: it would get kids into better schools, give all schools incentives to perform, and promote social equity.

Photograph by Ralph Mercer.

Critics dispute all this. In their view, going private has little to do with school quality. The real motivations are largely social—and pernicious. Private schools have special appeal to people with money and education, who want their children separated from ordinary kids. They have special appeal to whites, who want to avoid blacks. And they have special appeal to the devoutly religious, who want schools of their own. So far, critics argue, the social downside has been limited because only 10 percent of American children go private. But if choice were expanded, pernicious motivations would be unleashed, and the education system would become more inequitable, more segregated, and more penetrated by religion.

These perspectives on parental choice couldn’t be more different or more fundamental to an assessment of vouchers. The question is: How do they seem to square with the facts? I have carried out research, based on a nationally representative survey of American parents, that provides some tentative answers. The details are reported in my book Schools, Vouchers, and the American Public (forthcoming in early 2001). Here I will briefly highlight some of the more basic themes and findings.

The Logic of Choice

Before turning to the data, let’s begin by thinking rather abstractly for the moment about the demand for private schooling. There is a logic to it, and this logic should tell us something about what to expect.

Under the current system, going private is costly, and parental choice is governed by a simple calculus. Parents will tend to go private if they can afford the tuition, and if the value they associate with going private—whether it derives from performance, religion, ideology, race, or other concerns—exceeds the costs. This calculus does not tell us what parents actually value. They may value performance, or they may not. They may value racial separation, or they may not. These are empirical issues that can be answered only by looking at the data. The simple fact that parents must weigh the benefits of going private against the financial costs, however, points to certain inferences that should guide our thinking.

Most obviously, private schools under the present system are more accessible to the financially well off, because they are better able to afford the tuition. The same applies for the well educated, both because education is highly correlated with income and because better educated parents tend to be more motivated by educational concerns. For these reasons, the current system should indeed promote a class bias in the types of parents going private, much as the critics of choice contend (and as supporters recognize). This bias would only be enhanced, of course, if some of these parents were elitist and sought separation for its own sake. But even if none were elitist, a class bias would be an inherent part of the system.

Now let’s consider what would happen if choice were vastly expanded, and parents were allowed—by means of vouchers, say—to send their children to private schools at no cost. The most immediate implication is that even the poorest parents—many of whom, by no coincidence, are trapped in our nation’s worst schools—would now find it within their means to pursue private schooling for their kids. Unless “other factors” (discrimination, for instance, or lack of information) prevented them from following through on their demand, then, many more low-income parents would probably go private than currently do, and the income biases associated with the current system would be reduced, perhaps drastically.

Biases due to education (controlling for income) might remain, because the better educated parents at all income levels would continue to be more motivated to seek out new opportunities. But in the aggregate, with income and education so highly correlated, and with information about private options increasingly available over time, many more people with low educational attainment would tend to find their way into the private sector—lowering the average level of parent education in that sector and thus leading to moderation on this score as well.

Here too we cannot say what parents might value about going private. But we can say that, whatever they value, the parents who currently go private are likely to be somewhat more “extreme” on these values than other parents are (because they care about them enough to pay for them)—and that, when the costs of choice are reduced, more people with these same values will be able to go private. It is reasonable to suggest, then, that the public parents who indicate an interest in going private should turn out to be similar on value grounds to the parents who already go private. They should just be more moderate.

In sum, if we think in analytical terms about the demand side of school choice, there are logical reasons for having certain expectations about what would happen if choice were expanded. Under the current system, in which choice is costly, private school choice can be expected to produce social biases that mirror some of the concerns of voucher critics. An expansion of choice, however, should not make these biases worse, as critics tend to argue. Rather, it is more reasonable to expect that just the opposite will happen, and that the biases of the existing system will actually be moderated, perhaps substantially.

The Data

The analysis is based on a 1995 survey designed to explore the voucher issue. The sample consists of 4,700 randomly selected adults from across the United States, including oversamples of 1,200 parents and 1,000 inner-city parents. With statistical weighting, the sample as a whole yields a representative cross-section of the population. The survey is supplemented by a separately collected data set on the demographics and academic performance of the school districts in which each respondent lives.

With these data, the desire of parents to go private (or not) can be explored by reference to a range of possible influences. Among the more notable are the following (the actual analysis includes many more than are listed here):

Background: income, education, religion, race, and political party.

Context: how advantaged the school district is (as measured by an index of test scores and socioeconomic characteristics) and whether the respondent chose where to live based on the schools.

Attitudes: support for diversity (racial integration), a perception of inequity (that the public schools provide a lower quality education for low-income and minority kids), support for voluntary prayer in the schools, support for greater parent influence, desire for smaller schools, belief in what I call the “public school ideology” (which measures a normative attachment to public schooling and its ideals), a belief in markets (that choice and competition are likely to make schools more effective), and a concern that moral values are poorly taught in the public schools.

Performance: an index of four items. One asks respondents whether the public schools are doing fine or are in need of major or minor change. Another asks them to rate the local public schools on a scale from 1 to 10. Another asks them whether their community is proud of its public schools. And still another asks them to compare the academic performance of the local public schools to private schools.

Who Goes Private under the Current System, and Why?

Now let’s take a first look at the evidence. Even voucher advocates would agree that, because private school choice is costly under the current system, parents who go private are likely to be more socially advantaged than parents who remain in the public schools. A simple descriptive comparison of parents in the two sectors documents as much. On average, private parents have higher incomes and more education than public parents, and they are more likely to be white. They also display the religious and partisan characteristics commonly associated with private schooling: they are more likely to be Catholic, born-again Christian, and Republican.

The existence of social biases, however, does not necessarily mean that the rest of the critics’ indictment, about pernicious motivations, is supported by the evidence. In fact, a multivariate (probit) analysis, incorporating all the background, context, attitude, and performance variables outlined earlier, suggests that it isn’t. Two findings stand out.

• Attitudes toward race (diversity) appear to have little to do with why parents go private: there is no indication that whites go private in order to flee blacks and other minorities.

• Of all the influences on parental choice, by far the most powerful is school performance: The less satisfied parents are with the performance of the public schools, the more likely they are to go private. The notion that private parents are really motivated by social concerns, and that performance matters little to them, misses the mark entirely.

This analysis also reveals that, aside from attitudes toward race, all the other attitudes in the model—inequity, public school ideology, prayer, moral values, school size, and markets (parent influence could not be included in this part of the analysis, for measurement reasons)—proved relevant to why these parents currently go private. The question is: do these same sorts of values and beliefs seem to be relevant for public school parents who, given the choice, would be interested in switching to private schools? We’ll soon find out.

Do Public Parents Want to Go Private?

Let’s focus our attention now on public school parents and see what we can learn about the reasons some of them are interested in going private.

The first step is to get a sense of how many public parents would like to go private if given a chance. What we are getting at here is a kind of hidden demand. For it is clear that parents who currently go private are doing so because they want to—but it is not clear that parents who currently send their kids to public school are doing what they want. Many might prefer to go private, but be unable to do so for cost reasons.

What is the extent of this hidden interest in private schools? The survey asks public parents the following question: “If you could afford it, would you be interested in sending your children to a private or parochial school?” The results are striking. They show that most public parents, 52 percent, would be interested in going private if money were not a problem, compared with 43 percent who say they would stay in the public sector. This is consistent with a 1999 survey by Public Agenda, which asked public parents a similar question and found that 57 percent were interested in going private.

The desire to go private is even stronger among low-income inner-city parents. In this group, 67 percent said they would be interested in leaving the public system. This is an early indication that, as advocates claim, choice has special appeal to the disadvantaged—and is not a policy whose support is grounded in elitism.

The notion that there is widespread interest among parents in going private is often disputed by critics. They cite figures that seem to show the opposite. A well-publicized report by the Carnegie Foundation for the Advancement of Teaching, for example, asked public parents the following question: “Is there some other school to which you would like to send your child? This could be public or private, inside or outside of your district, with your child’s grade level.” A full 70 percent said they would not switch, and just 19 percent indicated they would switch to a private school. Phi Delta Kappa (PDK) has included the same sort of question and gotten similar answers in its polls.

Properly interpreted, there is no conflict between these “opposing” sets of results. The Carnegie and PDK questions implicitly ask respondents if they know of some other school they regard as preferable to their current school. But most public parents have no incentive to be well informed about specific private schools (or even other public schools), so it is not surprising that they can’t point to specific schools where they’d like to send their kids. When asked whether they are interested in going private, however, most parents make it clear that they are. Both measures are useful and valid. They are just measuring different things.

Why Do Public Parents Want to Go Private?

Now let’s explore the desire to go private in greater depth. The question is: what would happen if choice were no longer costly, and all parents were given the option of sending their kids to private schools? Which parents would be most likely to go private, and why? Here are the highlights of a multivariate (probit) analysis that takes account of all the factors discussed earlier.

First, as the descriptive results for inner-city parents tend to suggest, an interest in private schools is especially high among low-income and minority (black and Hispanic) parents generally. When choice is freely available and income no longer a constraint, private schools have disproportionate appeal to those who are less well off, and whose need for new opportunities is clearly much greater. This association of choice with the disadvantaged is reinforced by the results for district context: it is the public parents in disadvantaged districts who are the most interested in going private.

Still, the more traditional influences associated with the current system continue to be relevant when choice is expanded. The public parents who want to go private tend to be better educated, Catholic, born-again, and Republican—suggesting that many of the same sorts of values and beliefs are at work.

The education result, it is important to note, reinforces a basic theme from the broader research literature: the parents who want to go private tend to be low-income and minority, but also (controlling for income and race) better educated. Choice advocates laud the equity-promoting effects of expanded choice and point to polls and programs showing that low-income families have a strong interest in going private. Yet opponents point out that, even within programs restricted to the poor, it is the better educated, more motivated poor who take greatest advantage of choice. And this, critics argue—not without reason—may give rise to certain inequities. These results provide support for both sets of claims.

Now let’s consider what values and beliefs seem to affect the desire to go private among public parents. Several findings deserve emphasis:

• Racial attitudes do not appear to play a significant role. This suggests that the critics’ claims about the pernicious effects of race, while perhaps justified decades ago, are probably wide of the mark today.

• Aside from race, all of the attitudes in the model—regarding inequity, public school ideology, prayer, moral values, parent influence, school size, and markets—appear to have an influence, and in the direction choice advocates would expect. Public parents appear to be influenced by the same basic values and beliefs that are important to current private parents. The consistent relevance of these concerns suggests that there is a common structure to the way parents think about their choice of schools.

• Of all the attitudes in the model, attitudes toward inequity stand out as the most salient. This dovetails nicely with our earlier results, which associate choice with the plight of the disadvantaged. Overall, as perceptions of inequity rise from low to high, the probability that a public parent will be interested in going private increases by 17 percent. If we break respondents down by income, moreover, we find that concerns about inequity are far more important to parents who are disadvantaged than they are to other parents. Among low-income parents, a growing sense of inequity makes them 26 percent more likely to be interested in private schools, as compared with shifts of 12 percent for middle-income parents and 11 percent for upper-income parents. The equity issue, then, seems to matter a great deal to disadvantaged parents, and they appear to connect it to private-school choice in a way that is entirely consistent with the argument voucher advocates have been making for the past decade: that choice is a way of promoting social equity.

• For parents as a whole, public school ideology is almost as important as equity concerns in shaping the desire to go private. Parents who score high on public school ideology are 13 percent less likely to be interested in going private than parents who score low. This underlines the pervasive role that normative commitments play in wedding parents to the public system and making them resistant to arguments for vouchers.

• Performance is by far the most powerful influence on the desire to go private. When satisfaction with public school performance drops from high to low, the probability that a public parent is interested in going private increases by 37 percent—which dwarfs the effects of all other variables. Performance looms as the number one consideration for parents. The familiar arguments to the contrary appear to be quite wrong.

Low-Income, Inner-City Parents

Now let’s take a closer look at inner-city parents. When we restrict the analysis to this population, one finding stands in sharp contrast to those we uncovered for parents generally. Among inner-city parents, it appears that race does matter, and in just the way critics have argued: White parents who are opposed to diversity are especially interested in going private. The obvious interpretation is that, because race is a more salient issue in the inner city than elsewhere, many whites see private schools as a way to avoid integration with minorities.

This is the first evidence (within our own study) that separatism and possibly even bigotry may be motivating some of the parental interest in private schools. And it makes sense that such effects would show up, if they show up anywhere, for low-income whites in the inner city—for these are the whites who are most directly affected by policies of diversity. Whites in the suburbs, and whites with money, are much more removed from the reality of integration.

There is a more benign interpretation, however. It may be that our diversity variable has little to say about racism per se, but is really measuring the extent to which respondents value diversity. People who score low are not necessarily racists. They just don’t value diversity as much as people who score high. Thus the model’s results may simply be telling us that whites who support diversity are especially inclined to stay in the public sector. Whites who lack that positive motivation are less wedded to the public schools and more open to private options, but this does not mean they are racists.

We should be careful, then, about jumping to conclusions. Nonetheless, it appears that race is relevant to the way inner-city whites approach going private, and this raises a red flag that choice advocates and program designers need to be concerned about. On this count, the critics may be right.

Aside from this finding about race, two other findings from this analysis of inner-city parents need to be underlined. The first is that their desire to go private is heavily influenced by their sense that the current system is inequitable. Indeed, race aside, inequity is far more powerful than any other attitude in the model—and indicates, once again, that disadvantaged parents make a strong connection between equity and school choice.

The second finding concerns performance. The critics of choice have long argued that low-income parents from the inner city are particularly unlikely to be guided by performance criteria. They construct an image of parents who need the help of school administrators and government agencies, and, if left to their own devices, would make ill-informed decisions that are not motivated by school quality. The empirical results, however, again show that performance is the most powerful factor in the entire analysis. When satisfaction with school performance drops from high to low, the probability that a low-income parent will express interest in going private increases by 30 percent, which is quite a large shift indeed. In this key respect, they look a lot like all the other parents. They are primarily interested in getting good schools for their children.

The New Public and Private Sectors

What would the public and private sectors look like under a system of expanded choice? Our analysis gives us a basis for drawing some provocative inferences.

Let’s begin by recognizing that, in reality, only some of the “swing” parents—the public parents interested in going private—would actually make the switch. The fact is, of course, that only some would actually want to: for they have merely indicated an interest in going private, not that they would actually do so, and there may be all sorts of reasons they would eventually stay put. Moreover, the private sector would have to expand tremendously to absorb them all, and it could not immediately do this. For the short run, demand would exceed supply, leaving many swing parents in the public sector.

In the analysis that follows, I assume that the most interested swing parents are the ones who make the switch. These I identify as the parents who, in the prior probit analysis, are estimated to be at the high end—the upper half—of the interest scale. Given this scenario, half of the swing parents switch from public to private, half remain in the public sector—and the public sector thereby retains about 75 percent of its families overall. Empirically, this seems very plausible.

Suppose now that the new private system consists of the existing private parents plus the swing parents most interested in going private, while the new public system consists of the swing parents who are less interested plus the remaining public parents. How do the new systems compare to the original ones?

Even when just half of the swing parents go private, an expansion of choice dramatically transforms the private sector along almost every social dimension. Compared to existing private parents, the new recruits are substantially lower in income and education (see Figure 1), more likely to come from disadvantaged districts, and more likely to be black or Hispanic. When these recruits become part of the new private sector, the usual social biases associated with private schooling are vastly reduced. Indeed, minorities now make up 33 percent of the transformed private sector (see Figure 2). As these changes almost dictate, there is also a shift in political partisanship: for with the socially disadvantaged comes a big influx of Democrats, and the original Republican bias is reduced considerably.

There is one social bias, however, that is reduced only slightly: religion. The parents making up the new private sector are almost as likely to be Catholic as current private parents are, and they are just as likely to be born-again. Even with the influx of disadvantaged parents, therefore, the private sector retains much of its religious character.

This same expansion of choice does not lead to major changes in the public sector. It remains pretty much as before, except that it is somewhat higher in income, contains proportionately fewer minorities, and retains more parents who are from better districts. Again, this happens because many of the disadvantaged choose to leave, and the parents who stay tend to be rather advantaged. But overall, the public sector is not affected by the expansion of choice to nearly the extent the private sector is—and the reason is pretty obvious. The current private sector is small and relatively select (due to the tuition requirement), and it is transformed by the influx of new recruits. The public sector is already large and heterogeneous, and it remains so even after many parents leave. It is affected at the margins.

Given the changes occurring in both sectors, how does an expansion of choice appear to affect the social gap between public and private? The transformation of the private sector does not succeed in eliminating all vestiges of the original gap. The new private sector is still somewhat higher in education and income than the public sector, and more Catholic and Republican. This is due to the new recruits’ (many of them disadvantaged) having been averaged in with current private parents (many of them fairly advantaged), and there are not enough of the former—under my assumption—to outweigh the latter.

Nonetheless, the changes are considerable. The effect of choice is to reduce the social differences between public and private—and thus to promote moderation—on every one of the dimensions we’ve considered. Moderation is weakest for religion, testifying to the tenacity of religion in the private-school equation. But the degree of moderation is quite pronounced for income and education. The income gap in particular is dramatically reduced, leaving only slight differences between public and private. On other social grounds, moreover, the expansion of choice goes further than this and actually reverses the traditional association of private schooling with social advantage. Most important, parents in the new private sector are actually more likely to be black or Hispanic than public sector parents are.

These results, of course, are not chiseled in stone. With somewhat different assumptions about how many parents switch from public to private and exactly who they are, the details of the analysis would be somewhat different. It is reasonable to suggest, however, that its basic thrust would remain essentially the same: that choice tends to break down existing biases of social advantage.

Conclusion

For the most part, the evidence from this study tends to support the claims of voucher advocates and to contradict those of critics. The appeal of private schools is especially strong among parents who are low in income, minority, and live in low-performing districts: precisely the parents who are the most disadvantaged under the current system. They would be the ones who disproportionately take advantage of an expansion of choice in education—and their shift from public to private should tend to produce a very substantial measure of social moderation, rather than the worsening of social biases that critics say would occur.

Critics also argue that performance has little to do with why parents find private schools attractive, that the real reasons are rooted in elitism, racial separation, and religion. It is true that religious and moral values play important roles in the current system and would continue to do so under a system of expanded choice. But the evidence suggests that school performance is the single most important factor in the choice to go private—and that elitism and racial separation have little to do with it.

Some of the concerns critics raise, however, do find limited support in the data. One is that racial motives may play a role among low-income whites in the inner city. The other is that, even among the disadvantaged, those with higher levels of education are more likely to go private (although I should emphasize that, because disadvantaged people are so interested in going private and because they are poorly educated as a group, the aggregate effect of choice is to lower the average education levels of private-sector parents, not to raise them). Both of these results point to possible problems that choice advocates and program designers need to be aware of in pursuing voucher plans that are truly equitable.

We also need to recognize that there is more to the critics’ argument than our data can address. The focus here has been on the demand for private schools, and thus on what parents want. But parents who want to go private might not be able to do so in their everyday lives for reasons that choice advocates too often dismiss. In the real world, even if everyone had the right to choose their schools, parents who are educated and financially well off are likely to be more motivated than other parents, to have better information about their alternatives, to have more resources at their disposal for getting their way, to have better social connections and more attractive opportunities, and to have children who are easier and less costly to teach. On the supply side, moreover, private schools may find these advantaged parents (and their children) desirable, and may discriminate against poor and minority families.

These are legitimate concerns. Critics are right to emphasize them, and choice advocates need to take them into account as they think about the proper design of choice systems. With the right designs, these problems may be mitigated and parental demand more freely pursued. In any event, this is an analysis of the demand side of the equation only, and what it has to tell us about the social effects of choice should be understood to hold as long as “other things are equal.” It tells just part of what must ultimately be a much larger story.

Nonetheless, the part that it tells is quite fundamental—and quite positive for the choice movement, which clearly has much to build on in attracting parents to its cause.

—Terry M. Moe is a professor of political science at Stanford University and a senior fellow at the Hoover Institution.

The awarding of scholarships by lottery created a rare opportunity in educational research: a field experiment in which students were assigned randomly to both public and private schools, thus allowing me to test the effects of receiving a voucher and, more generally, to compare the performance of public and private schools. The study used both standardized test scores and surveys of parents and students to evaluate the effect of the scholarship program on both academic performance and student and parental satisfaction.

Data

Only students enrolled in grades 2 through 8 were tested for this study. These students fell into three categories: 1) those who won a scholarship and used it to enroll in a private school (“choice students”); 2) those who won a scholarship but either moved out of the area or elected to remain in a public school for a variety of reasons (“noncomplying students”); and 3) those who didn’t win a scholarship (“control students”).

Overall, 388 students used a scholarship to enroll in private school; 413 students won the lottery but did not enroll in private school; and 342 students didn’t win a scholarship. Near the end of the 1999-2000 academic year, all of these students and their parents were invited to attend testing sessions where parents completed surveys while students took the Iowa Test of Basic Skills (ITBS) survey version. Students in grades 4 through 8 also completed a survey. Of the 1,143 students who were sent invitations to attend a testing session, 452, or 40 percent, participated in the study (within the three groups, 53 percent of the choice students, 20 percent of the noncomplying students, and 49 percent of the control students participated in the study).

Most of the private schools at which students used the scholarships operate with less than half as much per-pupil spending as the public schools.

Even as favorable a research design as a lottery is susceptible to the problems of nonparticipation and noncompliance. If students with similar demographic characteristics drop out of the study in large enough numbers, they can upset the balance between the experimental and control groups, rendering them less comparable. But my data show that the incomes of the participating and nonparticipating families were roughly equal for both the lottery winners and losers, as well as for the choice, control, and noncomplying students. In other words, while those who participated in my study differed somewhat from those who did not, the differences don’t appear to have biased the comparability of the groups.

My primary interest lies in identifying the effect of using a scholarship to attend private school, not the effect of a student’s being offered a scholarship but not using it. I therefore want to compare the choice students, the students who used a scholarship to attend private school, with the control and noncomplying students, the two groups who entered the lottery but ultimately stayed in public schools. Overall, the choice students and the comparison groups were quite similar in their demographic characteristics, though clearly not identical. The test-score data were adjusted statistically to account for any observed differences between the two groups, such as level of family income-an important predictor of academic performance-that might have biased the results. In calculating the results, I controlled for a host of background characteristics, including the mother’s educational level, her race, the student’s family income, whether a student lived in a two-parent household, and the student’s sex.

Concern about the unobserved differences between families who send their children to public and private schools has always limited scholars’ ability to draw conclusions from evaluations of public- and private-school performance. Even after adjusting for observed demographic differences, researchers always wondered whether unobserved differences that were not being accounted for, such as parental motivation or the intellectual richness of home life, played a larger role than the schools themselves in causing differences in academic performance between public and private schools.

In this case, however, the application process and lottery have produced comparison groups that are already quite similar on observed as well as (in all likelihood) unobserved characteristics. All families were motivated enough to complete an application for a scholarship. Only low-income families were eligible for a scholarship. A lottery was used to select which students would be offered scholarships, creating, as the statistical analysis has confirmed, two groups that were nearly identical. While noncompliance and nonparticipation have created differences between the two groups, they are similar enough that adjusting for observed characteristics is likely to produce highly reliable results.

Results

After one year, the results show that students who used a scholarship to attend a private school scored 5.9 percentile points higher on the math section of the ITBS than comparable students who remained in public schools. Choice students scored 6.5 percentile points higher than their public school counterparts in reading after one year (see Figure 1).

Using a statistical technique known as instrumental analysis to adjust for the potential bias of noncompliance yields results that remain strong and positive. The results of this analysis show that, after only one year’s time, attending a private-school improved student performance on standardized tests in math and reading by between 5.4 and 7.7 percentile points.

On average, a scholarship raised students from the 30th percentile to the 37th percentile. This is a fairly large gain approximately 0.25 standard deviation in math and reading. To put this gain in perspective, the difference nationwide between minority and white students is approximately 1.0 standard deviation. The benefits of the Charlotte CSF program are roughly one-quarter as large at the end of only one year.

Another important measure of the scholarship program’s performance is parental opinion. While the desire to affirm their decision may distort parents’ judgment, their intimate knowledge of (and interest in) their children’s well-being puts them in a good position to assess the benefit of a particular program. Our survey asked parents to assign their child’s school a letter grade, A through F. Nearly twice as many choice parents gave their child’s school an A (53 percent) as did public-school parents (26 percent). Choice parents were also far more likely to report being “very satisfied” with virtually all aspects of their children’s school: its safety, teacher quality, class size, clarity of school goals, teaching moral values, academic quality, teachers’ respect for students, and so on (see Figure 2).

Surveys of students in grades 4 through 8 found much higher levels of satisfaction among choice students than among public-school students. Roughly 40 percent of choice students gave their private school an A, compared with 32 percent of public- school students. Responding to a question asking how they feel about going to school each day, 24 percent of the public-school students said they didn’t want to go, compared with 9 percent of private- school students. And 24 percent of nonscholarship students said they didn’t feel safe at school, compared with 9 percent of choice students (see Figure 2).

Parental reports confirmed their children’s perceptions about safety at school. More than a third of public-school parents reported problems with fighting in school (36 percent), compared with 16 percent of choice parents. One-quarter of public-school parents reported problems with racial conflict, compared with 12 percent of choice parents. Of public-school parents, 22 percent reported problems with guns or weapons at their children’s elementary schools, compared with 11 percent of choice parents. Keep in mind that none of these children is beyond 8th grade.

School Facilities and Services

Rising test scores and high levels of parental and student satisfaction indicate more-luxurious private schools with better resources, right? Far from it. Most of the private schools at which students used the CSF scholarships operate with less than half as much per-pupil spending as the public schools. Tuition at most of the private schools is less than $3,000. Additional fundraising brings no more than a few hundred dollars per student.

The private schools actually offer sparser facilities and fewer services than the public schools. For example, only 70 percent of choice parents reported their school’s having a library, compared with 90 percent of public-school parents. Only 63 percent of choice parents said their school had a gym, compared with 91 percent of public-school parents. In terms of services, only 18 percent of choice parents said that a program for students learning to speak English was made available to them, compared with 50 percent of public-school parents. Only 49 percent of choice parents reported a program for learning disabilities being available, compared with 71 percent of public-school parents. Only 51 percent of choice parents reported that their school had a program for gifted students, compared with 72 percent of public-school parents. Choice parents were also less likely to report the existence of a counselor, a nurse, a music program, an art program, or prepared lunches at their schools.

Some amenities were equally or more available at the private schools. Private and public schools were equally likely to have a computer lab. And the private schools were equally likely to offer individual tutors and more likely to offer after-school programs. When parents report that they are more satisfied with the private-school facilities, they clearly must be focusing on the features they deem most important. Private schools appear to have far fewer resources, but they concentrate those resources on providing the facilities and services that parents value most.

Making Money Matter

Why do parents like these poorly funded, amenity-starved schools so much? What might account for the improvement in their children’s academic performance? While this study was not designed to address these questions fully, I can speculate based on the data I collected. Some of the most important differences between the private and the public schools pertain to the quality and motivation of their teachers. Parents gave high marks to the quality of instruction at the private schools. So, too, did the students. Choice students are almost twice as likely to report that their teachers are “interested in students” as are public-school students. Choice students are also significantly more likely to report that their teachers listen to them, that teachers are fair, and that students get along with teachers.
Despite having less money for salaries and benefits, private schools appear to be better able to recruit quality teachers and to dismiss poor ones. They may attract higher quality teachers because they can offer positive working conditions, an organization with a clear sense of mission, and greater autonomy in the classroom. The ability of public schools to offer such perks is hampered by layers of bureaucratic regulations and management, an unfortunate by-product of political governance. Teacher union and school district rules also make the removal of poor teachers much more difficult in public than in private schools.

The private and public schools in the study also differ in the size of their student populations and average class size. The median choice student is enrolled in a school that has between 151 and 300 students. The median public-school student attends a school with between 451 and 600 students. The median choice student sits in a class with between 11 and 15 students, while the median public-school student is in a class with between 21 and 25 students. Educational researchers are increasingly recognizing that smaller districts, schools, and perhaps classes seem to improve student performance and satisfaction. Smallness may encourage the development of a sense of community and common purpose, possible keys to school success. And smallness obviates the need for rigid rules that restrict the autonomy of principals and teachers.

Some critics of school choice have suggested that small classes in private schools “explain” the achievement benefits of private-school scholarships and voucher programs. If public schools had the additional resources necessary to reduce class size, so the thinking goes, they too would raise student performance. This, of course, begs the question: Why are private schools, despite having far fewer resources, able to provide significantly smaller classes than public schools? And what assurance is there that additional funds for public schools will lead to reduced class sizes-and not to higher salaries or more non-teaching staff?

Nevertheless, class size, when factored into my analysis, was not significantly related to student achievement. In other words, class size did not “explain” the achievement benefits of receiving a scholarship to attend private school in Charlotte.

There was little evidence to suggest that private schools were creaming the best students or dumping the worst.

Creaming and Dumping

Another oft-heard excuse for the success of private schools is that they are able to select their students by accepting the finest students (or creaming) and dumping the undesirable ones.
In my sample, there was little evidence suggesting that private schools were creaming the best students or dumping the worst. First, hardly any private schools asked applicants to take an admissions test. Of those families who were unable to place their children in the schools of their choice, more than three-fifths cited as the main obstacle the financial constraint of having to bridge the gap between the $1,700 scholarship and the full tuition. Parental reports indicate that an admissions test prevented only two of the lottery winners from attending their chosen private school.

Second, there is no evidence that private schools expelled undesirable students or asked them not to return. Any choice parents whose children switched schools during the academic year were asked why they left their original private schools. None reported that they switched schools because their children were expelled. Of those parents who were in doubt as to whether they would return to the same school next year, none reported that their children were asked not to return. In short, there is virtually no evidence that the choice schools academically screened their students for admission or expelled-or “counseled out”-students they found undesirable.

Parents were also asked whether their children had any physical handicaps, learning disabilities, or difficulty learning to speak English. Very few reported physical handicaps-only 3 percent of choice parents and 2 percent of public-school parents. Similarly low percentages of choice and public-school parents reported that English was not their children’s native language. However, choice parents reported fewer children with learning disabilities (4 percent) than public-school parents (13 percent) reported. Given their lower levels of funding, fewer private schools offer special programs for learning disabilities. The disparity also may be explained by differing incentives to label children “learning disabled.” Public schools receive additional resources for students labeled learning disabled, and they may be able to exempt learning disabled students from accountability testing.

The higher share of learning disabled children in public schools also may be evidence of mere parental choice. Parents of children with special needs are more likely to choose schools that have the funds to offer special programs. A fair test of whether private schools are avoiding learning disabled students would compare the rates of learning disabilities when private schools are given the same additional resources to serve those children that public schools receive. Otherwise, the evidence on learning disability rates is ambiguous.

In general, students who used their scholarships to enroll in a private school were more likely to be minority and considerably less advantaged than the typical Charlotte student. Three-quarters of the choice students were African-American, compared with a little more than a third of all students in the Charlotte-Mecklenburg school district. As of 1990, the average family income in Charlotte was nearly $34,000, almost $10,000 more than the average family income of choice students ten years later. In 1999, 32 percent of choice families reported receiving some kind of public assistance, such as food stamps or welfare; only 5 percent of Charlotte households were on public assistance in 1990. Even after their first year in private school, choice students were still scoring well below the national average on standardized tests (although they were scoring significantly better than they would have had they not received a scholarship). It is clear that private schools are accepting the challenge of educating these disadvantaged students, not creaming off the best or dumping the worst.

The Next Level

It is important to understand that the privately funded scholarship program in Charlotte departs from a model, publicly funded, school choice program in a number of ways. First, the $1,700 scholarship always required a significant supplement from the family to meet the tuition expense. A voucher that is set at the level of per-pupil spending in the local school district is likely to cover the full cost of tuition at most private schools. This may alter the results we would expect to see from gaining access to private schools. The infusion of cash that a publicly funded voucher would provide to private schools might make them even more effective. But easing the financial burdens that families face might attract more lower-income and less-motivated parents who might act as a drag on test-score improvements.

Second, privately funded scholarships place little or no regulation on the activities of private schools. By contrast, publicly funded vouchers are likely to carry a variety of curbs that might distract private schools from their core mission and tie up resources that would otherwise have gone to instruction. In the end, however, the regulations might also improve the program by ensuring equal access and by requiring that schools provide accurate information on student performance and faculty qualifications.

Third, there was enough spare capacity in existing private schools to accommodate the small number of CSF scholarship winners. A larger, publicly funded school choice program would require the creation of new private schools. These schools might be better or worse than existing private schools.

There is no way to address these issues fully without expanding the number and scale of publicly funded voucher programs. There is now a body of evidence on the effects of school choice: the positive findings from Charlotte comport with the positive results of privately funded programs in New York; Washington, D.C.; and Dayton, as well as pilot voucher programs in Milwaukee and Cleveland. It is not yet clear whether these results will translate to a broader scale. But the existing evidence is encouraging enough to justify trying.

-Jay P. Greene is a senior fellow at the Manhattan Institute for Policy Research. To view his study in its entirety, log on to www.edmattersmore.org.

The American education system, uniquely decentralized among industrial nations, has been continually roiled by tides of local experimentation, especially during the past 20 years. The spread of whole-school reform models such as Success for All; the imposition of standards and high-stakes tests; the lowering of class sizes and slicing of schools into smaller, independent academies; the explosion of charter schools and push for school vouchers–all these reforms signal a vibrantly democratic school system.

Experimentation, however, means more than simply changing the way we do things. It also means systematically evaluating these alternatives. To scholars, experimentation further suggests: 1) conducting studies in laboratories where external factors can be controlled in order to relate cause more directly to effect; or 2) randomly choosing which schools, classrooms, or students will be exposed to a reform and which will be exposed to the alternative with which the reform is to be compared. When well executed, random assignment serves to rule out the possibility that any post-reform differences observed between the treatment and control groups are actually due to pre-existing differences between the two groups rather than to the effects of the reform. The superiority of random assignment for drawing conclusions about cause and effect in nonlaboratory settings is routinely recognized in both the philosophy of science literature and in methods texts in health, public health, agriculture, statistics, microeconomics, psychology, and those parts of political science and sociology that deal with improving the assessment of public opinion.

Since most education research must take place in actual school settings, random assignment would seem to be a highly appropriate research tool. However, though the American education system prizes experimentation in the sense of trying many new things, it does not promote experimentation in the sense of using random assignment to assess how effective these new things are. One review showed that not even 1 percent of dissertations in education or of the studies archived in ERIC Abstracts involved randomized experiments. A casual review of back issues of the premier journals in the field, such as the American Educational Research Journal or Educational Evaluation and Policy Analysis, tells a similar story. Responding to my query, a nationally recognized colleague who designs and evaluates curricula replied that in her area randomized experiments are extremely rare, adding, “You can’t get districts to randomize or partially adopt after a short pilot phase because all parents would be outraged.”

Very few of the major reform proposals currently on the national agenda have been subjected to experimental scrutiny. I know of no randomized evaluations of standards setting. The “effective schools” literature includes no experiments in which the supposedly effective school practices were randomly used in some schools and withheld from others. Recent studies of whole-school-reform programs and school management have included only two randomized experiments, both on James Comer’s School Development Program, which means that the effects of Catholic schools, Henry Levin’s Accelerated Schools program, or Total Quality Management have never been investigated using experimental techniques. School vouchers are a partial exception to the rule; attempts have been made to evaluate one publicly funded and three privately funded programs using randomized experiments. Charter schools, however, have yet to be subjected to this method. On smaller class sizes, I know of six experiments, the most recent and best known being the Tennessee class-size study. On smaller schools I know of only one randomized experiment, currently under way. In fact, most of what we know about education reforms currently depends on research methods that fall short of the technical standard used in other fields.

Equally striking is that, of the few randomized experiments cited above, nearly all were conducted by scholars whose training is outside the field of education. Educators Jeremy Finn and Charles Achilles began the best-known class-size experiment, but statisticians Frederick Mosteller, Richard Light, and Jason Sachs popularized the study, and economist Alan Krueger has conducted an important secondary analysis. Political scientist John Witte conducted the Milwaukee voucher study, while political scientists Jay Greene and his colleagues and economist Cecelia Rouse reanalyzed the data. Sociologists and psychologists conducted the Comer studies. Economists James Kemple and JoAnn Leah Rock are running the ongoing experiment on academies within high schools. Political scientist William Howell and his colleagues did the work on school-choice programs in Washington, D.C.; New York City; and Dayton, Ohio. Scholars with appointments in schools of education, where we might expect the strongest evaluations of school reform to be performed, evidence a 20-year near-drought when it comes to randomized experiments.

Such distaste for experiments contrasts sharply with the practices of scholars who do school-based empirical work but don’t operate out of a school of education. Foremost among these are scholars who research ways to improve the mental health of students or to prevent violence or the use of tobacco, drugs, and alcohol. These researchers usually have disciplinary backgrounds in psychology or public health, and they routinely assign schools or classrooms to treatments randomly. Randomized experiments are commonplace in some areas of contemporary research on primary and secondary schools. They’re just not being done by researchers who were trained in education schools.

Dealing with Complexity

In schools of education, the intellectual culture of evaluation actively rejects random assignment in favor of alternatives that the larger research community has judged to be technically inferior. Education researchers believe in a set of mutually reinforcing ideas that provides what for them is an overwhelming rationale for rejecting experiments on any number of philosophical, practical, or ethical grounds. Any Ph.D. from a school of education who was exposed to the relevant literature on evaluation methods has encountered arguments against experiments that appeared cogent and comprehensive. For older education researchers, all calls to conduct formal experiments probably have a “déjà vu” quality, reminding them of a battle they thought they had won long ago–the battle against a “positivist” view of science that privileges the randomized experiment and its related research and development model whose origins lie in agriculture, health, public health, marketing, or even studies of the military. Education researchers consider this model irrelevant to the special organizational complexity of schools. They prefer an R&D model based on various forms of management consulting.

In management consulting, the crucial assumptions are that 1) each organization possesses a unique culture and set of goals; therefore, the same intervention is likely to elicit different results depending on a school’s history, organization, personnel, and politics; and 2) suggestions for change should creatively blend knowledge from many different sources–from general organizational theories, from deep insight into the district or schools under study, and from “craft” knowledge of what is likely to improve schools or districts with particular characteristics. Scientific knowledge about effectiveness is not particularly prized in the management-consulting model, especially if it is developed in settings different from those where the knowledge is to be applied.

As a central tool of science, random assignment is seen as the core of an inappropriate worldview that obscures each school’s uniqueness, that oversimplifies the complicated nature of cause and effect in a school setting, and that is naive about the ways in which social science is used in policy debates. Most education evaluators see themselves as the vanguard of a post-positivist, democratic, and craft-based model of knowledge growth that is superior to the elitist scientific model that, they believe, has failed to create useful and valid knowledge about improving schools. Of the reasons critics articulate for rejecting random assignment as an evaluation tool, some are not very credible, but others are and should inform the design of future studies that use random assignment. Let’s deal with some of the major objections in turn.

The world is ordered more complexly than a causal connection from A to B can possibly capture. For any given outcome, randomized experiments test the influence of only a few potential causes, often only one. At their most elegant, they can responsibly test only a modest number of interactions between different treatments or between any one treatment and individual differences at the school, classroom, or individual level. Thus, randomized experiments are best when the question of causation is simple and sharply focused.

Lee Cronbach, perhaps the most distinguished theorist of educational evaluation today, argues that in the real world of education too many factors influence the system to isolate the one or two that were the primary causes of an observed change. He cannot imagine an education reform that fully explains an outcome; at most there will be just one cause of any change in this outcome. Nor can he imagine an intervention so general in its effects that the size of a cause-effect relationship remains constant across different populations of students and teachers, across different kinds of schools, across the entire range of relevant outcomes, and across all time periods. Experiments cannot faithfully represent a real world characterized by multivariate, nonlinear (and often reciprocal) causal relationships Moreover, few education researchers have much difficulty detailing contingencies likely to limit the effectiveness of a proposed reform that were never part of a study’s design.

There is substance to the notion that randomized experiments speak to a simple, and possibly oversimplified, theory of causation. However, many education researchers speak and write as though they accept certain contingency-free causal connections–for example, that small schools are better than large ones; that time on task raises achievement; that summer school raises test scores; that school desegregation hardly affects achievement; and that assigning and grading homework improves achievement. They also seem to be willing to accept some propositions with highly circumscribed causal contingency–for instance, that reducing class size increases achievement (provided that it is a “sizable” change and that the reduction is to fewer than 20 students per class); that Catholic schools are superior to public ones in the inner-city but not in suburban settings. Commitment to a full explanatory theory of causation has not precluded some education researchers from acting as if very specific interventions have direct and immediate effects.

Quantitative research has been tried and has failed. Education researchers were at the forefront of the flurry of social experimentation that took place at the end of the 1960s and through the 1970s. Quantitative studies of Head Start, Project Follow Through, and Title I concluded that, for all three programs, there were no replicable effects of any magnitude that persisted over time. Such results provoked hot disputes over the methods used, and many educational evaluators concluded that quantitative evaluation of all kinds had failed. Some evaluators turned to other methods of educational evaluation. Others turned to the study of school management and program implementation in the belief that poor management and incomplete implementation explained the disappointing results. In any event, dissatisfaction with quantitative evaluation methods grew.

However, none of the most heavily criticized quantitative studies involved random assignment. I know of only three randomized experiments on education reform available at the time. One was of the second year of “Sesame Street,” where cable capacity was randomly assigned to homes in order to promote differences in children’s opportunity to view the show. The second experiment was the widely known Perry Preschool Project in Ypsilanti, Michigan. The third involved only 12 youngsters who were randomly assigned to a desegregated school. Only the desegregation study involved primary or secondary schools. Thus it was not accurate to claim in the 1970s that randomized experiments had been tried and had failed. Only nonexperimental quantitative studies had been done, and few of these would pass muster today as even high-quality quasi-experiments.

Random assignment is not politically, administratively, or ethically feasible in education. The small number of randomized experiments in education may reflect not researchers’ distaste for them but a simple calculation of how difficult they are to mount in the complex organizational context of schools. School district officials do not like the focused inequities in school structures or resources that random assignment usually generates, fearing backlash from parents and school staff. They prefer it when individual schools can choose which reforms they will implement or when changes are made on a district-wide basis. Some school staff members also have administrative concerns about disrupting routines and ethical concerns about withholding potentially helpful treatments from students and teachers in need.

Surely it is not easy to implement randomized experiments of school reform. In many of the recent experiments, schools have dropped out of the experiment in different proportions, often because a new principal wanted to change what his predecessor had recently done, including eliminating the reform under study. Then there are the cases of possible treatment crossover, as happened in one of my own studies in Prince George’s County, Maryland. One principal in an experimental school was married to someone teaching in a control school, and they discussed their professional life at home; one control principal really liked the reform under study and tried to bring parts of it to his school; and the daughter of one of the program’s senior officials taught in a control school. In a similar vein, the Tennessee class-size experiment compared classrooms within the same schools. What did Tennessee teachers in the larger classes make of the situation whereby some colleagues in the same school taught smaller classes at the same grade level? Were they dispirited enough to work less? To avoid such possibilities, most public-health evaluations of prevention programs (such as those aimed at reducing drug use) use comparisons between schools instead of between classrooms within the same school. All randomized experiments in education have to struggle with issues like these.

What does it take to mount randomized experiments? Political will plays an important role. In the health sciences, random assignment is common because it is institutionally supported by funding agencies and publishing outlets and is culturally supported through graduate training programs and the broadly accepted practice of clinical trials. Public-health researchers have learned to place a high priority on clear causal inferences, a priority reinforced by their funders (mostly the National Institutes of Health, the Centers for Disease Control, and the Robert Wood Johnson Foundation). The health-related studies conducted in schools tap into this institutional and cultural structure. Similar forces operate with the rapidly growing number of studies of preschool education that use random assignment. Most are the product of congressional requirements to assign at random; the high political and scholarly visibility of the Perry Preschool and Abecedarian projects that used random assignment; and the involvement of researchers trained in psychology and microeconomics, fields where random assignment is valued.

Contrast this with educational evaluation. Reports from the Department of Education’s Office of Educational Research and Improvement (OERI) are supposed to detail what is known to work. However, neither the work of educational historian Maris Vinovskis nor my own reading of OERI reports suggests that any privilege is being accorded to random assignment. At a recent foundation meeting on teaching and learning, a representative of nine regional governors discussed the lists of best practices that are being widely disseminated. He did not care, and he believed that the governors do not care, about the technical quality of the designs generating these lists; the major concern is that educators can deliver a consensus on each practice. When asked how many of these best practices depended on randomized experiments, he guessed it would be close to zero. Several nationally known education researchers were present. They too replied that random assignments probably played no role in generating these best-practice lists. No one seemed to feel any distress at this.

Random assignment is premature because it assumes conditions that do not yet pertain in education. As the research emphasis shifted in the 1970s to understanding schools as complex social organizations with severe organizational problems, randomized experiments must have seemed premature. A more pressing need was to understand management and implementation, and to this end, more and more political scientists and sociologists of organizations were recruited into schools of education. They brought with them their own strongly held preference for qualitative methods and their memories of the wars between quantitative and qualitative methods in their own disciplines.

However, school research need not be predicated only on the idea of schools as complex organizations. Schools were once conceptualized as the physical structure containing many self-contained classrooms in which teachers tried to deliver effective curricula using instructional practices that demonstrably enhance students’ academic performance. This approach privileged curriculum design and instructional practice over the schoolwide factors that have come to dominate understandings of schools as complex organizations–factors like strong leadership, clear and supportive links to the world outside of school, a building-wide community focused on learning, and the pursuit of multiple forms of professional development.

Many important consequences have flowed from the intellectual shift in how schools are conceptualized. One is the lesser profile accorded to curriculum and instructional practice and to what happens once the teacher closes the classroom door; another is the view that random assignment is premature, given its dependence on expert school management and high-quality program implementation; and another is the view that quantitative techniques have only marginal usefulness for understanding schools, since a school’s governance, culture, and management are best understood through intensive case studies.

However, the aim of experiments is not to explain all sources of variation; it is to probe whether the school reform idea makes a difference at the margin, despite whatever variation exists among schools, teachers, students, or other factors. It is not an argument against random assignment to claim that some schools are chaotic, that implementation of a reform is usually highly variable, and that treatments are not completely faithful to their underlying theories. Random assignment does not need to be postponed while we learn more about school management and implementation.

Nonetheless, the more we know about these matters, the better we can randomize and the more management and implementation issues can be worthy objects of study within experiments. Advocates of random assignment will not be credible in educational circles if they assume that reforms will be implemented uniformly. Experimenters need to be forthright that school-level variation in implementation quality will often be very large. It is not altogether clear that schools are more complex than other settings where experiments are routinely done–say, hospitals–but most school researchers seem to believe this, and it seems a reasonable working assumption.

Thirty years after vouchers were proposed, we still have no clear answer about them. Thirty years after James Comer began his work that has resulted in the School Development Program, and again we have no clear answer. Almost 20 years after Henry Levin began Accelerated Schools; here too we have no answer. While premature experimentation is indeed a danger, these time lines are inexcusable. The federal Obey-Porter educational legislation cites Comer’s program as a proven program worth replicating elsewhere and provides funds for this. But when the legislation passed, the only available evidence about the program consisted of testimony; a dozen or so empirical studies by the program’s own staff that used primitive quasi-experimental designs; and the most-cited single study confounded the court-ordered introduction of the program with a simultaneously ordered reduction in class sizes of 40 percent. To be restricted to such evidence when making a decision about federal funding verges on the irresponsible.

Unlike medicine or public health, education has no tradition of multisite experiments with national reach. Single experiments of unclear reach, done only in Milwaukee, Washington, Chicago, and Tennessee, are what we typically find. Moreover, some kinds of school reform have no fixed protocol, and it is possible to imagine implementing vouchers, charter schools, or programs like Comer’s or Total Quality Management schools in many different ways. Indeed, the Comer programs in Prince George’s County, Chicago, and Detroit are different from one another in many major specifics. The nonstandardization of many treatments requires even larger samples than those typically used in medicine and public health. Getting cooperation from so many schools is not easy, given the history of local control in education and the absence of a tradition of random assignment. Still, larger individual experiments can be conducted than are being done today.

Random assignment is not needed because there are other less irritating methods for generating knowledge about cause and effect. Most researchers who evaluate education reforms believe there are superior alternatives to the randomized experiment. These methods are superior, they believe, because they are more acceptable to school personnel, because the knowledge they generate reduces enough uncertainty about causation to be useful, because the knowledge is relevant to a broader array of important issues than merely identifying a causal connection, and because schools are especially likely to use the results for self-improvement. No single alternative is universally recommended, and here I’ll discuss only two: intensive qualitative case studies and quasi-experiments.

• Intensive case studies. Cronbach asserted that the appropriate methods for educational evaluation are those of the historian, journalist, and ethnographer, not the scientist. Most educational evaluators now seem to prefer case-study methods for learning about reforms. They believe that these methods are superior because schools are less squeamish about allowing ethnographers through the door than experimentalists. They also believe that qualitative studies are more flexible. They provide simultaneous feedback on the many different kinds of issues worth raising about a reform–issues about the quality of implementation, the meaning various actors ascribe to the reform, the primary and secondary effects of the reform, its unanticipated side effects, and how different subgroups of teachers and students are affected. Entailed here is a flexibility of purposes that the randomized experiment cannot match, given its limited central purpose of facilitating clear causal inference.

A further benefit relates to schools actually using the results. Ethnography requires attention to the unfolding of results at different stages in a program’s implementation, thus generating details that can be fed back to school personnel and that also help explain why a program is effective. A crucial assumption is that school staff are especially likely to use a study’s results because they have a better ongoing relationship with qualitative researchers than they would have with quantitative ones. Of course, the use in question is highly local, often specific to a single school, while the usual aspiration for experiments is to guide policy changes that will affect large numbers of districts and schools.

The downside of case studies is the question of whether this process reduces enough uncertainty about causation to be useful. With qualitative methods it is difficult to know just how the group under study would have changed had the reform not been in place. The rationale for preferring an experiment over an intensive case study has to be the value of a clear causal inference, of not being wrong with the claim that a reform is effective or not. Of course, one can have one’s cake and eat it too, for there are no compelling reasons why case study methods cannot be used within an experiment to extend its reach. While black-box experiments that generate no knowledge of process may be common, they are not particularly desirable. Nor are they the only kinds of experiments possible.

• Quasi-experiments. Quasi-experiments are like randomized experiments in purpose and in most of their structural details. The defining difference is the absence of random assignment and hence of a demonstrably valid causal counterfactual. The essence of quasi-experimentation is the search, more through design than statistical adjustment, to create the best possible approximation of this missing counterfactual. However, quasi-experiments are second best to randomized experiments in the clarity of causal conclusions. In some quarters, quasi-experiment has come to connote any study that is not an experiment or any study that includes some type of nonequivalent control group or pretreatment observation. Indeed, many of the studies calling themselves quasi-experiments in educational evaluation are of types that theorists of quasi-experimentation reject as usually inadequate. To judge by the quality of the educational evaluation work I know best–on school desegregation, Comer’s School Development Program, and bilingual education–the average quasi-experiment in these fields inspires little confidence in its conclusions about effectiveness. Recent advances in the design and analysis of quasi-experiments are not getting into research evaluating education.

Moving Forward

It will be difficult to persuade the current community of educational evaluators to begin doing randomized experiments solely by informing them of the advantages of this technique, by providing them with lists of successfully completed experiments, by telling them about new methods for implementing randomization, by exposing them to critiques of the alternative methods they prefer, and by having prestigious persons and institutions outside of education recommend that experiments be done. The research community concerned with evaluating education reforms is a community in which all parties share at least some of the beliefs outlined above. They are convinced that anyone pursuing a scientific model of knowledge growth is an out-of-date positivist seeking to resuscitate debates that are rightly dead.

Some rapprochement might be possible. At a minimum, it would require advocates of experimentation to be explicit about the real limits of their preferred technique, to engage their critics in open dialogue about the critics’ objections to randomization, and to assert that experiments will be improved by paying greater attention to program theory, implementation specifics, quantitative and qualitative data collection, causal contingency, and the management needs of school personnel as well as of central decisionmakers.

Though it is desirable to enlist the current community of educational evaluation specialists in supporting randomized experiments, it is not necessary to do so. They are not part of the tiny flurry of controlled experiments now occurring in schools. Moreover, in several substantive areas Congress has shown its willingness to mandate carrying out controlled studies, especially in early-childhood education and job training. Therefore, end runs around the education research community are conceivable. This suggests that future experiments could be carried out by contract research firms, by university faculty members with a policy science background, or by education faculty who are now lying fallow. It would be a shame if this occurred and restricted our access to those researchers who know best about micro-level school processes, about school management, about how school reforms are actually implemented, and about how school, state, and federal officials tend to use education research. It would be counterproductive for outsiders to school-reform research to learn anew the craft knowledge insiders already enjoy. Such knowledge genuinely complements controlled experiments.

-Thomas D. Cook is a professor of sociology, psychology, education, and social policy at Northwestern University. This article is adapted from a chapter that will appear in Evidence Matters (Brookings, forthcoming).

The Florida A-Plus program is a school accountability system with teeth. Each public school is assigned a grade based on the performance of its students on the Florida Comprehensive Assessment Test (FCAT) in reading, math, and writing. Reading and writing FCATs are administered in the 4th, 8th, and 10th grades; students take the math FCAT in the 5th, 8th, and 10th grades. The scale-score results from these tests are divided into five categories. The letter grade that each school receives is determined by the percentage of its students scoring above the thresholds established by these five categories or levels. If a school receives two F grades in a four-year period, its students are offered vouchers that they can use to attend a private school. They are also offered the opportunity to attend a better-performing public school.

The FCAT was first administered in the spring of 1998. So far, only two schools in the state, both located in Escambia County, have received two failing grades, the second coming during the 1999 round of testing in both cases. Students in both schools were offered vouchers, and nearly 50 students and their families chose to attend one of a handful of nearby private schools, most of which were religiously affiliated. No additional schools were subject to the voucher provision after the 2000 administration of the FCAT because none failed for a second time.

The theory undergirding this system is that schools in danger of failing will improve their academic performance to avoid the political embarrassment and potential loss in revenues from having their students depart with tuition vouchers. Whether the theory accords with the evidence is the issue addressed here. Perhaps the threat of vouchers being offered to students will provide the impetus for reform. But it is also plausible that schools will develop strategies for improving the grade they receive from the state without actually improving the academic performance of students. Perhaps schools will not have the resources or flexibility to adopt necessary reforms even if they have the incentives to do so. Perhaps the incentives of the accountability system interact with the incentives of school politics to produce unintended outcomes.

The evidence suggests that the theory holds true: that the A-Plus program has been successful at motivating failing schools to improve their academic performance. The gains, moreover, seem to reflect real improvement rather than a mere manipulation of the state’s testing and grading system.

If a school receives two F grades in a four-year period, its students are offered vouchers that they can use to attend a private school. They are also offered the opportunity to attend a better-performing public school.

The Literature

The question of whether testing and accountability systems are an effective reform tool has seldom been the subject of rigorous research. Most research attention has been devoted to evaluations of the accountability system in Texas. The Texas Assessment of Academic Skills (TAAS) has been in existence for a decade and is the most comprehensive of all state testing systems. Students in Texas are tested in 3rd through 8th grades in math and reading. In addition, students must pass an exit exam first offered in 10th grade in order to graduate. The state is also phasing in requirements that students pass exams in order to be promoted to the next grade.

The comprehensive nature of Texas’s accountability system and the fact that its governor was a candidate for the presidency attracted considerable attention to the TAAS. The most systematic research on TAAS appeared in two somewhat contradictory reports issued by the RAND Corporation (for a critique of both reports, see Eric Hanushek’s “Deconstructing RAND” in the Spring 2001 issue, available on-line at www.educationnext.org). In the first report, released in July of 2000, David Grissmer and his colleagues analyzed scores from the National Assessment of Educational Progress (NAEP), a test administered by the U.S. Department of Education, in order to identify state policies that may contribute to higher academic performance. They found that states like Texas and North Carolina, with extensive accountability systems, were among the highest-scoring and fastest-improving states after demographic factors were controlled for. The report featured a lengthy comparison of student performance in California, which has an underdeveloped accountability system and weak academic performance, and Texas to highlight the importance of TAAS in improving academic achievement, as measured by NAEP.

The second report, released in October of 2000 by Stephen Klein and his colleagues, cast doubt on the validity of TAAS scores by suggesting that the results do not correlate with the test results of other standardized tests. Because the other standardized tests are “low-stakes tests,” without any reward or punishment attached to student or school performance, the authors reason that there are few incentives to manipulate the results or cheat, making the low-stakes test results a reliable measure of student performance (although it is also possible that schools and students won’t prepare enough for a low-stakes test to demonstrate their true abilities). By contrast, schools and students might have incentives and opportunities to manipulate the results of high-stakes tests, like TAAS. The dissonance between the different tests, the authors argue, should at least raise a red flag regarding the gains observed on TAAS. Klein and his colleagues also analyzed NAEP results in Texas, and, contrary to the findings of Grissmer and his colleagues, concluded that Texas’s performance on NAEP was not exceptionally strong.

Klein and his colleagues, however, cannot rule out alternative explanations for the weak correlation between TAAS results and the results of low-stakes standardized tests. It is possible that TAAS, which is based on the mandated Texas curriculum, tests different skills than those tested by the national standardized tests. Both could produce valid results and still be weakly correlated with one another if they are testing different things. It is also possible that the pool of standardized tests that were available to the RAND researchers were not representative of Texas as a whole. The standardized test results that were compared with TAAS results were only from 2,000 non-randomly selected 5th-grade students from one part of Texas. If this limited group of students were not representative of all Texas students, it would be inaccurate to draw any conclusions about TAAS as a whole.

Another examination of NAEP scores in Texas, which I conducted, showed that NAEP improvements were exceptionally strong in Texas while the TAAS accountability system was in place. The disparate findings regarding the relationship between Texas’s scores on TAAS and NAEP can be partially explained by differences in the time periods and grade levels examined, and by the presence or absence of controls for student demographics. For now it is enough to say that there is some ambiguity regarding any conclusions that can be drawn from a comparison of NAEP and TAAS results. This ambiguity is in part a result of the fact that NAEP is administered infrequently and only in certain grade levels.

A more recent collection of studies edited by Martin Carnoy of Stanford University and issued by the Economic Policy Institute finds that the accountability systems in Texas, North Carolina, and Florida (before the adoption of A-Plus) all motivated failing schools to produce significant gains. Unfortunately, none of the studies released by the Economic Policy Institute confirm the validity of the state testing results by comparing them to the results on national exams. It is possible that the critics of testing are right, that some or even all of the gains measured only by state tests are the product of teaching to the test, cheating, or other manipulations of the testing system. In addition, the pre-A-Plus Florida analysis reported by the Economic Policy Institute is plagued by several research design flaws. For example, the study compares results from schools that took several different standardized tests without making any effort to ensure that the results are comparable. And because only pass rates were available, the scale scores analyzed were estimated based on a series of assumptions.

The research presently available on the potential of vouchers to improve achievement in public schools is also less than conclusive. Recent studies by economist Caroline Minter Hoxby, as discussed in this issue, have attempted to address this question by examining the consequences of variation in the extent of choice currently available in the United States. They suggest that areas with more choice and competition experience better academic outcomes than areas with less choice and competition. While these results support the contention that vouchers would improve the quality of education for the entire education system, it remains to be seen whether even the prospect of competition can provoke a public school response.

Studying the A-Plus accountability and choice system in Florida therefore offers the possibility of some important contributions to the existing research literature. An evaluation of A-Plus can reveal whether the prospect of competition, in the form of vouchers offered to students at chronically failing schools, represents an effective incentive for improvement. Unlike other studies of accountability systems, the ability to validate the scores used in the A-Plus system by comparing them with performance on nationally normed exams offers the possibility of dispelling concerns about whether the observed gains are real or the products of teaching to the test, cheating, or manipulation of the testing system.

Validating the FCAT Results

The first section of the analysis addresses the question of whether Florida’s test is a valid test of students’ academic abilities. Given the concerns raised by the Klein study regarding the validity of the TAAS exams in Texas, I decided to use the same analytical technique as Klein: comparing results on the FCAT with results on low-stakes standardized tests given at around the same time and in the same grade.

During the spring of 2000, Florida schools administered both the FCAT and a version of the Stanford 9, which is a widely used and respected nationally normed standardized test. Performance on the FCAT determined a school’s grade from the state and therefore determined whether students would receive vouchers. Performance on the Stanford 9 carried no similar consequences, so schools and students had little reason to manipulate, cheat, or teach to the Stanford 9. If the results of the Stanford 9 are similar to the results of the FCAT, the FCAT is likely to be a valid measure of academic achievement. If the results are not similar, it is possible that the FCAT results are not a valid measure of student performance.

To achieve the same gain in math that the threat of vouchers accomplished, Florida schools would need to increase per-pupil spending by $3,484 at schools that had earned an F. This would be an increase of more than 60 percent in education spending.

The results of this analysis suggest that the FCAT results are valid measures of student achievement. Schools with the highest scores on the FCAT also had the highest scores on the Stanford 9 tests that were administered around the same time in the spring of 2000. Likewise, schools with the lowest FCAT scores tended to have the lowest Stanford 9 scores. If the correlation were 1.00, the results from the FCAT and Stanford 9 test would be identical. As it turns out, the correlation coefficient was 0.86 between the 4th grade FCAT and Stanford 9 reading test results. In 8th grade the correlation between the high-stakes FCAT and low-stakes standardized reading test was 0.95. In 5th-grade math, the correlation coefficient was 0.90; in 8th-grade math, the correlation was 0.95; and in 10th-grade math, the correlation was 0.91. In other words, the results of the two tests are quite similar. (It was not possible to verify the validity of the FCAT writing test with this technique because no Stanford 9 writing test was administered.)

In the second RAND study of TAAS in Texas, Klein and his colleagues never found a correlation of more than 0.21 between the school-level results from TAAS and the school-level results from low-stakes standardized tests. In this analysis there was never a correlation between FCAT and the Stanford 9 below 0.86.

To exclude the possibility that teaching to the test, cheating, or manipulation occurred only among schools that were previously failing, I also examined the correlations between the FCAT and Stanford 9 results among this subset of schools. This revealed that even among previously failing schools the correlations between the two test results remain very high, ranging from 0.77 to 0.99. It appears as if the pressures placed on previously failing schools did not lead them to distort their test results.

The Prospect of Vouchers

Now that the validity of the FCAT as a measure of student performance has been established, the question of whether vouchers inspired improvement among Florida’s failing schools can be studied. The greatest improvements should be seen among schools that had already received one F grade from the state, since their students would become eligible for vouchers if they received a second F. To test this hypothesis, average FCAT scale-score improvements for schools were broken out by the grade they received the year before.

In fact, the incentives appear to operate as expected. Schools that had received F grades in 1999 experienced the largest gains on the FCAT between 1999 and 2000. The year-to-year changes in school-level FCAT results did not differ systematically according to whether the school had received a grade of A, B, or C from the state. Schools that had received D grades and were close to the failing grade that could precipitate vouchers’ being offered to their students, by contrast, appear to have achieved somewhat greater improvements than those achieved by the schools with higher state grades. Schools that received F grades in 1999 experienced increases in test scores that were more than twice as large as those experienced by schools with higher state-assigned grades.

On the FCAT reading test, which uses a scale with results between 100 and 500, schools that had received an A grade from the state in 1999 improved by an average of 2 points between 1999 and 2000 (see Figure 1). Schools that had received a B grade improved by 5 points. Those earning a C in 1999 increased by 5 points. By contrast, schools with a D grade in 1999 improved by 10 points. Schools with F grades in 1999 showed an average gain of 18 points, equal to 0.8 standard deviations. In other words, the lower the grade in 1999, the greater the improvement in 2000.

A similar pattern emerged in the FCAT math results. Schools earning an A grade experienced an average 11-point gain. Schools with a B gained 9 points. Schools with C grades in 1999 showed gains of 12 points, on average, between 1999 and 2000. Schools earning D grades improved by 16 points, while schools that received F grades in 1999 made gains of 26 points, equal to 1.25 standard deviations.

The FCAT writing exam, whose scores range from 0 to 6, also shows larger gains for schools earning an F grade in 1999. Schools with an A grade in 1999 improved by 0.4 points on the writing test; B schools had an average gain of 0.4 points; and C schools gained 0.5 points. D schools improved 0.5 points, while F schools demonstrated an average gain of .9 points, equal to an astounding 2.2 standard deviations.

Alternative Explanations?

The fact that gains among schools facing the prospect of vouchers were nearly twice as large as the gains achieved by other schools might be at least partially attributable to other factors. One possible factor is regression to the mean, the statistical tendency for very low or very high scores to move closer to the group average when retested. This common dynamic could account for at least some of the extraordinary gains realized by previously failing schools. It is also plausible that the extraordinary gains of failing schools were the result of their being provided with additional resources not available to other schools. And some observers have speculated that the exceptional gains observed in Florida could be explained by a change in rules regarding the test scores of high-mobility students who move in and out of schools and districts often.

To test these alternative explanations I compared the improvements recorded by F-level schools that had above-average initial scores for their category with D-level schools that had below-average initial scores for their category. The intuition here is that high-scoring F schools and low-scoring D schools are very much alike initially, yet one group is subject to the accountability system’s punishments (the F label and the prospect of vouchers), while the other group of schools is not. In many ways this comparison approximates a randomized experiment. Because the two groups were so close to the threshold dividing D and F schools, chance may explain to a fair degree why these schools received one grade or the other. This is not to say that grading systems are inherently arbitrary; it is only recognizing the reality that luck is an important factor at the margins.

The initial similarity between the two groups of schools allows us to be confident that any difference in the gains realized by high-scoring F schools and low-scoring D schools is the result of the accountability system and not other factors. Regression to the mean cannot explain the gains of high-scoring F schools relative to low-scoring D-schools because both groups begin with similarly low scores. In fact, because the letter grade is based on the percentage of students scoring above certain thresholds and not on the average score in each school, the high-scoring F schools actually have slightly higher initial reading and math scores than do the low-scoring D schools. In addition, statistical techniques can control for the influence of differences in the background of students in each group or in the additional resources provided to each group.

Comparing the demographic characteristics of high-scoring F schools and low-scoring D schools confirms that the two groups are quite similar. They also do not differ significantly in their initial per-pupil spending, average class size, percentage of students receiving subsidized school lunches, percentage of students with limited English proficiency or disabilities, and the mobility of their student populations.

Note that the comparison between high-scoring F schools and low-scoring D schools is likely to underreport the true effect of labeling schools as failing and forcing them to face the prospect of vouchers. The comparison only measures the amount by which certain F schools outperform certain D schools, ignoring the possibility that D schools are also inspired to improve for fear of failing for the first time. Indeed, simply assigning grades to schools may inspire them to improve in order to get better grades. All schools face this incentive to some degree.

Nevertheless, high-scoring F schools did experience gains larger than their low-scoring D counterparts. After controlling for average class size, per-pupil spending in 1998-99, the percentage of students with disabilities, the percentage of students receiving a free or reduced- price school lunch, the percentage of students with limited English proficiency, and student mobility rates, high-scoring F schools achieved gains that were 2.5 points greater than their below-average D counterparts in reading (see Figure 2). The math results show that the prospect of vouchers inspired additional gains of 5.2 points. On the writing test, which has a scale of 1 to 6, the effect was 0.2 points, although, since the validity of the FCAT writing test cannot be confirmed, this finding is less definitive. Therefore, schools that received an F grade-and faced the prospect of vouchers should they receive another F-experienced gains superior to those made by schools at a similar level of performance but that did not face the threat of vouchers.

The larger gains made by schools facing the threat of vouchers cannot be explained by spending increases. While F schools did receive additional resources-about $600 per pupil in additional funding, compared with about $200 per pupil in D schools-taking this additional spending into account does not alter the extra gains achieved by schools that faced the prospect of vouchers. This is an especially important finding because the additional resources obtained by F schools may have been at least partially the result of the threat of vouchers. That is, school districts may have allocated more money to failing schools or failing schools may have been more aggressive in their grant writing precisely because they were taking action to avoid receiving a second F. The fact that including additional resources in the analysis does not diminish the magnitude of the motivational effect of vouchers suggests that the results are quite robust. Furthermore, the fact that controlling for the rate of student mobility does not have any effect on the results suggests that the exceptional gains achieved by F schools were not caused by a change in the rules concerning the treatment of high-mobility students.

Conclusion

To put the magnitude of the voucher effect into perspective, the same models can be used to calculate how much additional spending it would take to produce gains as large as those produced by labeling schools and threatening them with vouchers. According to the models comparing high-scoring F schools with low-scoring D schools, to achieve the same 5-point gain in math that the threat of vouchers accomplished, Florida schools would need to increase per-pupil spending by $3,484 at previously failing schools. This would be an increase of more than 60 percent in education spending. To realize the same gain as the A-Plus program accomplished in reading, Florida schools would need to spend $888 more per pupil, more than a 15 percent increase in per-pupil spending. To produce the same gain in writing scores, per-pupil spending would have to be increased by $2,805, more than a 50 percent increase.

For many years policymakers have focused on providing schools with enough resources to educate students. The evidence from the A-Plus accountability and choice program suggests that policymakers must also ensure that schools are provided with the appropriate incentives to use their resources effectively. Grading schools and using vouchers as a sanction for repeated failure inspires improvement at schools in a way that simply providing additional resources cannot. The evidence from Florida also suggests that the gains produced by such an accountability system are real indicators of improvement in learning, and not simply teaching to the test, cheating, or other manipulations of the testing system. Whether the same gains could have been produced using alternative sanctions is unknown. But the fact is that vouchers were used, and they were unquestionably effective.

-Jay P. Greene is a senior fellow at the Manhattan Institute for Policy Research.

Federal funding and direction routinely lead to research and development breakthroughs in science, medicine, and defense, among other fields. Witness the recent debate over federal funding of stem-cell research, during which supporters of stem-cell research emphasized that federal funding tends to draw the best scholars to a field. So why do research programs funded by the U.S. Department of Education continue to disseminate research that is barely respected by serious scholars? Many academics and policymakers consider the education research sponsored by the National Science Foundation or the National Institutes of Health to be more rigorous and scientifically sound than the research emanating from the Department of Education’s Office of Educational Research and Improvement (OERI). Likewise, the department’s Planning and Evaluation Service (PES), tasked with evaluating the performance of federal education programs like Title I, has a mixed record of developing sound and timely information on program quality.

As policymakers look to redesign or replace many of the basic federal education programs established during the 1960s, they face mounds of Department-produced research to guide them, but much of it is of limited value. This is a problem that has cut across the decades, during both Republican and Democratic administrations. With OERI up for reauthorization by Congress, this seems like the perfect time to revisit the federal role in research and development.

The Centers and Labs

Federal education research and development has been fragmented and disorganized for most of its history. In an effort to address these problems, in the mid-1960s the government created two complementary sets of institutions intended to promote more coordinated, long-term, large-scale education research and development: the Research and Development Centers and the Regional Educational Laboratories. However, most of their work turned out to be (and continues to be) short term and small scale. They tend to focus on small, individual research projects rather than supporting large-scale development and evaluations of alternative techniques and programs for helping all children to succeed in school.

The value of their research and development efforts has been routinely criticized, yet the share of research funding devoted to the centers and laboratories rose under both the National Institute of Education and its successor, OERI. The labs used their lobbying arm, the Council for Educational Development and Research, to protect their budgets even as funding for research proposed by individual scholars and other initiatives was disappearing or was dramatically curtailed during the Reagan years. As funding for OERI grew through the 1990s, a large share continued to go to the centers and labs or to activities only loosely related to research or development, such as distributing funds for introducing more technology in the classrooms (see Figure 1).

Neither the recent OERI third-year assessment of the centers nor the PES third-year review of the labs was a thorough or totally objective evaluation of the R&D contributions of these institutions. The results were also not made available to the public or to policymakers in a timely or convenient manner. Yet in the closing months of the Clinton administration, OERI made a last-minute decision to fund some of the existing centers for another five years and offered new five-year contracts to the labs-when the agency hasn’t even been reauthorized by Congress yet. It might make some sense to extend existing center or lab grants for another year while awaiting OERI reauthorization, but the decision to make long-term commitments to these institutions, based on inadequate evaluations and before Congress has even signaled its willingness to continue to fund them, seems quite misguided.

To be sure, the sheer paucity of funding places a serious limit on the quality of federally funded education research and evaluation, yet this hardly explains all the problems of education research and evaluation today. OERI has frequently suffered from unstable and intellectually weak leadership and the absence of a distinguished research staff. After Sharon Robinson completed three-and-a-half years as head of OERI, the agency went through four different heads in less than a year. During the Clinton administration, OERI personnel were cut by 25 percent, worsening the agency’s shortage of capable, innovative researchers. Furthermore, the agency has not taken advantage of its opportunities to hire qualified researchers. In the waning days of the Clinton administration, OERI appointed full-time directors for its five research institutes, but only one of these individuals has extensive research training and scholarly experience.

As a result, neither OERI nor Planning and Evaluation Service is providing a significant number of scientifically sound and educationally relevant program evaluations. The education marketplace is crowded with competing “whole school reform” models, such as Success for All, Accelerated Schools, and Core Knowledge, all promising remarkable gains in student achievement, especially among disadvantaged students. If one were to define the federal government’s role in education, wouldn’t one of its core functions be studying whether any of these programs actually live up to their claims? OERI has yet to systematically examine which models of education interventions are especially effective at improving student achievement in different settings. Nor has PES, which has the primary responsibility for conducting program evaluations, been able to accomplish this important task. Consequently, school districts lack the kind of research-based evidence necessary to choose among such programs; often the only information they have comes from the program developers themselves. Even worse, the federal government, through the Obey-Porter legislation, is giving aid to districts to implement many of these programs even though few of them are backed by any solid research.

Likewise, a variety of proposed reforms-charter schools, expanded standardized testing programs, class-size reduction, and school vouchers, to name a few-are currently being considered by legislators nationwide. Yet researchers all too often must turn to private foundations to secure enough funding to evaluate these reforms; OERI has almost no presence in many of these debates. What is the purpose of the Department of Education if not to provide scientifically reliable research information to guide legislators on the merits of various policy choices?

Protecting Research Independence

Federally funded education research needs to be objective and trustworthy, yet periodic political intrusions by Congress and the White House continue to muddy some of the key operations of OERI and of Planning and Evaluation Service. For example, congressional mandates specifying how OERI must spend its R&D funds hamper the agency’s ability to operate efficiently and effectively. Since the mid-1970s a few members of Congress from both parties have allied themselves with the largest beneficiaries of federal research contracts (the labs and centers) and ensured that these organizations receive a substantial portion of federal education funding for R&D.

The executive branch is equally guilty of periodically interfering with the research process. For instance, Pascal “Pat” Forgione was widely regarded as a conscientious and effective commissioner of the National Center for Education Statistics (NCES), the statistical branch of OERI. The national center’s Advisory Council on Education Statistics had strongly urged the secretary of education to reappoint him. Yet he was not renominated, and some speculate that the real cause of Forgione’s departure was his public protest of then-vice president Al Gore’s inappropriate politicization of the Department’s release of reading scores from the National Assessment of Educational Progress (NAEP). Rather than wait for NCES to release the NAEP results objectively, as was the normal and expected procedure, Gore decided to use the occasion to provide his own interpretation of the data-an interpretation that many observers thought was too slanted toward the administration’s own policy interests. Likewise, the Department of Education’s attempts during the late 1990s to use OERI staff and resources in developing the controversial voluntary national tests in reading and math, without congressional authorization, led some legislators to question the research agency’s independence.

The Department of Education itself is no stranger to politicking. When it suits its policy purposes, Planning and Evaluation Service has publicized evaluation results early-before the actual reports have even been made public. In 1993, PES published selected excerpts from the preliminary results from Prospects, the national evaluation of Chapter 1 (later renamed Title I), before the full results of the preliminary assessment had been made public. At other times, the agency has delayed much too long in releasing unfavorable results that might have challenged current initiatives and priorities. For instance, the $9 million Longitudinal Evaluation of School Change and Performance was designed to evaluate the effectiveness of the current Title I program. Though they could have, the Department of Education and its contractors did not provide those results in time for deliberations during the 106th Congress, when Title I was up for reauthorization.

The political independence of OERI needs to be safeguarded. Research and evaluation need to be carried out objectively and reported accurately-even if the results challenge some of the policies of the current administration. One solution many analysts have suggested is to create an altogether independent agency. This would be a wise first step. There are some drawbacks to having the agency responsible for education research located outside the Department of Education, but they are outweighed by the benefits of freeing the research unit from political interference and enabling it to develop more rigorous and scientifically sound research practices. Therefore, an independent agency, reporting to the President, should be created to handle education research, development, evaluation, and statistics-something similar to the National Endowment for the Humanities or the National Science Foundation. The proposed organization would house a series of quasi-independent and separate agencies, such as the National Center for Education Research, the National Center for Development and Evaluation, the National Center for Education Statistics, and the National Assessment Governing Board.

The National Center for Education Research would be led by a commissioner of research appointed to a six-year term. It is imperative that the commissioner of research have a distinguished background in research, development, or evaluation. The size of the center’s staff should be restored to at least the 1992 levels of OERI staff, and additional distinguished research and development professionals should be recruited. The R&D centers, located in the new agency, need to grow and become more focused if they are to be relevant sources of research. Some centers survive on an annual budget of only $1.5 million to $2.5 million, hardly enough to support a legitimate large-scale program of research. An individual R&D center’s budget should be at least $5.0 million annually. Moreover, the centers need to develop a coherent, focused, coordinated five-year program of research. They should not, as is now the case, have 20 to 30 different small-scale, uncoordinated projects scattered among a half-dozen different centers across the nation.

Another independent agency needs to be created to initiate and oversee serious efforts to develop effective educational strategies and evaluate federal education programs-a National Center for Development and Evaluation, overseen by a commissioner appointed for a six-year term. Large-scale, rigorous development programs have not been a part of the department’s agenda. Likewise, the Department of Education has not conducted many scientifically sound, nonpartisan, objective program evaluations during the past two decades.

The program development and evaluation effort should be overseen by an independent, objective group of experts who will not only provide technical support, but will also ensure that the design, implementation, and interpretation of the work is scientifically sound as well as useful to educators and policymakers. Moreover, they will help to ensure that the national center’s development projects and evaluations will be readily and equally available to everyone-and not just to those who control the Department of Education at the time.

The new National Center for Development and Evaluation should solicit and implement large-scale, systematic development projects-a role that has been largely missing from the Department of Education’s agenda in recent decades. This initiative might focus its energies initially on three to five long-term projects in areas such as developing reading improvement programs or helping at-risk children make successful transitions from early-childhood programs into regular classrooms. Any research outfit, including the federally funded centers and labs, could compete for these demonstration projects. The open competition would not only spur existing education research and development providers to develop better proposals, but it might also attract interest from other major social science research organizations, such as the Manpower Development Research Corporation, the RAND Corporation, or the Urban Institute.

1 The Regional Educational Laboratory at AEL Inc. 2 Northeast and Islands Regional Educational Laboratory at Brown University 3 Laboratory for Student Success 4 Mid-continent Research for Education and Learning 5 North Central Regional Educational Laboratory 6 Northwest Regional Educational Laboratory 7 Pacific Resources for Education and Learning 8 Southwest Educational Development Laboratory 9 The Regional Educational Laboratory at SERVE 10 Western Regional Education Laboratory at WestED 11 National Research and Development Centers Primary Site Location Center for Research on Education, Diversity, and Excellence-University of California, Santa Cruz 12 Center for Research on the Education of Students Placed At-Risk-Johns Hopkins University 13 Center for Research on Evaluation, Standards, and Student Testing-University of California, Los Angeles 14 Center for the Improvement of Early Reading Achievement-University of Michigan 15 Center for the Study of Teaching and Policy-University of Washington 16 National Center for Early Development and Learning-University of North Carolina at Chapel Hill 17 National Center for Improving Student Learning and Achievement in Mathematics and Science-University of Wisconsin 18 National Center for Postsecondary Improvement-Stanford University 19 National Center for the Study of Adult Learning and Literacy-Harvard University 20 National Center on Increasing the Effectiveness of State and Local Education Reform Efforts-University of Pennsylvania 21 National Research and Development Center on English Learning and Achievement-University of Albany 22 National Research Center on the Gifted and Talented-University of Connecticut

The new center would also be responsible for interpreting the effectiveness of the various strategies. The program evaluations will vary according to the types of information analysts are looking for. The most rigorous and statistically reliable studies will use randomized-assignment control groups-though the much higher costs associated with randomized experiments will limit the number of these studies.

The collection, analysis, and distribution of education statistics also need to be improved and protected from undue political interference. The National Center for Education Statistics should be continued, protected, and expanded as an independent entity with a commissioner of education statistics appointed for a six-year term. And the National Assessment Governing Board (NAGB), which now shares responsibility for overseeing NAEP, should be continued as an independent group and given additional resources to improve the quality of its work. In the past there has been some tension and confusion between NAGB and NCES over who administers the NAEP contract, how the NAEP scores are presented in terms of achievement levels, as well as when and how the results are disseminated to the public. The National Assessment Governing Board should be given the responsibility not only for setting NAEP policies, but also for administering and monitoring the NAEP contract, and releasing the results to the public. Currently, the National Assessment Governing Board sets NAEP policies, but does not monitor the contract for the program, and it should be given more control over the administration of NAEP so that the oversight and coordination of these vital assessments can be improved.

The Department of Education should continue to provide information and technical assistance to states, school districts, and local schools by funding education experts to help educators improve their overall operations or classroom teaching. But this important technical assistance work of OERI’s labs and the comparable and overlapping work of the recently created comprehensive regional assistance centers (currently overseen by the Office of Elementary and Secondary Education) should be merged and transferred to a new office of reform assistance and dissemination under an assistant secretary for planning and dissemination in the Department of Education. Moreover, states and local districts ought to be given more flexibility in deciding how to spend federal funds designated for technical assistance. They should be able to choose from among a variety of providers rather than being forced to receive those services from either the labs or the comprehensive regional assistance centers.

Same Old Song?

There has been no shortage of thoughtful suggestions for reforming federal R&D over the past three decades. Almost all reformers seem to call for more research funding; researchers with more training; more permanent and distinguished OERI leaders; more strategic planning to meet the needs of classroom teachers and students; more long-term, scientifically sound R&D projects; and preservation of the intellectual and political independence of the agency. Sound familiar? Indeed, most of these recommendations have made their way into the language of OERI’s periodic congressional reauthorizations.

Yet little seems to have changed. Structural weaknesses in the design of OERI, inadequate funding, and excessive congressional micromanagement all contribute to the inability of OERI to transform itself. But the agency’s leadership over the past 25 years must share some of the responsibility for its shortcomings; OERI directors have not always tried to recruit distinguished researchers or insisted on high-quality work from all of the agency’s grantees and contractors. Nor have all members of the education research community been sufficiently committed to making OERI a distinguished agency-especially when it meant subjecting their own federally sponsored work to rigorous evaluations or facing more frequent competitions for their funding.

Many in the education community are becoming impatient with the recitation of familiar promises to improve research and development in the near future. Unless policymakers transform OERI into a first-rate, high-quality R&D operation, some members of Congress might consider shifting some of the funding and responsibilities currently under OERI’s watch to other research and statistical agencies.

There are promising success stories in the remaking of federal efforts in education research. In the mid-1980s, the Department of Education commissioned the National Academy of Sciences to undertake a thorough review of the National Center for Education Statistics. The review panel was so disappointed with the NCES’s work that it recommended the center’s dissolution if corrective measures were not taken immediately. A few dedicated and talented individuals rose to the challenge. Working closely with the appropriate OERI staff as well as with several influential members of Congress, they managed within the space of only a few years to create an organization that is now acknowledged as a distinguished and effective federal statistical agency. Revamping OERI will take no less commitment and effort, but the result could be a rigorous and effective research agency that is able to really improve education for all of America’s children.

-Maris A. Vinovskis is a professor of history at the University of Michigan and the author of Revitalizing Federal Education Research: Improving the Regional Educational Laboratories, the R&D Centers, and the “New” OERI (University of Michigan Press, 2001).

In the summer of 1991, in response to President George H.W. Bush’s major education initiative, CEOs from a number of major corporations established the New American Schools Development Corporation (NASDC, later shortened to New American Schools, or NAS), a privately funded, nonprofit organization devoted to supporting the design and dissemination of “whole school reform” models. NASDC’s founders envisioned a complete overhaul of American education stimulated by the spread of these innovative designs. As one put it, school reformers who hoped to receive NASDC grants had to “cast aside their old notions about schooling–to start with a clean sheet of paper, and be bold and creative in their thinking, and to give us ideas that address comprehensive, systemic change for all students for whole schools.” To President Bush, NASDC represented a major step toward an “educational revolution” that would “seek nothing less than a new generation of schools.”

The idea was to apply a research -and -development model to a sector that too often fell for romantic–and untested–notions of how schools and learning should be structured. NASDC’s early leaders were determined to apply a no-nonsense business approach to their efforts, to create an organization that was as lean and agile as the corporations they led. New American Schools would be less bureaucratic and more aggressive in responding to new ideas than the typical government agency or major foundation. Moreover, NASDC would be a sort of venture capitalist for education, constantly evaluating its investments and continuing to fund only those designs that proved their effectiveness.

Initially, the plan for NASDC was to complement the creation of 535 federally funded “break the mold” schools by 1996. Each of the 435 congressional districts in the country would be home to one of these schools; two more schools per state would be funded as well. Each school would receive a one-time federal grant of $1 million in start-up funds. The privately funded NASDC would support R&D teams tasked with designing innovative reform models to be implemented in these schools.

NASDC’s original plan didn’t include supporting the implementation of these designs. But as the political environment soured early on, it became increasingly clear that NASDC wouldn’t have 535 federally funded laboratories in which to test its designs. In light of this, NASDC had to become more than just an R&D center; it would have to support not only the R&D efforts of the design teams, but also the dissemination and implementation of their ideas. Moreover, if it was to keep its promise of acting as a kind of venture capitalist for education reform, at some point these design teams would need to become self-sustaining, taking fees from school districts in return for their expertise.

Over a decade, this mandate would transform New American Schools from a privately funded R&D center that would support only effective whole-school reform models into a savvy service provider for research teams that needed to market and fund their designs. Their ability to sell those designs, in turn, was heavily dependent on the ability of New American Schools to lobby Congress for more federal funding of whole-school reform. In some ways, these changes strengthened NASDC’s claim of being a bold new venture in education reform, since few privately funded initiatives had ever committed to so ambitious a program. But along the way New American Schools lost sight of its mission and became more of a typical Washington insider than an education revolutionary.

Progressivism Rides Again

NASDC’s initial funding competition sought innovative ideas for creating schools in which all students would reach “world class standards in English, mathematics, science, history, and geography” and be “prepared for responsible citizenship, further learning, and productive employment.” The “request for proposals,” or RFP, process also required that the designs be replicable in other communities: “This is not a request to establish ‘model’ schools. The designs must be adaptable so that they can be used by many communities to create their own schools. The important thing is that long after NASDC has disappeared from the scene, its legacy for new designs will remain.” NASDC funds would support the start-up of these break-the-mold schools, but once they were running it was expected that they would operate on the same budgets as conventional schools.

Once NASDC chose the 11 winning proposals (see sidebar below), it established a three-phase process for refining, implementing, and marketing these designs. Phase one provided one year of funding, during which the teams fine-tuned their original proposals “into workable designs for school transformation.” At the end of phase one, NASDC dropped two of the designs, the Bensenville New American Schools and the Odyssey Project. Phase two provided an additional two years of NASDC funding for the design teams to implement (and further refine) their ideas and approaches. At the end of phase two, NASDC determined that two more designs, Community Learning Centers and Los Angeles Learning Centers, would not receive further support (although the Los Angeles group eventually rejoined New American Schools with a new name, Urban Learning Centers, and a more focused mission). Finally, phase three envisioned a two-year scaling-up process in which the remaining designs would be put into practice in a “large number of schools across the country.” NASDC stuck to its timetable for phases one and two, but the scale-up phase, which began in mid-1995, has taken longer and, in essence, continues today.

The popular and media response to the NASDC initiative was generally upbeat and positive, but a small, vocal group of critics repeatedly questioned NASDC and the designs. Initially the negative assessments came primarily from liberal critics of the Bush administration. Typical was a 1992 article in the Nation, in which the authors declared: “Most of the educational R&D teams endorsed by the corporation comprise an incestuous circle of right-wing ideologues and privatization advocates, teacher-hating technocrats and recession-rocked military contractors, their funding made palatable to the press by token support for established and respected liberal school-reform advocates.”

Others argued that NASDC had misplaced priorities and ignored such pressing issues as equalizing funding for poor districts and addressing the needs of minorities. Former U.S. commissioner of education Harold Howe II, for example, derided NASDC’s business-oriented approach, declaring, “You can bet that, when these parties get together, they will spend little time in making a diverse society work.”

Historians David Tyack and Larry Cuban wondered whether the New American Schools initiative was just the latest example of “ambitious reformers” promising “to create sleek, efficient school machines ‘light years’ ahead of the fusty schools of their times,” yet ultimately being overcome by the education system’s unreceptive environment. They argued that such efforts to reinvent schools have in practice “often resembled shooting stars that spurted across the pedagogical heavens, leaving a meteoric trail in the media but burning up and disappearing in the everyday atmosphere of the schools.”

These criticisms were provocative, but ultimately proved less than prophetic. Contrary to Howe’s prediction, the New American Schools designs have focused overwhelmingly on improving education for urban and minority students. And NASDC and its design teams, rather than “disappearing in the everyday atmosphere of the schools,” have survived for a decade, mainly due to their ability to adapt to the changing political and pedagogical environment.

Ironically enough, the attacks from conservatives have proven more prescient. They maintained that the RFP process was skewed to ensure the choice of safe rather than break-the-mold designs. The competition was more like those run by government agencies or major foundations than an agile start-up venture. This gave an advantage to experienced grant writers, and most of the winning proposals came from established education reformers and their organizations.

The competition rounded up not only many of the usual educational suspects but the usual educational ideas, too. Lynn Olson of Education Week pointed out that one “striking feature of the award-winning designs is how many ideas they have in common.” A more striking feature was how many of those ideas were deeply influenced by the questionable theories of the progressive education movement of the early 20th century, the original break-the-mold movement in American education. Indeed, most of the winning proposals read like echoes of progressive manifestos of the 1920s. Six of the eleven designs, for example, expressly declared that they were “learner-centered.” Eight sought to change the relationship between teachers and students by transforming the teacher into a “coach,” “facilitator,” or “guide.” All promised to meet “world class” curriculum standards, but only two (Expeditionary Learning and Modern Red Schoolhouse) focused their programs on students’ explicitly mastering academic disciplines. Nearly all the designs promoted interdisciplinary curricula to avoid what they saw as a key problem of traditional schools: teaching knowledge and skills in isolation.

Related to this interdisciplinary focus was the widely shared commitment to use the progressive-inspired “project method” to engage the interest of students. Eight NASDC designs proclaimed that project-based education would make learning exciting and relevant to students in ways that traditional textbook-centered education never could. Eight proposals envisioned using the community as a classroom to break down barriers between school and society, an idea championed by John Dewey, the great patriarch of progressive education. Like Dewey and other progressives, most of the proposals also argued that their design would not lead to the “piling up of facts,” but would produce critical thinkers.

The problem, as critics such as Robert Maynard Hutchins, E. D. Hirsch Jr., and Diane Ravitch have repeatedly shown, is that apart from a small number of schools serving highly motivated, well-to-do students, progressive education has rarely fulfilled its pedagogical promise. These critics have argued that progressive ideas often contribute to the evisceration of academic curricula and narrow the scope of student learning to entertaining but largely vapid topics. Especially relevant to the NASDC designs was the late Jeanne Chall’s finding, after surveying a host of research studies, that, on average, schools guided by progressive ideas have been less successful in raising academic achievement, especially among disadvantaged children.

It was, in fact, controversies over progressive ideas that were at least partly responsible for the Bensenville and Odyssey Project designs’ collapse in the face of angry community challenges. Both designs were grassroots efforts developed by local communities; neither was guided by or associated with the work of a national education leader. Still, among the most contentious elements of both designs was their progressive reorganization of the schools. A sociological study of the North Carolina-based Odyssey Project found that parents were bothered by the lack of course grades and college prep courses in the Odyssey curriculum. They saw Odyssey as an attempt to shift the curriculum to a more vocation-oriented track that would ensure a “compliant, trained workforce” for local industry.

These developments should have sent a warning to NASDC leaders about the politically charged and therefore tenuous nature of such broad efforts at education reform. These debacles showed that the success of NASDC-style reform–no matter how good the design nor how much money available–would depend ultimately on keeping all the community’s stakeholders committed to change over the long haul.

Regardless, phase one had gone generally well for the other nine design teams, and NASDC leaders plowed ahead undaunted. As businessmen, they recognized that any new venture has some failures and losses. More troubling for them at the time were two emerging problems–NASDC’s precarious financial situation and concerns about the organization’s survival following the defeat of George Bush by Bill Clinton in 1992.

Scale-Up

In its first two years of operation, NASDC raised only about $50 million of a planned $200 million. Concerns about fund-raising deepened with fears that NASDC was too closely tied to the Bush administration to win the new president’s support.

In May 1993, however, both President Clinton and Secretary of Education Richard Riley stated that the NASDC’s reform efforts dovetailed nicely with the new administration’s education agenda. In fact, this was no real stretch for Clinton. He had represented the nation’s governors at the 1989 meeting at which Bush and the governors had agreed on this agenda, and his administration embraced many of the same goals.

Yet Clinton’s support had little immediate effect on fund-raising, and NASDC watchers began asking whether the organization could remain in business much longer. In response, the NASDC board elected David Kearns CEO. Kearns, who had previously served as deputy secretary of education in the Bush administration and as CEO of Xerox, was widely respected in political, education, and business circles. The board hoped he would find ways to shore up the organization’s precarious financial situation. In January 1994, these hopes were borne out when philanthropist Walter Annenberg donated $50 million to NASDC–a life-saving development for the organization. Without these funds it was unlikely that NASDC could have made the crucial transition from serving a modest number of schools to its broader scale-up to the state and district level.

Further enhancing NASDC’s national profile was the fact that its key scale-up idea–moving from reforming individual schools to transforming entire systems–was quite similar to the idea of systemic reform that was one of the cornerstones of the Clinton administration’s education policy. NASDC was now freed from the allegation that it was little more than a partisan GOP initiative. In the process, however, NASDC also appeared to be distancing itself from its “revolutionary” free-market approach to education reform and was increasingly looking like a respectable if rather corporate member of the education establishment.

With its status bolstered, NASDC turned to its biggest challenge: scale-up. Between 1992 and 1995, nine NASDC-supported design teams had implemented their plans in 148 schools. Phase three called for a dramatic increase, with projections reaching into the thousands. At the time, however, none of the designs had demonstrated that it would lead to dramatic improvements in student achievement (although Robert Slavin’s Success for All program, the cornerstone of the Roots and Wings initiative, had shown promising results).

Nevertheless, NASDC moved forward with its scale-up plan. Agreements were ultimately reached with the states of Kentucky and Maryland and nine large urban districts: Cincinnati, Broward County (Fort Lauderdale), Dade County (Miami), Memphis, Philadelphia, Pittsburgh, San Antonio, San Diego, and Seattle (and a group of contiguous school systems). By mid-1997, two years into phase three, the number of participating schools had climbed to 553.

Memphis would become NASDC’s signature district. In 1997 its student population was more than 80 percent African-American, and more than 60 percent of all students qualified for free or reduced-price lunch. Only 40 percent of its 9th graders were passing the state’s minimum-competency test. However, Superintendent Naomi “Gerry” House was determined to use whole-school reform to try to solve Memphis’s education ills. By 1997, Memphis had almost reached the NASDC goal for jurisdictions participating in the scale-up of having 30 percent of its schools adopting some form of whole-school design. Encouraging news about student achievement soon followed. A study comparing 25 NASDC school sites in 1995-96 with a control group of non-NASDC schools found “significantly greater gains” in the redesigned schools.

Meanwhile, less encouraging news was emanating from the RAND Corporation’s studies of NASDC’s phases one and two. RAND, with which NASDC had been working from the beginning, found that one of the main problems in the early stages was the grating between these innovative designs and the superstructure of state and district policies. This spelled trouble for the scale-up phase, when entire systems would be expected to convert to these new designs while keeping such major influences as union-negotiated contracts and state policies largely intact. For instance, in a RAND survey of principals in NASDC-supported schools, a large majority reported that “standardized, multiple-choice tests are misaligned with the classroom practices of reforming schools.” This is a common criticism among all educators, but here the NASDC principals were voicing a more pointed concern, namely that there was a mismatch between the student-centered approaches of the design teams and the more traditional evaluation practices used by states and districts.

Moreover, a 1995 survey of parents in several NASDC jurisdictions found that they were deeply divided about reform, with over a third of the respondents wanting “the best of new and innovative methods”; just under a third wanting “tried and true” basics; and the remaining third wanting “a combination of old and new methods.” Getting such communities to commit to long-range plans for whole-school transformations would be a formidable challenge. This indicated a process of change that was more incremental and political than the NASDC teams had expected, as they would soon learn in Memphis. The early phase indicated that NASDC’s original perspective was somewhat simplistic, even naive. NASDC could not simply apply the business metaphor–”develop and test a new product and then go sell it”–to educational change.

As a result, by 1995, NASDC leaders rarely spoke about creating break-the-mold schools; the words “education revolution” had disappeared entirely. The realities of education politics were forcing NASDC to adapt and adjust its approach and expectations.

Congress to the Rescue

By mid-1997, NASDC, now named New American Schools, or NAS, was well into becoming more of a clearinghouse than an initiator of reform and was changing the design teams into market-oriented purveyors of education improvement. NAS increasingly focused on helping the teams become self-sufficient organizations capable of marketing and supporting their products. For some teams, this change was relatively painless. Success for All, the parent organization of Roots and Wings, was already a fee-for-service institution with substantial experience in this area. But others, such as the Modern Red Schoolhouse, had virtually no business or marketing experience. Thus, during phase three, NAS began sending financial and marketing experts to the design teams to help them develop business plans and marketing strategies. At the same time, NAS began addressing the problem of creating markets for the designs.

The most immediate problem for NAS and the teams was finding funds that districts could use to pay for their services, since design-team fees for the first year of implementation alone could range from $50,000 to more than $500,000 per school. This effort was especially crucial given that so many of the clients targeted by NAS were low-performing, financially strapped urban systems.

Help came in the form of the Comprehensive School Reform Demonstration Program, better known as the Obey-Porter Amendment. Passed by Congress in 1997, it provided about $150 million for competitive grants to aid schools in adopting “proven” whole-school reform models. Supporters of the amendment routinely cited the NAS designs as representative of the kinds of programs they hoped school districts would adopt. In fact, NAS marketing efforts and Obey-Porter funds have brought a five-fold increase since 1997 in the number of schools using NAS designs, to some 3,000 schools in 2001. Not surprisingly, NAS had lobbied Congress on behalf of the amendment.

One could view Obey-Porter as helping NAS fulfill its original role, namely acting as a private R&D agency whose designs would be implemented in schools that received federal start-up funds. Yet if Obey-Porter aided NAS in fulfilling an important goal, it also indicated that NAS was changing from an agency committed to operating outside the constraints of Washington into one that depended on Washington for the success of its venture.

However, the most troubling aspect of Obey-Porter and, by implication, of the new role played by NAS design teams in education reform, was not the dependency on federal funding but the claims of “proven effectiveness” that the law explicitly made on behalf of the designs. With the possible exception of Roots and Wings, at the time that Obey-Porter was enacted the NAS designs still couldn’t provide substantial evidence of effectiveness.

Disappointing Results

Indeed, NAS officials’ reaction to the 1999 American Institutes of Research (AIR) study An Educator’s Guide to Schoolwide Reform would reveal an organization that had not come to terms with the importance of evaluating student achievement in the reform process. Of 24 whole-school reform designs, AIR found that only three had “strong” positive effects on student achievement: Direct Instruction, High Schools That Work, and Success for All, none of which were NAS designs. Of the eight NAS programs, AIR rated Expeditionary Learning “promising” and Roots and Wings (the NAS version of Success for All) “marginal” in boosting achievement. All the others received no rating because there weren’t enough data.

New American Schools president John Anderson’s reaction, in a 1999 column for Education Week, was to maintain that the AIR review was flawed in its approach and conclusions. “Not surprisingly,” he argued, “AIR took a traditional approach, gathering the relatively small body of published research on the programs as its evidence of effectiveness. But this sort of reliance on limited and often time-consuming research as a way of assessing comprehensive designs may be outmoded.” Anderson held that “traditional” evaluation took too long and ignored data on progress that schools were gathering on an almost daily basis. In addition, he wrote, “Even the most careful research on a handful of campuses and a handful of matched schools could not give us a picture of what happens when models are used in hundreds of schools.” Anderson seemed troubled by the attention AIR gave to student achievement in making its assessments. “Beyond academic results,” he asserted, “design teams should be judged on other practices involving their relationship with schools and school districts. Practices involving implementation, communication, and data collection are all vital to viable campus change.”

Was Anderson implying that the NAS designs were so novel that standard approaches to research could not capture what they were doing? Was he questioning all traditional external reviews of the NAS programs? If so, these concerns didn’t stop him from citing data from a traditionally designed study on student achievement in Memphis as evidence that the designs in that city were working.

Meanwhile, the data finally produced by RAND on the performance of NAS schools in scale-up jurisdictions relative to other schools in the same district showed mixed results. About half of the NAS schools in Cincinnati, Memphis, Miami, and Kentucky (the cities where RAND studied math performance) outperformed the district average in mathematics, while 50 percent or more of the schools in Cincinnati, Philadelphia, San Antonio, and Washington (the cities where RAND studied reading performance) improved in reading. In other words, about half of the NAS schools in the cities studied by RAND only met the district average or underperformed relative to other district schools. RAND identified Roots and Wings as the most consistent design, with 10 of 21 schools making progress in both reading and mathematics relative to the district. Modern Red Schoolhouse did even better, with 7 of 11 of its schools improving in math and 8 in reading. Other designs showed considerable variation. Of the Audrey Cohen schools, for example, 5 of 8 schools improved in math, but only 2 in reading. Similarly, 9 of 24 ATLAS schools and only 4 of 16 Expeditionary Learning schools improved in math.

The inescapable finding from RAND was that, despite millions of dollars and enormous effort, over the course of a decade what NAS had brought about was at best incremental change and modest achievement gains. RAND’s mixed assessment was: “Dramatic achievement gains were not made, although some designs fared better than others.”

Another blow to NAS’s performance record was dealt in June 2001, when the Memphis school district announced that it was abandoning the district’s six-year experiment in whole-school reform. In the spring of 2000, when Johnnie Watson took over as superintendent from Gerry House, the great promoter of whole-school reform, he found a deeply troubled district. In the late 1990s, House had mandated that all of Memphis’s more than 160 schools adopt a reform model, a policy that angered and alienated many teachers. Worse, an internal district study found that after six years of reform and the expenditure of some $12 million, Memphis students had exhibited virtually no gains on state tests. Moreover, the study revealed that only 3 of 18 reform models implemented in the city had boosted student achievement; none of the three was a NAS design. Indeed, one NAS design, ATLAS, had “the most negative impact” on student achievement of any of the reform models. The report found that even Success for All, which was being used in almost 25 percent of the Memphis schools, performed poorly. There have been questions about the quality of the study, but this merely underscores one of NAS’s fundamental problems: the lack of independent, reliable research on the effectiveness of its designs, a deficiency that allows decisions like Memphis’s to be made more on the basis of politics than sound evidence about what works.

These events should signal to NAS that the future of whole-school reform rests as much on the fluid and unpredictable nature of educational politics, changes in leadership, and the sufferance of parents and teachers as it does on successful marketing, buy-in from key administrators, and substantial federal funds. Moreover, the Memphis experience highlights the importance of the bottom line–student achievement–for all such reform efforts.

How will Memphis’s rejection of whole-school reform affect NAS? While it is certainly too soon to offer a definitive answer, it seems likely that NAS will weather this storm, although its momentum may be somewhat slowed. The flow of Obey-Porter and, increasingly, Title I funds to support whole-school reform will sustain NAS and its design teams for many years to come.

Lost Chances

At its inception New American Schools held the promise of being an extremely exciting research-and-development initiative in education. It would sponsor the creation of innovative designs, pilot test them in a select group of schools, and decide whether they were effective enough to warrant wide dissemination. Its evolution into a financier, lobbyist, and marketing shop for a variety of as-yet-unproven whole-school designs must be considered at best a modest contribution to the cause of education improvement. In its current incarnation, NAS is almost indistinguishable from the foundations and government agencies that have dominated school reform efforts for decades. It has become part of the very education establishment its leaders once sought to circumvent.

More important than the changed status of NAS is the increasing prominence of NAS-style whole-school reform as one of the most popular strategies for education change. NAS has played a key role in that development. Yet its own experience in Memphis raises questions about whether the commitment to whole-school reform is the best strategy for improving educational outcomes in troubled urban schools. With so many unmanageable and unpredictable variables, whole-school reform will always be tenuous, because it attempts to simply glide by many of the institutional constraints–the variety of programs operating within a school, all with different goals; the requirements of the administrators’ and teacher unions’ contracts–that make large-scale change so tortuous. University of Michigan historian Maris Vinovskis and other scholars have suggested that less ambitious and less costly efforts such as more learning time during the summer might yield gains as great as or greater than comprehensive school reform.

One of Johnnie Watson’s first moves after ending Memphis’s experiment with whole-school reform was to order a new district-wide curriculum. A primary reason was the problem the district had experienced in adopting so many different designs. One teacher noted: “Since Memphis has such a mobile student population students fell even further behind because nothing was the same from school to school.” High rates of student mobility are an ever-present feature of all urban school districts. Yet the NAS design teams did not appear to have taken that vital fact of urban life and education into consideration when they developed their programs. The seeds for significant gains may lie in such small insights about student mobility rates and district-wide curricula.

-Jeffrey Mirel is a professor of educational studies and history at the University of Michigan and the author of The Rise and Fall of an Urban School System: Detroit, 1907-81 (University of Michigan Press, 1993).