Subjective performance evaluation holds the promise of measuring what matters. When evaluators are allowed to exercise their own judgment, rather than following a formal decision rule, however, the subjective measure may be corrupted by such behaviors as favoritism. One type of subjective evaluation, onsite inspection, is nonetheless used in many school systems around the world. In-class evaluations by external assessors have been proposed recently in the United States for the K–12 sector, as well as for the Head Start preschool program. Yet there is very little evidence to date on the validity of inspection ratings and the effectiveness of inspection-based accountability systems in improving school quality.

This study evaluates a subjective performance-evaluation regime in place in the English public school system since the early 1990s. Under this regime, independent inspectors visit schools, assess schools’ performance, and disclose their findings on the Internet. Inspectors combine hard metrics, such as test scores, with softer ones, such as observations of classroom teaching, in order to arrive at an overall judgment of school quality. Schools that receive a fail rating may be subject to sanctions, such as more frequent and intensive inspections.

I provide evidence on the effectiveness of several aspects of the inspections system. First, I demonstrate that inspection ratings can aid in distinguishing between more- and less-effective schools, even after controlling for test scores and various other school characteristics. Second, exploiting a natural experiment, I show that a fail inspection rating leads to test-score gains for primary school students that remain evident even after the students move into secondary schools. I find no evidence that schools that receive a fail rating are able to inflate test-score performance by gaming the system, suggesting that oversight by inspectors may mitigate such strategic behavior.

The English School Inspection System

The English public schooling system combines centralized testing with school inspections. Over the period relevant to this study, tests took place when students were age 7, 11, 14, and 16; these are known as the Key Stage l to Key Stage 4 tests, respectively. Successive governments have used the results of Key Stage tests, especially Key Stages 2 and 4, as performance measures when holding schools to account.

Inspectors examine students’ work and engage in discussions with students and parents.

Since the early 1990s, a government agency called the Office for Standards in Education, or Ofsted, has inspected all English public schools. Ofsted has three primary functions: 1) to offer feedback to the school principal and teachers; 2) to provide information to parents to aid their decisionmaking process; and 3) to identify schools that suffer from “serious weakness.” Although Ofsted employs its own in-house team of inspectors, the agency contracts out the majority of inspections to a handful of private-sector and nonprofit organizations via a competitive bidding process. Ofsted retains responsibility for setting overall strategic goals and objectives, putting in place a framework to guide the inspection process, and monitoring the quality of inspections.

Over the time period covered by this study, schools were generally inspected once during each three- to six-year inspection cycle. An inspection involves an assessment of a school’s performance on academic and other measured outcomes, followed by an onsite visit to the school, typically lasting one or two days for primary schools. Inspectors arrive at the school on very short notice (maximum of two to three days), which should limit schools’ ability to make last-minute preparations for the visit. Inspections take place throughout the academic year, September to July.

During the onsite visit, inspectors collect qualitative evidence on performance and practices at the school. A key element of this is classroom observation. In addition, inspectors hold in-depth interviews with the school leadership, examine students’ work, and engage in discussions with students and parents. The evidence gathered by the inspectors during their visit, as well as test-performance data, form the evidence base for each school’s inspection report. The school receives an explicit headline grade, ranging between l (“Outstanding”) and 4 (“Unsatisfactory,” also known as a fail rating). The full inspection report is made available to students and parents and is posted on the Internet.

There are two categories of fail, a moderate fail (known as “Notice to Improve”) and a more severe fail (“Special Measures”), which carry different sanctions. Schools that receive a moderate fail rating are subject to additional inspections, with an implicit threat of a downgrade to the severe fail category if inspectors judge improvements to be inadequate. Schools that receive the severe fail rating may experience more dramatic consequences: these can include changes in the school leadership team and the school’s governing board, increased resources, as well as increased oversight from the inspectors.

Over the period, September 2006 to July 2009, 13 percent of schools received the best rating, “Outstanding”; 48 percent received a “Good” rating; 33 percent received a “Satisfactory” rating; and 6 percent received a “Fail” rating. The fail group included 4.5 percent of schools receiving the moderate fail rating and 1.5 percent of schools receiving the severe fail rating.

Official policy statements indicate that inspectors place substantial weight on test scores, which is borne out by analysis of the data. A decline of 10 national percentile points on a school’s test performance in the year before inspection is associated with a 3 percentage point rise in the likelihood of being rated fail, taking into account the proportion of students eligible for free lunch, as well as the local authority in which the school is located. Nevertheless, test scores are not the only measure inspectors use to rate schools. Around 25 percent of schools that had scored in the bottom quarter nationally on the test were rated Outstanding or Good during the 2006 to 2009 period.

Validating Inspection Ratings

I first investigate whether inspection ratings convey any information on school quality beyond what is captured by test-score rankings. The critical question is whether inspectors visiting the school are able to gather and summarize information about school quality that is not already publicly available. If inspectors rely mostly or exclusively on test scores to arrive at the overall rating, then these ratings will not provide new information to educators, parents, and policymakers.

I test the validity of the inspection ratings by examining to what extent these ratings can forecast measures of school quality not observed by the inspectors, after taking into account the measures they do observe. I construct two measures of school quality—student perceptions of teacher practices and parent satisfaction—using data from the Longitudinal Study of Young People in England (LSYPE), a major survey supported by the Department for Education. Students age 14 are asked how likely teachers are to: take action when a student breaks rules, make students work to their full capacity, keep order in class, assign homework, check that any homework that is assigned is done, and grade students’ work. Parents are asked about their satisfaction with the interest teachers show in the child, school discipline, child’s school progress, and feedback from teachers.

I combine the student questions into a single measure of student perceptions of teacher practices and the parent questions into a single measure of parent satisfaction. I then examine whether these survey measures, which are not observed by the inspectors, are higher in schools that received better inspection ratings, controlling for various characteristics of the schools and survey respondents. For this analysis, school characteristics taken into account include national percentile test rank, the proportion of students eligible for a free lunch, whether the school is secular or religious, and the local education authority in which it is located. Student factors include prior test score, gender, ethnic background, parents’ education, income and economic activity, and whether the family receives government benefits.

Click to enlarge

My results confirm that lower inspection ratings are associated with sharply declining school quality as measured by student perceptions of teacher practices. The strength of this relationship may be gauged by comparing the change in quality associated with changes in the school’s position in the national test-score ranking: the results show that an increase of 50 percentile points is associated with an increase of 0.15 standard deviations in student perceptions of teacher practices (see Figure 1). A two-unit improvement in the inspection rating, such as from Satisfactory to Outstanding, is associated with an even larger increase of 0.21 standard deviations.

Results for the parent satisfaction measure are very similar to those reported for the teacher practices measure. A two-unit increase in the inspection rating is associated with an increase of 0.17 standard deviations in the parent satisfaction measure. The relationship between test scores and parental satisfaction, however, is statistically insignificant after controlling for inspection ratings. In short, this analysis confirms that inspection ratings can help detect differences in teacher practice and parental satisfaction among schools with similar test-score rankings and socioeconomic composition.

The Effect of a Fail Inspection on Test Scores

What is the effect of a fail inspection on students’ subsequent test scores? The challenge to answering this question is that receiving a fail rating is based at least partly on past test performance. Schools that have a bad year on the standardized test are more likely to receive a fail rating when they are next inspected. If the low score is due in part to bad luck, the score is likely to increase the next year, toward the school’s typical performance. Thus, schools that receive fail ratings may appear to improve in the following year for reasons other than the fail rating.

I address this concern by comparing schools inspected early in the year to those inspected late in the year. This analysis exploits a specific feature of the English testing system, namely, that the age-11 tests take place each year over five days in the second week of May. The results are released in mid-July. Schools that are inspected and receive a fail rating early in the academic year can respond to that rating and potentially improve their scores by the time of the May test. But schools that are failed later in the year—in particular, those that are failed after mid-May—cannot. I therefore estimate the effect of receiving a fail rating by comparing the May test results for schools inspected very early in the same academic year, the treatment group, with a comparison group of schools inspected after the test is taken in early May but before the results are released in July. The key idea is that inspectors have the same information on past test scores for both groups of schools.

I conduct this analysis using mathematics and English test scores for schools failed in one of the four academic years, 2005–06 to 2008–09. The key comparison is between students enrolled in schools that received a fail rating in the early part of the academic year, September to November (the treatment group) with those attending schools that received a fail rating late in the academic year, mid-May to mid-July (the control group). It is important to bear in mind that this methodology does not compare the effect of attending a school that received a fail rating with the effect of attending a school that received a higher rating.

The validity of this approach is supported by the fact that the treatment and comparison groups in general have very similar student and school characteristics. The proportion of students receiving a free school lunch, the proportion of students who are white British, student performance on the age-11 test in the prior year, and the school’s inspection rating from the previous inspection round are all similar, on average, in the treatment and control schools.

The results indicate that the effect of receiving a fail rating is to raise standardized test scores in a school by 0.12 standard deviations in math and by 0.07 to 0.09 standard deviations in English. These gains, which roughly equate to between one-third and one-half a year of typical instruction, are especially noteworthy given that they can only reflect the efforts of schools made between an inspection in the period from September to November and the tests administered in May, a maximum of eight months.

Testing for Strategic Behavior

An outstanding question is whether these improvements reflect strategic behavior by schools that face strong incentives to improve their test scores. These strategies could include the removal of low-performing students from the testing pool, teaching to the test, and targeting students close to the mandated proficiency threshold. I conduct three tests for the presence of these types of strategic responses.

First, I examine to what extent gains in test scores following the fail rating are accounted for by selectively removing low-performing students. Specifically, I examine whether the results change when I adjust my results to account for differences in student characteristics, including prior (age 7) test scores; gender; eligibility for free lunch; special education needs; month of birth; whether first language is English; ethnic background; and census information on the home neighborhood deprivation index. I find that controlling for these factors in the analysis has little impact on the estimated effect of receiving a fail rating. In other words, it doesn’t appear that schools try to game the system by systematically discouraging certain groups of students from taking the exam.

Second, I investigate whether there is any evidence that teachers target students on the margin of attaining “Level 4” proficiency; the percentage of students attaining that proficiency level is the key government target for age-11 students. Following a fail rating, the incentives to maximize students passing over the threshold are more intense than prior to the fail rating. Schools may therefore try to target resources toward students on the margin of attaining this threshold, to the detriment of students far below and far above.

I address this issue by examining whether the fail rating effect varies by students’ prior ability and find a strong inverse relationship between prior ability and the effects of attending a school that received a fail rating. The fail rating effect for students with test scores in the bottom quarter prior to the treatment year is 0.20 and 0.14 standard deviations in mathematics and English, respectively (see Figure 2). Students in the middle of the prior test-score distribution also experience substantial gains of roughly 0.10 to 0.12 standard deviations in math and 0.08 to 0.10 standard deviations in English. The gains for students with prior scores in the top quarter are the smallest, at 0.05 and 0.03 standard deviations in mathematics and English, respectively.

Why are the effects of a fail rating largest for students with low prior test scores? One potential explanation relates to differences within the schools in the degree to which parents are able to hold teachers accountable. Parents of children scoring low on the age-7 test are poorer than average and may be less able to assess their child’s progress and the quality of instruction provided by the school. Teachers may exert lower levels of effort for students whose parents are less vocal about quality of instruction. My results suggest that, following a fail rating and the subsequent increased oversight of schools, teachers increase their effort. This rise in effort may be greatest where previously there was the greatest slack.

Finally, I examine whether any gains in test scores in the year of the fail rating are sustained in the years following the inspection. This provides an indirect test of the extent of teaching to the test, as gains due to crude test-prep strategies are less likely to persist over time than gains produced by improved instruction. Specifically, I examine whether the effects on age-11 test scores can be detected when the students are tested again at age 14, three years after the students have left the primary school. This is a fairly stringent test of gaming behavior, because prior research has found evidence of “fade-out” of test-score gains even when there are no strong incentives to boost test scores artificially.

The results show that a fail rating raises average math and English test scores by 0.05 standard deviations three years after leaving the primary school. These medium-term gains are largest for lower-performing students, in line with earlier results showing large gains for these groups in the year of inspection.

Conclusion

How best to design incentives for public organizations such as schools is a fundamental public policy challenge. One solution, performance evaluation on the basis of test scores, is prevalent in many countries. This paper evaluates an alternative approach, school inspections, which may better capture the multifaceted nature of education production. A key concern under such a regime is that it is open to manipulation.

My first set of results demonstrates that inspector ratings are correlated with student- and parent-reported measures of school quality, even after controlling for test-score results and other school characteristics. In other words, inspectors are able to discriminate between more- and less-effective schools, and, significantly, report their findings even when the stakes are high. Simply disseminating inspection ratings and reports may therefore better inform consumers and other decisionmakers in the education sector.

My main finding is that receiving a fail inspection rating leads to test-score improvements of around 0.1 standard deviations. There is little evidence to suggest that schools are able to inflate test performance artificially by gaming the system. If inspectors are able to evaluate actual practices and instructional quality at the school, both before and after an inspection, then inspections may well have a mitigating effect on such unintended responses.

Finally, the data reveal that the fail rating effects are especially large for students with low prior test scores. The gains are large when compared to other possible policy interventions, such as the effects of attending a school with higher average achievement levels or enrolling in a charter school. These results are consistent with the view that children of low-income parents, arguably the least vocal in holding teachers accountable, benefit the most from inspections. Consequently, the findings of this study may be especially relevant in the current policy environment where, first, there is heightened concern about raising standards for this group of children and, second, these students are hard to reach using other policy levers.

Iftikhar Hussain is a lecturer in the Department of Economics at the University of Sussex.

Alyesha Taveras (left) graduated from high school in 2012 and is currently enrolled at Seton Hall University.

As the controversy raged in the late 1990s, a group of philanthropists created the New York School Choice Scholarships Foundation (SCSF), which offered three-year vouchers worth up to $1,400 annually to as many as 1,000 low-income families with children who were either entering 1st grade or were public school students about to enter grades two through five. Due to excess demand, SCSF established a lottery for interested families. If a family met the eligibility criteria and won the SCSF lottery, all of that family’s children entering grades one through five would receive a voucher. Recipients could attend any one of the hundreds of participating private schools, religious or secular, within New York City.

According to the Archdiocese of New York, average tuition in the city’s Catholic schools, the city’s largest private provider, was $1,728, which was 72 percent of the total per-pupil cost of $2,400 to educate a child at these schools. The scholarship would thus cover only a portion of the costs of the private education of eligible students. SCSF initially committed to making the scholarships available for a period of three years.

SCSF asked an independent research team to conduct an experimental evaluation of the impact of the intervention on student achievement and other outcomes, such as school climate and school quality, as reported by the students’ parents or other guardians. More than 20,000 students expressed interest in a voucher and were invited to one of five separate eligibility verification and testing sessions. To participate in the lottery, students other than those who had yet to begin 1st grade were required to take a standardized test. While students were taking the test, the adult accompanying the child answered questions about the child’s family background and the current school the child attended. All families were asked to supply identifying information for each child applying for a scholarship, including full name and date of birth.

Families who won the voucher lottery were told that scholarship renewal was dependent on participation in annual testing at a designated site other than the child’s school. Families who lost the lottery were compensated for participating in subsequent testing sessions, and their children were given additional chances to win the lottery. Those who won a subsequent lottery were dropped from the evaluation control group. Those families who won the lottery but who did not make use of the scholarship were also compensated for participating in subsequent testing sessions. The original evaluation identified, after three years, large positive effects of the voucher opportunity on the test scores of African Americans but not on the test scores of students from other ethnic groups.

In this paper, we extend the original evaluation of the SCSF program by estimating impacts of the offer of a voucher on college enrollment. Our results provide the first experimental evidence of the effects of a voucher intervention on this outcome. The study is also notable for obtaining information on college enrollments for 99 percent of study participants, greatly reducing the potential for bias due to attrition from the evaluation. We find large positive impacts on college enrollment for African American students but not for Hispanic students. Impact data for the small group of students from other backgrounds are too noisy to produce reliable evidence.

Evidence on College Enrollment

Chelsea Gil and Geanylyn Romero both graduated from high school in 2012 and are currently enrolled at the Borough of Manhattan Community College.

Few experimental evaluations have estimated the long-term impacts of interventions taking place during the regular years of schooling. Public school choice for disadvantaged students in the Charlotte-Mecklenburg school district in North Carolina was shown to reduce incarceration rates, especially among high-risk students (see “Does School Choice Reduce Crime?” research, Spring 2012). Another study found that class-size reduction in Tennessee’s K–3 classrooms increased college enrollment rates by about 6 percentage points among African American students, although no impacts were observed for white students.

The scarcity of experimental studies of long-term outcomes is especially true when it comes to school voucher research. One recent study using data from Washington, D.C., did identify positive impacts of a voucher program on high-school graduation rates. No studies have yet reported impacts on college enrollment, due in part to the challenges of following students long enough and obtaining accurate information on their postsecondary careers.

Fortunately, almost all colleges and universities in the United States, representing more than 96 percent of all college students, now submit enrollment information to the National Student Clearinghouse (NSC). We used the names and dates of birth of SCSF scholarship applicants, collected at eligibility verification sessions, to match them to NSC records. The information needed to make this match was available for 2,637 of the 2,666 students in the original sample.

Methods

Our primary outcome of interest is the overall (part-time and full-time) college enrollment within three years of expected (i.e., on-time) high-school graduation. We focus on this three-year window (the exact dates of which vary according to the student’s grade when enrolling in the study) because the most recent enrollment data available are for fall 2011 and the youngest cohort was expected to graduate high school in 2009. We also report the effects of the voucher offer on full-time enrollment; enrollment in four-year colleges; enrollment in private colleges; and enrollment in selective colleges.

Jason Tejada is currently in his senior year at Columbia University, on a full scholarship.

We identify students as not having enrolled in college if they are not matched to any NSC records. Some measurement error of college enrollment is possible. For example, a student who enrolled in college but whose birth date was incorrect in our records would be counted as a nonenrollee. This type of measurement error is unlikely to bias our estimates because there is no reason to believe it is related to whether a student won the school-choice lottery. Our results could be biased, however, if being offered a voucher affected enrollment in the small share of colleges that do not participate in the NSC.

We estimate the effects on college enrollment of simply being offered a voucher, even if it is not used to enroll in a private school, as well as the effects of actual voucher use. The effect of the voucher offer is referred to as an intent-to-treat (ITT) estimate, as offering a voucher to a family is an attempt by SCSF to induce the family to make use of a private school. The ITT effect includes both the effect of voucher use for those who used it and any effects on those who were offered the voucher but declined. The impact of actually using the voucher is referred to as a treatment-on-treated (TOT) estimate, as it identifies the effects on those actually treated, that is, those who used the voucher to attend a private school. The TOT analysis assumes that winning the lottery had no impact on college enrollment among students who never used a voucher.

Of the 2,637 students included in our analysis, 1,358 students won the lottery and were therefore assigned to the treatment group. The remaining 1,279 students were assigned to the control group. All the students who applied for a voucher were socioeconomically disadvantaged, as only low-income families were eligible to participate. Nearly half of the students came from families in which neither parent had attended college. The vast majority of students were African American or Hispanic; the performance of the average student when tested before names were entered into the lottery was between the 17th and 25th percentile of students nationwide.

Although African American and Hispanic students had fairly similar scores on the baseline achievement test, students in these groups differed in a number of respects. While 42 percent of all students in the control group enrolled in college within three years of expected high-school graduation, only 36 percent of African American students in the control group did so, compared to 45 percent of Hispanic students. African American students in the treatment and control groups were more likely than Hispanic students to be male and more likely to have a parent with a college education. They were also more likely to come from one-child families and from families with four or more children.

As would be expected in a randomized experiment, students in the treatment and control groups—both overall and across African Americans and Hispanics—had similar characteristics on average. In other words, it appears that the randomization worked in producing groups of students that were comparable before the intervention began.

Most, but not all, students offered a voucher used it at some point. The share of lottery winners using the scholarship they were offered declined from 74 percent in the first year after the initial offer to 55 percent in the third year. Over the first three years after the initial offer, the average member of the treatment group used a scholarship for 1.9 years. Among students who used the scholarship for any of the first three years, the average length of time a scholarship was used was 2.5 years. SCSF later extended its initial three-year commitment and, over all of the years observed in our data, the average member of the treatment group used a scholarship for 2.6 years. Among those who used the scholarship for at least 1 year, the average is 3.4 years. Scholarship usage patterns did not vary notably between African American and Hispanic students.

Results

We find that the offer of a voucher increased college enrollment within three years of the student’s expected graduation from high school by 0.7 percentage points, an insignificant impact. This finding, however, masks substantial variation in impacts among students from different ethnic groups. We find evidence of large, statistically significant impacts on African Americans, but fairly small and statistically insignificant impacts on Hispanic students. We discuss results for the small number of students from other groups below.

Click to enlarge

The SCSF-NSC linked data indicate that a voucher offer increased the college-enrollment rate of African Americans by 7 percentage points, an increase of 20 percent. If an African American student used the scholarship to attend private school for any amount of time, the estimated impact on college enrollment was 9 percentage points, a 24 percent increase over the college enrollment rate among comparable African American students assigned to the control group (see Figure 1). This corresponds to 3 percentage points for every year the voucher was used.

The impact of a voucher offer on the college-enrollment rate of Hispanic students is a statistically insignificant 2 percentage points. Although that estimate is much smaller than the one observed for African Americans, the impacts on the two ethnic groups are not significantly different from one another.

We obtain similar results for full-time college enrollment. Among African Americans, 26 percent of the control group attended college full-time at some point within three years of expected high-school graduation. The impact of a voucher offer was to increase this rate by 7 percentage points, a 25 percent increment. Among students using the voucher to attend a private school, the estimated impact was 8 percentage points, or roughly 31 percent. No statistically significant impact on full-time college enrollment was evident for Hispanic students.

Only 9 percent of the African American students in the control group attended a private four-year college. The offer of a voucher raised that proportion by 5 percentage points, an increase of 58 percent. That extraordinary increment may reflect the tight connections between private elementary and secondary schools and private institutions of higher education.

The percentage of African American students in the control group who attended a selective four-year college was 3 percent. That increased by 4 percentage points if the student received the offer of a voucher, a better than 100 percent increment in the percentage enrolled in a selective college, a very large increment from a very low baseline. Once again, no impacts were detected for Hispanic students.

Explaining Group Differences

The estimated impact of the voucher offer on college enrollment was roughly 5 percentage points greater for African American students than for Hispanic students, raising the question of why such a difference is observed between these two groups, both of which came from socioeconomically disadvantaged families.

We do not know for sure why larger impacts were observed for African American students than for Hispanic students, but it appears that the African American students in the study had fewer educational opportunities in the absence of a voucher. As noted above, Hispanic students were considerably more likely to attend college in the absence of a voucher opportunity. There is also some evidence that the public schools attended by Hispanic students were superior to those attended by African American students. When asked to rate the overall quality of the child’s school at baseline, the parents of Hispanic students gave an average rating of 2.63 (on a 4-point, GPA-type scale), compared to the 2.29 rating given by the parents of African Americans.

Given this disparity, it is not surprising that the impact of a voucher offer on school quality (as reported by parents) was generally larger for African American students than it was for Hispanic students. Survey data from the first-year follow-up indicate that a voucher offer reduced the number of reported problems at the school attended by 1.1 (out of 6 problems listed) for African Americans but by only 0.5 problems for Hispanics. Also, Hispanic parents in the control group continued to rate their children’s schools more favorably than did African American parents. All in all, it seems that the voucher option was less critical for Hispanic students than for African American students.

A possible alternative explanation focuses on motivations for moving from public to private school. Many Hispanic families may have been seeking a voucher opportunity for religious reasons, while most African American families had secular education objectives in mind. Eighty-five percent of Hispanic students were Catholic, the same religion as that of the most extensive network of private schools in New York City. Only 19 percent of African American families said their religious affiliation was Catholic. Sixty-five percent said they were Protestant, but there are very few Protestant and other non-Catholic religious schools in New York City.

Further, 71 percent of the Hispanic respondents said they attended religious services weekly, while only 47 percent of African American respondents said they did. When treatment-group parents with children in private schools were asked in the third-year follow-up study which type of school their child was attending, 93 percent of Hispanic respondents said it was a Catholic school and 71 percent of the African American respondents gave the same response. In that same follow-up survey, 39 percent of the Hispanic respondents listed religious considerations as one of the reasons they had sought a scholarship, compared to just 33 percent of African American respondents (though this last difference is not statistically significant).

The small group of students in the study from other ethnic backgrounds was diverse and less likely to use the voucher when it was offered to them, so we are hesitant to interpret their results. The group consists of 196 treatment and 127 control students, including 91 white students, 14 Asian students, 78 students from another background, and 140 students for whom information on ethnicity was not supplied. For this group as a whole, the estimated impact of the voucher offer on college enrollment within three years of expected graduation has a negative sign but is imprecisely estimated.

If we separate out white or Asian students, other-race students, and those for whom information on race is unavailable, the estimated effects of the voucher offer are all negative, but only the effect for white or Asian students is statistically significant. This group includes only 105 students, however, and we find that the treatment and control groups did not have similar characteristics at the beginning of the study. Consequently, we do not place much weight on this negative effect.

Conclusions

The magnitude of the voucher impact on African American students may seem unexpectedly large given the modest nature of the intervention: a partial-tuition scholarship of no more than $1,400 annually. Among all those offered a voucher, the average length of time a voucher was used was less than three years.

The impact is not substantially greater than that observed in other studies, however. Using a similar definition of scholarship use (receipt of any scholarship assistance), the evaluators of the federally funded Washington, D.C., voucher program estimated a positive impact of 21 percent on the high-school graduation rates of study participants, 88 percent of whom were African Americans. That is just short of the 24 percent impact on college-going for the New York City African American students in our study.

The impacts on college enrollment we estimate are somewhat larger than those of the much more costly class-size intervention in Tennessee. Susan Dynarski and her colleagues find that being assigned to a smaller class in the early elementary grades increased college enrollment rates among African Americans by 19 percent (6 percentage points on a base of 31 percent). Reduction of class size in Tennessee cost roughly $12,000 per student, whereas the SCSF voucher intervention cost the foundation about $4,200 per student, but reduced costs to the taxpayer by lowering the number of students who required instruction in public schools. Had the government paid for the voucher, the expenditure could have taken the form of a simple transfer from the public sector to the private sector, which in the long run need not add to the per-pupil cost of education.

The impact of the voucher offer we observe for African American students is also much larger than the impact of exposure to a highly effective teacher. Raj Chetty and his colleagues (see “Great Teaching,” research, Summer 2012) report that being assigned to an elementary school teacher who is 1 standard deviation more effective than the average teacher boosted college enrollment for students in a very large city by 0.5 percentage points at age 20, relative to a base of 38 percent, an increment of 1.25 percent. If one extrapolates that finding (as those researchers do not) to three years of highly effective teaching, the impact is 3.75 percent. The 24 percent impact we identify for African American students is many times as large.

The reader should be cautioned, however, that the results from any experiment cannot be easily generalized to other settings. For example, scaling up voucher programs would surely change the social composition of private schools. To the extent that student learning depends on peer characteristics, the impacts reported here could change. But the results of this investigation nonetheless advance our understanding of the effects of school choice policies by providing the first experimentally generated information on the long-term impact of a voucher intervention.

Matthew M. Chingos is a fellow in the Brookings Institution’s Brown Center on Education Policy. Paul E. Peterson is professor of government and director of the Program on Education Policy and Governance at Harvard University and senior fellow at the Hoover Institution at Stanford University.

Costs are no less a concern in K–12 education. Until the 2008 financial crisis and the subsequent slowdown in U.S. economic growth, per-pupil expenditures on elementary and secondary education had been steadily rising. The number of school personnel hired for every 100 students more than doubled between 1960 and the first decade of the 21st century. But in the past few years, local property values have stagnated and states have faced intensifying fiscal pressure. As a result, per-pupil expenditures have for the first time in decades shown a noticeable decline, and pupil-teacher ratios have begun to shift upward (see “Public Schools and Money,” features, Fall 2012). With the rising cost of teacher and administrator pensions, the squeeze on school districts is expected to continue.

A subject of intense discussion is whether advances in information technology will, under the right circumstances, permit increases in productivity and thereby reduce the cost of instruction. Greater, and smarter, use of technology in teaching is widely seen as a promising way of controlling costs while reducing achievement gaps and improving access. The exploding growth in online learning, especially in higher education, is often cited as evidence that, at last, technology may offer pathways to progress (see Figure 1).

However, there is concern that at least some kinds of online learning are of low quality and that online learning in general depersonalizes education. It is important to recognize that “online learning” comes in a dizzying variety of flavors, ranging from simply videotaping lectures and posting them online for anytime access, to uploading materials such as syllabi, homework assignments, and tests to the Internet, all the way to highly sophisticated interactive learning systems that use cognitive tutors and take advantage of multiple feedback loops. Online learning can be used to teach many kinds of subjects to different populations in diverse institutional settings.

Click to enlarge

Despite the apparent potential of online learning to deliver high-quality instruction at reduced costs, there is very little rigorous evidence on learning outcomes for students receiving instruction online. Very few studies look at the use of online learning for large introductory courses at major public universities, for example, where the great majority of undergraduate students pursue either associate or baccalaureate degrees. Even fewer use random assignment to create a true experiment that isolates the effect of learning online from other factors.

Our study overcomes many of the limitations of prior studies by using the gold standard research design, a randomized trial, to measure the effect on learning outcomes of a prototypical, interactive online college statistics course. Specifically, we randomly assigned students at six public university campuses to take the course in a hybrid format, with computer-guided instruction accompanied by one hour of face-to-face instruction each week, or a traditional format, with three to four hours of face-to-face instruction each week. We find that learning outcomes are essentially the same: students in the hybrid format pay no “price” for this mode of instruction in terms of pass rates, final-exam scores, or performance on a standardized assessment of statistical literacy. Cost simulations, although speculative, indicate that adopting hybrid models of instruction in large introductory courses has the potential to reduce instructor compensation costs quite substantially.

Research Design

Our study assesses the educational outcomes generated by what we term interactive learning online (ILO), highly sophisticated, web-based courses in which computer-guided instruction can substitute for some (though usually not all) traditional, face-to-face instruction. Course systems of this type take advantage of data collected from large numbers of students in order to offer each student customized instruction, as well as to enable instructors to track students’ progress in detail so that they can provide more targeted and effective guidance.

We worked with seven instances of a prototype ILO statistics course at six public university campuses (including two separate courses in separate departments on one campus). The individual campuses include, from the State University of New York (SUNY): the University at Albany and SUNY Institute of Technology; from the University of Maryland: the University of Maryland, Baltimore County, and Towson University; and from the City University of New York (CUNY): Baruch College and City College.

We examine the learning effectiveness of a particular interactive statistics course developed at Carnegie Mellon University (CMU), considered a prototype for ILO courses. Although the CMU course can be delivered in a fully online environment, in this study most of the instruction was delivered through interactive online materials, but the online instruction was supplemented by a one-hour-per-week face-to-face session in which students could ask questions or obtain targeted assistance.

The exact research protocol varied by campus in accordance with local policies, practices, and preferences, but the general procedure followed was 1) at or before the beginning of the semester, students registered for the introductory statistics course were asked to participate in our study and offered modest incentives for doing so; 2) students who consented to participate filled out a baseline survey; 3) study participants were randomly assigned to take the class in a traditional or hybrid format; 4) study participants were asked to take a standardized test of statistical literacy at the beginning of the semester; and 5) at the end of the semester, study participants were asked to take the standardized test of statistical literacy again, as well as to complete another questionnaire.

Of the 3,046 students enrolled in these statistics courses in the fall 2011 semester, 605 agreed to participate in the study and to be randomized into either a hybrid- or traditional-format section. An even larger sample size would have been desirable, but the logistical challenges of scheduling at least two sections (one hybrid section and one traditional section) at the same time, to enable students in the study to attend the statistics course regardless of their (randomized) format assignment, restricted our prospective participant pool to the limited number of “paired” time slots available. Also, student consent was required in order for researchers to randomly assign them to the traditional or hybrid format. Not surprisingly, some students who were able to make the paired time slots elected not to participate in the study. All of these complications notwithstanding, our final sample of 605 students is in fact quite large in the context of this type of research.

The baseline survey administered to students included questions on students’ background characteristics, such as socioeconomic status, as well as their prior exposure to statistics and the reason for their interest in possibly taking the statistics course in a hybrid format. The end-of-semester survey asked questions about their experiences in the statistics course. Students in study-affiliated sections of the statistics course took a final exam that included a set of items that was identical across all the participating sections at that campus. The scores of study participants on this common portion of the exam were provided to the research team, along with background administrative data and final course grades of all students (both participants and, for comparison purposes, nonparticipants) enrolled in the course.

The participants in our study are a diverse group. Half come from families with incomes less than $50,000 and half are first-generation college students. Less than half are white, and the group is about evenly divided between students with college GPAs above and below 3.0. Most students are of traditional college-going age (younger than 24), enrolled full-time, and in their sophomore or junior year.

The data indicate that the randomization worked properly in that traditional- and hybrid-format students in fact have very similar characteristics overall. The 605 students who chose to participate in the study also have broadly similar characteristics to the other students registered for introductory statistics. The differences that do exist are quite small. For example, participants are more likely to be enrolled full-time but only by a margin of 90 versus 86 percent. Their outcomes in the statistics course are also comparable, with participants earning similar grades and being only slightly less likely to complete and pass the course than nonparticipants.

An important limitation of our study is that while we were successful in randomizing students between treatment and control groups, we could not randomize instructors in either group and thus could not control for differences in teacher quality. Instructor surveys reveal that, on average, the instructors in traditional-format sections were much more experienced than their counterparts teaching hybrid-format sections (median years of teaching experience was 20 and 5, respectively). Moreover, almost all of the instructors in the hybrid-format sections were using the CMU online course for either the first or second time, whereas many of the instructors in the traditional-format sections had taught in this mode for years.

The “experience advantage,” therefore, is clearly in favor of the teachers of the traditional-format sections. The questionnaires also reveal that a number of the instructors in hybrid-format sections began with negative perceptions of online learning, which may have depressed the performance of the hybrid sections. The hybrid-format sections were somewhat smaller than the traditional-format sections, however, which may have conferred some advantage on the students randomly assigned to the hybrid format.

Learning Outcomes

Our analysis of the experimental data is straightforward. We compare the outcomes for students randomly assigned to the traditional format to the outcomes for students randomly assigned to the hybrid format. In a small number of cases—4 percent of the 605 students in the study—participants attended a different format section than the one to which they were randomly assigned. In order to preserve the randomization procedure, we associated students with the section type to which they were randomly assigned. This is sometimes called an “intent to treat” analysis, but in this case it makes little practical difference because the vast majority of students complied with their initial assignment.

Our analysis controls for student characteristics, including race/ethnicity, gender, age, full-time versus part-time enrollment status, class year in college, parental education, language spoken at home, and family income. These controls are not strictly necessary, since students were randomly assigned to a course format. We obtain nearly identical results when we do not include these control variables, just as we would expect given the apparent success of our random assignment procedure.

We first examine the impact of assignment to the hybrid format, relative to the traditional format, on students’ probability of passing the course, their performance on a standardized test of statistics, and their score on a set of final-exam questions that were the same in the two formats. We find no clear differences in learning outcomes between students in the traditional- and hybrid-format sections. Hybrid-format students did perform slightly better than traditional-format students on the three outcomes, achieving pass rates that were about 3 percentage points higher, standardized-test scores about 1 percentage point higher, and final-exam scores 2 percentage points higher, but none of these differences is statistically significant (see Figure 2).

Click to enlarge

It is important to note that these non-effects are fairly precisely estimated. This precision implies that if there had been pronounced differences in outcomes between traditional-format and hybrid-format groups, it is highly likely that we would have found them. In other words, we can be quite confident that the actual effects were in fact close to zero, and therefore differ from a hypothetical finding of “no significant difference” that may result from excessively noisy data or an insufficiently large sample.

We also calculate results separately for subgroups of students defined in terms of various characteristics, including race/ethnicity, gender, parental education, primary language spoken, score on the standardized pretest, hours worked for pay, and college GPA. We do not find any consistent evidence that the hybrid-format effect varies by any of these characteristics. There are no groups of students that benefited from or were harmed by the hybrid format consistently across multiple learning outcomes.

In addition, we examine how much students liked the hybrid format of the course, and find that students gave the hybrid format a modestly lower overall rating than their counterparts gave the traditional-format course (the rating was about 11 percent lower). By similar margins, hybrid students report feeling that they learned less and that they found the course more difficult. But there were no notable differences in students’ reports of how much the course raised their interest in the subject matter.

We also asked students how many hours per week they spent outside of class working on the statistics class. Hybrid-format students report spending 0.3 hours more each week, on average, than traditional-format students. This difference implies that in a course where a traditional section meets for three hours each week and a hybrid section meets for one hour, the average hybrid-format student would spend 1.7 fewer hours each week in total time devoted to the course, a difference of about 25 percent. This result is consistent with nonexperimental evidence that ILO-type formats can achieve the same learning outcomes as traditional-format instruction in less time, which has potentially important implications for scheduling and the rate of course completion.

Potential Savings

In other sectors of the economy, the use of technology has increased productivity, measured as outputs divided by inputs, and often increased output as well. Our study shows that a leading prototype hybrid-learning system did not increase outputs (student learning) but could potentially increase productivity by using fewer inputs.

It would seem to be straightforward to compare the side-by-side costs of the hybrid version of the statistics course and the traditional version. The problem, however, is that contemporaneous comparisons can be nearly useless in projecting long-term costs, because the costs of doing almost anything for the first time are very different from the costs of doing the same thing numerous times. This is especially true in the case of online learning, where there are substantial start-up costs that have to be considered in the short run but are likely to decrease over time. For example, the development of sophisticated hybrid courses will be a costly effort that would only be a sensible investment if the start-up costs were either paid for by others (foundations and governments) or shared by many institutions.

There are also transition costs entailed in moving from the traditional, mostly face-to-face model to a hybrid model that takes advantage of more sophisticated ILO systems employing computer-guided instruction, cognitive tutors, embedded feedback loops, and some forms of automated grading. Instructors need to be trained to take full advantage of such systems. On unionized campuses, there may also be contractual limits on section size that were designed with the traditional model in mind but that do not make sense for a hybrid model. It is possible that these constraints would be changed in future contract negotiations, but that too will take time.

We address these issues by conducting cost simulations based on data from three of the campuses in our study. Our basic approach is to start by looking, in as much detail as possible, at the actual costs of teaching a basic course in traditional format (usually, but not always, the statistics course) in a base year. Then, we simulate the prospective, steady-state costs of a hybrid version of the same course. These exploratory simulations are based on explicit assumptions, especially about staffing, which allow us to see how sensitive our results are to variations in key assumptions.

We did exploratory simulations for two types of traditional teaching models: 1) students taught in sections of roughly 40 students per section, and 2) students attending a common lecture and assigned to small discussion sections led by teaching assistants. We focus on instructor compensation because these costs comprise a substantial portion of the recurring cost of teaching and are the most straightforward to measure. We compare the current compensation costs of each of the two traditional teaching models to simulated costs of a hybrid model in which most instruction is delivered online, students attend weekly face-to-face sessions with part-time instructors, and the course is overseen by a tenure-track professor.

These simulations are admittedly speculative and subject to considerable variation depending on how a particular campus organizes its teaching, but they suggest that significant cost savings are possible. In particular, we estimate savings in compensation costs for the hybrid model ranging from 36 percent to 57 percent compared to the all-section traditional model, and 19 percent compared to the lecture-section model.

These simulations confirm that hybrid learning offers opportunities for significant savings, but that the degree of cost reduction depends (of course) on exactly how hybrid learning is implemented, especially the rate at which instructors are compensated and section size. A large share of cost savings is derived from shifting away from time spent by expensive professors toward both computer-guided instruction that saves on staffing costs overall and time spent by less-expensive staff in Q and A sessions.

Our simulations substantially underestimate the savings from moving toward a hybrid model in many settings because we do not account for space costs. It is difficult to put a dollar figure on space costs because capital costs are difficult to apportion accurately to specific courses, but the difference in face-to-face meeting time implies that the hybrid course requires 67 to 75 percent less classroom use than the traditional course.

In the short run, institutions cannot lay off tenured faculty or sell or demolish their buildings. In the long run, however, using hybrid models for some large introductory courses would allow institutions to expand enrollment without a commensurate increase in space costs, a major savings relative to what institutions would have to spend to serve the same number of students with a traditional model of instruction. In other words, the hybrid model need not just “save money”; it can also support an increase in access to higher education. It serves the access goal both by making it more affordable for the institution to enroll more students and by accommodating more students because of greater scheduling flexibility. This flexibility may be especially important for students who have to balance family and work responsibilities with course completion, as well as for students who live far from campus.

Conclusions

In the case of online learning, where millions of dollars are being invested by a wide variety of entities, we should perhaps expect that there will be inflated claims of spectacular successes. The findings in this study warn against too much hype. To the best of our knowledge, there is no compelling evidence that online learning systems available today—not even highly interactive systems, which are very few in number—can in fact deliver improved educational outcomes across the board, at scale, on campuses other than the one where the system was born, and on a sustainable basis.

This is not to deny, however, that these systems have great potential. Our study demonstrates the potential of truly interactive learning systems that use technology to provide some forms of instruction, in properly chosen courses, in appropriate settings. We find that such an approach need not affect learning outcomes negatively and conceivably could, in the future, improve them, as these systems become ever more sophisticated and user-friendly. It is also entirely possible that by reducing instructor compensation costs for large introductory courses, such systems could lead to more, not less, opportunity for students to benefit from exposure to modes of instruction such as independent study with professors, if scarce faculty time can be beneficially redeployed.

What would be required to overcome the barriers to adoption of even simple online learning systems—let alone more sophisticated systems that are truly interactive? First, a system-wide approach will be needed for a sophisticated customizable platform to be developed, made widely available, maintained, and sustained in a cost-effective manner. It is unrealistic to expect individual institutions to make the up-front investments needed, and collaborative efforts among institutions are difficult to organize, especially when nimbleness is needed. In all likelihood, major foundation, government, or private-sector investments will be required to launch such a project.

Second, as ILO courses are developed in different fields, it will be important to test them rigorously to see how cost-effective they are in at least sustaining and possibly improving learning outcomes for various student populations in a variety of settings. Such rigorous testing should be carried out in large public university systems, which may be willing to pilot such courses. Hard evidence will be needed to persuade other institutions, and especially leading institutions, to try out such approaches.

Finally, it is hard to exaggerate the importance of confronting the cost problems facing American public education at all levels. The public is losing confidence in the ability of the higher-education sector in particular to control costs. All of higher education has a stake in addressing this problem, including the elite institutions that are under less immediate pressure than others to alter their teaching methods. ILO systems can be helpful not only in curbing cost increases (including the costs of building new space), but also in improving retention rates, educating students who are place-bound, and increasing the throughput of higher education in cost-effective ways.

We do not mean to suggest that ILO systems are a panacea for this country’s deep-seated education problems. Many claims about “online learning” (especially about simpler variants in their present state of development) are likely to be exaggerated. But it is important not to go to the other extreme and accept equally unfounded assertions that adoption of online systems invariably leads to inferior learning outcomes and puts students at risk. We are persuaded that well-designed interactive systems in higher education have the potential to achieve at least equivalent educational outcomes while opening up the possibility of freeing up significant resources that could be redeployed more productively.

Extrapolating the results of our study to K–12 education is hardly straightforward. College students are expected to have a degree of self-motivation and self-discipline that younger students may not yet have achieved. But the variation among students within any given age cohort is probably much greater than the differences from one age group to the next. At the very least, one could expect that online learning for students planning to enter the higher-education system would be an appropriate experience, especially if colleges and universities continue to expand their online offerings. It is not too soon to seek ways to test experimentally the potential of online learning in secondary schools as well.

William G. Bowen is senior advisor to Ithaka S+R (the strategy and research arm of ITHAKA). Matthew M. Chingos is senior research consultant at Ithaka S+R and a fellow at the Brookings Institution’s Brown Center on Education Policy. Kelly A. Lack is a research analyst at Ithaka S+R, where Thomas I. Nygren is a former business analyst.

Data from the Bureau of Labor Statistics (BLS) show that school district costs for teachers’ health insurance rose at an average annual rate of 4 percent above inflation from 2004 to 2012. In 2004, health insurance costs tacked 11.4 percent onto teacher earnings; in 2012, they added 15.5 percent. At roughly $560 per pupil per year, the national average masks wide variation across states, as districts in some states have relatively low insurance costs while costs borne by districts in other states are quite high. The data do not include health costs for other school employees and retirees, which can be quite substantial.

In this study, we examine BLS data to compare the costs to districts for teacher health insurance with similar costs to private-sector employers. We find that insurance costs for teachers are 26 percent higher than they are for private-sector professionals, and this is partly explained by greater unionization in the public sector. We also examine data newly available from Wisconsin to quantify the impact of that state’s recent change in collective bargaining law: we find a reduction in district costs of 13 to 19 percent, the result of lower-cost policies and higher teacher contributions.

Comparing Employer Costs

We begin with a basic, high-level question: How do employer health care costs for teachers compare with those for private-sector professionals? The most comprehensive national data published on employer costs, the BLS National Compensation Survey (NCS), provide estimates of employer insurance costs on a “per-hour-worked” basis for 180 groups of employees, broken down by occupational groups, industries, ownership (private industry or state and local government), and other characteristics. These data do not separate health from other insurance costs (life and disability) for teachers, but these other components are small (approximately 5 percent of the total), so this does not significantly affect our results.

We focus our comparisons on K–12 teachers and private-sector professionals. Using unpublished data provided to us by the BLS, we multiply the hourly employer insurance costs by the number of hours worked to obtain annual costs for each group of workers. Some 97 percent of K–12 teachers work full-time, while 83 percent of private-sector professionals do so. Because part-time workers are less likely than full-time workers to have health insurance from their employers, we adjust the private-sector comparison data to match the percentage of teachers who work full time.

Click to enlarge

We estimate from these data that the national average of annual employer insurance costs in 2012 was $8,559 for K–12 teachers, and $6,803 for private-sector professionals. The difference between the figures has increased since 2004. Annual employer insurance costs for K–12 teachers rose 67 percent, compared to 49 percent for private professionals. The gap between employer costs was just 12 percent in 2004 but rose to 26 percent by 2012 (see Figure 1).

Our estimates for employer insurance costs average the expenditures across those employees who are covered by an employer’s plan and those who are not. Employees may not be covered either because no plan is offered (an issue for part-time employees in particular) or because the employee chooses not to participate (e.g., because coverage is available through a spouse’s employer). According to the NCS Employee Benefit Survey (EBS), 87 percent of K–12 teachers participate in a health insurance plan (medical, dental, vision, or prescription drug) through their employer, compared to 80 percent of private-sector professionals (our estimate, adjusting for the part-time percentage). Consequently, the difference between teachers and private-sector workers in employer health cost per participating employee is 16 percent ($9,838 vs. $8,492).

The EBS also collects data on premiums for medical insurance (a slightly narrower category than health insurance). The medical premiums are broken out by single and family coverage, so these data allow us to examine the cost of comparable policies. We find that for single coverage, employer costs for private-sector professionals are 82 percent of those for teachers ($4,496 vs. $5,494), but for family coverage, private-sector costs are 104 percent of those for districts ($11,116 vs. $10,728), slightly higher. This is a notable shift in the last few years. As recently as 2009, the employer cost for single coverage was $1,361 higher for teachers than for private-sector professionals, compared to $998 today, and for family coverage it was $29 higher for teachers instead of $388 lower. This suggests that some school districts have begun to adjust their policies toward private-sector norms.

Click to enlarge

Employee Contributions and Total Premiums

The EBS data on medical insurance also include information on employee contributions. Together with employer costs, these data indicate that, for both single and family plans, total premiums are higher for teachers than they are for private-sector professionals. For single coverage, teachers pay a smaller share (13 percent) than do private professionals (19 percent). For family coverage, teachers contribute more (34 vs. 29 percent), which is enough to cover the higher cost of their plan. In other words, the total premium for teachers’ family coverage is more expensive than it is for private-sector professionals, but the share coming from teachers more than covers the difference (see Figure 2).

In addition to premiums, employees incur out-of-pocket costs, such as deductibles and co-payments. The EBS data indicate that one reason teachers’ insurance plans are more expensive is that features of the plans (such as lower deductibles) reduce out-of-pocket costs. Although it is accurate to say that teachers pay more to get more in the way of family coverage, it is more precise to state that they pay more up front in premiums and then pay less out-of-pocket.

Union vs. Nonunion Employees

The NCS data allow us to compare medical insurance coverage and premiums for union vs. nonunion workers, where union status is defined by whether the employee belongs to a collective bargaining unit. These breakouts are not available for K–12 teachers or private-sector professionals, but they are available for the state and local government (public) sector and the private sector. The comparisons are still informative because teachers’ health care costs track those of the public sector to some extent.

These data indicate that about 95 percent of union workers have access to employer-provided medical insurance in both the public and private sectors, and their participation rate is essentially the same in both sectors (78 to 79 percent). Nonunion workers are less likely than union workers to participate in a medical plan through their employer, in large part because their employer is less likely to offer them one. The difference from union workers is smaller in the public sector, however, where the nonunion participation rate is 68 percent, compared to 48 percent in the private sector.

In the public and private sectors, for both single and family coverage, the employer cost is higher for union workers than for nonunion workers. The total premium is significantly higher in all cases except for family coverage in the private sector, where it is about the same for union and nonunion workers. Finally, employee contributions are lower for union workers, except for single coverage in the public sector.

These patterns are the same for the state and local government sector vs. the private sector, with union and nonunion combined: higher employer costs, higher total premiums, and lower employee contributions, for both types of coverage. The unionization rate is higher for the public sector than for the private sector (50 percent vs. 14 percent in the EBS data), suggesting that unionization explains some portion of each of these patterns (see Figure 3).

Click to enlarge

But these are not the patterns we observed between K–12 teachers and private-sector professionals: they are similar for single coverage but not for family coverage. Whatever impact unionization may have, there are other factors at play.

There is one state in which we have a seemingly natural experiment in changing teacher union strength: Wisconsin. If union strength results in higher employer costs, higher total premiums, and smaller employee contributions, then the removal of teacher health benefits from collective bargaining in Wisconsin might be expected to have the opposite effect: lower employer costs, lower total premiums, and larger employee contributions. This is exactly what happened.

Wisconsin Before and After Act 10

Wisconsin was the first state in the nation with public-sector collective bargaining and has long had one of the nation’s strongest teachers unions. It has also long been a state with very expensive teacher medical insurance. Average district costs in 2011 were $8,311 and $19,356 for single and family coverage, respectively. These costs were about 50 percent and 80 percent higher than the 2011 national averages for teachers, which were $5,500 and $10,723. Although Wisconsin is in a region with higher-than-average medical premiums, this geographic factor accounts for only a minor part of the gap between Wisconsin’s district costs and the national average.

Wisconsin’s high district costs reflected both the choice of expensive plans and low teacher contributions. In 2011, teachers made no contribution at all for single coverage in 43 percent of the state’s districts, nor for family coverage in 31 percent. By comparison, the noncontributory rates in 2011 among teachers in the national data discussed above were 39 percent and 16 percent, respectively. Among private-sector professional employees, the noncontributory rates for single and family plans were lower yet, 17 percent and 9 percent.

Act 10, proposed by Governor Walker and enacted by the legislature in 2011, removed benefits from local collective bargaining, thereby giving districts greater freedom to shop for less-expensive plans and to negotiate premiums. The law also allowed districts to establish higher employee contributions. Among the provisions of Act 10 was a 12 percent floor on the employee contribution rate, which applied directly only to the state-administered plan, but now serves as a benchmark that many school districts have followed.

These changes were intended to achieve savings on district benefit costs, through adoption of plans with lower premiums and increased teacher contributions. We examine the change in medical insurance costs for the school year ending in 2012, the first to be affected by Act 10, using data from the Wisconsin Association of School Boards (WASB). These results may not represent the total impact, as not all districts have renegotiated insurance contracts. Some are under contracts with insurers predating Act 10, including those with pre–Act 10 collective bargaining agreements that have not yet expired.

Click to enlarge

We calculate estimates of yearly changes using only districts for which data are available in consecutive years. The main finding from the WASB data is a sharp drop in employer costs in 2012 after years of steady growth. District payments for their employees’ medical care increased every year from 2003 to 2011. But from 2011 to 2012, average district costs for family coverage fell by an estimated $2,010, while district costs for single coverage declined by $1,042 (see Figure 4).

These figures underestimate the district savings attributable to Act 10, since premiums were steadily rising prior to Act 10 and were expected to continue doing so. When we account for this expected growth (using average growth from 2007 to 2011), we estimate savings of $2,614 for family coverage and $1,304 for single coverage. These estimates represent declines of 13 to 19 percent from the projected district costs for 2012.

Districts saved on teacher medical insurance costs in 2012 for two reasons: reductions in total premiums and increases in the portion paid by teachers. As discussed above, Act 10 did not directly raise teacher contributions, but the 12 percent minimum it established for the state plan set a standard that districts were now free to follow. For single coverage, between 2003 and 2011 the average share of medical insurance paid by teachers drifted up slightly, from about 3 to 4 percent, followed by a jump to more than 10 percent in 2012. Similarly, for family coverage, the average teacher contribution drifted up slightly over the period, to about 5 1/2 percent, and then jumped in 2012 to more than 10 percent. These figures now place Wisconsin in the vicinity of the national average contribution rate for teachers with single coverage of 13 percent, but still far below the average for family coverage of 34 percent.

In dollar terms, teacher contributions for family coverage rose by $939 in 2012, relative to the previous trend, while total premiums for family coverage declined by $1,674. Our estimate of $2,614 for the impact of Act 10 on district costs reflects these changes. The estimated impact on total premiums accounted for two-thirds of the reduction in district costs, and the act’s impact on employee contributions comprised the other third. We find a similar breakdown for single coverage.

These data have two important limitations. First, they likely understate the share of district savings attributable to higher employee costs because some (maybe most) of the reduction in total premiums is due to a rise in employee out-of-pocket payments (such as higher deductibles). Second, these data do not tell us anything about the quantity and quality of health care provided. Efficiency may have been enhanced as employees paid more of the cost and as employers became free to shop around, but we have no hard data on this.

As a check on the WASB data, we examined data from the Wisconsin Department of Public Instruction (DPI) on districts’ fringe benefit costs for teachers. Unlike the WASB data, these data are available for all districts but do not separate out health benefits from other fringe benefits, including retirement contributions, Social Security, and life insurance. The impact of Act 10 captured by these data will therefore include not only the effect on health insurance, but also the shift of about one-half of retirement contributions from employer to employee as mandated by Act 10.

The DPI data show a steady rise in fringe benefit costs from 1998 to 2011, in both dollar amounts and as a percentage of teacher salary, with the latter measure rising from 34 percent to 51 percent over the period. After Act 10, the average benefit rate dropped 8 percentage points to 43 percent. This is still quite high by comparison with the private sector, but markedly reduced. It is likely that at least one-half and perhaps two-thirds of the $4,500  drop in district fringe-benefit costs reflects the shift in retirement contributions, but virtually all of the remainder represents the reduction in district health-benefit costs. Thus the DPI data suggest a drop of $1,500 to $2,200 in average annual district health costs per teacher.

The DPI and WASB estimates show broadly consistent evidence of a large first-year impact of Act 10 on district costs for teacher health insurance, but we can only speculate on what the future effect will be. As mentioned above, some districts have not yet been able to use their new powers because of unexpired collective bargaining contracts or insurance policies, so there are more savings to be had. Many of the underlying drivers of rising health-care costs are independent of Act 10, and over the long term these will push Wisconsin employer costs back up, but from a significantly lower starting point. Moreover, as districts gain more experience in the open health care market, unfettered by collective bargaining, it is possible that they will be able to lower the rate of growth.

It is important to note that even with the dramatic savings from Act 10, district costs and total premiums in Wisconsin are still well above the national average for teachers. Indeed, by some estimates, prior to Act 10, a number of Wisconsin districts had insurance plans that were set to trigger the federal tax on “Cadillac plans” under the Affordable Care Act of 2010, scheduled to begin for 2018. This may still be true. Thus, there will be continuing pressure to reduce costs toward the national average, especially if and when the luxury tax is implemented.

Conclusion

The national data from the Bureau of Labor Statistics indicate that annual employer insurance costs are 26 percent higher for teachers than for private-sector professionals; adjusting for higher participation rates among teachers reduces the difference to 16 percent. Direct estimates of employer costs for medical plans present a mixed picture: higher employer costs for single coverage but not for family coverage. For both categories, total medical premiums are higher for teachers than they are for private-sector professionals, but for family coverage the teachers incur the extra expenditures themselves.

Unionization is associated with higher total premiums, higher employer costs, and lower employee contributions in both the public and private sectors. This suggests that the high unionization rate among teachers plays an important role in their employers’ higher average cost. Equally important, differences in teacher union strength across states help explain the wide variation in employer and employee health-insurance costs. In some nonunion states, teacher medical benefits are not particularly generous, owing to either low-cost plans (e.g., those with high deductibles) or high teacher contributions. In Arkansas, teachers typically pay 65 or 70 percent of the premiums for family coverage (the national average is 34 percent). In other states, with strong unions, such as Wisconsin, district insurance costs can be very expensive. It is in those states that the opportunities for district cost reduction are most promising, as data from Wisconsin so clearly show.

District cost reduction would ideally derive from changes that enhance efficiency, such as greater competition for health insurance. There should be no illusions that such efficiencies will come easily. In all likelihood, a great deal of any district cost reduction will take the form of higher teacher payments toward their health care through higher contributions and increased out-of-pocket expenses. This raises the question of the role of teacher health benefits in the total compensation package. The overall size of the package will continue to be the subject of debate. It is worth briefly commenting, however, on the importance of the structure of the package.

There are three reasons that efficiency might be enhanced by reallocating some of the compensation package from employer-paid health benefits to salary. First, efficiency in health-care expenditures is more likely enhanced when employees pay for services, since price signals provide the consumer with appropriate incentives. Second, shifting compensation back to salary (in the aggregate) provides greater opportunity for districts to use salary differentials to retain and recruit higher-quality teachers. Finally, as a matter of consumer choice, not all employees may want their employers to devote, say, $20,000 out of a $70,000 compensation package to medical insurance. Take-up rates well below 100 percent suggest that many teachers ascribe less value to the medical benefits offered than they cost. Thus, both efficiency (in attracting recruits) and equity (toward non-participants) might be enhanced by such a shift. Employers can offer greater choice among health plans of varying cost, with lower subsidies, fixed in size, and higher salaries that allow employees to choose how much they want to spend on higher-cost plans. As districts under fiscal distress increasingly turn to cost-cutting measures, such potential efficiency enhancements will become all the more important.

Robert Costrell is professor of education reform and economics at the University of Arkansas and fellow at the George W. Bush Institute. Jeffery Dean is distinguished doctoral fellow at the University of Arkansas. This paper is drawn from a chapter in A Bigger Bang for Education’s Bucks (George W. Bush Institute, forthcoming).

It is widely believed that a good principal is the key to a successful school. No Child Left Behind encouraged the replacement of the principal in persistently low-performing schools, and the Obama administration has made this a requirement for schools undergoing federally funded turnarounds. Foundations have invested millions over the past decade in New Leaders for New Schools, an organization that recruits nontraditional principal candidates and prepares them for the challenges of school leadership. And the recently launched George W. Bush Institute is making the principalship a focus of its activities. Yet until very recently there was little rigorous research demonstrating the importance of principal quality for student outcomes, much less the specific practices that cause some principals to be more successful than others. As is often the case in education policy discussions, we have relied on anecdotes instead.

This study provides new evidence on the importance of school leadership by estimating individual principals’ contributions to growth in student achievement. Our approach is quite similar to studies that measure teachers’ “value added” to student achievement, except that the calculation is applied to the entire school. Specifically, we measure how average gains in achievement, adjusted for individual student and school characteristics, differ across principals—both in different schools and in the same school at different points in time. From this, we are able to determine how much effectiveness varies from one principal to the next.

Our results indicate that highly effective principals raise the achievement of a typical student in their schools by between two and seven months of learning in a single school year; ineffective principals lower achievement by the same amount. These impacts are somewhat smaller than those associated with having a highly effective teacher. But teachers have a direct impact on only those students in their classroom; differences in principal quality affect all students in a given school. We also investigate one widely discussed mechanism through which principals affect student achievement: the management of teacher transitions. Importantly, because high teacher turnover can be associated with both improvement and decline in the quality of instruction, the amount of turnover on its own provides little insight into the wisdom of a principal’s personnel decisions. We confirm, however, that teachers who leave schools with the most-successful principals are much more likely to have been among the less-effective teachers in their school than teachers leaving schools run by less-successful principals. The final component of our analysis considers the dynamics of the principal labor market, comparing the effectiveness of principals who move on to those who stay in their initial schools. Constrained by salary inertia and the historical absence of good performance measures, the principal labor market does not appear to weed out those principals who are least successful in raising student achievement. This is especially true in schools serving disadvantaged students. This is troubling, as the demands of leading such schools, including the need to attract and retain high-quality teachers despite less desirable working conditions, may amplify the importance of having an effective leader.

The Texas Database

Our analysis relies on administrative data constructed as part of the University of Texas at Dallas (UTD) Texas Schools Project. Working with the Texas Education Agency (TEA), this project has combined different data sources to create matched data sets of students, teachers, and principals over many school years. The data include all Texas publicschool teachers, administrators, staff, and students in each year, permitting accurate descriptions of the schools led by each principal.

The Public Education Information Management System (PEIMS), TEA’s statewide database, reports key demographic data, including race, ethnicity, and gender for students and school personnel, as well as student eligibility for subsidized lunch (a standard indicator of poverty). PEIMS also contains detailed annual information on teacher and administrator experience, salary, education, class size, grade, population served, and subject. Importantly, this database can be merged with information on student achievement by school, grade, and year. Beginning in 1993, Texas schools have administered the Texas Assessment of Academic Skills (TAAS) each spring to eligible students in grades 3 through 8. Our analysis therefore focuses on principals in elementary and middle schools, for whom it is possible to develop performance measures.

The personnel data combine time as a teacher and as an administrator into total experience, so it is not possible to measure tenure as a principal accurately for those who became a principal prior to the initial year of our data (the 1990–91 school year). We therefore concentrate on the years from 1995 to 2001. Over this period, we are able to observe 7,420 individual principals and make use of 28,147 annual principal observations.

Measuring Principal Quality

The fundamental challenge to measuring the impact of school leaders is separating their contributions from the many other factors that drive student achievement. For example, a school that serves largely affluent families may create the illusion that it has a great principal, when family backgrounds are the key cause of high achievement. Alternatively, a school that serves disadvantaged students may appear to be doing poorly but in fact have a great principal who is producing better outcomes than any other principal would.

Our basic value-added model measures the effectiveness of a principal by examining the extent to which math achievement in a school is higher or lower than would be expected based on the characteristics of students in that school, including their achievement in the prior year. Put another way, it examines whether some schools have higher achievement than other schools that serve similar students and attributes that achievement difference to the principal. This approach is very similar to that employed in studies that measure teacher quality using databases tracking the performance of individual students over time.

The main concern with this approach is that there may be unmeasured factors that affect school performance. Our data contain only basic information on student background characteristics, such as gender, race or ethnicity, and eligibility for subsidized lunch. As a result, we cannot control for more nuanced measures of students and their families, such as motivation or wealth. We are, however, able to control for students’ test scores from the previous year, which may well capture a lot of the characteristics that we cannot measure directly. Moreover, there are also school factors not under the direct control of the school, including the quality of teachers inherited by the principal. Below we describe alternative approaches to isolating the contributions of the current principal.

In estimating principal effectiveness, we want to minimize the influence of specific circumstances and look at the underlying stable differences in impacts. This issue is important because a principal’s impact may vary with tenure in a school. A principal’s impact on the quality of the teaching staff (whether negative or positive), for example, probably increases over time as the share of teachers who were hired on her watch rises. To account for any differences in effectiveness that are related to tenure as a principal in a given school, we begin our analysis by focusing on data from the first three years a principal leads a school.

This first analysis indicates that the standard deviation of principal effectiveness is 0.21 standard deviations of test scores (see Table 1). This is a very large figure, perhaps unbelievably large, implying that a principal at the 75th percentile of this effectiveness measure shows average achievement gains of 0.11 standard deviations (relative to the average principal), while one at the 25th percentile shows average losses of 0.15 standard deviations. These differences are even more pronounced in high-poverty schools, for which the gap between the 25th and 75th percentile principal is more than one-third of a standard deviation. On average across all schools, the impact of having a principal 1 standard deviation more effective than the average principal is as much as seven additional months of learning in a single academic year.

As noted above, this initial estimate of the variability in principal effectiveness may partly reflect differences in school characteristics that are not under the principal’s control, such as the quality of the school building, or decisions made by district administrators as well as unmeasured parental influences. As a result, it may overestimate the amount of influence principals actually have.

We begin to address this issue by measuring principal effectiveness based only on comparisons of within-school differences in student achievement growth over time. In simplest terms, we compare average student achievement gains in the same school under different principals. This method eliminates the influence of any student, school, or neighborhood characteristics that do not change over time. Its main drawback is that it ignores all differences in principal effectiveness between schools, potentially underestimating the amount of variation in principal quality. For example, if each school tends to attract principals who are similar in quality whenever it searches for a new principal, this approach will understate the true extent of variation in principal effectiveness.

We conduct this second analysis using all of the principals in our data, not just those in their first three years leading a school, because the numbers of schools with two principals observed in their first three years is quite small. (Note that re-doing the prior analysis using data on all principals does not significantly alter the results presented above.) Restricting the analysis to comparisons within schools, however, cuts our estimate of the variation in principal effectiveness in half. Even this reduced estimate is substantial, however, indicating that a 1-standard-deviation increase in principal effectiveness raises school average achievement by slightly more than 0.10 standard deviations. This impact is roughly comparable to that observed for variations in teacher effectiveness in studies that use the same kinds of within-school comparisons.

Our first two methods involved estimating effectiveness measures for individual principals and then calculating the standard deviation of those measures. Although any unmeasured school factors that are unrelated to principal quality would not bias these results, such factors would inflate our estimates of the variation in principal quality based on these approaches. We therefore employ a third approach that gauges the amount of variation in principal effectiveness directly by measuring the additional fluctuation in school average achievement gains when a new principal assumes leadership, as compared to typical fluctuations from year to year.

Focusing on the additional variation in school average achievement gains around principal transitions reduces the magnitude of the estimates. Nonetheless, the results remain educationally significant: a 1-standard-deviation increase in principal quality translates into roughly 0.05 standard deviations in average student achievement gains, or nearly  quality between schools and again ignores any tendency for a given school to attract principals of similar quality over time, suggesting that it likely understates principals’ actual impact.

Teacher Turnover

The results presented so far rely on indirect measures of principal impact, namely, student learning gains during a principal’s tenure in a school. The data do not include any observations about what a principal actually does, or fails to do, to improve learning. We now turn to an analysis of the interactions of principals with teaching staff, which bears directly on a number of current policy debates.

A primary channel through which principals can be expected to improve the quality of education is by raising the quality of teachers, either by improving the instruction provided by existing teachers or through teacher transitions that improve the caliber of the school’s workforce. Teacher turnover per se has received considerable policy attention, largely because of the well-documented difficulties that new teachers experience. The potential benefits of reducing turnover nonetheless hinge on the effectiveness of both entering and exiting teachers.

We expect highly rated principals to be more successful both at retaining effective teachers and at moving out less-effective ones. Less highly rated principals may be less successful in raising the quality of their teaching staffs, either because they are less skilled in evaluating teacher quality, place less emphasis on teacher effectiveness in personnel decisions, or are less successful in creating an environment that attracts and retains better teachers. Although better principals may also attract and hire more-effective teachers, the absence of reliable quality measures for new teachers and the fact that many principals have little control over new hires lead us to focus specifically on turnover.

Unfortunately, our data do not contain direct information on personnel decisions that would enable us to separate voluntary and involuntary transitions, and existing evidence suggests that teachers rather than principals initiate the majority of transitions. In addition, the Texas data do not match students to individual teachers, meaning that we must draw inferences about teacher effectiveness from average information across an entire grade.

With detailed information on teacher effectiveness and transitions, we could investigate whether better principals are more likely to dismiss the least-effective teachers and reduce the likelihood that the more-effective teachers depart voluntarily. In the absence of such information, however, we focus on the relationship within schools between the share of teachers that exits each grade and the average value-added to student achievement in the grade. We examine how this varies with our measures of principal quality based on student achievement gains. For example, in a school where 5th-grade students learn more than 4th-grade students, we would expect a good principal to make more changes to the 4th-grade teaching staff.

The results of this analysis confirm that the relationship between higher teacher turnover and lower average valueadded in a given grade is stronger as principal quality rises. This pattern of results is consistent with the theory that management of teacher quality is an important pathway through which principals affect school quality. The fact that less-effective teachers are more likely to leave schools run by highly effective principals also validates our measure of principal quality. If our measure was just capturing random noise in the data rather than information about true principal quality, we would not expect it to be related to teacher quality and turnover.

Principal Transitions and Quality

Along with teacher turnover, instability of leadership is often cited as an impediment to improving high-poverty and low-performing schools. Consistent with these concerns, we find that Texas schools with a high proportion of low-income students are more likely to have first-year principals and less likely to have principals who have been at the school at least six years than those serving a less-disadvantaged population. Sorting schools by initial achievement rather than poverty level produces even larger differences (see Figure 1). The proportion of principals in their first year leading a school is roughly 40 percent higher in schools in the bottom quartile of average prior achievement than in schools in the top quartile; the proportion of principals that have been at their current school at least six years is roughly 50 percent higher in schools with higherachieving students.

Yet the import of leadership turnover also depends on whether highor low-quality personnel are leaving, something prior research has been unable to address. We therefore examine whether the likelihood that a principal leaves following the third year in a school varies with her effectiveness and with the share of low-income students in the school. We observe principals making a variety of career decisions: remaining in the same school as principal, becoming a principal at another school in the same district, becoming a principal in another district, moving into a central office position, or exiting the public schools entirely. We divide principals into four equal-sized groups based on estimates of their effectiveness using the first of the three methods described above. We also limit the data to include only principals with fewer than 25 years of total experience in order to minimize complications introduced by the decision to retire.

Our results confirm that the least-effective principals are least likely to remain in their current position and most likely to leave the public schools entirely. With the exception of the schools with the lowest poverty level, however, there is not a consistent relationship between the likelihood of remaining on as principal and principal quality (see Figure 2). In high-poverty schools, for example, principals in the middle two quartiles of effectiveness are substantially more likely to remain than those in the bottom quarter. The most effective principals are more likely to remain in the same position than those in the bottom quartile, but are considerably more likely to move on than those in the middle of the quality distribution.

Another result emerging from this analysis that is troubling from a policy perspective is the frequency with which low-performing principals move to principal positions at other schools. This trend is particularly striking in high-poverty schools, where more than 12 percent of poor performers annually make such a move. In contrast, less than 7 percent of the poorest performers in more-affluent schools become principals at other schools. This may reflect the fact that it is challenging in high-poverty schools to separate the effects of school circumstances from the quality of the principal, leading district administrators to give principals from high-poverty schools a chance at a different school.

The simple conclusion, nonetheless, is that the operation of the principal labor market does not appear to screen out the least-effective principals. Instead, they frequently move to different schools, perhaps reflecting the bargain necessary to move out an ineffective leader in a public-sector organization. Potentially, this is where the superintendent enters the picture. Making good decisions on the retention and assignment of principals may be among the distinguishing characteristics of successful superintendents, a possibility that warrants additional study.

Conclusions

The role of principals in fostering student learning is an important facet of education policy discussions. Strong leadership is viewed as especially important for revitalization of failing schools. To date, however, this discussion has been largely uninformed by systematic analysis of principals’ impact on student outcomes.

Determining the impact of principals on learning is a particularly difficult analytical problem. Nevertheless, even the most conservative of our three methodological approaches suggests substantial variation in principal effectiveness: a principal in the top 16 percent of the quality distribution will produce annual student gains that are 0.05 standard deviations higher than an average principal for all students in their school.

There are many channels through which principals influence school quality, although the precise mechanisms likely vary across districts with the regulatory and institutional structures that define principal authority. Because all principals participate in personnel decisions, we have focused on the composition of teacher turnover. For the best principals, the rate of teacher turnover is highest in grades in which teachers are least effective, supporting the belief that improvement in teacher effectiveness provides an important channel through which principals can raise the quality of education.

Finally, patterns of principal transitions indicate that it is the least and most effective who tend to leave schools, suggesting some combination of push and pull factors. This pattern is particularly pronounced in high-poverty schools. It is also worrisome that a substantial share of the ineffective principals in high-poverty schools takes principal positions in other schools and districts. Clearly, much more needs to be learned about the dynamics of the principal labor market. For student outcomes, greater emphasis on the selection and retention of high-quality principals would appear to have a very high payoff.

Gregory F. Branch is program manager at the University of Texas at Dallas Education Research Center. Eric A. Hanushek is senior fellow at the Hoover Institution of Stanford University. Steven G. Rivkin is professor of economics at University of Illinois at Chicago.

Students in urban high schools are of particular concern. Populated predominantly by low-income and minority students, these schools struggle with two related problems. First, many students do not earn passing grades in early courses that are thought to be prerequisites for more-advanced subjects. Second, students are at high risk of failing to earn their high school diplomas at all. In fact, only 65 percent of black and Hispanic students graduate high school, with little evidence that the graduation gap between them and white students has changed in the last few decades. One theory for these low high-school completion rates is that failures in early courses, such as algebra, interfere with subsequent course work, placing students on a path that makes graduation quite difficult.

One increasingly popular approach to improving students’ math skills is “algebra for all,” which encourages more students to take algebra and at earlier ages. The best study of this approach, using evidence from Charlotte, North Carolina (see “Solving America’s Math Problem,” features, Winter 2013), shows that pushing students into course work for which they are ill prepared actually harms their subsequent academic achievement. A potentially promising alternative, and one we focus on here, is “double-dose” algebra, in which struggling students are given twice as much instructional time as they would normally receive. The best study of this approach, by Takako Nomi and Elaine Allensworth, examined the short-term impact of such a policy in the Chicago Public Schools (CPS), where double-dose algebra was implemented in 2003. Under that policy, students scoring below the national median on the 8th-grade math exam were required to take two periods of algebra a day during 9th grade instead of one, with the second class providing support and extra practice. Students placed in the extra classes thus received substantially more algebra instruction than other students. Nomi and Allensworth reported no improvement in 9th-grade algebra failure rates as a result of this intervention, a disappointing result for CPS. The time frame of their study did not, however, allow them to explore longer-run outcomes of even greater importance to students, parents, and policymakers.

Our study extends this work to examine the impact of CPS’s double-dose algebra policy on such longer-run outcomes as advanced math course work and performance, ACT scores, high-school graduation rates, and college enrollment rates. Using data that track students from 8th grade through college enrollment, we analyze the effect of this innovative policy by comparing the outcomes for students just above and just below the double-dose threshold. These two groups of students are nearly identical in terms of academic skills and other characteristics, but differ in the extent to which they were exposed to this new approach to algebra. Comparing the two groups thus provides unusually rigorous evidence on the policy’s impact.

We find positive and substantial longer-run impacts of double-dose algebra on college entrance exam scores, high school graduation rates, and college enrollment rates, suggesting that the policy had significant benefits that were not easily observable in the first couple of years of its existence. The benefits of double-dose algebra were largest for students with decent math skills but below-average reading skills, perhaps because the intervention focused on written expression of mathematical concepts.

Double-Dose Algebra

Since the late 1990s, Chicago Public Schools has been attempting to increase the rigor of student course work and prepare students for college entrance. Starting with students entering high school in the fall of 1997, CPS eliminated lower-level and remedial courses so that all first-time freshmen would enroll in algebra in 9th grade, geometry in 10th grade, and algebra II or trigonometry in 11th grade. Soon after, it became apparent to CPS officials that many students were unable to master the new curriculum, resulting in very low passing rates in 9th-grade algebra. The cause of this high failure rate was thought to lie largely with the poor math skills with which students entered high school.

In response to the low passing rates, CPS launched the double-dose algebra policy for all students entering high school in the fall of 2003. Instead of reinstating the traditional remedial courses from previous years, CPS required enrollment in two periods of algebra for all first-time 9th graders testing below the national median on the math portion of the 8th-grade Iowa Tests of Basic Skills (ITBS). Students enrolled for two math credits, a full-year regular algebra class plus a full-year algebra support class. Prior to the double-dose policy, algebra curricula varied considerably across CPS high schools, due to the decentralized nature of the district. With the new policy, CPS offered teachers of double-dose algebra two specific curricula called Agile Mind and Cognitive Tutor, stand-alone lesson plans they could use, and three professional development workshops each year, where teachers were given suggestions about how to take advantage of the extra instructional time.

Though it is difficult to know precisely what occurred in these extra classes, students assigned to double-dose algebra reported more frequently writing sentences to show how they solved a math problem; explaining how they solved a problem to the class; writing math problems for other students to figure out; discussing possible solutions with other students; and applying math to situations in life outside of school. The additional time spent building verbal and analytical skills may have conferred benefits in subjects other than math.

CPS also strongly advised schools to schedule their algebra support courses in three specific ways. First, double-dose algebra students should have the same teacher for their two periods of algebra. Second, the two algebra periods should be offered consecutively. Third, double-dose students should take the algebra support class with the same students who are in their regular algebra class. Most schools followed these recommendations in the initial year. In the second year, schools began to object to the scheduling difficulties of assigning the same teacher to both periods, so CPS removed that recommendation.

The policy we study had many components. Assignment to double-dose algebra doubled the amount of instructional time and exposed students to the curricula and activities discussed above. The recommendation that students take the two classes with the same set of peers increased tracking by skill level. All of these factors were likely to, if anything, improve student outcomes. We will also show, however, that the increased tracking by skill placed double-dose students among substantially lower-skilled classmates than non-double-dose students, which could have hurt student outcomes. Our results will capture the net impact of all of these factors.

Data

Our analysis focuses on the first two cohorts of students subject to the double-dose algebra policy, those entering high school in the fall of 2003 and in the fall of 2004. These two cohorts included more than 41,000 students overall. Our primary results are based on the 11,507 students with 8th-grade math test scores within 10 percentile points of the cutoff used to assign students to double-dose algebra.

As discussed above, the implementation of the policy differed somewhat between the two cohorts. For the 2003 cohort, 80 percent of double-dose students had the same teacher for both courses, 72 percent took the two courses consecutively, and rates of overlap between the two classes’ rosters exceeded 90 percent. For the 2004 cohort, only 54 percent of double-dose students had the same teacher for both courses, and only 48 percent took the two courses consecutively. Overlap between the rosters remained, however, close to 90 percent. We find that the policy’s impacts were similar for the two cohorts, so we combine them for the purpose of presenting our results.

Longitudinal data from CPS enable us to track students from 8th grade through college enrollment. These data include demographic information, detailed high-school transcripts, numerous standardized test scores, and graduation and college enrollment information. We include in our analysis all students who entered 9th grade for the first time in the fall of 2003 or 2004, who had valid 8th-grade math test scores, and who enrolled in freshman algebra in a high school in which at least one classroom of students was assigned to double-dose algebra. These CPS students were primarily from disadvantaged socioeconomic backgrounds and minority groups. About 90 percent were black or Hispanic, and 20 percent received special education services. The average CPS student scored at approximately the 45th percentile (a little below average) on the nationally normed exam used to determine which students were required to take double-dose algebra.

We focus on two sets of student outcomes. The first set measures students’ academic achievement and includes grades, course work, and standardized test scores. For example, our data include pass rates for algebra and higher-level math courses and students’ scores on the ACT (a college-entrance exam). The second set captures educational attainment, including high school graduation and college enrollment rates. We consider students to be high school graduates if they received a regular CPS diploma within five years of starting high school. About 50 percent of CPS students in our data graduated high school within four years, with another 5 percent graduating in the fifth year. College enrollment is measured using data on CPS high-school graduates from the National Student Clearinghouse. We count students as college matriculants if they enrolled in college by October 1 of the fifth year after starting high school. Only 28 percent of students in our data both graduated from a CPS high school and enrolled in college within this time frame. Of these college matriculants, 46 percent enrolled in two-year colleges and 54 percent enrolled in four-year colleges.

Method

Given the substantial differences between students who were and were not assigned to double-dose algebra, simply comparing their later outcomes would likely produce misleading evidence on the policy’s impact. To eliminate this bias, we take advantage of the fact that students scoring below the 50th percentile on the 8th-grade ITBS math test were supposed to enroll in double-dose algebra. This rule allows us to isolate the impact of double-dose algebra by comparing the outcomes of students who scored just below the cutoff to those who scored just above the cutoff. These two groups of students were very similar—their scores differed by a tiny amount—but only one group was required to take double-dose algebra. And there were no differences in the outcomes of students scoring just below and above the assignment cutoff among earlier cohorts of CPS students, suggesting that any differences we identify can be attributed to the policy.

Overall, 55 percent of CPS students scored below the 50th percentile and thus should have been assigned to double-dose algebra, but only 42 percent were actually assigned to the support class. In addition, some students took double-dose algebra, even though they scored above the cutoff on the exam. As a result, the difference in double-dose assignment rates between students just below the cutoff and students just above the cutoff was about 40 percentage points—not 100 percentage points, as would be the case if the rule were followed without exceptions. We adjust our results to reflect the fact that some above-threshold students were double-dosed and some who scored below the threshold were not. Our results should therefore be interpreted as the effect of taking double-dose algebra among students who took it because they scored below the cutoff—not for some other reason, such as a desire for additional instruction in math.

Before turning to our results on student outcomes, we first examine how the double-dose algebra policy changed students’ freshman-year experiences. Students assigned to double-dose algebra obviously spent more class periods learning math. The second class did not replace core academic courses in English, social studies, or science; it did replace other courses, such as fine arts and foreign languages. The net result was a small increase in the total number of courses taken freshman year.

Students in the double-dose classes were also more likely to be grouped with classmates of similar academic skill. Assignment to double-dose decreased the average achievement of a student’s classmates by more than 19 percentile points, and increased the size of regular algebra classes by 2.4 students. Doubling instructional time in math by replacing other course work thus increased tracking by ability and class size.

Results

Double-dosing had an immediate impact on student performance in algebra, increasing the proportion of students earning at least a B by 9.4 percentage points, or more than 65 percent. It did not have a significant impact on passing rates in 9th-grade algebra, however, or in geometry (usually taken the next year). Double-dosed students were, however, substantially more likely to pass trigonometry, a course typically taken in 11th grade. The mean GPA across all math courses taken after freshman year increased by 0.14 grade points on a 4.0 scale.

As a whole, these results imply that the double-dose policy greatly improved freshman algebra grades for the higher-achieving double-dosed students, but had relatively little impact on passing rates for the lower-achieving students. The latter fact is one of the primary reasons that CPS has continued to refine its algebra instruction policy. There is, however, some evidence of improved passing rates and GPAs in later math courses, suggesting the possibility of benefits beyond 9th grade.

Though course work and grades matter for students’ academic trajectories, the subjective nature of course grading suggests that standardized tests may be a better measure of the impact of double-dosing on math skill. We do not find consistent evidence of impacts on student performance on the preliminary ACT (called PLAN) exam taken in the fall of 10th grade, but we do find impacts on the 11th-grade PLAN (see Figure 1). On that exam, double-dosing was found to increase algebra scores by 0.15 standard deviations and overall math scores by 0.16 standard deviations. Perhaps more importantly, a nearly identical effect is seen on the math portion of the ACT (taken in the spring of 11th grade), with double-dose algebra raising scores by 0.15 standard deviations on an exam used by many colleges as part of the admissions process. This is equivalent to closing roughly 15 percent of the black-white gap in ACT scores.

These results from standardized tests suggest that double-dosed students experienced few, if any, short-run achievement gains but did experience larger gains that persisted at least two years after the end of double-dose classes. Did these gains translate into improved educational attainment? We find that they did, with double-dosing increasing four- and five-year high-school graduation rates by 8.7 and 7.9 percentage points, respectively, a 17 percent improvement (see Figure 2).

Figure 2 also shows that double-dosed students were 8.6 percentage points more likely to enroll in college within five years of starting high school, a nearly 30 percent increase over the base college enrollment rate of 29 percent. Nearly all of this increase comes from enrollment in two-year colleges, with more than half of that resulting from part-time enrollment in such colleges. Given the relatively low academic skills and high poverty rates of double-dosed CPS students, it is unsurprising that double-dosing improved college enrollment rates at relatively inexpensive and nonselective two-year postsecondary institutions.

It is important to note that many of these results are much stronger for students with weaker reading skills, as measured by their 8th-grade reading scores. For example, double-dosing raised the ACT scores of students with below-average reading scores by 0.22 standard deviations but raised above-average readers’ ACT scores by only 0.09 standard deviations. The overall impact of double-dosing on college enrollment is almost entirely due to its 13-percentage-point impact on below-average readers (see Figure 3). This unexpected pattern may reflect the intervention’s focus on reading and writing skills in the context of learning algebra.

Finally, we consider the possibility that the increased focus on algebra at the expense of other course work may have affected achievement in other subjects. We find strong evidence that rather than harming achievement in reading and science, double-dosing had positive effects across the board. Double-dosed students scored nearly 0.20 standard deviations higher on the verbal portion of the ACT, were substantially more likely to pass chemistry classes usually taken in 10th or 11th grade, and earned modestly higher GPAs across all of their nonmath classes in the years after 9th grade. In other words, the skills gained in double-dose algebra seem to have helped students in other subjects and in subsequent years.

Conclusion

Our study provides the first evidence of positive and substantial long-run impacts of intensive math instruction on college entrance exam scores, high school graduation rates, and college enrollment rates. We also show that the intervention was most successful for students with relatively high math skills but relatively low reading skills. Although the intervention was not particularly effective for the average affected student, the fact that it improved high school graduation and college enrollment rates for even a subset of low-performing and at-risk students is extraordinarily promising when targeted at the appropriate students. In this case, those were students with only moderately low math skills but below-average reading skills.

This double-dose strategy has become an increasingly popular way to aid students struggling in mathematics. Today, nearly half of large urban districts in the United States report double math instruction as the most common form of support for students with lower skills. The central concern of urban school districts is that algebra may be a gateway for later academic success, so early high-school failure in math may have large effects on subsequent academic achievement and graduation rates. With the current policy environment calling for “algebra for all” in 9th grade or earlier, effective and proactive intervention is particularly critical for those who lack foundational mathematical skills. A successful early intervention may be the best way to boost students’ long-term academic success.

Kalena Cortes is assistant professor at the Bush School of Government and Public Service at Texas A&M University. Joshua Goodman is assistant professor of public policy at the Harvard Kennedy School. Takako Nomi is assistant professor of education at St. Louis University.

In the research reported here, we study one approach to teacher evaluation: practice-based assessment that relies on multiple, highly structured classroom observations conducted by experienced peer teachers and administrators. While this approach contrasts starkly with status quo “principal walk-through” styles of class observation, its use is on the rise in new and proposed evaluation systems in which rigorous classroom observation is often combined with other measures, such as teacher value-added based on student test scores.

Proponents of evaluation systems that include high-quality classroom observations point to their potential value for improving instruction (see “Capturing the Dimensions of Effective Teaching,” Features, Fall 2o12). Individualized, specific information about performance is especially scarce in the teaching profession, suggesting that a lack of information on how to improve could be a substantial barrier to individual improvement among teachers. Well-designed evaluation might fill that knowledge gap in several ways. First, teachers could gain information through the formal scoring and feedback routines of an evaluation program. Second, evaluation could encourage teachers to be generally more self-reflective, regardless of the evaluative criteria. Third, the evaluation process could create more opportunities for conversations with other teachers and administrators about effective practices.

In short, there are good reasons to expect that well-designed teacher-evaluation programs could have a direct and lasting effect on individual teacher performance. To our knowledge, however, ours is the first study to test this hypothesis directly. We study a sample of midcareer elementary and middle school teachers in the Cincinnati Public Schools, all of whom were evaluated in a yearlong program, based largely on classroom observation, sometime between the 2003–04 and 2009–10 school years. The specific school year of each teacher’s evaluation was determined years earlier by a district planning process. This policy-based assignment of when evaluation occurred permits a quasi-experimental analysis. We compare the achievement of individual teachers’ students before, during, and after the teacher’s evaluation year.

We find that teachers are more effective at raising student achievement during the school year when they are being evaluated than they were previously, and even more effective in the years after evaluation. A student instructed by a teacher after that teacher has been through the Cincinnati evaluation will score about 11 percent of a standard deviation (4.5 percentile points for a median student) higher in math than a similar student taught by the same teacher before the teacher was evaluated.

Our data do not allow us to identify the exact mechanisms driving these improvements. Nevertheless, the results contrast sharply with the view that the effectiveness of individual teachers is essentially fixed after the first few years on the job. Indeed, we find that postevaluation improvements in performance were largest for teachers whose performance was weakest prior to evaluation, suggesting that rigorous teacher evaluation may offer a new way to think about teacher professional development.

Evaluation in Cincinnati

The data for our analysis come from the Cincinnati Public Schools. In the 2000–01 school year, Cincinnati launched the Teacher Evaluation System (TES) in which teachers’ performance in and out of the classroom is assessed through classroom observations and a review of work products. During the yearlong TES process, teachers are typically observed in the classroom and scored four times: three times by an assigned peer evaluator—a high-performing, experienced teacher who previously taught in a different school in the district—and once by the principal or another school administrator. Teachers are informed of the week during which the first observation will occur, with all other observations unannounced. Owing mostly to cost, tenured teachers are typically evaluated only once every five years.

The evaluation measures dozens of specific skills and practices covering classroom management, instruction, content knowledge, and planning, among other topics. Evaluators use a scoring rubric based on Charlotte Danielson’s Enhancing Professional Practice: A Framework for Teaching, which describes performance of each skill and practice at four levels: “Distinguished,” “Proficient,” “Basic,” and “Unsatisfactory.” (See Table 1 for a sample standard.)

Both the peer evaluators and administrators complete an intensive TES training course and must accurately score videotaped teaching examples. After each classroom observation, peer evaluators and administrators provide written feedback to the teacher and meet with the teacher at least once to discuss the results. At the end of the evaluation school year, a final summative score in each of four domains of practice is calculated and presented to the evaluated teacher. Only these final scores carry explicit consequences. For beginning teachers (those evaluated in their first and fourth years), a poor evaluation could result in nonrenewal of their contract, while a successful evaluation is required before receiving tenure. For tenured teachers, evaluation scores determine eligibility for some promotions or additional tenure protection, or, in the case of very low scores, placement in a peer assistance program with a small risk of termination.

Despite the training and detailed rubric provided to evaluators, the TES program experiences some of the leniency bias typical of other teacher-evaluation programs. More than 90 percent of teachers receive final overall TES scores in the highest two categories. Leniency is much less frequent in the individual rubric items and individual observations. We hypothesize that this microlevel evaluation feedback is more important to lasting performance improvements than the final, overall TES scores.

Previous research has found that the scores produced by TES predict student achievement gains (see “Evaluating Teacher Effectiveness,” research, Summer 2011). Student math achievement was 0.09 standard deviations higher for teachers whose overall evaluation score was 1 standard deviation higher (the estimate for reading was 0.08). This relationship suggests that Cincinnati’s evaluation program provides feedback on teaching skills that are associated with larger gains in student achievement.

As mentioned above, teachers only undergo comprehensive evaluation periodically. All teachers newly hired by the district, regardless of experience, are evaluated during their first year working in Cincinnati schools. Teachers are also evaluated just prior to receiving tenure, typically their fourth year after being hired, and every fifth year after achieving tenure.

Teachers hired before the TES program began in 2000–01 were not initially evaluated until some years into the life of the program. Our analysis only includes these pre-TES hires: specifically, teachers hired by the district in the school years from 1993–94 through 1999–2000. We further focus, given available data, on those who were teaching 4th through 8th grade in the years 2003–04 through 2009–10. We limit our analysis to this sample of midcareer teachers for three reasons. First, for teachers hired before the new TES program began in 2000–01, the timing of their first TES review was determined largely by a “phase-in” schedule devised during the program’s planning stages. This schedule set the year of first evaluation based on a teacher’s year of hire, thus reducing the potential for bias that would arise if the timing of evaluation coincided with, for example, a favorable class assignment. Second, because the timing of evaluation was determined by year of hire, and not experience level, teachers in our sample were evaluated at different points in their careers. This allows us to measure the effect of evaluation on performance separate from any gains that come from increased experience. Third, the delay in first evaluation allows us to observe the achievement gains of these teachers’ students in classes the teachers taught before the TES assessment so that we can make before-and-after comparisons of the same teacher.

Additionally, our study focuses on math test scores in grades 4–8. For most other subjects and grades, student achievement measures are simply not available. Students are tested in reading, but empirical research frequently finds less teacher-driven variation in reading achievement than in math, and ultimately this is the case for the present analysis as well. While not the focus of our research, we briefly discuss reading results below.

Data provided by the Cincinnati Public Schools identify the year(s) in which a teacher was evaluated by TES, the dates when each observation occurred, and the scores. We combine these TES data with additional administrative data provided by the district that allow us to match teachers to students and student test scores. As we would expect, the 105 teachers in our analysis sample are a highly experienced group: 66.5 percent have 10 to 19 years of experience, compared to 29.3 percent for the rest of the district. Teachers in our analysis are also more likely to have a graduate degree and be certified by the National Board for Professional Teaching Standards, two characteristics correlated with experience.

Methodology

Our objective is to measure the impact of practice-based performance evaluation on teacher effectiveness. Simply comparing the test scores of students whose teachers are evaluated in a given year to the scores of other teachers’ students would produce misleading results because, among other methodological issues, less-experienced teachers are more likely to be evaluated than more-experienced teachers.

Instead, we compare the achievement of a teacher’s students during the year that she is evaluated to the achievement of the same teacher’s students in the years before and after the evaluation year. As a result, we effectively control for any characteristics of the teacher that do not change over time. In addition, we control for determinants of student achievement that may change over time, such as a teacher’s experience level, as well as for student characteristics, such as prior-year test scores, gender, racial/ethnic subgroup, special education classification, gifted classification, English proficiency classification, and whether the student was retained in the same grade.

Our approach will correctly measure the effect of evaluation on teacher effectiveness as long as the timing of a teacher’s evaluation is unrelated to any student characteristics that we have not controlled for in the analysis but that affect achievement growth. This key condition would be violated, for example, if during an evaluation year or in the years after, teachers were systematically assigned students who were better (or worse) in ways we cannot determine and control for using the available data. It would also be violated if evaluation coincided with a change in individual teacher performance unrelated to evaluation per se. Below, we discuss evidence that our results are not affected by these kinds of issues. We also find no evidence that teachers are systematically assigned students with better (or worse) observable characteristics in their evaluation year compared to prior and subsequent years.

Results

We find suggestive evidence that the effectiveness of individual teachers improves during the school year when they are evaluated. Specifically, the average teacher’s students score 0.05 standard deviations higher on end-of-year math tests during the evaluation year than in previous years, although this result is not consistently statistically significant across our different specifications.

These improvements persist and, in fact, increase in the years after evaluation (see Figure 1). We estimate that the average teacher’s students score 0.11 standard deviations higher in years after the teacher has undergone an evaluation compared to how her students scored in the years before her evaluation. To get a sense of the magnitude of this impact, consider two students taught by the same teacher in different years who both begin the year at the 50th percentile of math achievement. The student taught after the teacher went through the TES process would score about 4.5 percentile points higher at the end of the year than the student taught before the teacher went through the evaluation.

We also find evidence that the effects of going through evaluation in the TES system are not the same for all teachers. The improvement in teacher performance from before to after evaluation is larger for teachers who received relatively low TES scores, teachers whose TES scores improved the most during the TES year, and especially for teachers who were relatively ineffective in raising student test scores prior to TES. The fact that the effects were largest for teachers who, presumably, received more critical feedback and for those with the most room for improvement strengthens our confidence in the causal interpretation of the overall results.

Our findings remain similar when we make changes to our methodological choices, such as varying the way we control for teacher experience, not controlling for teacher experience, and not controlling for student characteristics. We also examine whether our results could be biased by a preexisting upward trend in each teacher’s performance unrelated to experience or evaluation, and find no evidence of such a trend. Finally, we find no evidence that our results reflect teacher turnover from school to school or from grade to grade that causes them not to appear in our data in later years (for example, by moving to a nontested grade or leaving the Cincinnati Public Schools).

In contrast to the results for math achievement, we do not find any evidence that being evaluated increases the impact that teachers have on their students’ reading achievement. Many studies find less variation in teachers’ effect on reading achievement compared to teachers’ effect on math achievement, a pattern that is also evident in our data from Cincinnati. Some have hypothesized that the smaller differences in effectiveness among reading teachers could arise because students learn reading in many in- and out-of-school settings (e.g., reading with family at home) that are outside of a formal reading class. If teachers have less influence on reading achievement, then even if evaluation induces changes in teacher practices, those changes would have smaller effects on achievement growth.

Discussion

The results presented here—greater teacher performance as measured by student achievement gains in years following TES review—strongly suggest that teachers develop skills or otherwise change their behavior in a lasting manner as a result of undergoing subjective performance evaluation in the TES process. A potential explanation for these results is that teachers learn new information about their own performance during the evaluation and subsequently develop new skills. New information is potentially created by the formal scoring and feedback routines of TES, as well as increased opportunities for self-reflection and for conversations regarding effective teaching practice in the TES environment.

Moreover, two features of this study—the analysis sample of experienced teachers and Cincinnati’s use of peer evaluators—may increase the saliency of these hypothesized mechanisms. First, the teachers we study experienced their first rigorous evaluation after 8 to 17 years on the job. Thus they may have been particularly receptive to and in need of information on their performance. If, by contrast, teachers were evaluated every school year (as they are in a new but similar program in Washington, D.C.), the effect resulting from each subsequent year’s evaluation might well be smaller. Second, Cincinnati’s use of peer evaluators may result in teachers being more receptive to feedback from their subjective evaluation than they would be were the feedback to come solely from their supervising principals.

Teachers also appear to generate higher test-score gains during the year they are being evaluated, though these estimates, while consistently positive, are smaller. These improvements during the evaluation could represent the beginning of the changes seen in years following the review, or they could be the result of simple incentives to try harder during the year of evaluation, or some combination of the two.

A remaining question is whether the effects we find are small or large. A natural comparison would be to the estimated effects of different teacher professional-development programs (in-service training often delivered in formal classroom settings). Unfortunately, despite the substantial budgets allocated to such programs, there is little rigorous evidence on their effects. There are, however, other results from research on teacher effectiveness that can be used for comparison. First, the largest gains in teacher effectiveness appear to occur as teachers gain on-the-job experience in the first three to five years. Jonah Rockoff reports gains of about 0.10 student standard deviations over the first two years of teaching when effectiveness is measured by improvements in math computation skills; when using an alternative student math test measuring conceptual understanding, the gains are about half as large. Second, Kirabo Jackson and Elias Bruegmann find that having more effective teacher peers improves a teacher’s own performance; a 1-standard-deviation increase in teacher-peer quality is associated with a 0.04-standard-deviation increase in student math achievement. Compared to these two findings, the sustained effect of TES assessment is large.

But are these benefits worth the costs? The direct expenditures for the TES program are substantial, which is not surprising given its atypically intensive approach. From 2004–05 to 2009–10, the Cincinnati district budget directly allocated between $1.8 and $2.1 million per year to the TES program, or about $7,500 per teacher evaluated. More than 90 percent of this cost is associated with evaluator salaries.

A second, potentially larger “cost” of the program is the departure from the classroom of the experienced and presumably highly effective teachers selected to be peer evaluators. The students who would otherwise have been taught by the peer evaluators will likely be taught by less-effective, less-experienced teachers; in those classrooms, the students’ achievement gains will be smaller on average. (The peer evaluator may in practice be replaced by an equally effective or more effective teacher, but that teacher must herself be replaced in the classroom she left.)

While this second cost is more difficult to calculate, it is certainly offset by the larger gains made by students in the evaluated teachers’ classrooms. Those students are scoring, on average, 10 percent of a standard deviation better than they would have otherwise, and since each peer evaluator evaluates 10 to 15 teachers each year, those gains are occurring in multiple teachers’ classrooms for a number of years.

The results of our study provide evidence that subjective evaluation can improve employee performance, even after the evaluation period ends. This is particularly encouraging for the education sector. In recent years, the consensus among policymakers and researchers has been that after the first few years on the job, teacher performance, at least as measured by student test-score growth, cannot be improved. In contrast, we demonstrate that, at least in this setting, experienced teachers provided with unusually detailed information on their performance improved substantially.

American public schools have been under new pressure from regulators and constituents to improve teacher performance. To date, the discussion has focused primarily on evaluation systems as sorting mechanisms, a means to identify the lowest-performing teachers for selective termination. Our work suggests optimism that, while costly, well-structured evaluation systems can not only serve this sorting purpose but can also enhance education through improvements in teacher effectiveness. In other words, if done well, performance evaluation can be an effective form of teacher professional development.

Eric S. Taylor is a doctoral student at Stanford University. John H. Tyler is professor of education, economics, and public policy at Brown University. This article is based in part on a forthcoming study in the American Economic Review.

Birdette Hughey is the 2011 Mississippi Teacher of the Year.

In February 2012, the New York Times took the unusual step of publishing performance ratings for nearly 18,000 New York City teachers based on their students’ test-score gains, commonly called value-added (VA) measures. This action, which followed a similar release of ratings in Los Angeles last year, drew new attention to the growing use of VA analysis as a tool for teacher evaluation. After decades of relying on often-perfunctory classroom observations to assess teacher performance, districts from Washington, D.C., to Los Angeles now evaluate many of their teachers based in part on VA measures and, in some cases, use these measures as a basis for differences in compensation.

Newspapers that publish value added measures no doubt relish the attention they generate, but the bigger question in our view is whether VA should play any role in the evaluation of teachers. Advocates argue that the use of VA measures in decisions regarding teacher selection, retraining, and dismissal will boost student achievement, while critics contend that the measures are a poor indicator of teacher quality and should play little if any role in high-stakes decisions. The Obama administration has thrown its weight squarely behind the advocates, launching a series of programs that encourage states to develop evaluation systems based substantially on VA measures.

The debate over the merits of using value added to evaluate teachers stems primarily from two questions. First, do VA measures work? In other words, do they accurately capture the effects teachers have on their students’ test scores? One concern is that VA measures will incorrectly reward or penalize teachers for the mix of students they get if students are assigned to teachers based on characteristics that VA analysis typically ignores.

Second, do VA measures matter in the long run? For example, do teachers who raise test scores also improve their students’ outcomes in adulthood or are they simply better at teaching to the test? Recent research has shown that high-quality early-childhood education has large impacts on outcomes such as college completion and adult earnings, but no study has identified the long-term impacts of teacher quality as measured by value added.

We address these two questions by analyzing school-district data from grades 3–8 for 2.5 million children, linked to information on their outcomes as young adults and the characteristics of their parents. We find that teacher VA measures both work and matter. First, we find that VA measures accurately predict teachers’ impacts on test scores once we control for the student characteristics that are typically accounted for when creating VA measures. Second, we find that students assigned to high-VA teachers are more likely to attend college, attend higher-quality colleges, earn more, live in higher socioeconomic status (SES) neighborhoods, and save more for retirement. They are also less likely to have children during their teenage years.

Teachers in all grades from 4 to 8 have large impacts on their students’ adult lives. On average, a 1-standard-deviation improvement in teacher value added (equivalent to having a teacher in the 84th percentile rather than one at the median) in a single grade raises a student’s earnings at age 28 by about 1 percent. Replacing a teacher whose value added is in the bottom 5 percent with an average teacher would increase students’ total lifetime incomes by more than $1.4 million for a typical classroom (equivalent to $250,000 in present value). In short, good teachers create substantial economic value, and VA measures are useful in identifying them.

Our findings address the three main critiques of VA measures raised in a recent Phi Delta Kappan article by Stanford education professor Linda Darling-Hammond and her colleagues. We show directly using quasi-experimental tests that standard VA measures are not biased by the students assigned to each teacher. Hence, value-added metrics successfully disentangle teachers’ impacts from the many other influences on student progress. We also show that although VA measures fluctuate across years, they are sufficiently stable that selecting teachers even based on a few years of data would have substantial impacts on student outcomes such as earnings.

Students assigned to high-VA teachers are more likely to attend college, earn more, live in higher socioeconomic status neighborhoods, and save more for retirement.

Data

We draw information from two sources: school-district records on students and teachers, and information on the same students and their parents from administrative data sources such as tax records. The school-district data contain student enrollment history, test scores, and teacher assignments from the administrative records of a large urban school district. These data span the school years 1988–89 through 2008–09 and cover roughly 2.5 million children in grades 3 through 8.

The school-district data include approximately 18 million test scores. Test scores are available for English language arts and math for students in grades 3–8 from the spring of 1989 to 2009. In the early part of the sample period, these tests were specific to the district, but by 2005–06 all tests were statewide, as required under the No Child Left Behind law. In order to calculate results that combine scores from different tests, we standardize test scores by subject, year, and grade. The district data also contain other information on students, such as race or ethnicity, gender, and eligibility for free or reduced-price lunch (a standard measure of poverty).

Our data on students’ adult outcomes include earnings, college attendance, college quality (measured by the earnings of previous graduates of the same college), neighborhood quality (measured by the percentage of college graduates in their zip code), teenage birth rates for females (measured by claiming a dependent born when the woman was still a teenager), and retirement savings (measured by contributions to 401[k] plans). Parent characteristics include household income, marital status, home ownership, 401(k) savings, and mother’s age at child’s birth.

Having spent a single year in the classroom of a teacher with value added that is 1 standard deviation higher increases earnings at age 28 by about 1 percent. If that 1 percent advantage were to remain stable throughout an individual's career, it would add up to about $25,000 in total earnings.

Do Value-Added Measures Work?

Value-added analysis aims to isolate the causal effects teachers have on student achievement by comparing how well their students perform on end-of-year tests relative to similar students taught by other teachers. These comparisons take into account students’ test scores in the prior year as well as their race or ethnicity, gender, age, suspensions and absences in the previous year, whether they repeated a grade, special education status, and limited English status. We also control for teacher experience as well as for class and school characteristics, including class size and the academic performance and demographic characteristics of all students in the relevant classroom and school.

Many other researchers use methods for measuring teacher value added that are similar to ours, so it is not surprising that we obtain similar results. For example, we find that a 1-standard-deviation increase in teacher value added corresponds to increases in student math and English scores of 12 and 8 percent of a standard deviation, respectively. In both subjects, this difference is equivalent to approximately three months of additional instruction.

Can we take this as evidence of teachers’ causal impact on student test scores? Recent studies by economists Thomas Kane, Doug Staiger, and Jesse Rothstein, among others, have reached divergent conclusions about whether VA measures should be interpreted in this way. In particular, critics contend that VA measures are likely to be biased as a result of the way that students are assigned to teachers. For example, some teachers might be consistently assigned students with higher-income parents (which typically cannot be accounted for by school districts when generating VA measures because they do not collect precise data on family income). We implement two new tests to determine whether VA estimates are biased.

Our first test examines whether in fact high-VA teachers tend to be assigned students from more-advantaged families. We calculate an overall measure of parents’ socioeconomic status, combining the parental characteristics listed above. Not surprisingly, parent socioeconomic status is strongly predictive of student test scores, and, looking at simple correlations, we find that less-advantaged students do tend to be assigned to teachers with lower VA measures. However, controlling for the limited set of student characteristics available in school-district databases, such as test scores in the previous grade, is sufficient to account for the assignment of students to teachers based on parent characteristics. That is, if we take two students who have the same 4th-grade test scores, demographics, classroom characteristics, and so forth, the student assigned to a teacher with higher VA in grade 5 does not systematically have different parental income or other characteristics.

This first test shows that any bias in VA estimates due to the omission of parent characteristics that we are able to observe is minimal. The possibility remains, however, that students are assigned to teachers based on unmeasured characteristics unrelated to parent socioeconomic status. For example, principals may consistently assign their most-disruptive students to teachers whom they believe are up to the challenge. Alternatively, principals might assign these same students to their least-effective teachers, whom they are not worried about losing. Our second test seeks to determine the amount of bias introduced by this kind of sorting.

To do so, we exploit the fact that adjacent grades of students within the same school are frequently assigned to teachers with very different levels of value added because of idiosyncrasies in teacher assignments and turnover. During our analysis period, roughly 15 percent of teachers in our data switched to a different grade within the same school from one year to the next, 6 percent of teachers moved to a different school within the same district, and another 6 percent left the district entirely. These year-to-year changes in the teaching staff at a given school generate differences in value added that are unlikely to be related to student characteristics.

To illustrate, suppose a high-VA 4th-grade teacher enters a school at the beginning of a school year. If VA estimates capture teachers’ true impact on their students, students entering grade 4 in that school should have higher year-end test scores than those of the previous cohort. And the size of the change in test scores across these consecutive cohorts should correspond to the change in the average value added across all teachers in the grade. For example, in a school with three equal-sized 4th-grade classrooms, the replacement of a teacher with a VA estimate of 0.05 standard deviations with one with a VA estimate of 0.35 standard deviations should increase average test scores among 4th-grade students by 0.1 standard deviations.

In fact, that is exactly what we find, as shown in Figure 1. To construct this figure, we first define the top 5 percent of teachers as “high VA” and the bottom 5 percent as “low VA.” Figure 1 displays average test scores for cohorts of students in the years before and after a high-VA teacher arrives. We see that end-of-year test scores in the subject and grade taught by that teacher rise immediately by about 4 percent of a standard deviation. This impact on average test scores is commensurate in magnitude with what we would have predicted given the increase in average teacher value added for the students in that grade.

We obtain parallel findings when we examine the departure of high-VA teachers and the entry and exit of low-VA teachers. When a high-VA teacher leaves a given subject-grade-school combination, test scores of subsequent students in that subject, grade, and school fall. Likewise, students benefit from the departure of a low-VA teacher and are harmed by the arrival of a low-VA teacher.

Together, these results provide direct evidence that removing low-VA teachers (bottom 5 percent) and retaining high-VA teachers (top 5 percent) improves the academic achievement of students. But what about the remaining 90 percent of teachers? When we perform a similar analysis for all teachers, we again find that changes in the quality of the teaching staff strongly predict changes in test scores across consecutive cohorts of students in the same school, grade, and subject. Moreover, in middle schools, where students usually learn math and English from different teachers, we confirm that the arrival or departure of math teachers affects math scores but not English scores (and vice versa).

Using these techniques, we can calculate the amount of bias in our VA estimates. We find that the degree of bias is, on average, less than 2 percent. We therefore conclude that standard VA estimates accurately capture the impact that teachers have on their students’ test scores. Although the results could differ in other settings, our method of using natural teacher turnover to evaluate bias in VA estimates can be easily implemented by school districts to evaluate the accuracy of their VA models.

Do Value-Added Measures Matter?

Even though value-added measures accurately gauge teachers’ impacts on test scores, it could still be the case that high-VA teachers simply “teach to the test,” either by narrowing the subject matter in the curriculum or by having students learn test-taking strategies that consistently increase test scores but do not benefit students later in their lives. To address this issue, we measure the relationship between teachers’ value added and their students’ outcomes in adulthood. We compare students who were assigned high-VA vs. low-VA teachers in grades 4–8 and study their outcomes in adulthood.

We find that high-VA teachers raise students’ chances of attending college at age 20 (see Figure 2a). A student assigned to a teacher with a VA 1 standard deviation higher is 0.5 percentage points more likely to attend college at age 20 (an increase of 1.3 percent). Students of higher-VA teachers also attend higher-quality colleges, as measured by the average earnings of previous graduates of those colleges.

A person’s income doesn’t begin to stabilize until their late twenties, so our analysis of earnings focuses on the year when students were 28, the oldest age at which we observe a sufficiently large number of students. We find that having spent a single year in the classroom of a teacher with value added that is 1 standard deviation higher increases earnings at age 28 by $182, or about 1 percent (see Figure 2b). If that 1 percent advantage were to remain stable throughout an individual’s career, it would add up to about $25,000 in total earnings.

In addition to improved earnings, we also find that improvements in teacher value added significantly reduce the likelihood that female students will have a child during their teenage years, increase the socioeconomic status of the neighborhoods in which students live in adulthood, and raise 401(k) retirement savings rates. Moreover, it is likely that improved education would yield benefits that we are not able to measure but have been shown by other studies, such as reduced crime and improved citizenship.

To sum up, our evidence confirms that the students of high-VA teachers benefit not just by scoring higher on math and reading tests at the end of the school year, but also through improved outcomes later in life. The size of these effects may seem small, but recall that they reflect the impact of a higher-VA teacher for a single year and could compound over time to the extent that students are exposed to multiple high-VA teachers. As important, a single high-VA teacher has this effect not only on a single student but rather on an entire classroom—and often on many classrooms of students over the course of a career.

Wanda Booth, Florida's 2011 Charter School Teacher of the Year, works with students. Teachers in all grades have large impacts on their students' adult lives.

Policy Implications

In a recent article (see “Valuing Teachers,” features, Summer 2011), Eric Hanushek argues in favor of dismissing the bottom 5 percent of teachers based on their VA scores. While such a policy would have many costs and benefits that are beyond the scope of our study, we can illustrate the magnitudes implied by our analysis by calculating its impacts on students’ earnings. Our estimates imply that replacing a teacher whose value added is in the bottom 5 percent with an average teacher would increase students’ cumulative lifetime income by a total of $1.4 million per classroom taught. This gain is equivalent to $267,000 in present value at age 12, discounting at a 5 percent interest rate. However, it is important to realize there is uncertainty in VA measures, which are estimates that may be based on only a few classrooms of students, so the gains from removing teachers identified as ineffective based on a limited number of years of data are smaller. We estimate the gains from “deselecting” the bottom 5 percent of teachers to be approximately $135,000 in present value based on one year of data and $190,000 based on three years of data. These benefits, while still large, would have to be weighed against any costs associated with the policy, such as teachers demanding higher pay to compensate them for the risk of dismissal.

We also measure the expected gains from policies that pay higher salaries or bonuses to high-VA teachers in order to increase retention rates. The gains from such policies appear to be only somewhat larger than their costs. Although the benefit from retaining a teacher whose value added is at the 95th percentile after three years is nearly $200,000 per year, most bonus payments end up going to high-VA teachers who would have stayed even without the additional payment. Replacing low-VA teachers is therefore likely to be a more cost-effective strategy to increase teacher quality in the short run than paying to retain high-VA teachers. In the long run, higher salaries could attract more high-VA teachers to the teaching profession, a potentially important benefit that we do not measure here.

While these calculations illustrate the magnitudes of teachers’ impacts on students, they do not by themselves offer a blueprint for the design of optimal teacher evaluations, salaries, or merit-pay policies. Teachers were not evaluated based on test scores in the school district and time period we study. VA measures may not be as useful for identifying teachers with positive long-term impacts on their students if teachers respond to their use in evaluation systems by engaging in practices such as teaching to the test or even outright cheating. In addition, our analysis does not compare value added with other measures of teacher quality, like evaluations based on classroom observation, which might be even better predictors of teachers’ long-term impacts than VA scores.

In summary, our research demonstrates that good teachers are of great value to their students, and that VA measures are a potentially valuable tool for measuring teacher performance. The most important lesson we draw is that finding policies to raise the quality of teaching is likely to yield substantial economic and social benefits.

Raj Chetty is professor of economics at Harvard University. John N. Friedman is assistant professor of public policy at Harvard Kennedy School. Jonah E. Rockoff is associate professor of business at Columbia University’s Graduate School of Business. For further information on the study, see http://obs.rc.fas.harvard.edu/chetty/value_added.html.

Commentary

In light of the widespread attention given to the Chetty, Friedman, and Rockoff research, Education Next asked four experts to comment on the study’s implications for teacher policy.

Implications for Policy Are Not So Clear - By Douglas Harris
Profound Implications for State Policy - By Chris Cerf and Peter Shulman
More Evidence Would Be Welcome - By Dale Ballou
Low-Performing Teachers Have High Costs - By Eric A. Hanushek

Proponents of later start times, who have received considerable media attention in recent years, argue that many students who have to wake up early for school do not get enough sleep and that beginning the school day at a later time would boost their achievement. A number of school districts have responded by delaying the start of their school day, and a 2005 congressional resolution introduced by Rep. Zoe Lofgren (D-CA) recommended that secondary schools nationwide start at 9:00 or later. Despite this attention, there is little rigorous evidence directly linking school start times and academic performance.

In this study, I use data from Wake County, North Carolina, to examine how start times affect the performance of middle school students on standardized tests. I find that delaying school start times by one hour, from roughly 7:30 to 8:30, increases standardized test scores by at least 2 percentile points in math and 1 percentile point in reading. The effect is largest for students with below-average test scores, suggesting that later start times would narrow gaps in student achievement.

The primary rationale given for start times affecting academic performance is biological. Numerous studies, including those published by Elizabeth Baroni and her colleagues in 2004 and by Fred Danner and Barbara Phillips in 2008, have found that earlier start times may result in fewer hours of sleep, as students may not fully compensate for earlier rising times with earlier bedtimes. Activities such as sports and work, along with family and social schedules, may make it difficult for students to adjust the time they go to bed. In addition, the onset of puberty brings two factors that can make this adjustment particularly difficult for adolescents: an increase in the amount of sleep needed and a change in the natural timing of the sleep cycle. Hormonal changes, in particular, the secretion of melatonin, shift the natural circadian rhythm of adolescents, making it increasingly difficult for them to fall asleep early in the evening. Lack of sleep, in turn, can interfere with learning. A 1996 survey of research studies found substantial evidence that less sleep is associated with a decrease in cognitive performance, both in laboratory settings and through self-reported sleep habits. Researchers have likewise reported a negative correlation between self-reported hours of sleep and school grades among both middle- and high-school students.

I find evidence consistent with this explanation: among middle school students, the impact of start times is greater for older students (who are more likely to have entered adolescence). However, I also find evidence of other potential mechanisms; later start times are associated with reduced television viewing, increased time spent on homework, and fewer absences. Regardless of the precise mechanism at work, my results from Wake County suggest that later start times have the potential to be a more cost-effective method of increasing student achievement than other common educational interventions such as reducing class size.

Wake County

The Wake County Public School System (WCPSS) is the 16th-largest district in the United States, with 146,687 students in all grades for the 2011–12 school year. It encompasses all public schools in Wake County, a mostly urban and suburban county that includes the cities of Raleigh and Wake Forest. Start times for schools in the district are proposed by the transportation department (which also determines bus schedules) and approved by the school board.

Wake County is uniquely suited for this study because there are considerable differences in start times both across schools and for the same schools at different points in time. Since 1995, WCPSS has operated under a three-tiered system. While there are some minor differences in the exact start times, most Tier I schools begin at 7:30, Tier II schools at 8:15, and Tier III at 9:15. Tiers I and II are composed primarily of middle and high schools, and Tier III is composed entirely of elementary schools. Just over half of middle schools begin at 7:30, with substantial numbers of schools beginning at 8:00 and 8:15 as well. The school day at all schools is the same length. But as the student population has grown, the school district has changed the start times for many individual schools in order to maintain a balanced bus schedule, generating differences in start times for the same school in different years.

The only nationally representative dataset that records school start times indicates that, as of 2001, the median middle-school student in the U.S. began school at 8:00. More than one-quarter of students begin school at 8:30 or later, while more than 20 percent begin at 7:45 or earlier. In other words, middle school start times are somewhat earlier in Wake County than in most districts nationwide. The typical Wake County student begins school earlier than more than 90 percent of American middle-school students.

Data and Methods

The data used in this study come from two sources. First, administrative data for every student in North Carolina between 2000 and 2006 were provided by the North Carolina Education Research Data Center. The data contain detailed demographic variables for each student as well as end-of-grade test scores in reading and math. I standardize the raw test scores by assigning each student a percentile score, which indicates performance relative to all North Carolina students who took the test in the same grade and year. The second source of data is the start times for each Wake County public school, which are recorded annually and were provided by the WCPSS transportation department.

About 39 percent of WCPSS students attended magnet schools between 2000 and 2006. Since buses serving magnet schools must cover a larger geographic area, ride times tend to be longer for magnet school students. As a result, almost all magnet schools during the study period began at the earliest start time. Because magnet schools start earlier and enroll students who tend to have higher test scores, I exclude magnet schools from my main analysis. My results are very similar if magnet school students are included.

The data allow me to use several different methods to analyze the effect of start times on student achievement. First, I compare the reading and math scores of students in schools that start earlier to the scores of similar students at later-starting schools. Specifically, I control for the student’s race, limited English status, free or reduced-price lunch eligibility, years of parents’ education, and whether the student is academically gifted or has a learning disability. I also control for the characteristics of the school, including total enrollment, pupil-to-teacher ratio, racial composition, percentage of students eligible for free lunch, and percentage of returning students. This approach compares students with similar characteristics who attend schools that are similar, except for the fact that some schools start earlier and others start later.

The results produced by this first approach could be misleading, however, if middle schools with later start times differ from other schools in unmeasured ways. For example, it could be the case that more-motivated principals lobby the district to receive a later start time and also employ other strategies that boost student achievement. If that were the case, then I might find that schools with later start times have higher test scores, even if start times themselves had no causal effect.

To deal with this potential problem, my second approach focuses on schools that changed their start times during the study period. Fourteen of the district’s middle schools changed their start times, including seven schools that changed their start times by 30 minutes or more. This enables me to compare the test scores of students who attended a particular school to the test scores of students who attended the same school in a different year, when it had an earlier or later start time. For example, this method would compare the test scores of students at a middle school that had a 7:30 start time from 1999 to 2003 to the scores of students at the same school when it had an 8:00 start time from 2004 to 2006. I still control for all of the student and school characteristics mentioned earlier.

As a final check on the accuracy of my results, I perform analyses that compare the achievement of individual students to their own achievement in a different year in which the middle school they attended started at a different time. For example, this method would compare the scores of 7th graders at a school with a 7:30 start time in 2003 to the scores of the same students as 8th graders in 2004, when the school had a start time of 8:00. As this suggests, this method can only be used for the roughly 28 percent of students in my sample whose middle school changed its start time while they were enrolled.

Results

My first method compares students with similar characteristics who attend schools that are similar except for having different start times. The results indicate that a one-hour delay in start time increases standardized test scores on both math and reading tests by roughly 3 percentile points. As noted above, however, these results could be biased by unmeasured differences between early- and late-starting schools (or the students who attend them).

Using my second method, which mitigates this bias by following the same schools over time as they change their start times, I find a 2.2-percentile-point improvement in math scores and a 1.5-point improvement in reading scores associated with a one-hour change in start time.

My second method controls for all school-level characteristics that do not change over time. However, a remaining concern is that the student composition of schools may change. For example, high-achieving students in a school that changed to an earlier start time might transfer to private schools. To address this issue, I estimate the impact of later start times using only data from students who experience a change in start time while remaining in the same school. Among these students, the effect of a one-hour later start time is 1.8 percentile points in math and 1.0 point in reading (see Figure 1).

These estimated effects of changes in start times are large enough to be substantively important. For example, the effect of a one-hour later start time on math scores is roughly 14 percent of the black-white test-score gap, 40 percent of the gap between those eligible and those not eligible for free or reduced-price lunch, and 85 percent of the gain associated with an additional year of parents’ education.

The benefits of a later start time in middle school appear to persist through at least the 10th grade. All students in North Carolina are required to take the High School Comprehensive Test at the end of 10th grade. The comprehensive exam measures growth in reading and math since the end of grade 8 and is similar in format to the end-of-grade tests taken in grades 3–8. Controlling for the start time of their high school, I find that students whose middle school started one hour later when they were in 8th grade continue to score 2 percentile points higher in both math and reading when tested in grade 10.

I also looked separately at the effect of later start times for lower-scoring and higher-scoring students. The results indicate that the effect of a later start time in both math and reading is more than twice as large for students in the bottom third of the test-score distribution than for students in the top third. The larger effect of start times on low-scoring students suggests that delaying school start times may be an especially relevant policy change for school districts trying to meet minimum competency requirements (such as those mandated in the No Child Left Behind Act).

Why Do Start Times Matter?

The typical explanation for why later start times might increase academic achievement is that by starting school later, students will get more sleep. As students enter adolescence, hormonal changes make it difficult for them to compensate for early school start times by going to bed earlier. Because students enter adolescence during their middle-school years, examining the effect of start times as students age allows me to test this theory. If the adolescent hormone explanation is true, the effect of school start times should be larger for older students, who are more likely to have begun puberty.

I therefore separate the students in my sample by years of age and estimate the effect of start time on test scores separately for each group. In both math and reading, the start-time effect is roughly the same for students age 11 and 12, but increases for those age 13 and is largest for students age 14 (see Figure 2). This pattern is consistent with the adolescent hormone theory.

To further investigate how the effect of later start times varies with age, I estimate the effect of start times on upper elementary students (grades 3–5). If adolescent hormones are the mechanism through which start times affect academic performance, preadolescent elementary students should not be affected by early start times. I find that start times in fact had no effect on elementary students. However, elementary schools start much later than middle schools (more than half of elementary schools begin at 9:15, and almost all of the rest begin at 8:15). As a result, it is not clear if there is no effect because start times are not a factor in the academic performance of prepubescent students, or because the schools start much later and only very early start times affect performance.

Of course, increased sleep is not the only possible reason later-starting middle-school students have higher test scores. Students in early-starting schools could be more likely to skip breakfast. Because they also get out of school earlier, they could spend more (or less) time playing sports, watching television, or doing homework. They could be more likely to be absent, tardy, or have behavioral problems in school. Other explanations are possible as well. While my data do not allow me to explore all possible mechanisms, I am able to test several of them.

I find that students who start school one hour later watch 12 fewer minutes of television per day and spend 9 minutes more on homework per week, perhaps because students who start school later spend less time at home alone. Students who start school earlier come home from school earlier and may, as a result, spend more time at home alone and less time at home with their parents. If students watch television when they are home alone and do their homework when their parents are home, this behavior could explain why students who start school later have higher test scores. In other words, it may be that it is not so much early start times that matter but rather early end times.

Previous research tends to find that students in early-starting schools are more likely to be tardy to school and to be absent. In Wake County, students who start school one hour later have 1.3 fewer absences than the typical student—a reduction of about 25 percent. Fewer absences therefore may also explain why later-starting students have higher test scores: students who have an early start time miss more school and could perform worse on standardized tests as a result.

Conclusion

Later school start times have been touted as a way to increase student performance. There has not, however, been much empirical evidence supporting this claim or calculating how large an effect later start times might have. My results indicate that delaying the start times of middle schools that currently open at 7:30 by one hour would increase math and reading scores by 2 to 3 percentile points, an impact that persists into at least the 10th grade.

These results suggest that delaying start times may be a cost-effective method of increasing student performance. Since the effect of later start times is stronger for the lower end of the distribution of test scores, later start times may be particularly effective in meeting accountability standards that require a minimum level of competency.

If elementary students are not affected by later start times, as my data suggest (albeit not definitively), it may be possible to increase test scores for middle school students at no cost by having elementary schools start first. Alternatively, the entire schedule could be shifted later into the day. However, these changes may pose other difficulties due to child-care constraints for younger students and jobs and afterschool activities for older students.

Another option would be to eliminate tiered busing schedules and have all schools begin at the same time. A reasonable estimate of the cost of moving start times later is the additional cost of running a single-tier bus system. The WCPSS Transportation Department estimates that over the 10-year period from 1993 to 2003, using a three-tiered bus system saved roughly $100 million in transportation costs. With approximately 100,000 students per year divided into three tiers, it would cost roughly $150 per student each year to move each student in the two earliest start-time tiers to the latest start time. In comparison, an experimental study of class sizes in Tennessee finds that reducing class size by one-third increases test scores by 4 percentile points in the first year at a cost of $2,151 per student per year (in 1996 dollars). These calculations, while very rough, suggest that delaying the beginning of the school day may produce a comparable improvement in test scores at a fraction of the cost.

Finley Edwards is visiting assistant professor of economics at Colby College.

Click to enlarge

Not all school systems are moving in that direction. In Cambridge, Massachusetts, a district with surprisingly low student performance given the substantial per-pupil resources at its command, the school committee has decided to try to boost student achievement by abandoning its K–8 model in favor of having separate middle schools that serve grades 6 through 8 (though, in an unusual twist, each of the latter will be housed in the same facility as an elementary school).

In short, policymakers nationwide continue to wrestle with a basic question: At what grade level should students move to a new school? In the most common grade configuration in American school districts, public school students make two school transitions, entering a middle school in grade 6 or 7 and a high school in grade 9. This pattern reflects the influence of enrollment pressures and pedagogical theories that, over the past half century, all but eliminated the K–8 school from the American education landscape. A small fraction of students do attend public schools encompassing grades K–8, 6–12, or even K–12, however. We exploit this variation by comparing the achievement trajectories of Florida students entering a middle school or a high school to those of their peers who do not make those transitions.

Our study extends research conducted in New York City (see “Stuck in the Middle,” research, Fall 2010), in which Jonah Rockoff and Benjamin Lockwood found that entering a middle school causes a sharp drop in student achievement relative to the performance of those remaining in K–8 schools. It is hard to know whether one can generalize from results from the nation’s largest city (and school district), however, especially when it employs a complex procedure for assigning students to middle schools. Also, the New York City study was unable to follow students after 8th grade, making it impossible to know whether the negative impacts that were observed were temporary or extended into high school. This is a critical question inasmuch as a key rationale for middle school is its potential for easing the transition to high school. What is lost at the first transition may be more than gained at the second, which is presumably less abrupt for the middle-school child than for the one entering high school directly from an elementary-school environment.

To explore these issues, we use statewide data covering all students in Florida public schools who were in grades 3 to 10 between 2000 and 2009. Although a large majority of Florida students enter a middle school in grade 6, some do so in grade 7. Still others attend K–8 schools and avoid the middle-school transition altogether. To determine whether entering a middle school in grade 6 or grade 7 has any effect on achievement, we examine whether students experience a drop in test scores relative to students in K–8 schools that coincides with their transition to the new school. In the same way, we compare the learning trajectories of students entering high school in grade 9 to those of students who attend K–12, 6–12, or 7–12 schools in order to determine whether high-school transitions affect achievement.

Our results cast serious doubt on the wisdom of the middle-school experiment that has become such a prominent feature of American education. We find that moving to a middle school causes a substantial drop in student test scores (relative to that of students who remain in K–8 schools) the first year in which the transition takes place, not just in New York City but also in the big cities, suburbs, and small-town and rural areas of Florida. Further, we find that the relative achievement of middle-school students continues to decline in the subsequent years they spend in such schools. Nor do we find any sign that the middle-school students catch up with those who remained in the K–8 environment once all of them have entered high school. On the contrary, students entering a middle school in grade 6 are more likely not to be enrolled in any Florida public school as 10th graders (despite having been enrolled in grade 9), a strong indication that they have dropped out of school by that time.

We also find that the transition to high school causes a small drop in student achievement for all students who make this transition (as distinct from those in schools with 6–12 grade configurations). However, this drop holds far less policy significance both because of its size and because the decline does not appear to persist beyond grade 9.

The achievement drops we observe as students move to both middle and high schools suggest that moving from one school to another (or simply being in the youngest grade in a school) adversely affects student performance. The size and persistence of the effect of entering a middle school, however, suggests that such transitions are particularly damaging for adolescent students or that middle schools provide lower-quality education than K–8 schools provide for students at the same point in their education.

Data and Method

We draw the data for our analysis from the Florida Department of Education’s PK-20 Education Data Warehouse. The data contain state math and reading test scores for all Florida students attending public schools in grades 3 to 10 from the 2000–01 through 2008–09 school years. They also include information on the school each student attends and its location as well as student characteristics such as ethnicity, gender, special education classification, and eligibility for a free or reduced-price lunch.

We use different samples of students for different parts of our analysis. First, to estimate the effect of entering a middle school in grade 6 or 7, we examine only students enrolled in grade 3 between 2001 and 2004 who completed the state test in both math and reading in each of the subsequent five years. Second, to investigate whether the effects of middle-school entry persist through grades 9 and 10, we examine only students enrolled in grade 3 in 2001 or 2002 who were tested in both subjects each of the following seven years. Finally, to estimate the effect of entering high school in grade 9, we examine students enrolled in grade 6 between 2001 and 2005 who were tested in both math and reading in the following four years.

Our strategy for identifying the effects of alternative grade configurations on student achievement parallels and extends that of Rockoff and Lockwood’s study of New York City middle schools mentioned above. Specifically, we examine changes in individual students’ achievement over time, focusing on differences in the timing of students’ entry into middle school that result from the grade configuration of the school the student attended in 3rd grade. For example, we are interested in whether students who attended a K–6 school in 3rd grade experience a drop in their achievement in 7th grade relative to students who attended a K–8 school in 3rd grade and thus did not switch schools between grades 6 and 7.

The key assumption of our methodology is that there are no unobserved differences between students who in 3rd grade attended schools that had these different grade configurations that affect achievement precisely in the year when students enter middle school. In other words, we are assuming that the negative effect of a transition is not anticipated by parents and reflected in the choice of a school with a particular grade configuration in grade 3. We conduct an analogous analysis of high-school entry, taking advantage of the different grade configurations of the schools students attended in 6th grade.

Because we compare the achievement of individual students to themselves over time, our analysis takes into account all student characteristics (both observed and unobserved) that do not change over time. In addition, we also control for whether the individual student had been retained in a grade, whether the student had ever been retained, and whether the student attends a charter school (which in Florida are more likely than traditional public schools to have K–8 configurations).

The Middle-School Cliff

We find that students who will enter a middle school in 6th or 7th grade have positive achievement trajectories in math and reading from 3rd grade to 5th, relative to their counterparts who will never enter a middle school because they attend a school that continues through 8th grade. Achievement in both subjects falls dramatically in 6th grade for students who enter middle school in that grade. Students who will enter middle school in grade 7 continue to improve relative to their K–8 peers through grade 6, but experience a sharp drop in achievement upon entering middle school in grade 7.

Specifically, we find math achievement falls by 0.12 standard deviations and reading achievement falls by 0.09 standard deviations for transitions at grade 6 (see Figure 1). Students who make the transition at grade 7 experience even larger drops in their achievement of 0.22 and 0.15 standard deviations in math and reading, respectively. National data indicate that student achievement increases by roughly 0.30 standard deviations in math and 0.25 standard deviations in reading each year for typical 6th- and 7th-grade students. The drops in achievement we observe for students entering middle schools therefore amount to between 3.5 and 7 months of expected learning over the course of a 10-month school year.

Just as troubling is the fact that these students’ relative performance in both subjects continues to decline in subsequent middle-school grades. After three years in a middle school, students who entered in 6th grade score 0.23 standard deviations in math and 0.14 standard deviations in reading worse than we would have expected had they attended a K–8 school. After two years in a middle school, students who entered in 7th grade underperform by 0.31 standard deviations in math and 0.15 standard deviations in reading.

We also find little evidence that students who attend middle school make larger achievement gains than their peers in grades 9 and 10, by which time most Florida students have entered high school. In addressing this issue we must limit our attention to the two cohorts of students entering 3rd grade prior to 2001 or 2002, whose progress we are able to follow through the 10th grade. Although the math achievement of students who entered middle school in 7th grade improves by 0.05 standard deviations in 9th grade relative to students who attended K–8 schools, the same pattern is not evident in reading or in either subject for the much larger group of students who entered middle school in 6th grade (see Figure 2). In other words, we can safely reject the hypothesis that students who attend middle schools benefit at the transition to high school from their previous experience with school transition or from the specific educational programs available in middle schools.

Investigating the transition to high school, we find that students moving to a new high school between grades 8 and 9 suffer a small drop in achievement of 0.03 standard deviations in math and 0.04 standard deviations in reading (relative to those in grade 6–12 schools or schools with another configuration that requires no transition at this point). However, their relative achievement trajectories become positive again after this drop at the transition point.

We supplement our analysis on math and reading achievement with similar analyses of the effects of entering a middle school on the probability of students’ not being enrolled in a Florida public school in 10th grade (a proxy for dropping out of high school by this time) and on being retained in 9th grade (often a strong predictor that a student will leave school prior to graduation). Our results suggest that entering a middle school in 6th grade increases the probability of early dropout by 1.4 percentage points (or 18 percent). Although entering a middle school in 7th grade does not appear to increase early dropout, it increases the probability that a student will be retained in 9th grade by 1 percentage point. Both results provide additional cause for concern with the middle-school model.

Is it possible that our results reflect differences across school districts that employ alternative grade configurations? We explore this question by conducting our test-score analysis separately for schools in Miami-Dade County. With more than 345,000 students, Miami-Dade is the largest district in Florida and offers a wide range of grade configurations for students up through grade 8. We find that the negative effects of entering a middle school for grade 6 or grade 7 are, if anything, even more pronounced in Miami-Dade County than they are statewide.

Not Just an Urban Problem

This result for Miami-Dade County raises the possibility that the negative effects of middle-school entry are only notable in urban settings. We address this issue by looking separately at the effects of entering a middle or high school across communities of varying sizes. Using Census Bureau classifications, we group students into three categories according to the location of the school they attended in 3rd grade: 1) a large or midsize city, 2) suburbia (specifically, the urban fringe of a large or midsize city), and 3) towns and rural areas. The results suggest that the negative effects of entering a middle school are most pronounced in cities, but they remain sizable even in rural areas, confirming that the negative effects of configurations that separate the middle-school grades are by no means limited to urban school districts.

We also examine whether the middle-school effect varies across subgroups of students defined in terms of prior test performance, ethnicity, and gender. Students whose 3rd-grade scores were below the statewide median saw substantially larger declines in math scores at both the middle- and high-school transition points than higher-achieving students. These patterns are consistent with the theory that lower-achieving students have access to fewer educational resources outside of school and may therefore be at higher risk of being adversely affected by school transitions. We find no clear indication that the negative effect differs in size for higher- and lower-achieving students in reading, however.

Results for students of different ethnicities follow a similar pattern. Grade configuration has a larger effect on the math scores of traditionally disadvantaged subgroups than on other students. Black students in particular demonstrate large relative gains in math achievement prior to entering a middle school but then suffer larger drops both at and following the transition. Again, however, we find only small and statistically insignificant differences between the effects estimated for students of different ethnicities in reading. We find no differences in the effects for girls and boys.

Potential Explanations

Our results confirm that transitions into both middle schools and high schools cause drops in student achievement but that these effects are far larger for students entering middle schools. One possible interpretation of this pattern is that school transitions are more disruptive for younger students, perhaps because they are more susceptible to the negative influence of older students. Yet our estimates suggest that the effect of middle-school entry on student achievement is larger for students entering in grade 7 than for students entering in grade 6. Moreover, the fact that relative achievement continues to decline after students’ initial entry into middle schools suggests that average educational quality in Florida is lower in stand-alone middle schools than in schools serving grades K–8.

To explore why this might be the case, we first examine several characteristics of Florida elementary, middle, and K–8 schools. The most striking difference across school types involves cohort sizes (the average number of students in each grade). Although middle schools offer far fewer grades than K–8 schools, Florida middle schools on average enroll 146 more students than their K–8 counterparts; as a result, typical grade cohorts are almost three times as large. Florida middle schools also spend 11 percent less per student and have higher student-teacher ratios than K–8 schools, suggesting a potential role for differences in available resources. In contrast, we find no evidence that differences in observed teacher characteristics could explain our findings. Average teacher experience and average teacher salaries are similar across school types, while the share of the school’s instructional staff without prior experience is modestly higher in K–8 schools.

We conduct two analyses to shed light on whether these observed differences between middle schools and K–8 schools are likely to contribute to differences in school quality. First, we rerun our test-score analysis while controlling for these differences and find a similar pattern of results. Second, we examine whether the size of the drop in relative achievement suffered by students entering middle school in grade 6 varied with the characteristics of the middle school they attended. The results of this analysis again provide little evidence that low middle-school quality stems from differences in the school characteristics we can observe.

Middle schools could also differ from K–8 schools in their educational practices in ways that lead to lower student-achievement gains. To explore this possibility, we draw on a unique survey of Florida school principals conducted in 2003–04 to document responses to the state’s high-stakes accountability system. Confidentiality requirements preclude us from linking survey responses to specific schools, but we can document any differences in the average responses offered by principals of different school types.

We find few significant differences in the educational practices of the two groups of schools in our study. In particular, we observe no differences in the length of the school day or in measures of the extent to which schools had adopted specific policies to help low-performing students, policies to improve the performance of ineffective teachers, and incentives to reward highly effective teachers. If anything, these measures suggest that middle schools are more likely to have policies aimed at improving student achievement. We also find no differences across school types when we measure the degree of teacher autonomy.

A final set of survey items asked not about specific policies or practices but about the school’s overall climate. On these items, middle-school principals expressed significantly lower levels of agreement with statements indicating that their new and veteran teachers were excellent. This suggests that teachers in these schools may be less well equipped to deal with the challenges presented by their students. More middle-school principals also agreed with the statement that parents are worried about violence in the school. Although differences on the remaining items were statistically insignificant, they consistently point in the direction of middle schools having less-favorable school climates than K–8 schools.

In short, we find little evidence that the negative effects of attending a middle school are attributable to differences in resources, cohort sizes, or educational practices. We do, however, find suggestive evidence that the overall climate for student learning is worse in middle schools than in schools that serve students from elementary school through the 8th grade. This suggests a final potential interpretation of our results that is directly related to the choice of grade configuration: students may benefit from being among the oldest students in a school setting that includes very young students, perhaps because they have greater opportunity to take on leadership roles. This interpretation could account for both the gains in relative achievement made by students in K–5 and K–6 schools prior to entering middle schools and the superior performance of K–8 students relative to their peers in middle schools. A possible, if unlikely, alternative explanation is that students entering schools with different grade configurations have different growth trajectories for reasons having nothing to do with their schooling environment.

Taken as a whole, our results suggest that school transitions lower student achievement but that attending middle schools in particular has adverse consequences for American students. Especially when considered along with those of other recent studies, our findings clearly support ongoing efforts in urban school districts to convert stand-alone elementary and middle schools into schools with K–8 configurations. They are also relevant to the expanding charter-school sector, which has the opportunity to choose grade configurations without the disruption caused by school closures. More research is needed to see whether policy or pedagogical innovations can mitigate the effects of middle school. In the meantime, policymakers should exercise caution before extending the middle-school experiment to school districts that still enjoy the K–8 configuration.

Martin R. West is assistant professor of education at the Harvard Graduate School of Education and deputy director of the Program on Education Policy and Governance (PEPG) at Harvard’s Kennedy School. Guido Schwerdt is a researcher at the Ifo Institute for Economic Research in Munich, Germany.

In this study, I investigate whether the opportunity to attend a school other than a student’s assigned neighborhood school reduces criminal activity, especially among disadvantaged youth. Many of the schools chosen by the students were “better” on traditional indicators, such as student test scores and teacher characteristics. All of them, however, were preferred by the applicant over the default option. The analysis therefore sheds light on whether efforts to expand school choice can be an effective crime-prevention strategy, particularly when disadvantaged students can gain access to “better” schools.

We know that criminal offenders often have low levels of education: only 35 percent of inmates in U.S. correctional facilities have earned a high school diploma, compared to 82 percent of the general population. Criminal activity is concentrated among minority males; it begins in early adolescence and peaks when most youth should still be enrolled in secondary school. The schools these young men would attend are typically in high-poverty urban neighborhoods, have high rates of violence and school dropout, and struggle to retain effective teachers. Such schools may be a particularly fertile environment for the onset of criminal behavior. Yet little research has been conducted to determine the effect of school quality on crime.

In this study I explore this question using data from the Charlotte-Mecklenburg (North Carolina) school district (CMS) to measure the impact of school quality on arrest and incarceration rates. I take advantage of the CMS districtwide open-enrollment school-choice plan, which until recently let students choose where they wanted to go to school and employed lotteries to admit students to oversubscribed schools. I compare the criminal activity of students who won the lottery to attend their first-choice school to that of students who lost the lottery.

I find consistent evidence that attending a better school reduces crime among those age 16 and older, across various schools, and for both middle and high school students. The effect is largest for African American males and youth who are at highest risk for criminal involvement. In general, high-risk male youth commit about 50 percent less crime as a result of winning the school-choice lottery. They are also more likely to remain enrolled in school, and they show modest improvements on measures of behavior such as absences and suspensions. Yet there is no detectable impact on test scores for any youth in the sample.

School Choice in CMS

With more than 150,000 students enrolled in 2008–09, Charlotte-Mecklenburg is the 20th largest school district in the nation. The CMS attendance area encompasses all of Mecklenburg County, including Charlotte and several surrounding cities. Overall, CMS is racially and demographically diverse. About 45 percent of the students in CMS middle and high schools in 2003 were African American, less than 10 percent were Hispanic (although the Hispanic population was growing rapidly over this period), and about 50 percent were eligible for free or reduced-price lunches. Individual CMS schools vary widely in demographic composition: CMS high schools in 2003 ranged from less than 10 percent to close to 90 percent nonwhite, and were also dissimilar in average test scores and rates of high school graduation.

From 1971 until 2001, CMS schools were forcibly desegregated under a court order. Students were bused all around the district to preserve racial balance in schools. After several years of legal challenges, the court order was overturned, and CMS was instructed that it could no longer determine student assignments based on race. In December 2001, the CMS school board instituted a policy of districtwide open enrollment for the 2002–03 school year. School boundaries were redrawn as contiguous neighborhood zones, and children who lived in each zone were guaranteed access to their neighborhood school. Under busing, schools were racially balanced, but the surrounding neighborhoods remained highly segregated. Thus the redrawing of school boundaries led to concentrations of minority students in some schools.

The first open-enrollment lottery took place in the spring of 2002. CMS conducted an extensive outreach campaign to ensure that choice was broad-based, and 95 percent of parents submitted at least one preferred school; parents could submit up to three (not including their neighborhood school). Admission for all students from outside the neighborhood zone was subject to grade-specific limits. The lottery process for oversubscribed grades gave preference first to students who previously attended the school and their siblings, then to low-income students applying to schools that previously did not have a majority of low-income students, and finally to students applying to a school within their “choice zone” (which would guarantee them access to district-provided transportation). I study the effects of winning a seat at a preferred school in the 2002 lotteries on student outcomes through 2009, seven years after the lotteries were conducted.

Because nearly all rising 12th graders received their first choice, I restrict my study to students in grades 6 through 11. I also exclude the 5 percent of students who were not enrolled in any CMS school in the previous year. About 60 percent of the remaining students chose (and were automatically admitted to) their neighborhood school. About 75 percent of applicants to nonguaranteed schools were in lottery priority groups in which the probability of admission was either zero or one. Even though these students chose a nonguaranteed school, there is no randomness in whether they were admitted, so I do not use them in the study. The resulting sample consists of 1,891 high-school students (grades 9–11) and 2,320 middle-school students (grades 6–8). Compared to all students in CMS, these students were more likely to be African American and eligible for free lunch; they also had lower test scores and higher rates of absence and out-of-school suspensions (see Figure 1).

Data

Since the mid-1990s, the North Carolina Department of Public Instruction (NCDPI) has required all districts to submit data that include demographic information, attendance rates, and behavioral outcomes, yearly test scores in math and reading for grades 3 through 8, and subject-specific tests for higher grades. I used these data, along with internal CMS files that contain student-identifying information such as name, date of birth, and exact address in every year. This information enabled me to match CMS students to arrest records from the Mecklenburg County Sheriff’s Office, which include all arrests of adults (age 16 and over in North Carolina) that occurred in the county.

I measure crime severity in two ways, both of which are intended to capture the idea that not all crimes are equal. First, I use estimates that economists have developed of the social cost of crimes, which include tangible costs, such as lost productivity and medical care, as well as intangible costs, such as impact on quality of life; these estimates are extremely high for fatal crimes. (The estimated social cost of murder is $4.3 million in 2009 dollars. The next costliest crime is rape, which is estimated at $125,000.) To avoid the results being driven entirely by a few murders, in my main analysis I limit the cost of murder to twice the cost of rape. The second measure of severity weighs crimes by the expected punishment resulting from a successful conviction. Neither measure accounts for justice system costs such as police or prisons.

Methodology

If the school lottery is truly random, the winners and losers will on average have identical observed and unobserved characteristics. With a large enough sample, a simple comparison of outcomes between winners and losers would identify the causal effect of winning the lottery. In reality, CMS conducted many lotteries (for each school and grade). The number of students in each lottery is relatively small, so my analysis combines data from all of the middle-school and all of the high-school lotteries. My results reflect the average difference in outcomes between winners and losers across all of the lotteries conducted at each level.

The result is the “intent-to-treat” effect of winning a lottery; it is an intent because students offered a place in their first-choice school did not always take it (for example, they may have moved out of the district). Students who won the lottery are more than 55 percentage points more likely than losers to attend their first-choice school in the first year, and on average spend an additional 1 to 1.5 years enrolled in that school overall. One can therefore obtain a rough estimate of the effect of actually attending the first-choice school (as a result of winning the lottery) by doubling the results presented below.

I examine the impact of winning the lottery on crime separately for groups of students with different propensities to commit crimes, with a focus on the highest-risk group. Because students with adult arrest records can be tracked all the way back to kindergarten in some cases, I use all of the potential predictors of criminal behavior—test scores, demographics, behavior, and neighborhood characteristics—to calculate an index of crime risk. The students in the top 20 percent of this crime-risk index are disproportionately African American males and eligible for free lunch (see Figure 1a). Their test scores are on average one standard deviation below the North Carolina state average, and they are absent and suspended many more days than the average student (see Figure 1b). Because high-risk students are overwhelmingly male, I exclude females from all of the analyses. The results comprise a final sample of 1,014 high-school students and 1,081 middle-school students.

High-school lottery winners attend schools that are demographically very similar to the schools attended by lottery losers, while middle-school winners attend schools that are less African American and higher income on average. All lottery winners travel farther to attend their first-choice school, but the distance is greater for high school students than for middle school students.

High-school lottery winners in the high-risk group and all middle-school lottery winners experience modest increases in standard measures of school quality. Their peers’ average test scores are about 0.15 standard deviations higher, and the new schools have higher-quality teachers, measured in terms of the fraction of teachers with less than three years’ experience, the fraction that are new to the school that year, the percentage of teachers with an advanced degree, and the share of teachers who attended a “highly competitive” college as defined by the Barron’s rankings. For youth in the high-risk group, the gain as measured by these quality indicators is roughly equivalent to moving from one of the lowest-ranked schools to one around the district average.

Results

I find that winning a lottery for admission to a preferred school at the high school level reduces the total number of felony arrests and the social cost of crime. Among middle school students, winning a school-choice lottery reduces the social cost of crime and the number of days incarcerated. Importantly, I find that these overall reductions in criminal activity are concentrated among students in the highest-risk group. Indeed, I find little impact either positive or negative of winning a school-choice lottery on criminal activity for the 80 percent of students outside of this group.

Consider first the results for high school students in the high-risk group. Among these students, winning admission to a preferred school reduces the average number of felony arrests over the study period from 0.77 to 0.43, a pattern driven largely by a reduction of 0.23 in the average number of arrests for drug felonies (see Figure 2). The average social cost of the crimes committed by high-risk lottery winners (after adjusting the cost of murders downward) is $3,916 lower than for lottery losers, a decrease of more than 35 percent. (Without adjusting for the cost of murder, I estimate the reduction in the social cost of crimes committed by lottery winners at $14,106.) High-risk lottery winners on average commit crimes with a total expected sentence of 35 months, compared to 59 months among lottery losers.

Among high-risk middle-school students, I find no effect of winning a school-choice lottery on the average number of felony arrests. Although the number arrests for violent felonies falls, this is offset by an increase in the number of property arrests. Because violent crimes carry greater social costs, however, winning a school-choice lottery reduces the average social cost of the crimes committed by middle school students by $7,843, or 63 percent. It also reduces the total expected sentence of crimes committed by each student by 31 months (64 percent).

An important limitation of this analysis is that I do not have access to data on juvenile crime. Especially for students in the middle school sample, this could mask big differences in juvenile offending in the early years after the lotteries were conducted. As an alternative, I examine the effect of winning the lottery on school disciplinary outcomes such as absences and suspensions, as well as on test scores. Among the high-risk group, lottery winners are absent slightly less than the lottery losers are. The effect on high school suspensions in 2003 is relatively large, but the other school discipline effects are small and statistically insignificant.

In contrast to the results for crime and disciplinary outcomes, I find no evidence that winning admission to a preferred school leads to test-score gains. But I do find some impacts on enrollment, grade progression, and grade attainment for high-risk youth. For example, high-risk middle-school lottery winners are 18 percentage points more likely than lottery losers to be enrolled in CMS in their 10th-grade year. The effect on 11th-grade enrollment is about half the size (9 percentage points), and there is no impact on persistence into 12th grade.

Despite the impacts on enrollment and progression, there is no detectable increase in high school graduation rates. Because I am limited to CMS administrative data, it is difficult to distinguish dropouts from subsequent GED recipients or transfers who may have graduated elsewhere. Administrative records are particularly problematic for high-risk youth, who sometimes disappear from CMS well before they are old enough to do so legally. The graduation rate is only about 25 percent among high-risk high-school students, and currently only about 10 percent among high-risk middle-school students, although some who are still enrolled may yet graduate. Additionally, a bit less than 10 percent of the high-risk middle-school sample never appears in any high school grade but subsequently appears in the arrest data. Because any intervention aimed at high school students would miss this group altogether, this suggests that high school might be too late for the youth at highest risk of criminal activity.

Explanations and Policy Implications

Overall, I find that winning the lottery to attend a first-choice school has a large impact on crime for high-risk youth. High-risk lottery winners experienced roughly a 50 percent reduction in the measures of criminal activity that weight crimes by their severity.

I consider four possible explanations for the reduction in crime among high-risk lottery winners. The first is incapacitation, which advances that winning the lottery entails longer bus rides to and from school, thus occupying youth during high-crime hours. The second is contagion, in which winning the lottery prevents crime by removing high-risk youth from crime-prone peers or neighborhoods, thereby reducing contemporaneous exposure of high-risk youth to criminogenic influences. These first two explanations would predict a strong initial effect that fades over time. If, for example, drug-market activity is concentrated within a few schools, we might expect large differences in criminality in the high school years that diminish as enrollment in the chosen school ends and lottery winners and losers return to the same neighborhoods. When I examine the effect of winning a school lottery separately at different points in time after the lotteries were conducted, however, I find larger effects in later years. I therefore conclude that there is little support for the incapacitation and contagion explanations since they do not fit the pattern of results over time.

A third possibility is that the reduction in crime comes from the skills students gain by attending a higher-quality school. If the schools attended by lottery winners do a better of job of teaching skills that increase students’ ability to find employment, they will stay enrolled in school longer, delaying the onset of criminality through the peak period of offending behaviors. Moreover, youth with more and better schooling will gain access to more and better opportunities for paid work, making crime less attractive. Based on a back-of-the-envelope calculation of the relationship between enrollment and criminal activity in my sample, I estimate that the effects of winning a school lottery on enrollment could potentially explain about 45 percent of the impact on criminal activity in the high school sample, but only about 10 percent in the middle school sample.

Alternatively, peer networks formed in middle or high school could have a persistent influence on adult criminality without affecting skills directly. In my own data, I find relatively little evidence that the propensity of a student’s peers to engage in criminal activity influences the degree to which he commits violent crimes. This may be due in part to the high rate of early dropout among violent felons. However, having crime-prone peers in middle school substantially increases the likelihood of committing a violent crime, especially for youth in the high-risk group. Based on this relationship, I estimate that changes in peers can explain roughly 9 percent of the impact on violent arrests in the middle school sample.

Regardless of the mechanisms by which admittance to a preferred school influences criminal activity, the fact that these impacts are concentrated among high-risk students has important implications for the design of school-choice programs. It may make sense for oversubscribed schools of choice to give preferential admission to students at greatest risk of criminal activity. To illustrate this point, I use my results to evaluate the consequences of two different types of lotteries: 1) those giving priority to the highest-risk students and 2) a simple lottery similar to those virtually all charter schools nationwide are required to use to admit students when the schools are oversubscribed. The actual CMS lottery system gave preferences to low-income students who applied to schools with a low fraction of low-income students. As a consequence, many poor (and high-crime risk) students were automatically admitted to schools while other students had to win the lottery.

If slots in oversubscribed schools were systematically allocated to the highest-risk students, the social cost of crime would fall by an additional 27 percent relative to the actual CMS assignment mechanism. A more realistic form of targeting is the method actually pursued by CMS, giving preference to low-income students within the lottery system. I estimate that this policy choice lowered the social cost of crime by about 12 percent, relative to a simple charter-style lottery with no preferential treatment. Although this analysis does not consider the possibility that a greater concentration of high-risk students could have adverse effects on other students, it nonetheless highlights the likely beneficial consequences of giving preference to disadvantaged students in the admissions process for oversubscribed schools.

Conclusion

In this study, I find that winning a lottery for admission to the school of choice greatly reduces criminal activity, and that the greatest reduction occurs among youth at the highest risk for committing crimes. The impacts persist beyond the initial years of school enrollment, seven years after the school-choice lottery was held. The findings suggest that schools may be an opportune setting for the prevention of future crime. Many high-risk youth drop out of school at a young age and are incarcerated for serious crimes prior to the age of high school graduation. For these youth, who are on the margins of society, public schools may present the best opportunity for intervention.

The end of busing and the implementation of open enrollment in CMS was a significant policy change. The four neighborhood high schools to which most of the lottery applicants were assigned lost more than 20 percent of their enrollment in a single year. In subsequent years, two of these schools were restructured as magnet schools offering specialized programs in a small school setting. Two middle schools that lost significant numbers of students were subsequently closed. The open enrollment policy thus sent a strong signal of parental demand to CMS that may have resulted in the shutting down or restructuring of low-performing schools. The No Child Left Behind Act of 2001 included a provision that allowed parents to transfer students from “persistently dangerous” public schools, but many states have set the legal threshold so high that very few schools qualify. The results here suggest that, to the extent that low-quality schools are also persistently dangerous, allowing students to leave them might benefit individual students as well as society as a whole.

David J. Deming is assistant professor of education at the Harvard Graduate School of Education. This article is adapted from a study in the November 2011 issue of the Quarterly Journal of Economics.

For nearly a decade, the No Child Left Behind Act (NCLB) has focused the attention of policymakers and researchers squarely on the achievement of low-performing students, with some apparent success. The math and reading scores on the National Assessment of Educational Progress of the nation’s lowest-achieving 10 percent of 4th and 8th graders have risen sharply since 2000, continuing a trend that began in the 1990s. Yet some may wonder about the potential cost of this focus on higher-achieving students, for whom improvements over the same time period have been modest. Among the questions related to this debate is whether additional programs and resources should be devoted to students on the higher end of the spectrum, those considered gifted.

Three million students in the United States are classified as gifted, yet little is known about the effectiveness of traditional gifted and talented (G&T) programs. In theory, G&T programs might help high-achieving students because they group them with other high achievers and typically offer specially trained teachers and a more advanced curriculum. While previous research indicates that ability grouping is in fact correlated with higher achievement, these findings could be misleading if students placed in high-ability classrooms were likely to be successful for reasons that researchers are unable to measure, such as stronger motivation. To our knowledge, no existing studies offer convincing evidence on the causal effect of G&T programs on student achievement.

Our research begins to fill this gap with two studies of the G&T programs available to high-achieving middle-school students in a large urban school district in the southwestern United States which, to preserve anonymity we shall refer to as LUSD. Since 2007, all 5th-grade students in LUSD have been evaluated to determine eligibility for gifted and talented programs starting in 6th grade. Those students who are deemed eligible often are grouped in classes with other gifted students. They are also permitted to apply for admission to two middle schools that have oversubscribed magnet G&T programs.

The two studies use different methods to ask distinct but closely related questions. The first exploits the fact that eligibility for G&T programming in LUSD is determined by a well-defined cutoff in students’ evaluation scores. By comparing students who score just above the cutoff to those who score just below, the study provides evidence on the effect of enrollment in a G&T program on achievement for those students on the margin of eligibility. The second study takes advantage of the randomized lotteries that determine admission to the district’s two premier magnet G&T programs. By comparing students who win the lottery and attend the magnet G&T schools to those who lose the lottery and attend other “neighborhood” programs, the research provides evidence on whether the magnet G&T programs provide any additional benefits.

The results of both studies will be discouraging for those hopeful that current G&T programs provide a means to accelerate the progress of our most capable students. The first shows that barely eligible students who participated in LUSD’s G&T curriculum for all of 6th grade and half of 7th grade exhibit no significant improvement in test scores across a range of subjects, despite their being surrounded by higher-achieving peers and taking more advanced courses. The lottery study corroborates these results, as students admitted to the G&T magnet schools show little improvement in test scores by 7th grade, despite having higher-achieving peers and being taught by more effective teachers. The lone exception is in science, where students admitted to G&T magnet schools performed at substantially higher levels.

It is difficult to know what accounts for these puzzling results. Our best guess, which we discuss in detail below, is that being placed with higher-achieving peers is not all that it is cracked up to be. Students admitted to both types of G&T programs suffer a large drop in their relative rank in terms of grades within their classes, which could have adverse consequences that offset any benefits of improvements in their educational environment. But we are getting ahead of ourselves. Let’s first take a closer look at the programs and the evidence on their effects.

Gifted Students in LUSD

LUSD is a large school district, with more than 200,000 students. The district is heavily minority and very low income; the minority population is more heavily Hispanic than African American. All LUSD students are evaluated for placement in middle-school G&T programs during 5th grade, including those who participated in the district’s G&T program in elementary school. In order to be deemed eligible for the middle school G&T program, a student must meet the eligibility criteria set forth in the “gifted and talented identification matrix.” The matrix converts scores on standardized tests—the Stanford Achievement Test for English-speaking students and the Aprenda exam for Spanish-speaking students with limited English proficiency—scores on the Naglieri Nonverbal Ability Test (NNAT), average course grades, teacher recommendations, and indicators for socioeconomic status into an overall index score.

While all students who meet these requirements qualify for the G&T program, not all end up being classified as G&T, because parents are allowed to opt out. Some students also enroll in the program initially but later withdraw. Schools in LUSD have a monetary incentive for attracting gifted students, as LUSD provides a funding boost of 12 percent over the average allotment for a regular student.

Gifted students in LUSD are far less likely to be economically disadvantaged and more likely to be white or Asian than other students in the district. They also perform at far higher levels on the Stanford Achievement Tests, which the district administers annually in five subjects: math, reading, language, social science, and science. Their advantage in math and reading test scores in 5th grade is roughly 0.7 of a standard deviation, which amounts to well over two years of academic progress (see Figure 1). By the time the same students have reached 7th grade, these gaps have widened to 1.5 standard deviations in math and 1.25 standard deviations in reading. While this pattern suggests that the students enrolled in the district’s G&T programs learn at a faster rate between 5th and 7th grade, it does not necessarily mean that the G&T programs are the cause. It is to that question we now turn.

Effects on Barely Eligible Students

Our first study examines the effects of participation in a G&T program on students who were just barely eligible to participate based on their overall index scores. We focus on students who were evaluated for G&T eligibility as 5th graders in the spring of 2008 for whom we are able to observe outcomes as 7th graders in the 2009–10 school year. Our outcome measures include Stanford Achievement Test scores and attendance rates, both of which are drawn from administrative data provided by the district. After restricting the sample to students near the G&T eligibility cutoff, we are able to examine these outcomes for roughly 2,600 students.

The method used in the study, known as regression discontinuity analysis, takes advantage of the fact that the district uses a strict numerical cutoff in the index score assigned to students as 5th graders in order to determine their eligibility to participate in the G&T program the following year. Because the students are unable to precisely manipulate their index scores, those scoring just below the eligibility cutoff should be very similar to those scoring just above the cutoff. We can therefore attribute any differences in student outcomes on either side of the cutoff to the effect of having being deemed eligible.

As noted above, not all eligible students end up participating in G&T programs due to factors such as a parent’s decision to opt out. Similarly, some students who do not initially qualify later become eligible through an appeals process that allows parents to submit an alternative standardized test score or through additional evaluations conducted in 6th grade. As a result, we use standard statistical techniques to account for the fact that the cutoff our regression discontinuity analysis exploits is “fuzzy” rather than sharp. This allows us to provide evidence on the effects of actual participation in the G&T program, not simply eligibility for it.

Before looking at student outcomes, we first used the same method to confirm that participation in the district’s standard G&T programs led to measurable differences in students’ educational experiences. Clearly, it did. The average achievement of the peers in G&T students’ classrooms were between 0.25 and 0.33 of a standard deviation higher in each core academic subject. Participation in the G&T program also increased the number of advanced courses in which students enrolled in 6th and 7th grade. We found no evidence, however, that the teachers to whom students in the G&T program were assigned were any more effective, as measured by their impact on student test scores.

Did these improvements in peer characteristics and curricular rigor translate into improved outcomes? Our results indicate that they did not (see Figure 2). Our estimates of the effects of G&T participation for barely eligible students are close to zero in all five subjects and are sufficiently precise to allow us to rule out with 90 percent confidence effects as small as 0.04 standard deviations (sd) in math, 0.07 sd in reading, 0.12 sd in language, 0.10 sd in social studies, and 0.19 sd in science. We also looked at the impact of G&T participation for specific student subgroups defined by gender, race/ethnicity, socioeconomic status, and whether the students had been classified as gifted in elementary school. We found little evidence of differential impacts for students in any of these groups.

The Effects of G&T Magnet Programs

Why does the G&T program in LUSD not yield benefits for students on the margin of eligibility? One reason could be that the qualification boundary is set so low that such students are not able to take advantage of the programs’ purported benefits. Our second analysis, which uses experimental research methods to study the effects of enrollment in the district’s G&T magnet programs, is intended to shed light on this concern.

LUSD has 41 middle schools, of which 8 have G&T magnet programs, and 2 of these are oversubscribed. As a result, the district uses lotteries to determine which students will be admitted as 6th graders. Our analysis compares the performance of students who win the lottery and attend one of the G&T magnet programs to those who lose the lottery and either attend a neighborhood G&T program in the district, a magnet school based on a different specialty, or a charter school. Because the lottery is random, any differences in outcomes between lottery winners and losers can be attributed to the effect of enrolling in the G&T magnet program rather than one of these alternatives. Moreover, the results of this analysis will apply to the entire population of students who chose to apply.

Our lottery analysis is based on the sample of LUSD 5th-grade students determined to be eligible for G&T programs in 2007–08 who applied for admission to one of the two middle schools with an oversubscribed G&T magnet program. This group includes 542 students, 394 of whom were offered admission and 148 of whom were not. We find no statistically significant differences in the observed characteristics of lottery winners and losers, suggesting that the lotteries were in fact conducted in a random way.

The students in the lottery differ both academically and demographically from the students who were included in the regression discontinuity study. Not only do the lottery students have higher test scores than students at the eligibility cutoff, but their test scores exceed those of the average G&T student in the district. Lottery participants are also less likely to be on subsidized lunch, and less likely to be minority.

Of the 542 lottery participants, only 440 students, including 331 winners (84 percent) and 109 losers (74 percent), remain in LUSD by 7th grade. Fortunately, the observed characteristics of lottery winners and losers who remain in the district continue to be very similar. Even so, when analyzing the data we control for students’ demographic characteristics and prior achievement, and use weights designed to make the final sample comparable in terms of its observed characteristics to the set of students that initially applied for the lottery.

One disadvantage of this second study is that the lottery losers have a range of alternative experiences and most participate in standard G&T programs, so the comparison group’s educational experience is less clear than it was in the regression discontinuity analysis. Nonetheless, our data confirm that students admitted to the G&T magnet schools with lotteries seem to have experienced large improvements in their educational environment. Winning the lottery increased the average achievement of students’ classroom peers by as much as a full standard deviation in some subjects. And in contrast to the G&T program as a whole, students admitted by lottery to G&T magnet program were assigned to more effective teachers.

Turning to student outcomes, however, our results provide little evidence that attending a G&T magnet program leads to improvements in student achievement (see Figure 3). The one exception is science test scores, for which we estimate a positive effect of 0.28 standard deviations. Due to the relatively small sample sizes, all of the effects are imprecisely estimated and do not allow us to definitively rule out reasonably large positive effects. Even so, the estimated effects for math, reading, and social studies are negative, and the estimated effect for language is effectively zero.

Discussion

It is difficult to understand why we find little evidence that G&T programs positively affect achievement. A common concern with studies of high-achieving students is that the available achievement measures may not be well suited to discern improvements for this group. This would be particularly worrisome if we were using a state accountability exam targeted toward low-achieving students, but it is less of an issue with the Stanford Achievement Tests. Indeed, we found little evidence of students performing near the maximum levels on these tests in either the regression discontinuity or lottery samples. Although it is possible that the additional course material taught in G&T classes is poorly aligned with topics covered in the achievement test, research documenting the benefits of being placed with higher-ability peers suggests that we should see improvements, even if that were the case.

The effect of being placed in a higher-ability classroom may not necessarily be positive, however, especially for a marginal G&T student. In particular, the drop in ranking relative to one’s peers may have a negative effect: a marginal G&T student is likely to go from being near the top of the regular class to being near the bottom of the G&T class. Even students in the middle of the G&T distribution are likely to experience a loss of ranking in the magnet G&T schools as compared to their neighborhood schools. It may be that students are demoralized by the drop in their relative rankings or that teachers provide more resources to students at the top of the class.

Substantial evidence from educational psychology indicates that students who are placed in higher-achieving groups can suffer psychological harm. A commonly used measure is a student’s “self-concept,” how a student perceives her abilities relative to an objective measure such as achievement. A 1995 study by Herbert Marsh and colleagues compared G&T students to observably similar students in mixed G&T and non-G&T classes and found that G&T students show declines in their math and reading self-concept. More recent research has documented lower self-concept and greater test anxiety among gifted students in ability-segregated classrooms.

Although we do not have direct evidence on student confidence, we can make use of student course grades and rank within the class to probe for evidence consistent with this kind of effect. We evaluate the impact of G&T program enrollment in the regression discontinuity study and of attending a G&T magnet in the lottery analysis. In both cases, we find clear reductions in student grades. For the regression discontinuity sample, grades fall by a statistically significant 4 points out of 100 (3 points changes a grade from a B+ to a B, for example) in math and by 2 to 3 points in other subjects, although these effects are not statistically significant for 7th grade. For the lottery analysis, the grade reductions are even more dramatic, with drops of 7 points in math, 8 in science, and 4 in social studies.

It is also useful to consider how students’ rankings within their peer groups differ by treatment status, as this provides a direct measure of how a student may perceive his position in the overall distribution of student ability. We assume that students mostly compare themselves to their schoolmates who take the same courses in the same grade. Thus, we rank students within each school, grade, and course by their final course grades and then convert these rankings to percentiles. The rankings based on 7th-grade courses exhibit notable drops when students cross the G&T eligibility threshold. Controlling for race, gender, economic disadvantage, LEP (Limited English Proficiency), and prior gifted status, marginal G&T students have a relative rank in 7th grade that is 13 to 21 percentiles lower than similar students who were not admitted. Attending a premier G&T magnet in 7th grade generates a nearly 30 percentile ranking drop in all four of the courses examined.

In short, the necessary conditions are clearly met for a drop in relative ranking to play a role in offsetting the expected positive impact of more rigorous courses, more effective teachers, and higher-achieving peers. The possibility that G&T students are subject to such a mechanism suggests potential constraints on the benefits of programs that provide more similar peers and an increase in traditional education inputs.

One should not conclude from the lack of achievement results, however, that the G&T programs should be scuttled. Our analysis occurs in a district with a large number of relatively high-quality magnet programs, and thus the alternatives to the G&T programs may be strong. There may also be benefits that we are not able to capture, such as impacts on SAT scores, graduation rates, and college attendance. Further, our study examines a G&T program in one district. Certainly, districts vary in the approaches they take to educating gifted students, so it may be that similar studies of programs in other districts would yield different results. Nonetheless, this study does raise questions about the efficacy of G&T programs and the traditional model of ability-segregated classrooms.

Sa Bui is a doctoral candidate in economics at the University of Houston, where Steven Craig is professor of economics and Scott Imberman is assistant professor of economics.

If principals have the authority to dismiss teachers, will they dismiss the less effective ones, or will they instead make perverse decisions by letting the good teachers go? Evidence from low-stakes surveys suggests that principals are able to identify the most and least effective teachers in their schools, as measured by their impact on student achievement (see “When Principals Rate Teachers,” research, Spring 2006). But would that ability influence their dismissal decisions?

On this topic, debate has been vigorous but research almost nil, in good part because teachers with tenure are not easily dismissed and principals take on that task only if they have a strong backbone or face an extremely urgent situation, or both. In some instances, however, principals have considerable latitude when it comes to dismissing teachers who have not been in service long enough to have earned tenure.

One such situation developed in Chicago in July 2004 when the Chicago Public Schools (CPS) and the Chicago Teachers Union signed a new collective-bargaining agreement that gave principals the flexibility to dismiss probationary (nontenured) teachers beginning in the 2004–05 school year for any reason and without the documentation and hearing process that is typically required for dismissals in other districts. Since CPS provided information that allowed me to link information on CPS teacher dismissals to several measures of teacher performance, I was able to study whether principals exercise their authority wisely. The procedures were fairly straightforward. By comparing the characteristics of dismissed versus nondismissed probationary teachers within the same school and year, I was able to determine just how much weight school administrators place on a variety of teacher characteristics, including their performance in the classroom.

I find that principals in Chicago do exercise their authority in  sensible ways. Principals are more likely to dismiss teachers who are frequently absent and who have previously received poor evaluations. They dismiss elementary school teachers who are less effective in raising student achievement. Principals are also less likely to dismiss teachers who attended competitive undergraduate colleges. It is interesting to note that dismissed teachers who were subsequently hired by a different school are much more likely than other first-year teachers in their new school to be dismissed again.

These results suggest that other school districts could possibly improve student achievement if they adopted policies similar to those applied in Chicago. To be clear, however, the analysis presented in this paper does not seek to evaluate the educational impact of this new policy. Instead, it uses the existence of the policy, in conjunction with detailed data on teachers and principals, to provide descriptive evidence on the relationship between the exercise of dismissal authority and teacher effectiveness.

Teacher Dismissals in Chicago

As in many public school districts, teacher layoffs and dismissals in CPS are highly regulated. Prior to 2004, virtually no teachers—not even probationary teachers—were dismissed for cause in CPS. Of course, it is likely that some teachers who switched schools or left CPS entirely were informally “counseled out” by school administrators. But it was impossible to distinguish these “involuntary” separations from truly voluntary attrition.

This situation changed with the signing of a new collective-bargaining agreement in 2004. Each February, principals are able to log into a district computer system that has a list of all of the probationary teachers in their school (i.e., those who have been teaching for fewer than five consecutive years during the period of my analysis). The principal can then check one of two boxes: renew or nonrenew. Although principals are required to provide district officials with at least one reason for the nonrenewal decision, they are not required to justify or explain their decision and they do not need to provide teachers with this reason. If a principal chooses nonrenew, the teacher may reapply to positions in other Chicago public schools. However, nonrenewed teachers are not guaranteed another job in CPS. The ease with which administrators can dismiss a probationary teacher, with a simple “click” of a button, is noteworthy. This policy change made Chicago the only large school district in the country to provide principals with this degree of flexibility over personnel decisions. Already since the conclusion of the analysis period for this study (2005 through 2007), this flexibility has diminished in several ways. For example, the probationary period has been reduced from 4 to 3 years, and principals who choose to nonrenew a teacher now must have conducted at least one formal observation of the teacher prior to nonrenewal.

Data

The data for my study of this policy change come from several sources. Teacher personnel files provide information on teacher background, current assignment, and, for probationary teachers, whether or not they were renewed. I supplement this with information on school demographics, principal characteristics from personnel files, and student test-score information.

I examine dismissal among probationary teachers in CPS in three consecutive school years: 2004–05, 2005–06, and 2006–07. The sample excludes individuals who were employed by the central office, including speech pathologists, nurses, counselors, and teachers working in administrative or professional development capacities. Moreover, I exclude teachers in a handful of “alternative” schools that serve severely disabled students or other special populations, as well as teachers on leave or who were employed less than half time. For a small number of teachers who taught subjects such as art or music in multiple schools, I include only the observation in the school that is listed as their “primary” appointment. The final sample consists of 16,246 elementary school teachers and 7,764 high school teachers spread across 588 schools.

Measures of Teacher Quality

This analysis incorporates three proxies for teacher performance. First, I use teacher absences because they are well measured, are easy to interpret, and impose substantial nonfinancial and financial costs on the school. The second measure is the formal performance rating that the principal gave the teacher in prior years. Traditionally, principals rate teachers every one to three years (depending on the tenure status of the teacher) on a four-point scale that indicates superior, excellent, satisfactory, or unsatisfactory performance. While there are no high stakes associated with these ratings (virtually no teachers receive an unsatisfactory rating), there is considerable variation across teachers in the top rating categories, and they arguably provide a sense of how the principal views the teacher. The third measure is a value-added estimate of teacher effectiveness. This measure is meant to capture the extent to which each teacher contributes to student achievement growth from one year to the next, as measured by the standardized tests taken by students in CPS. While this is an objective and direct measure of one important dimension of teacher effectiveness, only a fraction of teachers work in grades and subjects in which students take standardized tests. It is not possible to calculate value-added measures for many teachers in our sample, including teachers in grade 2 or below, most teachers in grades 10 or above, and any teacher in a noncore subject. Unlike some school districts, Chicago traditionally has not maintained reliable data linking teachers to classrooms, particularly at the elementary level. Working with CPS officials, however, I was able to obtain such links for a limited sample of teachers and years, thus allowing me to create value-added measures for part of my sample.

Methods

The primary goal of my analysis is to determine which teacher, principal, and school characteristics are associated with the likelihood that a teacher will be dismissed. I first compare the probability that a teacher is dismissed across schools and years in order to discern any differences related to school characteristics. Then to examine the influence of teacher characteristics on the likelihood of dismissal, I compare teachers within the same year and school to account for unobserved school-level factors that might be correlated with teacher characteristics and the probability of dismissal.

A concern with this approach is that if the analysis fails to include a teacher characteristic that a) principals consider in the dismissal decision and b) is correlated with one of the included variables, the estimate for the included characteristic may be biased. One potentially important variant of this concern involves the supply of teachers. If it is more difficult to find qualified teachers in certain subjects or grade levels, then the principal may be less likely to dismiss teachers in these areas. To the extent that teachers in harder-to-staff areas are concentrated among particular demographic groups, or tend to graduate from particular institutions, the results for these teacher characteristics could be misleading. Also, schools fund teachers from a variety of revenue streams, and it may be difficult for principals to reallocate positions across funds. For this reason, if a school experiences a decline in a particular revenue fund, the principal may be more inclined to dismiss teachers funded by this source.

To address these concerns, I account in all analyses for the teacher’s program area (for example, regular education grades 1 to 3, regular education grades 4 to 8, secondary math, secondary science, bilingual education, vocational education, etc.) and for the revenue source from which each teacher position is funded.

Of course, it is still possible that my results concerning specific teacher characteristics suffer from a standard omitted variable bias. For example, it may be the case that high rates of absenteeism are associated with a bad attitude or shirking in other dimensions, and it is these factors, rather than the absences per se, that the principal is reacting to in dismissing teachers with more absences. In this case, one may not be able to say anything definitive about principal views regarding teacher absenteeism itself, but rather about behaviors and characteristics associated with absenteeism, all of which presumably speak to performance in some form or another.

Dismissal Policy Impact

Each year under the new policy, roughly 11 percent of probationary teachers were dismissed, despite the fact that more than one-third of schools did not dismiss any teachers. The numbers of teachers who were nonrenewed in any given year likely overstates the impact of the policy because a number of young teachers would likely have left CPS in the absence of the policy, either voluntarily or due to subtle “encouragement” on the part of the principals. If the dismissal policy merely formalized previously informal dismissals, however, then one would not necessarily expect to find a substantial change in separations.

Comparing dismissal rates before and after implementation of the new policy provides insight on this issue. In the three years prior to the introduction of the policy, roughly 10 to 15 percent of first-year probationary teachers left CPS and an additional 4 percent moved to a different CPS school. In the years after the policy was in place, the corresponding rates were roughly 18 and 10 percent, respectively. Comparing the year immediately prior to establishment of the policy (2004) with the first two years of the policy’s implementation (2005 and 2006), it appears that the separation rate increased by roughly 9 percentage points (see Figure 1). In contrast, there was virtually no change among more-experienced teachers (i.e., those with 6 to 15 years of experience), who were not subject to the policy. The dismissal policy therefore appears to have had at least a modest impact on the number of teacher separations, although the impact is not as large as the overall nonrenewal numbers would suggest.

It is worth noting that more than half of the dismissed teachers were rehired the following year by another school in the district. For example, 50.6 percent and 56.4 percent of first-year probationary elementary and high school teachers, respectively, who were dismissed in spring 2005 were rehired by a CPS school in the fall. At least some of the dismissals under the policy were the result of position cuts, in which case the teacher’s former principal may have provided the teacher with a good recommendation; it is therefore not surprising that some fraction of dismissed teachers were rehired. It is also likely that some fraction of teachers dismissed due to poor performance were also rehired by other CPS schools.

Which school and principal characteristics are related to dismissal? In both elementary and secondary schools, principals in the district’s larger schools dismissed a smaller fraction of probationary teachers. In elementary schools, higher student achievement at the school is associated with a smaller fraction of probationary teachers being dismissed. Among high schools, however, schools with higher-achieving students dismissed a larger fraction of their probationary teachers. Principals who attended more competitive colleges and principals who were older dismissed a smaller proportion of teachers in both elementary and high schools. Male high-school principals dismissed a significantly smaller percentage of their teachers, while principal gender did not play as important a role at the elementary level. Finally, principals new to the building dismissed a substantially larger fraction of teachers in elementary schools, but not in high schools.

Teacher Characteristics

Turning to the characteristics of individual teachers, I find that prior-year principal evaluations and current-year teacher absences both influence the likelihood of dismissal (see Figure 2). Teachers who were rated satisfactory in the prior academic year were 22.1 percentage points more likely to be nonrenewed than teachers in the same school who were rated superior. Teachers rated excellent were 4.3 percentage points more likely to be dismissed than those rated superior. Given an average dismissal rate of roughly 11 percent, these results suggest that teacher performance as reflected in prior evaluations is strongly associated with dismissal. Teachers who were absent 11 to 20 times between September and March of the current year were also 11.3 percentage points more likely to be nonrenewed than their colleagues who were never absent. Teachers absent 6 to 10 days were 3.5 percentage points more likely to be dismissed.

The results also indicate that principals value teachers with stronger educational backgrounds as measured by college quality. For example, a teacher who attended a highly competitive college (with a Barron’s ranking of four) is nearly 3 percentage points (roughly 15 percent) less likely to be dismissed than a teacher who attended a least-competitive (unrated) college. On the other hand, on average, principals do not seem to value certification exam performance or advanced degrees, at least after taking into account the other available measures of teacher performance.

Interestingly, probationary teachers who were dismissed from another school in the prior year, and rehired by the current school, are substantially more likely to be dismissed a second time. For example, elementary school teachers who were dismissed from another school in the prior year were 4.9 percentage points (about 45 percent) more likely to be let go relative to first-year teachers in the school. In high school, previously dismissed teachers were 13.4 percentage points (more than 130 percent) more likely to be dismissed than first-year teachers. These results suggest that many of the initial nonrenewal decisions were not idiosyncratic, stemming from a particularly bad match, or based on temporary difficulties experienced by the teacher. Rather, they suggest that, at least in many cases, the initial nonrenewal decision reflected a concern with the teacher’s general productivity.

These results provide evidence that principals consider some measures of teacher performance and qualifications in making their dismissal decisions. To the extent that one views student achievement as the primary outcome of interest, however, one should directly assess how a teacher’s ability to improve student achievement influences the likelihood of dismissal. I provide some evidence on this issue by focusing on the relationship between teacher value-added and dismissal for the subsample of 803 elementary school and 1,134 high school teachers for which value-added measures are available.

For elementary schools, a one-standard-deviation increase in teacher value-added is associated with a 7.1-percentage-point (over 100 percent) decrease in the likelihood of dismissal (see Figure 3). In contrast, I find that teacher value-added has zero association with dismissal among the sample of 9th-grade core-subject teachers in high schools. One possible reason for the difference across grade levels is that the assessment used for the 9th-grade value-added measure is the PLAN test, which is given in the fall of a student’s 10th-grade year. PLAN is developed by ACT and is not tightly linked to any particular curriculum. Hence, because of both the timing of the exam and its content, the 9th-grade value-added measures may not capture teacher effectiveness as well as the elementary value-added measures.

Do Principals Discriminate?

One potential concern about policies like Chicago’s that provide principals with greater discretion in personnel decisions is that principals would dismiss teachers capriciously or on the basis of criteria unrelated to performance. Indeed, I find that several teacher demographics, including age, gender, and race, are associated with the likelihood of dismissal, even after controlling for the measures of teacher performance and qualifications described above. Principals are 3.8 percentage points more likely to dismiss male teachers than female teachers, an effect of more than 25 percent given the baseline dismissal rate of 10 to 12 percent. Principals are considerably more likely to dismiss older teachers. For example, teachers 36 to 50 years of age are 4 percentage points (33 percent) more likely to be dismissed than teachers age 22 to 28. The relatively small number of probationary teachers over age 50 is 10 percentage points (nearly 100 percent) more likely to face dismissal than their youngest counterparts. And black teachers are 2.1 percentage points less likely to be dismissed than their colleagues.

While these results raise some concerns, it would be incorrect to conclude on the basis of this evidence alone that principals in Chicago were acting in a discriminatory manner. The analysis reported here cannot control for many direct measures of teacher qualities that principals could legitimately consider in making a dismissal decision (e.g., energy, enthusiasm, ability to relate to children, familiarity with the best instructional practices). Moreover, the sample selection introduced by nonrandom hiring may lead to biased estimates of the relationship between dismissal and any easily observable, predetermined teacher characteristic such as age or gender. If, for example, male teachers were less productive on average than female teachers (or even if the principal believed this to be the case), then the marginal male teacher who was hired must be more attractive on some other, likely unobservable, dimension relative to the marginal female teacher hired.

In order to shed light on the issue of principal discrimination, I examine whether principals are more likely to dismiss teachers of a different gender, age, or race from their own. Although principals are no more likely to dismiss a teacher of the opposite gender, they are somewhat more likely to dismiss teachers of a different race. While these patterns could indicate discrimination, it is possible that they are explained by other factors. Given the widespread belief that same-race role models are crucial for low-income students, it would not be surprising if principals took into account the composition of their student body when making dismissal decisions. Indeed, insofar as prior research has demonstrated that, all else equal, students learn more when taught by a teacher of the same race, this might be a legitimate determination on the part of the principal. My results provide support for this hypothesis. I find that as the fraction of students in the school that share the race of the teacher rises, the likelihood that the teacher will be dismissed declines. Specifically, an increase of 50 percentage points in the fraction of students who share the teacher’s race decreases the likelihood that the teacher will be dismissed by slightly more than 1 percentage point, or 10 percent. More importantly, the evidence that principals are more likely to dismiss a teacher of a different race becomes statistically insignificant after controlling for this variable.

Finally, I find evidence that younger principals are more likely to dismiss older teachers than they are to dismiss younger teachers. There are no obvious explanations for this pattern, although one might speculate that younger principals may value different characteristics in a teacher than older principals. Regardless, this pattern does seem to warrant further exploration.

Conclusions

By comparing the characteristics of dismissed versus nondismissed probationary teachers within the same school and year, the analysis presented above provides a unique source of evidence on which teacher characteristics principals value most highly. I find that principals do consider teacher performance in determining which teachers to dismiss. Principals are significantly more likely to dismiss teachers who are frequently absent and who have received unsatisfactory evaluations in the past. Perhaps most telling, elementary school teachers who were dismissed had significantly lower impacts on student achievement in prior years than their peers who were not dismissed.

These results suggest that reforms along the lines of the Chicago policy could improve student achievement by providing principals with the tools to manage the quality of personnel in their classrooms. It should be noted, however, that many principals—including those in some of the worst-performing schools in the district—did not dismiss any teachers despite the new policy. The apparent reluctance of some Chicago principals to utilize the additional flexibility granted under the new contract may indicate that issues such as teacher supply and/or social norms governing employment relations are more important factors than policymakers have realized.

Brian Jacob is professor of education policy and economics at the University of Michigan. This article is based on a study that is forthcoming in Educational Evaluation and Policy Analysis.

Calls to reform teacher layoff policies have begun to appear with regularity in newspaper editorials, policy briefs, and statehouses—and for good reason. A growing body of research confirms that teacher quality is the most influential in-school factor driv­ing student achievement. That being the case, teacher dismissal policies and procedures can have profound implications for how much students learn.

Newly available data on “reduction-in-force” (RIF) notices received by teachers in Washington State shed light on the consequences of existing layoff policies for student achievement as well as the consequences of adopting alternatives. Our analysis of these data provides strong evidence that seniority plays an out­sized role in determining which teachers are targeted for layoffs, likely in part because collective bargaining agreements ordinarily require that the teachers last hired are the first to be fired. Those in subject areas with teacher shortages, such as mathematics and sci­ence, are less likely than other teachers to receive a lay­off notice, suggesting that districts have some degree of flexibility in their dismissal procedures. However, were districts to adopt policies that allowed admin­istrators to dismiss teachers according to their effec­tiveness rather than their seniority, they could lay off fewer teachers, achieve the same budgetary savings, and increase the overall efficacy of their teaching force.

Seniority-Based Layoff Policies

In the overwhelming majority of school-district collective bargaining agreements, “last in, first out” provisions make seniority the determining factor in which teachers are laid off. All of the 75 largest school districts in the nation use seniority as a factor in layoff decisions, and seniority is the sole factor determining the order of layoffs in more than 70 percent of these districts.

The situation in Washington State—the focus of this study—looks similar. A review of the collective bargaining agreements operating in Washington’s 10 largest school districts shows that all use seniority as a basis for determining layoffs, and 8 of these districts use seniority as the only determinant of which teachers get laid off.

There are notable examples of districts that do not rely solely on seniority. In 2004, the Chi­cago Public Schools changed its policies to allow principals’ evaluations of untenured teachers to influence layoff decisions (see “Principled Principals” research). And the Los Angeles Unified School District recently agreed to limit the use of seniority in layoff determinations as part of a settlement in a lawsuit brought by the American Civil Liberties Union (ACLU). Over the past two years, more than a dozen states have sought to change laws that make seniority the determining factor in layoff decisions; so far, Florida, Idaho, Utah, and Ohio have succeeded.

Driving these changes is a belief that seniority-based layoff policies may have negative consequences for student achieve­ment. First, to achieve a targeted budget reduction, school districts need to lay off a greater number of junior teachers than senior teachers (as junior teachers have lower salaries), meaning that a seniority-based layoff policy will cause class sizes to rise more than they would under an alternate arrange­ment. Second, the most-senior teachers may not be the most effective teachers. With a seniority-based layoff policy, school systems may be forced to cut some of their most promising new talent rather than dismiss more-senior teachers, who may not be terribly effective in raising student achievement. A final way in which seniority-based systems may have consequences for student achievement is that strict adherence to seniority would require at least some districts to lay off teachers in subject areas with teacher shortages, such as math and special education.

Beyond the effects of seniority-based layoffs on the teacher workforce as a whole are potential distributional conse­quences. In many districts, schools with high proportions of at-risk students tend to employ the most first- and second-year teachers. Under a seniority-based layoff policy, these schools stand to lose the largest share of their teachers.

Data

This study relies on a unique dataset from Washington State that links teachers to their schools and, in some cases, to their students; the dataset also includes information on those teachers who received RIF notices in the 2008–09 and 2009–10 school years. In the 2008–09 school year, 2,144 employees received a layoff notice and in 2009–10, some 450 employees received a notice.

Employees who received these notices can be linked with administrative records of their credentials, school assignments, academic degrees, and compensation. The administrative database we used provides a record of employees working in Washington State’s school districts and includes information such as their places of employ­ment, experience and degree, gender and race, and annual compensation levels.

We restrict our analysis to employees who were in a teach­ing position the year they received a layoff notice. Our final sample includes 1,717 teachers who received a layoff notice in 2008–09 and 407 teachers who received one in 2009–10, with 130 teachers who received a layoff notice in both school years. Overall, about 2 percent of teachers in the state received a layoff notice in either year. It is important to stress that not all these teachers were ultimately laid off, largely due to the influx of federal stimulus money. Of the 1,717 teach­ers who received a RIF notice in 2008-09, for example, 1,457 returned to the same district in 2009-10. We still focus on all RIF notices because they indicate the teachers who were targeted for layoffs, and thus tell us about the likely effects of the system that governs layoffs.

The database does not include a direct measure of a teach­er’s seniority in the current district, so we estimate seniority based on how many years the teacher has been employed by the same district. The credentials data include where each teacher was trained and in what areas each teacher holds endorsements. We create a measure of the selectivity of each teacher’s college and code each endorsement a teacher holds in any of 10 subject areas.

Information about the schools in which teachers are employed comes from two sources. Washington State Report Card data provide measures of racial composition, student-teacher ratios, the percentages of students enrolled in the free or reduced-price meals program, total enrollment, and the percentage of students who passed the reading and math Washington Assessment of Student Learning exams in each teacher’s school. We use the Common Core of Data to iden­tify teachers in urban areas, the grade level of each teacher’s school, and the per-pupil expenditure on instruction by each teacher’s district.

We can also link a subset of teachers to their students’ test-score performance, which allows us to use value-added models to estimate their teaching effectiveness. Our data on student achievement come from the Washington State Assessment of Student Learning, a statewide test given annually in 3rd through 8th grade as well as in 10th grade. The student database also includes information on race and ethnicity, free or reduced-price meal eligibility, and status in the following programs: Learning Assistance Program reading/math, Title I reading/math, Title I Migrant, Gifted/Highly Capable, State Transitional Bilingual Program, and Special Education.

Methods

We first examine the simple associations between the various teacher and school characteristics listed above and the likelihood of receiving a layoff notice. In order to provide a more detailed picture of the factors that are associated with teacher layoff notices, we then examine the effects of each of these various factors on the prob­ability that a teacher received a layoff notice, while con­trolling for the others. Of course, these relationships are correlations only and in theory may not represent causal relationships. However, we are confident that, despite the nonexperimental nature of this study, its findings none­theless provide an accurate picture of the causal impact of, for instance, a teacher’s credential on the likelihood of receiving a layoff notice.

The teacher characteristics that we examine include senior­ity in district, degree level (master’s or higher vs. bachelor’s), gender, race, college selectivity, and endorsement area. The school characteristics include whether it is in an urban area, grade level (e.g., high school), the number of students enrolled, student-teacher ratio, the percentage of students who are eli­gible for the free or reduced-price lunch program, the percent­age of minority students, and measures of student achievement in reading and math. In addition, we control for district-level characteristics, including total enrollment, per-pupil expendi­tures, and percentage of funding that comes from local, state, and federal sources.

These analyses identify the teacher, school, and district characteristics that are associated with layoff notices, but perhaps of greater interest is the relative effectiveness of teachers who receive layoff notices. For the subset of teach­ers who can be linked to students, we are able to estimate value-added measures of classroom performance for each teacher in each year. These indicate how well a teacher’s students did relative to other teachers’ students, controlling for prior student achievement and for student and fam­ily background characteristics (for example, age, race and ethnicity, disability, free or reduced-price lunch status, and parental education level).

Who Gets RIFed?

Not surprisingly, we find that most of the teachers receiving layoff notices are relatively junior. Approximately 60 percent of teachers receiving layoff notices have two or fewer years of experience, and approximately 80 percent have two or fewer years of seniority within their current district. It is interesting to note, however, that some teachers who receive layoff notices are well into their careers, implying that at least some districts in the state are making judgments about which teachers should be laid off based on criteria other than seniority.

Teachers who received layoff notices are also far less likely to hold an advanced degree. Consequently, there is an aver­age difference of about $15,000 in salary between teachers who did and did not receive notices. Had all 1,717 teachers who received layoff notices in 2008–09 actually been laid off, the salary savings in the state would have been $5,521,238. As noted earlier, one of the prevail­ing critiques of seniority-based layoffs is that it is necessary to lay off more teachers in order to attain a specified budget objective than it would be if districts used alternative criteria. If teach­ers were laid off at random (so that the laid-off teachers made the average salary in their dis­trict), we estimate that it would only be neces­sary to lay off 1,349 teachers in order to attain the same budgetary savings. This is roughly 20 percent less than the actual number of teachers who received layoff notices.

According to the 2006 report “Educator Supply and Demand in Washington State,” there are 14 endorsement areas for which there are “high degrees of shortage,” all of which fall into math, science, or special education. We classify any teacher with an endorsement in one of these areas accordingly. There is some evidence to suggest that school districts are choosing to retain teachers in subject areas with teacher shortages, with 13.3 per­cent of teachers that received layoff notices falling into such a category compared to 15.1 percent of teachers who did not receive a notice.

Teachers receiving a notice tended to be in smaller schools, but were not, in general, more likely to be teaching in schools with high proportions of minority students or lower test-score levels. However, school-level measures can mask a significant degree of teacher sorting across classrooms within schools. For the subset of teachers who can be linked to their students, we find that teachers who received a layoff notice are more likely to be teaching poor, non-white, and lower-scoring students than other teachers.

We next examine our value-added measures of teacher effectiveness and find that teachers who received layoff notices were about 5 percent of a standard deviation less effective on average than the average teacher who did not receive a notice. This result is not surprising given that teach­ers who received layoff notices included many first- and second-year teachers, and numerous studies show that, on average, effectiveness improves substantially over a teacher’s first few years of teaching.

Explaining RIFs

Our analysis of multiple factors indicates that, as expected, seniority plays an important role in determining whether teachers receive a layoff notice. We find additional evi­dence that districts are choosing to retain teachers thought to have advanced or atypical skills. On average, teachers with a master’s degree or an endorsement in a subject area with teacher shortages are about 0.6 percentage points less likely to receive a RIF notice. Conversely, teachers with endorsements in health, physical education, or the arts are far more likely to receive a layoff notice. Finally, we find evidence that school districts behave strategically by retaining teachers who have endorsements in multiple areas and therefore provide flexibility in terms of the classes they can teach. Perhaps surprisingly, controlling for district and school characteristics does not noticeably change the results reported above, and few of the school-level vari­ables identifying student demographics are predictors of which teachers receive layoff notices.

Finally, we ran our analysis including value-added measures of teacher effective­ness for the subset of teachers we are able to link to individual students. It is first worth noting that the inclusion of the teacher effec­tiveness measures does little to change the estimated effects of the teacher, school, and district characteristics discussed above. More importantly, the effects of the value-added measures (based on both math and read­ing scores) are close to zero, suggesting that effectiveness plays little or no role in deter­mining which teachers are targeted for lay­offs. And, these results were robust to a vari­ety of different ways of measuring teacher value added. In other words, the fact that teachers who received layoff notices were, on average, somewhat less effective than their peers is an artifact of the relationship between effectiveness and seniority.

Policy Implications

Our findings largely comport with what one would expect given seniority provisions in collective bargaining agree­ments. The surprise is that factors other than seniority do appear to influence which teachers are targeted for layoffs.

To get a more concrete sense of the extent to which various factors play into the targeting of teachers for layoffs, we ran simulations based on the effects calculated by our statistical model. First, we calculate the expected probability of a teacher with each combination of endorsement area and seniority level receiving a layoff notice. Although a teacher’s endorse­ment area does affect the likelihood of being laid off, the effect is far smaller than the influence of seniority. For instance, we estimate the probability that a first-year special education teacher receives a layoff notice is 6.2 percent, compared to 17 percent for a first-year health/physical education teacher. This difference is statistically significant, but it pales in com­parison to the difference in probability for a first-year teacher compared to a teacher with 12 or more years of seniority: The estimated probability of a teacher with 12 or more years of seniority receiving a layoff notice is less than one-quarter of 1 percent for every endorsement area (see Figure 1).

Next we examine the implications of employing an effec­tiveness-based layoff policy rather than the seniority-driven system currently in place. First, we calculate a value-added measure of effectiveness that com­bines data from all available years and both sub­jects (averaging math and reading). Teachers in each school district are then ranked accord­ing to this value-added score. Finally, starting with the least effective teachers in each district and moving up the effectiveness ladder, enough teachers are assigned to a hypothetical layoff pool to achieve a budgetary savings for each district that is at least as great as the budgetary savings each district would have seen had all the teachers who received a layoff notice in 2008–09 actually been laid off.

The overlap between the subgroup of teach­ers who received a layoff notice and the sub­group of teachers who received one in our simu­lation is relatively small—only 23 teachers (or 16 percent of the teachers for whom we could estimate value-added who received a layoff notice). Moreover, because the teachers who received layoff notices in our simulation were more senior (and had higher salaries) than the teachers who actually received layoff notices, the simulation results in far fewer layoffs. We calcu­late that districts would only have to lay off 132 teachers under an effectiveness-based system in order to achieve the same budgetary savings they would achieve with 145 layoff notices under today’s seniority-driven system, a difference of about 10 percent.

As expected, there are large differences in classroom effec­tiveness between teachers who actually received layoff notices and those who would have received them in our effectiveness-based simulation. The two groups differ by about 20 percent of a standard deviation in students’ math and reading achieve­ment (see Figure 2). The magnitude of the difference is strik­ing, roughly equivalent to having a teacher who is at the 16th percentile of effectiveness rather than at the 50th percentile. This difference corresponds to roughly 2.5 to 3.5 months of student learning.

Since there is little overlap between the samples under these different scenarios, we investigate the likelihood that different types of students might be disproportion­ally affected by one type of layoff system. For the subset of teachers who can be linked to student-level data, we consider the characteristics of the students whose teachers received a layoff notice under the actual system and in our simulation. We find that the probability that students in a particular subgroup have a teacher who received a layoff notice varies considerably from one subgroup to the next. In particular, black students are far more likely than other students to have been in a classroom of a teacher who received a layoff notice. The effectiveness-based layoffs result in fewer layoff notices and are much more equita­bly distributed across student subgroups; black students in particular are only marginally more likely to have been in a classroom with a teacher who received a layoff notice under this system.

Districts across the country are rethinking layoff strate­gies. This is sensible, because although the simplicity and transparency of a seniority-based system certainly has advantages, it is hard to argue that it is in the best interest of students. The effectiveness-based system in our simulation would result in a very different group of teachers targeted for layoffs than does the current system and in layoffs that affect different segments of the student population. Most importantly, the differences in the effectiveness of teach­ers laid off under each type of system have implications for student achievement.

Dan Goldhaber is director of the Center for Education Data and Research at the University of Washington Bothell and a co-editor of Education Finance and Policy. Roddy Theobald is a researcher at the Center for Education Data and Research and doctoral student in statistics at the University of Washington.

The working paper on which this article is based is available here.

An unabridged version of this article is available here.

In recent years, a consensus has emerged among researchers that teacher quality matters enormously for student performance. Students taught by more-effective teachers learn substantially more over the course of the year than students taught by less-effective teachers. Yet little is known about what makes for a more-effective teacher.

Most research on teacher effectiveness has focused on teacher attributes, finding that readily measurable characteristics such as experience, certification, and graduate degrees generally have little impact on student achievement. Relatively few rigorous studies look inside the classroom to see what kinds of teaching styles are the most effective. We tackle this underexplored area by investigating the relative effects of two teacher practices—lecture-style presentations and in-class problem solving—on the achievement of middle-school students in math and science.

Ever since John Dewey explored hands-on learning at the University of Chicago Laboratory School more than a century ago, lecture-style presentations have been criticized as old-fashioned and ineffective. It is said, for example, that lectures presume that all students learn at the same pace and fail to provide instructors with feedback about which aspects of a lesson students have mastered. Students’ attention may wander during lectures, and they may more easily forget information they encountered in this passive manner. Lectures also emphasize learning by listening, which may disadvantage students who favor other learning styles.

Alternative instructional practices based on active and problem-oriented learning presumably do not suffer from these disadvantages. But they may have their own shortcomings. Learning by problem-solving may be less efficient, as discovery and problem-solving often take more time than mastering information received from an authority figure. And incorrect or misleading information may be conveyed in conversations among students in middle schools.

Nonetheless, a number of small-scale studies have identified positive impacts of interactive teaching styles on student learning. As a consequence, prominent organizations such as the National Research Council and the National Council of Teachers of Mathematics, since at least 1980, have called for teachers to engage students in constructing their own new knowledge through more hands-on learning and group work. By the mid-1990s, in a study for the National Institute for Science Education, Iris Weiss could identify “some encouraging signs. The majority of elementary, middle, and high school science and mathematics classes worked in small groups at least once a week, and roughly one in four classes did so every day. Moreover, the use of hands-on activities had increased since the mid-1980s.” Even so, more than a decade later, traditional lecture and textbook methodologies continue to be a significant component of science and mathematics instruction in U.S. middle schools. A rigorous, large-scale study has yet to resolve a question that has divided pedagogical thinking for generations.

In our study, we examine whether student achievement in the United States is affected by the share of teaching time devoted to lecture-style presentations as distinct from problem-solving activities. Employing information on in-class time use provided by a nationally representative sample of U.S. teachers in the 2003 Trends in International Mathematics and Science Study (TIMSS), we estimate the impact of teaching practices on student achievement by looking at the differential effects on the same student of two different teachers, using two different teaching strategies. We find that teaching style matters for student achievement, but in the opposite direction than anticipated by conventional wisdom: an emphasis on lecture-style presentations (rather than problem-solving activities) is associated with an increase—not a decrease—in student achievement. This result implies that a shift to problem-solving instruction is more likely to adversely affect student learning than to improve it.

Data and Methodology

Our research draws on data from the 2003 Trends in International Mathematics and Science Study (TIMSS). The TIMSS data comprise information on students in two grades in a number of countries, but we utilize only information on 8th-grade students in the United States. Our sample includes 6,310 students in 205 schools with 639 teachers (303 math teachers and 355 science teachers, of which 19 teach both subjects). In addition to test scores in math and science, the TIMSS data include background information on students’ home and family life as well as data on teacher characteristics, qualifications, and classroom practices. School principals provide information on school characteristics.

Most important for our purpose, teachers were asked what proportion of time in a typical week students spent on each of eight in-class activities. The overall time in class apportioned to three of these activities—listening to lecture-style presentation, working on problems with the teacher’s guidance, and working on problems without guidance—likely provides a good proxy for the time in class in which students are taught new material. We divided the amount of time spent listening to lecture-style presentations by the total amount of time spent on each of these three activities to generate a single measure of how much time the teacher devoted to lecturing relative to how much time was devoted to problem-solving activities.

A change in our measure of teaching style can be interpreted as a shift from spending time on one practice to spending time on the other, holding constant the total time spent on both practices. For example, an increase of 0.1 indicates that 10 percentage points of total time devoted to teaching new material are shifted from teaching based on problem solving to giving lecture-style presentations. We combined the other teaching activities (besides lecturing and problem solving) into a separate measure of the share of total teaching time devoted to other activities and control for this measure throughout our analysis. We also control for the total number of minutes per week that the teacher reported teaching the math or science class, as more total instructional time could have an independent effect on student learning.

Although it is difficult to determine from the TIMSS data exactly how much time is spent on lecturing as distinct from problem-solving activities, it appears that teachers generally follow the advice given by progressive educators. On average, they allocate twice as much time to problem-solving activities as to direct instruction. Specifically, teachers devote about 40 percent of class time to problem-solving activities (with or without teacher guidance); during roughly 20 percent of class time, students listen to the initial presentation of material to be learned. The remainder of the class time is allocated to such tasks as class management, reviewing homework, re-teaching the material, and clarifying content (see Figure 1).

Teachers who spent more time lecturing were more likely to be male and under age 50. Interestingly, they were also less likely to have the maximum number of years of teacher training registered by the background survey or to have taken pedagogical or content knowledge classes in the prior two years (see Figure 2).

A key challenge in studying the effects of teaching practices is that teachers may adjust their methods in response to the ability or behavior of their students. If teachers tend to rely more on lectures when assigned more capable or attentive students, this would generate a positive relationship between the amount of time spent lecturing and student achievement, even in the absence of a true causal effect. Similarly, there could be unobserved differences between students whose teachers rely more and less heavily on lecturing if, for example, teachers in schools serving low-income students adopt different practices than teachers in other types of schools.

To address these concerns, we exploit the fact that the TIMSS study tested each student in both mathematics and science. This allows us to compare the math and science test scores of individual students whose teacher in one subject tended to emphasize a different teaching style than their teacher in the other subject. In other words, we ask, if a given student’s math teacher spent more (or less) time lecturing than his or her science teacher, does the student perform better or worse on the math test than on the science test?

Results

Contrary to contemporary pedagogical thinking, we find that students score higher on standardized tests in the subject in which their teachers spent more time on lecture-style presentations than in the subject in which the teacher devoted more time to problem-solving activities. For both math and science, a shift of 10 percentage points of time from problem solving to lecture-style presentations (e.g., increasing the share of time spent lecturing from 20 to 30 percent) is associated with an increase in student test scores of 1 percent of a standard deviation. Another way to state the same finding is that students learn less in the classes in which their teachers spend more time on in-class problem solving.

Importantly, the strength of the relationship increases when we restrict our analysis to the roughly one-third of students in the TIMSS sample who had the exact same peers in both their math and science classes. Among this group of students, a shift of 10 percentage points of time from problem solving to lecturing is associated with an increase in test scores of almost 4 percent of a standard deviation—or between one and two months’ worth of learning in a typical school year (see Figure 3). This pattern increases our confidence that the overall result does not reflect differences in the peer composition of students’ math or science classes. In fact, it suggests that peer effects may actually be leading us to understate the strength of the relationship between lecturing and student learning.

Do certain types of students benefit more from lectures than others? We find suggestive evidence that the relationship between lecture-style teaching and achievement is strongest among higher-achieving and more-advantaged students. For example, the positive effect is largest for students who report having more than one bookcase in the home, a rough indicator of the quality of their home environment. There is no evidence, however, that lower-achieving students or students from less-advantaged backgrounds learn less when their teachers emphasize lectures.

These patterns are consistent with the findings of a 1997 study by Dominic Brewer and Dan Goldhaber, which found that more in-class problem solving for American 10th-grade students in math is related to lower test scores on a standardized test. Because our results are based on comparisons of the same student in two different classes, however, they are less subject to the concern that teachers adjust their practices based on the students to which they are assigned. Furthermore, the other commonly investigated teacher characteristics (e.g., gender, experience, and credentials) do not show significant effects on student achievement in our analysis. This is in line with previous findings in the literature and underscores the importance of the statistical relationship between more lecture-style teaching and student achievement.

While the richness of the TIMSS data enables us to control for an unusually large set of teacher characteristics, our results could still be biased if teachers with different effectiveness levels are more likely to choose different teaching styles. For example, if more-effective teachers tend to spend more time lecturing because they are good at it and enjoy it, then our results could show a positive effect of lecture-style presentations, even if those teachers would have been even more effective had they devoted more time on problem-solving activities. Given the pedagogical emphasis on the use of problem-solving activities, it seems unlikely that the very best teachers would be using the less-effective teaching style (the only alternative explanation for our finding).

Still, it is important to keep in mind that our results are limited to student achievement as measured by the 2003 TIMSS test scores in 8th-grade math and science in the United States. Different results might be found for different subjects, grades, or tests. Depending on the teacher, the students, the content taught, or other factors, problem-solving activities could turn out to be the more effective style. Even though lecture-style teaching seems to be a more effective method in middle-school math and science, that does not mean it would be the preferable approach to elementary-school reading.

Also, our findings are based on student performance on the TIMSS math and science exams, which are designed to measure mastery of factual knowledge of the curricula that schools expect students to learn. Other tests intended to measure problem-solving ability and the competence to apply mathematical and scientific concepts in real-world settings (such as the Programme for International Student Assessment [PISA] administered by the Organization of Economic Cooperation and Development) might yield different results. Unfortunately, we are unable to ascertain whether this might be the case, as PISA did not ask teachers about their pedagogical approach.

Finally, our information on teaching practices, which is based on in-class time use reported by teachers, does not allow us to distinguish between different implementations of teaching practices. In other words, a certain teaching technique may be very effective if implemented in the optimal way. But the strength of our approach is that it examines which teaching style turns out to be effective, on average, for teachers in general. Optimal teaching methods that cannot be executed by teachers in general may do more harm than good.

Conclusion

Given the limitations of the data, our finding that spending increased time on lecture-style teaching improves student test scores results should not be translated into a call for more lecture-style teaching in general. But the results do suggest that traditional lecture-style teaching in U.S. middle schools is less of a problem than is often believed.

Newer teaching methods might be beneficial for student achievement if implemented in the proper way, but our findings imply that simply inducing teachers to shift time in class from lecture-style presentations to problem solving without ensuring effective implementation is unlikely to raise overall student achievement in math and science. On the contrary, our results indicate that there might even be an adverse impact on student learning.

Guido Schwerdt is a postdoctoral fellow at the Program on Education Policy and Governance (PEPG) at Harvard University and a researcher at the Ifo Institute for Economic Research in Munich, Germany. Amelie C. Wuppermann is a postdoctoral researcher at the University of Mainz, Germany.

An unabridged version of this article is available here.

The ubiquity of “satisfactory” ratings stands in contrast to a rapidly growing body of research that examines differences in teachers’ effectiveness at raising student achievement. In recent years, school districts and states have compiled datasets that make it possible to track the achievement of individual students from one year to the next, and to compare the progress made by similar students assigned to different teachers. Careful statistical analysis of these new datasets confirms the long-held intuition of most teachers, students, and parents: teachers vary substantially in their ability to promote student achievement growth.

The quantification of differences has generated a flurry of policy proposals to promote teacher quality over the past decade, and the Obama administration’s recent Race to the Top program only accelerated interest. Yet, so far, little has changed in the way that teachers are evaluated, in the content of pre-service training, or in the types of professional development offered. A primary stumbling block has been a lack of agreement on how best to identify and measure effective teaching.

A handful of school districts and states—including Dallas, Houston, Denver, New York, and Washington, D.C.—have begun using student achievement gains as indicated by annual test scores (adjusted for prior achievement and other student characteristics) as a direct measure of individual teacher performance. These student-test-based measures are often referred to as “value-added” measures. Yet even supporters of policies that make use of value-added measures recognize the limitations of those measures. Among the limitations are, first, that these performance measures can only be generated in the handful of grades and subjects in which there is mandated annual testing. Roughly one-quarter of K–12 teachers typically teach in grades and subjects where obtaining such measures is currently possible. Second, test-based measures by themselves offer little guidance for redesigning teacher training or targeting professional development; they allow one to identify particularly effective teachers, but not to determine the specific practices responsible for their success. Third, there is the danger that a reliance on test-based measures will lead teachers to focus narrowly on test-taking skills at the cost of more valuable academic content, especially if administrators do not provide them with clear and proven ways to improve their practice.

Student-test-based measures of teacher performance are receiving increasing attention in part because there are, as yet, few complementary or alternative measures that can provide reliable and valid information on the effectiveness of a teacher’s classroom practice. The approach most commonly in use is to evaluate effectiveness through direct observation of teachers in the act of teaching. But as “The Widget Effect” reports, such evaluations are a largely perfunctory exercise.

In this article, we report a few results from an ongoing study of teacher classroom observation in the Cincinnati Public Schools. The motivating research question was whether classroom observations—when performed by trained professionals external to the school, using an extensive set of standards—could identify teaching practices likely to raise achievement.

We find that evaluations based on well-executed classroom observations do identify effective teachers and teaching practices. Teachers’ scores on the classroom observation components of Cincinnati’s evaluation system reliably predict the achievement gains made by their students in both math and reading. These findings support the idea that teacher evaluation systems need not be based on test scores alone in order to provide useful information about which teachers are most effective in raising student achievement.

The Cincinnati Evaluation System

Jointly developed by the local teachers union and district more than a decade ago, the Cincinnati Public Schools’ Teacher Evaluation System (TES) is often cited as a rare example of a high-quality evaluation program based on classroom observations. At a minimum, it is a system to which the district has devoted considerable resources. During the yearlong TES process, teachers are typically observed and scored four times: three times by a peer evaluator external to the school and once by a local school administrator. The peer evaluators are experienced classroom teachers chosen partly based on their own TES performance. They serve as full-time evaluators for three years before they return to the classroom. Both peer evaluators and administrators must complete an intensive training course and accurately score videotaped teaching examples.

The system requires that all new teachers participate in TES during their first year in the district, again to receive tenure (usually in their fourth year), and every fifth year thereafter. Teachers tenured before 2000–01 were gradually phased into the five-year rotation. Additionally, teachers may volunteer to be evaluated; most volunteers do so to post the high scores necessary to apply for selective positions in the district (for example, lead teacher or TES evaluator).

The TES scoring rubric used by the evaluators, which is based on the work of educator Charlotte Danielson, describes the practices, skills, and characteristics that effective teachers should possess and employ. We focus our analysis on the two (out of four total) domains of TES evaluations that directly address classroom practices: “Creating an Environment for Student Learning” and “Teaching for Student Learning.” (The other two TES domains assess teachers’ planning and professional contributions outside of the classroom; scores in these areas are based on lesson plans and other documents included in a portfolio reviewed by evaluators.) These two domains, with scores based on classroom observations, contain more than two dozen specific elements of practice that are grouped into eight “standards” of teaching. Table 1 provides an example of two elements that comprise one standard. For each element, the rubric provides language describing what performance looks like at each scoring level: Distinguished (a score of 4), Proficient (3), Basic (2), or Unsatisfactory (1).

Data and Methodology

Cincinnati provided us with records of each classroom observation conducted between the 2000–01 and 2008–09 school years, including the scores that evaluators assigned for each specific practice element as a result of that observation. Using these data, we calculated a score for each teacher on the eight TES “standards” by averaging the ratings assigned during the different observations of that teacher in a given year on each element included under the standard. We then collapsed these eight standard-level scores into three summary indexes that measure different aspects of a teacher’s practice:

• The first, which we call Overall Classroom Practices, is simply the teacher’s average score across all eight standards. This index captures the general importance of the full set of teaching practices measured by the evaluation.

• The second, Classroom Management vs. Instructional Practices, measures the difference in a teacher’s rating on standards that evaluate classroom management and that same teacher’s rating on standards that assess instructional practices. A teacher who is more skilled at managing the classroom environment, as compared to her ability to engage in desired instructional activities, will receive a higher score on this index than a teacher who engages in these instructional practices but who is less skilled at managing the classroom.

• The third, Questions/Discussion vs. Standards/Content, measures the difference between a teacher’s rating on a single standard that evaluates the use of questions and classroom discussion as an instructional strategy, and that same teacher’s average rating on three standards that assess teaching practices that focus on classroom management routines, on conveying standards-based instructional objectives to students, and on demonstrating content-specific knowledge in teaching these objectives.

Our main analysis below examines the degree to which these summary indices predict a teacher’s effectiveness in raising student achievement. Note, however, that we did not construct the indices based on any hypotheses of our own about which aspects of teaching practice measured by TES were most likely to influence student achievement. Rather, we used a statistical technique known as principal components analysis, which identifies the smaller number of underlying constructs that the eight different dimensions of practice are trying to capture. As it turns out, scores on these three indices explain 87 percent of the total variation in teacher performance across all eight standards.

For all teachers in our sample, the average score on the Overall Classroom Practices index was 3.21, or between the “Proficient” and “Distinguished” categories. Yet one-quarter of teachers received an overall score higher than 3.53 and one-quarter received a score lower than 2.94. In other words, despite the fact that TES evaluators tended to assign relatively high scores on average, there is a fair amount of variation from teacher to teacher that we can use to examine the relationship between TES ratings and classroom effectiveness.

In addition to TES observation results, Cincinnati provided student data for the 2003–04 through 2008–09 school years, including information on each student’s gender, race/ethnicity, English proficiency status, participation in special education or gifted and talented programs, class and teacher assignments by subject, and state test scores in math and reading. This rich dataset allows us to study students’ math and reading test-score growth from year to year in grades four through eight (where end of year and prior year tests are available), while also taking account of differences in student backgrounds.

Our primary goal was to examine the relationship between teachers’ TES ratings and their assigned students’ test-score growth. This task is complicated, however, by the possibility that factors not measured in our data, such as the level of social cohesion among the students or unmeasured differences in parental engagement, could independently affect both a TES observer’s rating and student achievement. To address this concern, we use observations of student achievement from teachers’ classes in the one or two school years prior to and following TES measurement, but we do not use student achievement gains from the year in which the observations were conducted. (If some teachers are assigned particularly engaged or cohesive classrooms year after year, the results could still be biased; this approach, however, does eliminate bias due to year-to-year differences in unmeasured classroom traits being related to classroom observation scores.)

We restrict our comparisons to teachers and students within the same schools in order to eliminate any potential influence of differences between schools on both TES ratings and student achievement. In other words, we ask whether teachers who receive higher TES ratings than other teachers in their school produce larger gains in student achievement than their same-school colleagues.

Results

We find that teachers’ classroom practices, as measured by TES scores, do predict differences in student achievement growth. Our main results, which are based on a sample of 365 teachers in reading and 200 teachers in math, indicate that improving a teacher’s Overall Classroom Practices score by one point (e.g., moving from an overall rating of “Proficient” [3] to “Distinguished” [4]) is associated with one-seventh of a standard deviation increase in reading achievement, and one-tenth of a standard deviation increase in math (see Figure 1).

The specific point system that TES uses to rate teachers as Proficient and Distinguished is somewhat arbitrary. For a better sense of the magnitude of these estimates, consider a student who begins the year at the 50th percentile and is assigned to a top-quartile teacher as measured by the Overall Classroom Practices score; by the end of the school year, that student, on average, will score about three percentile points higher in reading and about two points higher in math than a peer who began the year at the same achievement level but was assigned to a bottom-quartile teacher.

This difference might not seem large but, of course, a teacher is just one influence on student achievement scores (and classroom observations are only one way to assess the quality of a teacher’s instruction). By way of comparison, we can estimate the total effect a given teacher has on her students’ achievement growth; that total effect includes the practices measured by the TES process along with everything else a teacher does. The difference between being taught by a top-quartile total-effect teacher versus a bottom-quartile total-effect teacher would be about seven percentile points in reading and about six points in math (see Figure 2). This total-effect measure is one example of the kind of “value-added” approach taken in current policy proposals.

From these data, we can also discern relationships between more specific teaching practices and student outcomes across academic subjects (see Figure 1). Among students assigned to different teachers with the same Overall Classroom Practices score, math achievement will grow more for students whose teacher is better than his peers at classroom management (i.e., has a higher score on our Classroom Management vs. Instructional Practices measure). We also find that reading scores increase more among students whose teacher is relatively better than his peers at engaging students in questioning and discussion (i.e., has a high score on Questions/Discussion vs. Standards/Content). This does not mean, however, that students’ math achievement would rise if their teachers were to become worse at a few carefully selected instructional practices. Although this might raise their Classroom Environment vs. Instructional Practices score it would also lower the Overall Classroom Practices score, and any real teacher is the combination of these three scores.

Do these statistics provide any insight that teachers can use to focus their efforts? First, our finding that Overall Classroom Practices is the strongest predictor of student achievement in both subjects indicates that improved practice in any of the areas considered in the TES process should be encouraged. In other words, the practices captured by the TES rubric do predict better outcomes for students. If, however, teachers must choose a smaller number of practices on which to focus their improvement efforts (for example, because of limited time or professional development opportunities), our results suggest that math achievement would likely benefit most from improvements in classroom management skills before turning to instructional issues. Meanwhile, reading achievement would benefit most from time spent improving the practice of asking thought-provoking questions and engaging students in discussion.

Can we be confident that the various elements of practice measured by TES are the reasons that students assigned to highly rated teachers make larger achievement gains? Skeptical readers may worry that better teachers engage in more of the practices encouraged by TES, but that these practices are not what make the teacher more effective. To address this concern, we take advantage of the fact that some teachers were evaluated by TES multiple times. For these teachers, we can test whether improvement over time in the practices measured by TES is related to improvement in the achievement gains made by the teachers’ students. This is exactly what we find. Since this exercise compares each teacher only to his own prior performance, we can be more confident that it is differences in the use of the TES practices themselves that promote student achievement growth, not just the teachers who employ these strategies.

Conclusion

Is TES worth the considerable effort and cost? Does the intensive TES process (with its multiple observations and trained peer evaluators) produce more accurate information on teachers’ effectiveness in raising student achievement gains than do more-subjective evaluations? In fact, studies of informal surveys of principals (see “When Principals Rate Teachers,” research, Spring 2006) and teacher ratings by mentor teachers find that these more-subjective evaluation methods have similar power to detect differences in teacher effectiveness as the TES ratings. These studies may lead some to question the need for the more detailed TES process. We contend, however, that evaluations based on observations of classroom practice are valuable, even if they do not predict student achievement gains considerably better than more subjective methods like principal ratings of teachers.

The additional information the TES system provides can be used in several important ways. First, the data gleaned from the observations allow researchers to connect specific teaching practices with student achievement outcomes, providing evidence of effective teaching practices that can be widely shared.

The TES program also has the advantage of furnishing teachers and administrators with details about the specific practices that contributed to each teacher’s score. The descriptions of practices, and different performance levels for each practice, that comprise the TES rubric can help teachers and administrators map out professional development plans. A school administrator who desires to differentiate the support she provides to individual teachers would benefit from knowing the components of each teacher’s overall scores. A teacher who would like to improve his classroom management skills may find that he has scored relatively low in a particular standard, and then take steps to improve his practice in response to that information.

Finally, scoring individual practices allows for understanding of more fine-grained variations in skill among teachers with similar overall ratings. It is notable, especially given “The Widget Effect” study, that nearly 90 percent of teachers in our sample received an overall “Satisfactory” rating (i.e., “Distinguished” or “Proficient” in Cincinnati’s terms). Still, there are readily discernible differences in mastery of specific skills within that 90 percent, and those differences in skills predict differences in student achievement.

There are other aspects of the Cincinnati system that may or may not account for the results we observed. First, the observers were external to the school and, in most cases, had no personal relationship with the person they were observing. Second, the observers were trained beforehand and were required to demonstrate their ability to score some sample videos in a manner consistent with expert scores. Simply handing principals a checklist with the same set of standards may not lead to a similar outcome.

The results presented here constitute the strongest evidence to date on the relationship between teachers’ observed classroom practices and the achievement gains made by their students. The nature of the relationship between practices and achievement supports teacher evaluation and development systems that make use of multiple measures. Even if one is solely interested in raising student achievement, effectiveness measures based on classroom practice provide critical information to teachers and administrators on what actions they can take to achieve this goal.

Thomas J. Kane is professor of education and economics at the Harvard Graduate School of Education. Eric S. Taylor is a doctoral student at the Stanford University School of Education. John H. Tyler is associate professor of education, economics, and public policy at Brown University. Amy L. Wooten is a doctoral student at the Harvard Graduate School of Education. Reflecting equal contributions to this work, authors are listed alphabetically. This article is based in part on a larger study which is forthcoming in the Journal of Human Resources.

American 15-year-olds continue to perform no better than at the industrial-world average in reading and science, and below that in mathematics. According to the results of the 2009 Program for International Student Assessment (PISA) tests, released in December 2010 by the Organisation for Economic Co-operation and Development (OECD), the United States performed only at the international average in reading, and trailed 18 and 23 other countries in science and math, respectively. Students in China’s Shanghai province outscored everyone.

Many have identified variations in teacher quality as a key factor in international differences in student performance and have urged policies that will lift the quality of the U.S. teaching force. To that end, President Barack Obama has called for a national effort to improve the quality of classroom teaching and repeatedly indicated his support for policies that would provide financial rewards for outstanding teachers.

In a March 2009 speech to the Hispanic Chamber of Commerce, he explained,

Good teachers will be rewarded with more money for improved student achievement, and asked to accept more responsibilities for lifting up their schools. Teachers throughout a school will benefit from guidance and support to help them improve.

In the administration’s Race to the Top initiative, the U.S. Department of Education encouraged states to devise performance pay plans for teachers in the hope that such an intervention could have a significant impact on student performance.

But is there anything in the data the OECD has accumulated to give policymakers reason to believe that merit pay works? Do the countries that pay teachers based on their performance score higher on PISA tests? Based on my new analysis, the answer is yes. A little-used survey conducted by the OECD in 2005 makes it possible to identify the developed countries participating in PISA that appear to have some kind of performance pay plan. Linking that information to a country’s test performance, one finds that students in countries with performance pay perform at higher levels in math, science, and reading. Specifically, students in countries that permit teacher salaries to be adjusted for outstanding performance score approximately one-quarter of a standard deviation higher on the international math and reading tests, and about 15 percent higher on the science test, than students in countries without performance pay. These findings are obtained after adjustments for levels of economic development across countries, student background characteristics, and features of national school systems.

I draw these conclusions cautiously, as my study is based on information on students in just 27 countries, and the available information on the extent of performance pay in a country is far from perfect. Further, the analysis is based on what researchers refer to as observational rather than experimental data, making it more difficult to make confident statements regarding causality.

It is possible that what I have observed is the opposite of what it seems: countries with high student achievement may find it easier to persuade teachers to accept pay for performance, thereby making it appear that merit pay is lifting achievement. More generally, both performance pay and higher levels of achievement could be produced by some set of factors other than all of those taken into account in the analysis. For example, performance pay could be more widely used in places where, as in Asia, cultural expectations for student performance are high, making it appear that performance pay systems are effective, when in fact both performance pay plans and student achievement are the result of underlying cultural characteristics. But even if my findings are not indisputable, I did carry out a variety of checks to see if any observable factor, such as Asian-European differences, could account for the conclusion. Thus far, I have been unable to find any convincing evidence that the findings are incorrect. Given that, let us take a closer look at what can be learned about the impact of performance pay from PISA data.

Prior Research

Standard economic theory predicts that workers will exert more effort when monetary rewards are tied to the amount of the product they produce. Not only does performance pay stimulate individual effort on the job, it is theorized, but jobs where rewards are tied to effort attract energetic, risk-taking employees who are likely to be more productive. This latter consideration, says Stanford economist Edward Lazear, “is perhaps the most important” way in which a merit pay plan can influence worker performance. But if economists expect positive results from merit pay, many educators believe that teachers are motivated primarily by the substantive mission of the teaching profession and that they do not respond to—indeed, they may resent and resist—monetary incentives that tie salary levels to performance indicators.

To see whether the education sector is an exception to general economic theory, a number of performance pay experiments have been carried out, and in Israel and India such studies have shown positive impacts on student achievement. Experimental studies have tracked only the short-term impact of merit pay, however, and so have not identified any long-term effects that might come from changes in the kinds of people who choose to go into this line of work. Conceivably, a merit pay system could discourage entry into the profession of potentially excellent teachers reluctant to subject themselves to the requirements of a pay-for-performance scheme. Alternatively, if performance pay makes teaching more attractive to talented workers, short-term evaluations could understate its benefits.

One way to capture the long-term effects of teacher performance pay, including changes in the characteristics of those choosing to become a teacher, is to compare countries with performance pay systems to those without. This is now possible because the OECD in 2005 administered a separate survey to each of its member countries concerning the teacher compensation systems in place during the 2003–04 school year, when the 2003 PISA study was conducted. The 2003 PISA provides test score results in math, reading, and science for representative samples of 15-year-olds within each country, or nearly 200,000 students altogether. (The relative performance of countries on the PISA changed only slightly between the 2003 and 2009 tests. For example, in no subject did the scores for the United States differ significantly between 2003 and 2009.)

The PISA study is particularly useful, because it also includes information on a wide variety of family, school, and institutional factors that are likely determinants of student achievement. My analysis adjusts, at the level of the individual student, for such characteristics as the student’s gender and age, preprimary education, immigration status, household composition, parent occupation, and parent employment status. Nine measures of school resources and location are available, including class size, availability of materials, instruction time, teacher education, and size of community. Country-level variables included in the analysis were per capita GDP, teacher salary levels, average expenditure per student, external exit exams, school autonomy in budget and staffing decisions, the share of privately operated schools, and the portion of government funding for schools.

The PISA sampling procedure ensured that a representative sample of 15-year-old students was tested in each country. The student sample sizes in the OECD countries range from 3,350 students in 129 schools in Iceland to 29,983 students in 1,124 schools in Mexico. I therefore use weights when conducting my analysis so that each country contributes equally to the estimated effect of performance pay on student achievement.

Measuring Teacher Performance Pay

The measure of performance pay available from the OECD survey is less precise than one would prefer. It simply asks officials in participating countries whether the base salary for public-school teachers could be adjusted to reward teachers who had an “outstanding performance in teaching.” While the survey asked about many other forms of salary adjustments, the study protocol reports that this was the only one that “could be classified as a performance incentive.”Among the 27 OECD countries for which the necessary PISA data are also available, 12 countries reported having adjustments of teacher salaries based on outstanding performance in teaching. The form of the monetary incentive and the method for identifying outstanding performance varies across countries. For example, in Finland, according to the national labor agreement for teachers, local authorities and education providers have an opportunity to encourage individual teachers in their work by personal cash bonuses on the basis of professional proficiency and performance at work. Outstanding performance may also be measured based on the assessment of the head teacher (Portugal), assessments performed by education administrators (Turkey), or the measured learning achievements of students (Mexico). Unfortunately, the coding of the measure does not allow my analysis to consider variation in the scope, structure, and incentives of performance-related pay schemes.

As an example of the limitation of this measure, note that the United States is coded as a country where teacher salaries can be adjusted for outstanding performance in teaching on the grounds that salary adjustments are possible for achieving the National Board for Professional Teaching Standards certification or for increases in student achievement test scores. That policy, however, affects only a few teachers in selected parts of the country. Given such weaknesses in the survey measure, it is all the more remarkable that I was able to detect impacts on student achievement.

Main Results

As noted above, my main analysis indicates that student achievement is significantly higher in countries that make use of teacher performance pay than in countries that do not use it. On average, students in countries with performance-related pay score 24.8 percent of a standard deviation higher on the PISA math test; in reading the effect is 24.3 percent of a standard deviation; and in science it is 15.4 percent (see Figure 1). These effects are similar to the impact identified in the experimental study conducted in India and about twice as large as the one found in a similarly designed Israeli study.

Figure 2 depicts the math result graphically. The figure’s vertical axis displays the average math test scores of students in each country after adjusting for all of the control variables in the model, with the exception of the variable measuring the use of performance pay. The horizontal axis in turn shows the performance pay variable, also after adjusting for those same control variables. The solid line on the figure shows the estimated relationship between these two variables across the 27 countries included in the analysis. It shows a clear positive association between the variation in country-average test scores and the variation in teacher performance pay that cannot be attributed to the other factors included in the analysis.

A lingering concern, however, is that the analysis may be contaminated by the fact that the very cultures that introduce merit pay are those that set high expectations for student achievement. The countries represent widely different cultures, including Asian ones, where expectations for students are often much higher than in Europe and North America. The best way to account for cultural differences among the continents of the world is to control in the analysis for the average effect of living on a particular continent, a strategy known to statisticians as continental fixed effects. Figure 1 thus also shows results based on models that include a fixed effect for each of the four continents with OECD countries: Europe, North America, Oceania, and East Asia. In these models, the effects of pay for performance are shown to be even larger than the results based on comparisons across continents. In other words, the findings cannot be attributed to cultural differences among the major regions of the world, because they are even larger when one looks only at patterns within these regions.

As a further test, I estimated the impact of performance pay for only the 21 participating European countries. Once again, the results showed even larger positive effects than those obtained for the full sample.

Other Sensitivity Tests

When findings are based on small samples, it is important to ascertain whether a conclusion is sensitive to the particular analysis being conducted. Even after conducting a preferred analysis that maximizes use of the information available and best conforms to underlying economic theory, it is important to make sure that the pattern that one has identified is not a statistical accident that readily disappears if a slightly different analysis is conducted. For this reason, I performed a variety of sensitivity tests for math achievement because the reliability of the math test across countries and cultures is usually considered higher than it is for reading or science. Remarkably, the relationship between performance pay and math achievement remained essentially unchanged, regardless of the sensitivity test that I ran.

My first sensitivity check focused on cultural differences among countries that were not captured by the continental fixed effects analysis. In this sensitivity check, I excluded two countries, Mexico and Turkey, which have particularly low levels of GDP per capita. Since it is known that the level of GDP is strongly correlated with educational performance, it may be that the inclusion of these two countries is producing misleading results. But dropping these countries hardly affects results.

The second sensitivity test excluded the level of educational attainment of the teachers, on the grounds that teacher quality might itself be affected by a country’s performance pay policies and therefore should not be used as a control variable. Excluding this variable did not materially change the results from those reported in Figure 1. In a third series of sensitivity tests, I excluded from the analysis one country at a time to make sure that the situation in no one country was driving the overall pattern of results. I found no evidence that that was happening.

The incidence of performance pay is, to some extent, clustered in two regions: Scandinavia (Denmark, Finland, Norway, and Sweden) and Eastern Europe (Czech Republic and Hungary). In a fourth set of sensitivity tests, I separately excluded the countries from these two regions from the analysis to see whether results were highly dependent on one or the other cluster. The results remained unchanged, indicating that neither of these regional clusters is solely responsible for the main result.

A fifth set of sensitivity tests was possible because I have information on other policies that lead to differential pay among teachers. Salaries may vary depending on 1) the teaching conditions and responsibilities (such as taking on management responsibilities, teaching additional classes, and teaching in particular areas or subjects), 2) teacher qualifications and training, or 3) a teacher’s family status and/or age. Since it is possible that student achievement is higher whenever pay schedules are flexible, regardless of the connection to teacher classroom effectiveness, I estimated the impact of each of these three sets of factors on math achievement. None showed a significant impact on performance, and the effect of performance pay remained large and significant, even when these other possible salary adjustments were included in the analysis.

In sum, the main results shown in Figure 1 survive a wide variety of sensitivity tests. That the results are robust to multiple model specifications provides strong evidence that performance pay helps to explain the variation in student performance on the PISA tests.

Differential Effects

With one exception (immigrants benefited less than native-born students from a performance pay regime), I found only small differences in the impact of performance pay on the math achievement of subgroups in the population. Since important differential effects were identified for only one subgroup, one cannot infer that the impact of performance pay on student math learning is concentrated on any particular group of students.

I did, however, find a surprising difference in the way in which a teacher’s education background affects math learning, depending on the presence of a pay-for-performance system. In countries with performance pay, teachers who have an advanced degree in pedagogy do not outperform those without such a degree (the only measure of a teacher’s education available in the PISA data base). However, in countries without performance pay, students learn more in math if they have a pedagogically trained teacher. Perhaps an incentive system washes out any differences that may be caused by variations in teacher training.

Conclusions

The analysis presented above represents the first evidence that, all other observable things equal, students in countries with teacher performance pay plans perform at a higher level in math, reading, and science. The differences in performance are large, ranging from 15 percent (in science) to 25 percent (in math and reading) of a standard deviation. Since one-quarter of a standard deviation is roughly a year’s worth of learning, it might reasonably be concluded that by the age of 15, students taught under a policy regime that includes a performance pay plan will learn an additional year of math and reading and over half a year more in science. However, this conclusion depends on the many assumptions underlying an analysis based on observational data.

Although these are impressive results, before drawing strong policy conclusions it is important to confirm the results through experimental or quasi-experimental studies carried out in advanced industrialized countries. Nothing in the PISA data allows us to identify crucial aspects of performance pay schemes, such as the way in which teacher performance is measured, the size of the incremental earnings received by higher-performing teachers, or very much about the level of government at which or the manner in which decisions on merit pay are made. Studies of such matters are probably better performed within countries, taking advantage of variation in policies within those countries. The study design also does not allow one to tease out the relative importance of the incentive to existing teachers of a performance pay plan as compared to the changes that may take place in teacher recruitment when compensation depends in part on merit rather than just on a standardized pay schedule. Since much more work needs to be done on all of these questions, a wit might insist that performance pay apply to scholars as well.

Ludger Woessmann is professor of economics at the University of Munich and head of the department of Human Capital and Innovation at the Ifo Institute for Economic Research.

An unabridged version of this article is available here.

Merit pay proponents argue that monetary incentives for better teaching can improve the quality of instruction in our nation’s classrooms. Yet only a handful of studies have evaluated the impact of teacher merit pay on student achievement. These studies offer no conclusive recommendations regarding the optimal role of merit pay in U.S. school systems, leaving policymakers largely dependent on studies on other countries for information about how best to implement merit pay programs.

Recently, the New York City Department of Education (DOE) conducted a policy experiment to test whether merit pay given to all teachers at an effective school could increase student achievement. The city’s School-Wide Performance Bonus Program, launched in 2007 and endorsed by both the DOE and the teachers union, was implemented in a randomly selected subset of the city’s most disadvantaged schools. The randomized design of school selection makes it possible to separate out the causal effect of this form of merit pay from myriad other influences on student learning.

Our analysis is based on data from the first two years of the bonus program. In interpreting our findings, it is important to appreciate the key features of the program’s structure. Teachers received bonuses based on the overall performance of all tested students in their school, rather than just on the performance of students in their own classrooms. According to proponents of group incentives, this design can minimize conflicts and foster a spirit of cooperation among teachers at participating schools. However, under group incentive schemes, individual teachers may not have sufficient motivation to improve their own performance if they know that their success in attaining a bonus depends heavily on the efforts made by other teachers. Especially in schools with a large number of teachers, it may be difficult to sustain a school-wide push to mobilize the efforts of most teachers. The New York City bonus program thus provides valuable information on the effects of a school-wide bonus plan.

Other specific characteristics of the bonus plan and the New York City context may also have influenced its effectiveness. If a school won a bonus, money was distributed among teachers and other school personnel by a committee consisting of two administrators and two teachers union representatives at the school. The bonus program was implemented alongside a new citywide accountability system that provided strong incentives to improve student achievement, regardless of whether a school was participating in the bonus program. Also, over the period we examine, all schools experienced increases in student achievement on the New York state test, leading some to suggest that the exam had grown easier (or at least easier to teach to). Roughly 90 percent of participating schools received a bonus in the second year of the program.

Did the group bonus program operating in this policy environment have an impact on student achievement? We find very little effect overall, positive or negative. There is some evidence, however, that the program had a positive impact in schools where teachers were few in number, an environment in which it may be easier for teachers to cooperate in pursuit of a common reward. This study leaves open the question of whether a bonus program that rewards teachers for their own specific effectiveness would be more successful.

The Program

In November 2007, the New York City DOE launched the School-Wide Performance Bonus Program, randomly selecting 181 schools serving kindergarten through 8th grade to participate from a group of 309 high-need schools. (Disadvantaged high schools were also randomly selected into the program; we focus only on elementary and middle schools since these are the grades for which we can measure math and reading achievement.) The remaining 128 schools that were not selected serve as the control group for the purposes of our evaluation. The 309 schools included in the study differed from other city schools in the following ways: They had a higher proportion of English Language Learners (ELL), special education, minority students, and students eligible for the Title I free or reduced-price lunch program, as well as lower average math and reading scores. Teachers in these schools had slightly less experience and slightly more absences than teachers in other schools. The schools were smaller and had fewer teaching staff than other New York City schools.

The bonus program was the product of lengthy negotiations between district administrators and the teachers union. As a result of these negotiations, schools had to gain the support of 55 percent of their full-time United Federation of Teachers (UFT) staff each year in order to participate. Out of the 181 schools selected for the program, 25 schools voted not to participate in the first year of implementation or withdrew from the program following an initial vote of approval, and three more schools pulled out before the second year. Additionally, at the discretion of the DOE, two schools initially assigned to the treatment group were moved to the control group, and four schools initially designated as control schools were moved to the treatment group and subsequently voted to participate in the program. Of course, the schools that elected not to take part in the program and those moved by the DOE may differ in important ways from schools that chose to participate. We therefore consider the treatment group to include all 181 schools originally deemed eligible for bonus payments and take into account the fact that not all of them were actually participating in the program when interpreting our results.

Schools that implemented the program could earn a lump-sum bonus for meeting school-wide goals. These goals were tied to the New York City accountability system and were mainly determined by student performance on state math and reading exams. Under this accountability system, schools receive scores and grades that summarize their overall performance on three sets of measures: school environment, student performance, and student progress. The school environment measure incorporates student attendance and the results from surveys of parents, teachers, and students. Student performance measures include average student achievement on reading and math exams, along with median proficiency and the percentage of students achieving proficiency. The student progress measure considers the average change in test scores from year to year and the percentage of students who made progress from one year to the next. The accountability system also gives “extra credit” for exemplary progress among high-need students. Schools received target scores based on their accountability grades, and schools with lower accountability grades needed to make larger improvements to reach their targets.

Schools participating in the bonus program received awards based on their progress toward meeting target scores. Schools that achieved their goals received bonuses equal to $3,000 per union teacher. Schools that fell short but manage to meet 75 percent of their goal received $1,500 per union teacher. Schools that did not achieve their target faced no consequences from the bonus program beyond the absence of incentive pay. For a sense of the strength of the incentive provided by the bonuses, the full $3,000 award represents a 7 percent increase in the salary of teachers at the bottom of the pay scale and a 3 percent increase for the most experienced teachers. In other words, these bonuses provided a substantial monetary benefit to most recipients.

Each participating school was required to develop a plan for distributing any lump-sum bonus awarded to the school. In the first year of the program, plans had to be submitted to the DOE after students took the state math and reading exams but before exam results were released and, thus, before schools knew whether they would receive a bonus. In every school, a four-member compensation committee, consisting of the principal, a second administrator, and two teachers elected by the school’s UFT members, determined how bonuses would be distributed. The DOE program guidelines placed only two restrictions on the schools’ bonus distribution plans: all union teachers had to receive a portion of the bonus payment and bonuses could not be distributed based on seniority. Otherwise, the committees had full discretion over bonus amounts and over whether other school employees would also receive funds. About half of the school committees chose to divide the award roughly equally among all recipients. In these schools, the difference between the highest and lowest bonus payment was less than $100. In the rest of the schools, the difference between the highest and the lowest bonus ranged from a low of $200 to a high of $5,000.

Of the 158 schools that voted to participate in the first year of the program, 87 (55 percent) received bonus payments. The bonus pool totaled $14.0 million in the first year and averaged $160,500 per school. In the second year of the program, the 2008–09 school year, 139 participating schools (91 percent) earned bonus awards, averaging $195,100 per school and totaling $27.1 million.

Little Difference for Students

Before we get to the detailed findings of our study, it is important to make clear the nature of the incentives NYC teachers and administrators faced over the period we examine. First, the 2007–08 school year was the first year of both the bonus program and a new citywide accountability system. The accountability system provided strong incentives to improve student achievement, regardless of whether a school was participating in the bonus program. For example, schools that earned A or B accountability grades were eligible for principal bonuses and additional funds when students transferred from schools receiving a poor grade. Schools that received D and F grades faced potential consequences, including principal removal and school closure. With this in mind, we see the results of our study as representing the effect of group-based teacher merit pay for schools that are already under accountability pressure. However, given that all school districts in the United States are subject to No Child Left Behind and many states have implemented their own accountability systems, this may be the most appropriate context in which to study the consequences of merit pay.

The second thing to keep in mind is that the power of the bonus program incentives was likely muted in the first year because of the timing of the program announcement. Eligible schools were notified in November of 2007, leaving relatively little time for teachers and administrators to alter their educational plans before accountability exams were administered in January for reading and March for math. As noted above, the percentage of schools that hit their achievement targets increased between the first, truncated year of the program and the second, when schools had more time to respond to the program incentives. But we caution readers to remember that this leap in bonus payouts is not, by itself, evidence that merit pay worked. It may instead reflect citywide performance improvements or, more pessimistically, that the New York state tests decreased in difficulty over this period. The most important comparison to make is between the treatment group schools eligible for the bonus program (most of which actually participated in the program) and the schools in the control group. Treatment-group schools need to at least outpace their counterparts in the control group over these two years for us to say that merit pay made a real difference for student achievement. It is this comparison that is at the heart of our analysis.

How did bonus program schools fare compared to schools in the control group? Both groups of schools saw an increase in the average math and reading scores during the first two years of the bonus program; treatment-group schools, however, did not experience a statistically significant improvement in average test scores relative to the schools in the control group. Nor did these results change notably when we 1) made adjustments for the small differences in treatment and control school characteristics that existed despite randomization between treatment-group and control-group schools, or 2) took into account whether treatment-group schools elected to participate in the bonus program. It is possible, of course, that looking at average student achievement could divert our attention from changes for particular groups of students. Were teachers, we wanted to know, focusing their attention on either high-achieving or low-achieving students in an effort to meet target scores? We used statistical techniques similar to the one we employed to examine changes in average scores to assess the effect of the bonus program on the percentage of students achieving proficiency on math and reading exams. Once again, we found no evidence that the bonus program led to changes in this measure of student achievement. Participation in the bonus program did not, for example, boost the percentage of students who scored at or above the level designated as “proficient” under New York state accountability standards. Bonus-program schools fared no better than schools in the control group, and in the second year of the program, treatment schools experienced a statistically significant, although quite small, decrease in math proficiency.

On a related note, the New York City accountability system and, as a byproduct, the bonus program, contain incentives to focus on particular groups of students, since improvements for some student groups matter more in the calculations of a school’s accountability grade. In addition to calculating overall achievement for all students in a school, components of the New York City accountability system take into account changes in the achievement of students who were in the lowest third of their grade in the prior year, those on the cusp of proficiency, and those close to the school’s median score, along with students who are designated as ELL and students who are enrolled in special education programs. Again, we found no evidence to suggest that the bonus program led to achievement gains for any of these groups of students. On average, students in these groups fared just as well whether they attended a school that was participating in the bonus program or one in the control group.

Limitations of Group Bonuses

Does evidence that the New York City bonus program did not lead to marked gains in student achievement, at least in the program’s first two years, mean that merit pay for teachers in general does not work? That is certainly one possible conclusion to be drawn from our findings. Another possibility is that this particular type of merit pay program, where bonuses are based on school-wide performance and teachers expect to receive bonus payments regardless of their effort, does not work in all schools. Group bonuses may weaken the incentives for individual teachers to increase effort devoted to raising student achievement to the point that the programs become ineffective. And perhaps this problem would be mitigated in programs in which rewards are more tightly coupled to the effort an individual makes in the classroom.

Think about two schools, one with many more teachers than the other, both participating in a school-wide merit pay program. In each school, the impact of an individual teacher’s effort on the expected bonus is determined by the number of other teachers with tested students, since bonus receipt is primarily based on student performance on math and reading exams. Because of this, a very good teacher with a large number of teaching colleagues can do less to raise school-wide student performance than a teacher of the same quality in a school with fewer teachers. In the school with more teachers, the diffusion of responsibility for test-score gains across many teachers may erode the incentive that any individual teacher has to increase effort in the classroom. Some teachers may conclude that exerting additional effort will produce little difference in the overall performance of the school. The central idea here is that teachers could face relatively strong or weak incentives under the same merit pay program as a result of the number of teachers at their school. With this logic in mind, we examined the effect of the New York City school-wide merit pay program at schools with different numbers of teachers with test-taking students. Did schools with fewer teachers show signs that teachers were responding to merit pay incentives?

We conducted a statistical analysis similar to our method for estimating the average effect of the bonus program across all New York City schools in the experiment. But this time, we looked for different effects on math scores in schools with more and fewer math teachers and different effects on reading scores on schools with larger and smaller cohorts of reading teachers.

It turns out that the effectiveness of school-wide bonus programs may, in fact, depend on the number of teachers with tested students in a school (see Figure 1). For schools in the bottom quartile of the number of teachers with tested students, that is, schools with approximately 10 or fewer such teachers in elementary and K–8 schools and five or fewer in middle schools, school-wide merit pay did lead to improved student achievement. We estimate that the New York City bonus program had a positive effect on student math achievement in these schools in both program years, although the estimated effect in the second year fell just short of conventional levels of statistical significance. Conversely, this analysis also indicates that the program may have slightly lowered student achievement in schools with larger teaching staffs. Math achievement gains attributable to the bonus program in schools with smaller teaching staffs were modest in size but meaningful. In the first year of the program, the bonus program boost to math scores was, by our estimates, 3.2 points on the New York state test, or 0.08 student-level standard deviations. To benchmark this effect against the magnitude of other familiar results, it is slightly smaller than the estimated 0.1 standard deviation gain in achievement that results from being assigned to a teacher at the 85th percentile of the effectiveness distribution rather than a teacher at the median.

The Devil in the Details

The New York City bonus-pay program provides us with a valuable opportunity to study the effect of merit pay for teachers in an experimental setting. We are a long way from amassing a convincing body of research on either side of the debate over merit pay in education, but what this experiment makes frustratingly clear for merit pay proponents is that the structure of the payment scheme can make a large difference. For merit pay to improve student outcomes, teachers must face strong incentives to improve their performance. Our study indicates that school-wide bonus programs may be able to provide those incentives in schools with relatively small teaching staffs. They may also be appropriate for schools characterized by a high degree of staff cohesion, in which teachers work collaboratively to improve student learning and it is difficult to isolate the performance of a single teacher. The early experience with the New York City School-Wide Performance Bonus Program suggests, however, that a heavy reliance on school-wide rewards may hamper the effectiveness of merit pay programs in schools with large teaching staffs that are not highly collaborative.

Sarena Goodman and Lesley Turner are PhD candidates in Columbia University’s Department of Economics. The randomization of schools participating in the School-Wide Performance Bonus Program was designed and conducted by Harvard University economist Roland Fryer.

An unabridged version of this article is available here.

An unabridged version of this article is available here.

Programs that enable students to attend private schools, including both vouchers and scholarships funded with tax credits, have become increasingly common in recent years. This study examines the impact of the nation’s largest private school scholarship program on the performance of students who remain in the public schools. The Florida Tax Credit Scholarship Program (FTC) was signed into law in 2001 and opened to students from low-income families in the 2002–03 school year. FTC provides corporations with tax credits for donations they make to scholarship funding organizations, the nonprofits that determine student eligibility for the program and issue scholarships. Corporations can receive dollar-for-dollar tax credits for up to 75 percent of their total state tax obligation each year.

Although little noticed, tax credit scholarship programs now send many more low-income students to private schools than do traditional school voucher programs. More than 104,000 students nationwide attended private schools through tax credit programs in the 2008–09 school year, while only 60,000 students used private school vouchers. Scholarship programs similar to Florida’s now operate in several states, including Arizona, Pennsylvania, Rhode Island, and Indiana, and are being considered in Maryland and New Jersey. The Florida program is set to expand dramatically in the coming years (see sidebar), making evidence of its consequences all the more timely.

One popular argument for expanding private school choice is that public schools will improve their own performance when faced with competition for students. Because state school funding is tied to student enrollment, losing students to private schools means losing revenue. The threat of losing students to private schools may give schools greater incentive to cultivate parental satisfaction by operating more efficiently and improving the outcomes valued by students and parents. Alternatively, private school vouchers and scholarships may have unintended negative effects on public schools: they may draw away the most involved families from public schools, community monitoring of those schools may diminish, and schools may reduce the effort they put into educating students.

It is notoriously difficult to gauge the competitive effects of private schools on public school performance. Private schools may be disproportionately located in communities with low-quality public schools, causing the relationship between private school competition and public school performance to appear weaker than it actually is. If, however, private schools are located in areas where citizens care a lot about educational quality, the relationship will appear stronger than it truly is.

This study takes advantage of the introduction of the FTC to provide new evidence on the effects of increased competition on student achievement in public schools. Before the program began in 2002, roughly 11 percent of students in Florida were enrolled in private schools. FTC targeted students from low-income families, only 5 percent of whom had been attending private schools.

We examine whether students in schools that face a greater threat of losing students to private schools as a result of the introduction of tax-credit funded scholarships improve their test scores more than do students in schools that face less-pronounced threats. We find that they do, and that this improvement occurs before any students have actually used a scholarship to switch schools. In other words, it occurs from the threat of competition alone.

Florida Tax Credit Scholarship Program

In order to be eligible for an FTC scholarship, students must meet the income guidelines (until recently, family incomes below 185 percent of the federal poverty line for new applicants) and either must have attended a Florida public school for the full school year before program entry or be entering kindergarten or first grade. With the exception of these early-grade private school students, students already attending private schools in Florida are not eligible for first-time scholarships. Students who enter a private school on a scholarship are eligible to retain their scholarships in future years, so long as their family income remains within the stated limits (until recent changes, below 200 percent of the federal poverty line).

Scholarships need not cover all of the costs of attending private schools, and parents are free to send their children to any private school regardless of the share of tuition and fees the scholarship covers. The scholarship is quite generous; it covers approximately 90 percent of tuition and fees at a typical religious elementary school in Florida and two-thirds of tuition and fees at a typical religious high school. As a result, the program greatly increased the accessibility of private schools to low-income families. In the first year, some 15,585 scholarships were awarded, increasing the number of low-income students attending private schools by more than 50 percent. For the 2009–10 school year, the FTC program awarded scholarships to 28,927 students (see Figure 1).

Data

Our analysis draws on several data sources. The Florida Department of Education’s Education Data Warehouse provides test scores from the Florida Comprehensive Achievement Test (FCAT) and demographic characteristics for all students in the public schools. We use test-score data from the 1999–2000 school year through 2006–07. Students classified as learning disabled were excluded from the analysis, as they are eligible for a more generous voucher through the McKay Scholarship Program, and the FTC program should therefore have had no effect on schools’ efforts to retain these students (see “The Case for Special Education Vouchers,” features, Winter 2010).

The Florida Department of Education publishes public and private school addresses, including latitude and longitude information for the public schools. The address information was geocoded to generate measures of the pressure that public schools face from private competitors. We first limited our attention to the 92 percent of public school students in the state attending schools with a private competitor within a five-mile radius. Because it is not obvious how best to gauge the amount of competition faced by the remaining schools, we constructed four different measures:

• Distance: the crow’s-flight distance between the physical addresses of each public school and the nearest private competitor. A private school qualifies as a competitor to a public school if it serves any of the grades taught in that public school.

• Density: the number of private competitors within a five-mile radius of the public school.

• Diversity: the number of different types of private schools within a five-mile radius of the public school. To generate this measure, we first identified 10 distinct types of private schools, defined by their stated religious (or secular) affiliation.

• Concentration: an index of how varied the private school competitors are for a given public school, based on the counts of different types of schools within a five-mile radius.

The distance and density measures gauge whether easier access to a private school of any type increased the competitive pressure on public schools when the new policy lowered the effective cost of attending private school for eligible students. The diversity and concentration measures capture the variety of options available to students; public schools in areas with more varied options should feel more competitive pressure in the wake of the policy change.

Methods

In order to determine the effect of scholarship-induced private school competition on public school performance, we examine whether students in schools that face a greater threat of losing students to private schools as a result of the introduction of tax-credit funded scholarships improve their test scores more than do students in schools that face a less-pronounced threat. Specifically, we look to see whether test scores showed greater improvement in the wake of the new policy for students attending public schools with more (or more varied) nearby private options that suddenly became more affordable for low-income students than did scores for students attending schools with fewer (or less varied) potential competitors.

This analysis is possible because of the considerable variation in potential competition faced by schools across the state of Florida. Prior to the introduction of the program, some communities in Florida had a much richer and more diverse set of private school options than did other communities. The overall share of low-income students attending private schools ranged from 1.4 percent in Punta Gorda to 7.9 percent in the Melbourne-Titusville-Cocoa-Palm Bay area. More importantly, by our measures, the amount of competition that specific public schools faced on the eve of the program also varied widely. Our density measure, for example, ranges from one private school within five miles to 60. The average Florida public school had roughly 14 private schools nearby, but more than 30 percent of schools had fewer than 2 or more than 30.

We isolate the effect of competition introduced by the program by comparing the performance of each school’s students to the performance of its students the year before the program was enacted. When making these comparisons, we take into account student characteristics that are associated with test scores, including gender and race/ethnicity, English-language-learner status, and eligibility for free or reduced-price lunch.

We also control for some characteristics of schools that could affect the degree of competitive pressure. Most importantly, we control for the letter grades that schools received from the state’s accountability system; schools with lower grades may feel particular pressure to increase their scores to avoid accountability sanctions, independent of the effects of the FTC program. We also control for the percentage of the school’s student body that was eligible for free or reduced-price lunch, as only these students were eligible for FTC scholarships.

There are three main ways in which a program that expands access to private schools could affect public school performance. Public schools could react to private school competition by altering their policies, practices, or effort; this is the direct competitive effect. Such a program could also affect public schools by changing the mix of students who attend them. A third possibility is that, so long as only a few students leave a public school with scholarships, the program could have effects on resources. Resource effects could be either negative (as total state aid decreases with the loss of students), or positive (as per-pupil resources might actually increase following small losses of students, due to the indivisibility of classroom teachers). We can eliminate the possibility of student-body composition and resource effects by concentrating solely on the FTC program’s effects during the 2001–02 school year, after the program’s announcement but before students could actually leave the public schools with a scholarship. During this academic year, students in the public schools were applying for private school scholarships for the following year.

Results

We find that all four measures of competition (distance, density, diversity, and concentration) are positively related to student performance on state math and reading tests. (Because we obtained similar results looking at performance in each subject separately, we focus our discussion on the average score across both subjects.) Each of our competition measures uses different units. We therefore report the estimated effects of a one standard deviation increase in the amount of competition faced by a given public school by each measure.

For every 1.1 miles closer to the nearest private school, public school math and reading performance increases by 1.5 percent of a standard deviation in the first year following the announcement of the scholarship program. Likewise, having 12 additional private schools nearby boosts public school test scores by almost 3 percent of a standard deviation. The presence of two additional types of private schools nearby raises test scores by about 2 percent of a standard deviation. Finally, an increase of one standard deviation in the concentration of private schools nearby is associated with an increase of about 1 percent of a standard deviation in test scores.

Although these effects are relatively small, they consistently indicate a positive relationship between private school competition and student performance in the public schools, even before any students leave for the private sector. That is, these results provide evidence that public schools responded to the increased threat of losing students to the private schools. The fact that we obtain quite similar results regardless of the specific measure used makes us confident that the findings are not driven by other factors that might distinguish public schools facing more or less competition based on a given measure. Indeed, in ongoing work we have also considered measures of competition based on the number of available slots in nearby private schools and on the number of nearby churches, and again find very similar results.

Moreover, it is important to recognize that the results reported above represent lower-bound estimates of the effects of competition on public school performance. They are based only on comparisons of schools with different levels of competition. If all public schools improved their performance in response to the scholarship program, this improvement would not be detected by our analysis.

One might expect that some public schools have a greater incentive to respond to potential competition associated with the availability of private school scholarships for low-income students than others. We consider two major reasons schools may face different incentives to react to competitive pressure. First, elementary and middle schools may have more of an incentive to respond to competitive pressure than high schools because the scholarships cover a greater share of private school tuition and fees in the early grades than they do in the high school years. Although the differences in the share covered might not matter for higher-income families, for many low-income families the difference in out-of-pocket expenses between an elementary or middle school and a high school is likely to be significant. Knowing this, public high schools might not react as strongly to competition from a private school scholarship program as would public elementary and middle schools. Consistent with this hypothesis, we find that the effect of competition is more than twice as large for elementary and middle schools as it is for high schools (see Figure 2).

Second, public schools that stand to lose the largest amounts of revenue if many of their scholarship-eligible students leave may be more responsive than those schools less likely to lose large amounts of revenue. All public schools may experience resource effects as a consequence of losing students to private schools through a scholarship program. However, those that are on the margin of receiving federal Title I aid have the largest incentive to retain students from low-income families. These federal resources, which average more than $500 per pupil, are directed to school districts, which then allocate them to the elementary and middle schools that low-income students attend. We find that public schools that are likely to receive Title I aid in the next year if they retain their low-income students, but not if they don’t, tend to improve disproportionately in the year following the program announcement, whereas schools whose Title I aid is unlikely to change respond much less noticeably or not at all.

We also investigate whether the estimated effects of the scholarship program persist in later years. After the first year of the analysis, resource and composition effects may occur as students who receive scholarships leave the public schools for private schools. We find that the effects of the voucher program grow stronger over time (see Figure 3), resulting perhaps from increased knowledge of the program, which might contribute to greater competitive pressure, or from the advent of composition and resource effects. While it is difficult to disentangle the reasons for this strengthening over time of the program’s estimated effects, these findings nonetheless suggest that our first-year results may understate the positive effect of the FTC program on public school performance

Conclusion

Our results indicate that the increased competitive pressure public schools faced following the introduction of Florida’s Tax Credit Scholarship Program led to general improvements in their performance. Both expanded access to private school options and greater variety of options that students have in terms of the religious (or secular) affiliations of private schools are positively associated with public-school students’ test scores following the introduction of the FTC program. The gains occur immediately, before any students leave the public schools with a scholarship, implying that competitive threats are responsible for at least some of the estimated effects. And the gains appear to be much more pronounced in the schools most at risk to lose students (elementary and middle schools, where the cost of private school attendance with a scholarship is much lower) and in the schools that are on the margin of Title I funding.

To be sure, our study has several limitations. First, our measures of competition reflect the state of the private school market in 2001, before private schools had a chance to respond to the FTC scholarship program. Although that ensures that the competition measure is not itself affected by postpolicy test scores, it does give a less accurate view of the competitive pressures faced by schools in subsequent years.

Second, our study is based on data from a single state. It is possible that the dynamics between competitive pressures and public-school students’ test scores are systematically different in Florida than they are in the rest of the nation. In particular, more than 90 percent of Florida’s students live in the state’s top 20 most populous metropolitan areas. In states with a greater share of the population in rural areas, the effects of a scholarship program may not exert the same degree of competitive pressure on public schools. It may also be the case that Florida’s diverse range of private school options and accompanying greater competition among the private schools limit the study’s generalizability. Nonetheless, our results indicate that private school competition, brought about by the creation of scholarships for students from low-income families, is likely to have positive effects on the performance of traditional public schools.

David Figlio is professor of education, social policy and economics at Northwestern University and research
associate at the National Bureau of Economic Research. Cassandra M.D. Hart is a doctoral student in the
Department of Human Development and Social Policy at Northwestern University.

An unabridged version of this article is available here.

Middle school. The very words are enough to make many Americans shudder with memories of social anxiety, peer pressure, bad haircuts, and acne. But could middle schools also be bad for student learning? Could something as simple as changing the grade configuration of schools improve academic outcomes? That’s what some educators have come to believe.

States and school districts across the country are reevaluating the practice of educating young adolescents in stand-alone middle schools, which typically span grades 6 through 8 or 5 through 8, rather than keeping them in K–8 schools. The middle-school model began to be widely adopted almost 40 years ago. Now, reformers in Massachusetts, Pennsylvania, Ohio, Tennessee, Oklahoma, Maryland, and New York, and the large urban districts of Cincinnati, Cleveland, Philadelphia, and Baltimore, are challenging the notion that grouping students in the middle grades in their own school buildings is the right approach.

Why the turn against middle schools? For more than three decades, American public education embraced this organizational model. Between 1970 and 2000, the number of public middle schools in the U.S. grew more than sevenfold, from just over 1,500 to 11,500. These new middle schools displaced both traditional K–8 primary schools and junior high schools (which first appeared a century  ago and served grades 7–8 or 7–9). From 1987 to 2007, the percentage of public-school 6th graders in K–6 schools fell from roughly 45 percent to 20 percent.

Neither the middle school nor the junior high has ever been popular among private schools, which educated only 2 percent of their 6th and 7th graders in these types of schools in 2007. And maybe the private schools have had it right all along. For the last two decades, education researchers and developmental psychologists have been documenting changes in attitudes and motivation as children enter adolescence, changes that some hypothesize are exacerbated by middle-school curricula and practices.

These findings are cause for concern, but there is reason to doubt their conclusions. Because the studies use data from a single school year to contrast students in middle schools and K–8 schools, most of the available research cannot reject the possibility that differences between the groups of students, rather than in the grade configuration of their schools, are actually responsible for the differences in behavior and achievement.

To provide more rigorous evidence on the effect of middle schools on student achievement, we turned to a richly detailed administrative dataset from New York City that allowed us to follow students from grade 3 through grade 8. Some of these children attended middle schools and some did not. Because we could follow the same children over a period of time, we could do a better job of ruling out the role of influences other than middle-school attendance on educational outcomes.

What we found bolsters the case for middle-school reform: in the specific year when students move to a middle school (or to a junior high), their academic achievement, as measured by standardized tests, falls substantially in both math and English relative to that of their counterparts who continue to attend a K–8 elementary school. What’s more, their achievement continues to decline throughout middle school. This negative effect persists at least through 8th grade, the highest grade for which we could obtain test scores.

We found that the middle-school achievement gap cannot be explained by a scarcity of financial resources for the schools. Instead, the cause is more likely to be related to other school characteristics, especially the fact that middle schools in New York City educate far more students in each grade. Although our conclusions about the reasons for the middle-school gap are tentative, we are quite confident that the evidence shows that middle schools are not the best way to educate students—at least in places like New York City.

Data and Methods

Our study was based on data for New York City school children who were in grades 3 though 8 during the 1998–99 through 2007–08 school years. We were able to follow students who entered 3rd grade between the fall of 1998 and the fall of 2002 for six years, until most had completed the 8th grade. We have data about the grade configuration and other characteristics of their schools, individual academic achievement as measured by annual standardized test scores in math and English, and a variety of personal characteristics. In particular, we know each student’s gender, ethnicity, whether they received free or reduced-price lunch through the federal lunch program, whether they were English language learners or received special education services, and their record of suspensions and absences from school.

Elementary schools in New York City typically serve students until grade 5 or grade 6, while a smaller portion of elementary schools run through grade 8. This means that most students move to a middle school in either grade 6 or grade 7, while some never move to a middle school. Of the 3rd graders in our initial sample of students, 62 percent were in a K–5 school, 24 percent were in a K–6 school, and 7 percent were enrolled in a K–8 school. The small fraction of remaining students attended K–3, K–4, or K–7 schools and are excluded from our analysis.

To isolate the impact of attending a middle school from the many other factors that influence student achievement, we combined two basic strategies. Most importantly, we tracked the performance of individual students over time to see how their performance evolved relative to that of their peers as they progressed from grades 3 to 8, in essence, using each student as his or her own control group. This step alone provides much stronger grounds for conclusions about the effects of attending a middle school than previous research.

A lingering concern, however, is the possibility that different types of students choose to attend middle schools than choose to continue in a K–8 school. If students do sort themselves into middle schools because of some unobserved characteristic that causes changes in academic achievement over time, we would incorrectly attribute differences in achievement to the middle schools instead of to characteristics of the students themselves. We reduced the likelihood of making this mistake by using a statistical technique that effectively takes the choice to switch schools out of the students’ (or parents’) hands. Specifically, we ran a statistical model that used the last grade served by the school that a student attended in grade 3 to predict whether the student attended a middle school. We then used that prediction to place each student into one of the two groups we are comparing, that is, students who attend middle schools and those who do not. Our key assumption in taking this approach is that there are no unobservable factors that cause a drop in student achievement at precisely the same time as students must leave the elementary schools they attended in grade 3. While we cannot definitively rule out the existence of such factors, we do not know of any plausible alternatives that would explain our findings.

The Middle-School Disadvantage

What determines a student’s level of academic achievement is complex. But the simple fact is that students who enter public middle schools in New York City fall behind their peers in K–8 schools.  This is true both for math and English achievement. Even more troubling, the middle-school disadvantage grows larger over the course of the middle-school years. With the transition into a middle school, students set out on a trajectory of lower achievement gains.

The achievement gap between middle-school students and K–8 students is put in stark relief in Figure 1, which displays our estimates of the impact of attending a middle school on student achievement as measured by standardized tests in math and English Language Arts. The graphs show how well students who attend a middle school perform relative to how we would expect them to perform if they attended a K–8 school. We report those differences, in standard deviations of student achievement in math and reading, for the 3rd through 8th grades. We separate students who enter a middle school in grade 6 from those students who enter a year later, in grade 7.

No matter whether students enter a middle school in the 6th or the 7th grade, middle-school students experience, on average, a large initial drop in their test scores. Even after accounting for a host of other factors that influence student achievement, students who eventually attend middle schools go from scoring better than their counterparts in K–8 schools in the year prior to transitioning to middle school to scoring below where we would expect if they were not attending a middle school. Math achievement for 6th graders transitioning to middle school falls by 0.18 standard deviations, and English achievement falls by 0.16 standard deviations. Contrast that decline with the 6th-grade test scores for students who will enter middle school the following year, in the 7th grade. Their test scores in both subjects continue to improve relative to their peers in K–8 schools. When these 6th graders move to a middle school in the 7th grade, however, we see the same dramatic fall in academic achievement: math scores decline by 0.17 standard deviations and English achievement falls by 0.14 standard deviations. Just how large are these effects? Consider that decrease in achievement associated with middle school entry—between 0.14 and 0.18 standard deviations—is roughly 20 to 25 percent of the achievement gap between poor and non-poor students (as measured by free lunch receipt) in New York City (about 0.7 standard deviations).

Moreover, these are not temporary dips followed by rebounds in learning. Throughout the middle-school years, students fall further behind. After two years in a middle school, on average a student who entered in the 7th grade will score 0.10 standard deviations in math and 0.09 standard deviations in English below what we would expect if he had gone to a K–8 school. After three years in a middle school, a student who entered in the 6th grade will underperform on 8th-grade assessments by 0.17 standard deviations in math and by 0.14 standard deviations in English.

A particularly distressing finding from our study is that students with lower initial levels of academic achievement fare especially poorly in middle school. To investigate the possibility of different effects on students with higher and lower initial achievement levels, we separated students into two groups: one group had grade 3 test scores above the citywide median, the other group scored below the median. Although we found substantial drops in achievement during middle school for both groups of students, the first-year drop and cumulative deficit were, respectively, 50 percent and more than 200 percent greater for students who start at the lower end of the achievement distribution.

We also found evidence that student absence rates increased when students entered middle schools and were significantly higher in grade 8 than for students who never entered a middle school (see Figure 2). More specifically, our estimates indicate that students were missing almost two additional days of school per year than would have been the case had they attended a K–8 school. Thus, increased absences may be one mechanism through which middle schools lower student achievement. There is little chance, however, that absences could explain a large share of the overall effect of attending a middle school.

To be sure, the population of public school children in New York City is different from that of many other school districts around the country. These differences might mean that middle-school attendance would have smaller or larger effects on other students than we estimate it to have on New York City’s public school children. For example, students with fewer educational resources at home may be more strongly affected by changes in their school environment. If that is the case, studying New York City students, who arguably come from less advantaged backgrounds than, say, the students in New York City suburbs, may have led us to find a larger middle-school effect than had we followed a more-affluent student population. While we encourage readers to be cautious about applying our findings without qualification to all public schools, we also encourage school districts to support research that can identify middle-school effects in other settings, especially since we find the consequences of attending a middle school for student achievement to be substantial and troubling.

Explaining the Trouble with Middle Schools

Why might New York City’s middle schools be detrimental to academic achievement? We find little support for the notion that differences in resources, such as per-pupil expenditures and class size, could explain the middle-school achievement gap. In middle schools serving grades 6–8 and grades 7–8, average per-pupil expenditures were $10,094 and $11,082, respectively, while per-pupil expenditures in K–8 schools were roughly equivalent, at $10,950. Nor do students experience a large decline in per-pupil spending when they move to a middle school. Average per-pupil expenditure in K–5 schools was $10,144 (compared to the $10,094 for grade 6–8 middle schools) and $9,680 in K–6 schools (compared to $11,082 in grade 7–8 middle schools).

Nor can we attribute the disparity we see to differences in class size. The average class size is slightly smaller for 5th graders in K–5 schools than for 6th graders in 6–8 schools (24 vs. 25 students); students in K–8 schools see similar growth in class size between grades 5 and 6. Class size is actually larger for grade 6 students in K–6 schools than for grade 7 students in 7–8 schools (24 vs. 23 students).

What about the possibility that the relative age of students in a school, especially during adolescence, can influence how students learn? In other words, does being the youngest students in a school have negative effects on the educational experience of those students? We could not find evidence in our data to support this explanation for the initial drop in test scores upon transitioning to a middle school. In our study sample, about one-third of new 7th graders moved out of a school serving grades K–6 and entered a middle school for 7th and 8th graders, becoming the youngest cohort in the school, while roughly half of new 7th graders entered a grade 6–8 middle school as part of the school’s middle cohort of students. We find that the effect of entering a middle school was essentially the same for both of these groups.

At least part of the problem with middle schools may be that they usually combine students from multiple elementary schools. In the New York City schools we studied, the average cohort size was 75 students in K–8 schools, 100 students in K–5 and K–6 schools, and over 200 students in middle schools for grades 6–8 and 7–8 (see Figure 3). We went back to our data and analyzed the effect of these cohort size differences on test scores. What we found was that cohort size has a pronounced influence on student achievement during these school years. We estimate that an 8th grader who attends school with 200 other 8th-grade students will score 0.04 standard deviations lower in both math and English than he would if he attended a school with 75 other 8th graders, the average cohort size for a K–8 school. This 0.04 standard deviation deficit represents roughly one-quarter of the largest test-score declines we attribute to middle-school attendance.

Given the data we have, we can only speculate about why it is harder to educate middle school–aged students in large groups. Developmental psychologists have shown that adolescent children commonly exhibit traits such as negativity, low self-esteem, and an inability to judge the risks and consequences of their actions, which may make them especially difficult to educate in large groups. The combining of multiple elementary schools and their students also disrupts a student’s immediate peer group. And middle schools often serve a more diverse student population than many students encountered in elementary school. Yet while it seems plausible that these changes in environment would matter, we could not find any evidence in our data that any one hypothesis can explain the drop in learning among students moving to middle schools.

Even though a full explanation of the middle-school achievement gap eludes us, there does seem to be a consensus among New York City students and their parents that educational quality in the city’s public middle schools is lower than in the boroughs’ K–8 schools. We reached this conclusion after examining responses to a citywide survey of parents of children in grades K–8 and students in grades 6 and higher, which was conducted at the end of the 2006–07 and 2007–08 school years as a part of the city’s new school accountability system.

On average, New York City parents of students in middle schools gave their schools lower marks on measures related to education quality than parents whose children attend K–8 schools. Figure 4 shows that parent evaluations of school safety, academic rigor, and overall educational quality was much lower among those whose children attended middle schools than among parents with children in K–5, K–6, and K–8 schools. It is important to note that this is not simply a product of the challenges of educating adolescents. There is little perceptible decline in satisfaction among parents in K–8 schools as their children age, a consistency we would not expect if educational quality simply cannot withstand the onslaught of puberty.

The students’ opinions are consistent with their parents’ assessments, although the lack of data on students below grade 6 prohibits us from more direct measurement of the degradation of education quality in middle schools. The clearest pattern that emerges from student reports is that 6th and 7th graders in middle schools think their schools have less academic rigor, less mature social behavior among the students, are less safe, and provide lower-quality education than do 6th graders in K–6 or K–8 schools.

The Longer View

We don’t yet know whether the troubling slide in test scores for middle-school students persists through the end of high school, a question that is certainly worth studying. Unfortunately, our data do not allow us to follow the students in our study further than grade 8. If the decline does continue, middle schools not only hurt student achievement in the short term but set students up for unnecessary longer-term disadvantages.

Of course, it is possible that transitioning to high school could be more difficult for students who come from K–8 schools than for middle school students. If K–8 students experience a larger drop in achievement upon entering high school, that could bring the two groups of adolescents back into parity. But it is hard to recommend closing the middle-school achievement gap by bringing everybody down. The better option is to address the trouble with middle schools—or do away with them altogether.

Jonah E. Rockoff is associate professor of business at the Columbia Graduate School of Business. Benjamin B. Lockwood is research coordinator at the Paul Milstein Center for Real Estate at the Columbia Graduate School of Business.

An unabridged version of this article is available here.

Never before have Americans had greater access to information about school quality. Under the federal No Child Left Behind Act (NCLB), all school districts are required to distribute annual report cards detailing student achievement levels at each of their schools. Local newspapers frequently cover the release of state test results, emphasizing the relative standing of their community’s schools. Meanwhile, new organizations like GreatSchools and SchoolMatters aggregate this information and make it readily available to parents online.

But do all these performance data inform perceptions of school quality? Or do citizens base their evaluations instead on such indicators as the racial or class makeup of schools, regardless of their relationship with actual school performance?

In discussions of parental choice in education, researchers have frequently speculated that parents would base their evaluations of schools primarily on the characteristics of their student bodies. Columbia University professor Amy Stuart Wells, for example, concluded that the decisions of St. Louis parents participating in a voluntary desegregation program were based “on a perception that county is better than city and white is better than black, not on factual information about the schools.” And even if some parents base their decisions on educational quality, many observers worry that low-income and minority parents will be less informed about or interested in school quality, placing their children at a disadvantage in the education marketplace.

The evidence on these questions available to date comes from small-scale studies of specific school districts, making it difficult to reach general conclusions about the degree to which parents and the public at large are well informed about the performance of local schools. We are now able to supplement that research with data from a nationally representative survey of parents and other adults conducted in 2009 under the auspices of Education Next and the Program on Education Policy and Governance (PEPG) at Harvard University. Because we knew the addresses of respondents in advance of the survey, we were able to link individual respondents to specific public schools in their community and to obtain their subjective ratings of those schools. We also gathered publicly available data on student achievement in the same schools, making it possible to compare respondents’ subjective ratings to objective measures of school quality.

Our results indicate that citizens’ perceptions of the quality of their local schools do in fact reflect the schools’ performance as measured by student proficiency rates in core academic subjects. Although citizens also appear to take into account the share of a school’s students who are poor when evaluating its quality, those considerations do not overwhelm judgments based on information about academic achievement.

Public Perception and Objective Quality Measures

The 2009 Education Next–PEPG Survey was administered to a nationally representative sample of 3,251 American adults, including an oversample of 948 residents of the state of Florida. The Florida oversample was conducted in order to link perceptions of school quality to the unusually rich information about school performance available in that state. The survey was administered over the Internet by the polling firm Knowledge Networks in February and March of 2009. (For methodological details and complete survey results, see “The Persuadable Public,” features, Fall 2009.)

Before conducting the survey, we geo-coded the address of each respondent to latitude-longitude coordinates and a census block. We also obtained latitude-longitude coordinates for every U.S. public school from the National Center for Education Statistics. Using census blocks to place respondents within school districts, we then linked each respondent to the closest elementary, middle, and high schools (up to five schools of each type) operated by the local school district.

The survey asked all respondents this question: “Each of the following schools in your area serves elementary-school students. Which one, if any, do you consider your local elementary school?” It then offered each respondent a personalized list of the five closest elementary schools from which to pick; respondents were also allowed to specify a school that did not appear on the list. After a specific elementary school had been identified, the survey asked the respondent to grade this school on a scale from A to F. This same process was then repeated for middle and high schools.

We converted the A to F grades that respondents assigned to the schools into a standard grade-point-average (GPA) scale (A=4 and F=0). Of the elementary and middle schools our survey respondents rated, 41 percent received a B grade, while 36 percent received a C. In contrast, only 14 percent of schools received an A grade, 7 percent a D, and 2 percent an F. This distribution corresponds to an overall GPA of 2.57, or just below a B-minus average. Interestingly, respondents assigned their local middle schools grades that were, on average, one-quarter of a letter grade lower than the grades they assigned their local elementary schools (see Figure 1).

We measured actual school quality as the percentage of students in a school who achieved “proficiency” in math and reading on the state’s accountability exams (taking the average proficiency rate across the two subjects). School-level data on student proficiency were drawn from SchoolDataDirect.org for the 2007–08 school year, the most recent year for which test-score data would have been publicly available when the survey was conducted. Although the rigor of state content standards and definitions of math and reading proficiency vary widely (see “State Standards Rise in Reading, Fall in Math,” features), we are able to adjust for these differences by limiting our comparisons to respondents within the same state when examining the relationship between proficiency levels and school ratings.

To be sure, the percentage of students achieving proficiency in core academic subjects is an imperfect measure of quality, even when comparing schools in the same state. Given the strong influence of out-of-school factors on student achievement, any quality measure based on the level of student performance at a single point in time will be heavily influenced by characteristics of a school’s student body. At the same time, proficiency rates are the only quality measure available for a national sample of schools. They are determined in part by the amount students learn in school, and research suggests that moving to a school with higher proficiency rates does produce achievement gains.

Nor do we wish to claim that any judgment of school quality that does not correspond to test-score performance is uninformed or irrational. The ability to promote math and reading achievement is hardly the only dimension along which citizens are likely to evaluate their local schools. But we suspect that high test scores go along with other aspects of school quality that citizens value in their schools, so that evidence of a connection between student achievement and public opinion likely indicates that parents and other members of the public have the information they need to make reasonable judgments about their schools.

National Evidence

These data enable us to provide the first evidence on the extent to which citizens’ subjective ratings of specific schools correspond to publicly available information on their actual performance. Because other school characteristics may also influence perceptions of school quality, we incorporated into our analysis data from the National Center for Education Statistics on the racial/ethnic composition of each school, the percentage of students eligible for free or reduced-price lunch (an indicator of poverty), average cohort size (our preferred measure of school size), and pupil-teacher ratio (a proxy measure of class size) in the 2007–08 school year. We exclude high schools when analyzing the data for the nation as a whole because proficiency data are unavailable for many of them, and when available, typically reflect the performance of only a single cohort of students. We also adjust for whether the respondent was evaluating an elementary or a middle school to account for the fact that middle schools received systematically lower grades from survey respondents.

Figure 2 presents the strength of the relationship between citizen ratings of school quality and each of these school characteristics after taking into account the other key variables built into our analysis. The values of each variable except the one identifying elementary schools have been standardized to illustrate their relative importance. (In technical terms, the relationships presented for these variables reflect the effect of an increase of one standard deviation in the value of the characteristic in question.) The figure confirms that student proficiency rates are a significant predictor of citizen ratings of school quality. An increase of 18 percentage points in percent proficient (i.e., one standard deviation) is associated with a rating that is on average 0.16 grade points higher, or about one-sixth of a letter grade.

Examining the racial/ethnic and class makeup of a school’s student body in isolation would suggest that both are important predictors of citizen ratings, a fact that may explain the common perception that this is the case. In particular, schools with 25 percentage points more African American students received ratings that were 15 percent of a letter grade lower, while schools with 24 percentage points more Hispanic students received ratings that were 16 percent of a letter grade lower. Schools with 26 percentage points more poor students received ratings that were one-quarter of a letter grade lower.

However, when these variables are considered simultaneously and alongside school performance and resource measures, only the poverty indicator retains predictive power. Neither the percentage of students who are African American nor the percentage who are Hispanic is systematically related to perceptions of school quality. The percentage of students who are poor remains an important predictor of citizen ratings, with a relationship essentially as strong as that for proficiency rates.

Even after controlling for proficiency rates and other school characteristics, middle schools receive ratings that are, on average, 18 percent of a letter grade lower than comparable elementary schools. In other words, proficiency rates explain some, but by no means all, of the lower perceived quality of middle schools. This finding is of interest given recent research suggesting that middle schools have adverse consequences for student achievement (see “Stuck in the Middle,” research). In contrast, neither school size nor pupil-teacher ratio are important determinants of perceptions of school quality. In fact, the weak relationship between pupil-teacher ratio and school ratings is in the opposite of the expected direction: schools with larger classes receive somewhat higher grades, perhaps because effective schools attract more families to the neighborhood.

As noted above, it has often been speculated that disadvantaged groups are less informed about school quality than more-advantaged groups. But we find that the relationship between school performance and citizen ratings is as strong for African American and Hispanic respondents as it is for whites. The relationship between school quality and citizen ratings is also essentially the same for high-income and more-educated respondents as it is for low-income and less-educated respondents.

We also consider whether the relationship between school performance and citizen ratings is stronger for parents of school-age children, who are arguably the most connected to their local schools, or for homeowners, whose property values are influenced by school quality. Perhaps surprisingly, homeowners are no more sensitive to differences in school quality than are other citizens. However, the relationship between proficiency rates and school ratings is more than twice as strong for parents of school-age children than for other respondents (see Figure 2). An increase of one standard deviation in percent proficient is associated with a rating from parents that is one-third of a letter grade higher, as compared with 16 percent of a letter grade higher for the public as a whole. Parents also give low-scoring schools far lower ratings than do other local residents, but this difference narrows and eventually reverses direction as proficiency rates increase (see Figure 3). Like those of other citizens, parents’ ratings of local schools are not influenced by the schools’ racial/ethnic composition, school size, or pupil-teacher ratios. However, parents do appear to be somewhat more responsive than other citizens to school poverty rates and take an especially dim view of middle schools, assigning them grades that are 39 percent of a letter grade lower than otherwise similar elementary schools.

Finally, we consider the issue of differences in school quality across states. Because NCLB allows each state to set its own standards for proficiency, schools in different states with the same percentage of students achieving proficiency may be of markedly different quality if one state has high standards and the other low. The national sample allows us to examine the degree to which citizen ratings of school quality are responsive to performance levels relative to the nation or simply to differences in performance within specific states. The National Assessment of Educational Progress (NAEP) conducted every two years by the U.S. Department of Education provides evidence on the average performance of 4th- and 8th-grade students in each state in mathematics and reading. We use data from the 2007 NAEP to see whether respondents in states with higher-scoring students rate their schools higher, on average, than respondents in states with lower NAEP scores. That is, if we compare respondents whose local schools have the same proficiency rate as measured by their state test, do the respondents in states with better schools, as measured by student performance on the NAEP, assign their school higher grades? We find no evidence that respondents in general, or even parents, have information about school quality beyond the information provided on the state assessments. In other words, citizens appear to be taking cues about school quality from local comparisons or from information provided by their state testing system without taking into account the relative rigor of state standards.

Levels or Growth?

Our analysis yields strong evidence that citizens, and especially parents of school-age children, rate schools in a way that lines up with publicly available information about school quality. As discussed previously, however, the percentage of students scoring at the proficient level on state tests is an imperfect indicator of school quality, contaminated as it is by the fact that student achievement is influenced by a host of factors outside of a school’s control. A better, if still an imperfect, measure of school quality is the amount of growth in student achievement from one year to the next. To examine the correspondence of citizen perceptions of school quality and measures of test-score growth, we turn to our representative sample of residents of Florida, where the state accountability system evaluates schools based on both test-score levels and test-score growth. Because high-school performance data are widely available in Florida, we are able to include high schools in this portion of the analysis.

Florida assigns schools letter grades based on a point system with eight main components, which we divide into two categories: level-related points (percentage proficient in math, English, writing, and science) and growth-related points (percentage making learning gains in math and reading and the percentage of the lowest 25 percent of students making gains in math and reading). The level variable is highly correlated with the school quality measure (percent proficient) used in the national analysis, but the correlation between the growth variable and percent proficient is considerably weaker.

Our basic strategy is to compare the ratings Florida residents assigned to their schools both to test-score levels and to test-score growth at those schools. Because measures of test-score growth are less stable over time than measures of test-score levels, we average the points awarded to each school based on levels and growth over the previous three years. Adjustments are also made for the same demographic and school characteristics as in the national analysis. To make the results as comparable as possible to those reported for the national sample, we also scale the point variables so that a one-unit increase in each variable corresponds to a shift of one standard deviation in the performance distribution of Florida public schools.

The results indicate that Florida residents’ perceptions of school quality are even more responsive to differences in student achievement levels than are those of the national public. An increase of one standard deviation in the level variable is associated with ratings that are almost one-third of a letter grade higher after taking into account other school characteristics. We also find that perceptions of school quality in Florida are unrelated to student demographic characteristics, including the percentage of students who are poor, once we take into account levels of student achievement. Although we cannot be sure, both Floridians’ greater responsiveness to test performance and their lack of responsiveness to student demographic characteristics could reflect the transparency and salience of the state’s high-profile school accountability system.

When both the test-score level and growth variables are examined simultaneously, however, the relationship between level-related points and citizen evaluations of schools is almost twice as strong as for growth-related points. This suggests that citizen ratings do reflect differences in the growth in student achievement across schools, but that this is primarily because of the correlation between achievement levels and achievement growth.

The Role of Accountability Systems

So far we have shown that citizens’ assessments of schools are strongly related to objective measures of performance made available by state accountability systems. Yet it is difficult to determine whether respondents’ apparent sensitivity to actual quality is the result of publicly available information or simply direct experience with schools. The fact that parental perceptions track actual school quality more closely than those of other citizens, but the perceptions of homeowners do not, suggests that direct interactions with a school may be a more important factor than simply having a vested interest in acquiring information about local school quality. But do accountability systems also play a role in shaping citizen perceptions?

Again, Florida provides an ideal case for more detailed analysis. As noted above, the Florida Department of Education uses the total number of points received (i.e., the sum of level- and growth-related points) to assign each school a letter grade between A and F. These grades receive considerable media attention in Florida, so we might expect citizen ratings to be correlated with them. This expectation is confirmed in the data: a school grade that is one point higher (again measured on a standard GPA scale) is associated with a respondent rating that is 0.2 grades higher.

To test the hypothesis that publicly available information has an impact over and above direct observation of school performance, we can compare the ratings given by respondents whose schools were very close to the cutoffs in the point system used by Florida to assign school grades. We know that schools with more points received higher ratings on average, but might also expect to see a “jump” in the average rating at these cutoffs. Because schools on either side of the cutoff should be of essentially the same quality, we can interpret any jump in the rating observed at the cutoff as the pure effect of information provided by the school grade on citizen perceptions of school quality.

We focus our attention on the B/C cutoff, because that is the only one for which we have enough respondents assigned to schools near the cutoff to yield results with a reasonable degree of precision. Comparing respondents’ ratings of schools on either side of this cutoff suggests a large positive effect of receiving the higher (B) grade, with an increase in the grades assigned to schools in the range of of 36 to 57 percent of a letter grade. That the publicized school grades have a direct effect on respondent ratings over and above the relationship between ratings and the underlying point variables suggests that the signals provided by the state’s school accountability system do in fact affect citizen perceptions of their local schools.

Implications

The findings reported above represent the first systematic evidence that Americans’ perceptions of the quality of their local public schools reflect publicly available information about the academic achievement of the students who attend them. Importantly, disadvantaged segments of the population are no less informed about school quality than other citizens. Although the mechanisms explaining this responsiveness are not entirely clear, our evidence suggests that both direct experience with schools and the public dissemination of performance data may play a role.

It is worth emphasizing several limitations on this evidence of responsiveness. First, the relationship between actual and perceived quality is modest for citizens as a whole, although it is quite strong for parents, who have the most opportunities to observe schools and arguably have the strongest incentives to be informed. Second, both parents and the public appear to be more responsive to the level of student achievement at a school than to the amount students learn from one year to the next. Finally, citizens appear sensitive to relative differences in school quality within their state (as reflected in school performance on state tests) but insensitive to information on school quality in the state as a whole (as measured by statewide performance on a national assessment).

Even so, at least two policy implications emerge from our results. First, our finding that accountability ratings influence citizens’ assessments of their local schools coupled with the fact that citizen ratings are more strongly associated with achievement levels than with achievement growth suggest that featuring growth measures more prominently in school accountability ratings could cause citizens to pay more attention to this barometer of school quality. Second, our finding that citizen ratings are associated with student performance on state tests but not with performance on a national assessment suggests that a closer alignment of state standards (or a move toward common standards across states) might help citizens form more accurate perceptions of their schools. In particular, it could lower perceptions of school quality in states where many students perform poorly relative to national norms but are deemed proficient by the state.

Matthew M. Chingos is a postdoctoral fellow at Harvard University’s Program on Education Policy and Governance. Michael Henderson is a doctoral candidate in Harvard’s Department of Government. Martin R. West is assistant professor of education at the Harvard Graduate School of Education and executive editor of Education Next.

How has the No Child Left Behind (NCLB) Act affected student achievement? This is no idle question, as the landmark federal law is long overdue for reauthorization. The Obama administration has recently urged Congress to add the issue to its already crowded 2010 agenda, even going so far as to include an additional $1 billion for K–12 education in its budget proposal if the law is reauthorized this year (a wholly symbolic gesture, given that it is Congress that sets spending levels, but one that indicates the administration’s priorities).

Yet heightened attention to NCLB has not produced consensus over its consequences for students. No Child Left Behind was a reauthorization of the Elementary and Secondary Education Act (ESEA), the central federal legislation relevant to K–12 schooling. NCLB dramatically expanded the law’s scope by requiring that states introduce school-accountability systems that applied to all public schools and students in the state. NCLB requires annual testing of students in reading and mathematics in grades 3 through 8 (and at least once in grades 10 through 12) and that states rate schools, both as a whole and for key subgroups, with regard to whether they are making adequate yearly progress (AYP) toward their state’s proficiency goals. Supporters and critics, in their various approaches to discerning NCLB’s impact, share a significant problem: because NCLB applies to all public school students, researchers lack a suitable comparison group and so have been unable to distinguish the law’s effects from the myriad other factors at work over the past eight years.

The new research we present below takes on this challenge. Our basic insight is that the test-based accountability provisions that are the defining characteristic of NCLB did not come from nowhere, but rather were modeled quite closely on reforms adopted by many states in the 1990s. For states with such accountability systems in place before 2002, NCLB’s most important components may have created some logistical headaches but were largely irrelevant. In contrast, NCLB forced the remaining states to enact accountability systems for the first time. We can therefore estimate the impact of NCLB’s accountability mandates by comparing test-score changes in states that did not have NCLB-style accountability policies in place when the law was implemented to test-score changes in those that did.

We find that the accountability provisions of NCLB generated large and statistically significant increases in the math achievement of 4th graders and that these gains were concentrated among African American and Hispanic students and among students who were eligible for subsidized lunch. We find smaller positive effects on 8th-grade math achievement. These effects are concentrated at lower achievement levels and among students who were eligible for subsidized lunch. We do not, however, find evidence that NCLB accountability had any impact on reading achievement among either 4th or 8th graders.

Assessing NCLB

The broad interest in understanding whether NCLB has influenced student achievement, both overall and for key subgroups, has motivated careful scrutiny of trend data from the National Assessment of Educational Progress (NAEP) and other sources. For example, the authors of a report commissioned by the U.S. Department of Education’s Institute of Education Sciences (IES) note that achievement trends on both state assessments and NAEP were “positive overall and for key subgroups” through 2005. Using more recent data, a report by the Center on Education Policy concludes that reading and math achievement as measured by state assessments has increased in most states since 2002 and that there have been smaller but similar patterns in NAEP scores. Both reports were careful to stress that these national gains are not necessarily attributable to the effects of NCLB.

Other studies have taken a less sanguine view of these achievement gains, arguing that they are misleading because states have made their assessment systems less rigorous over time. University of California scholar Bruce Fuller and colleagues, for example, document a growing disparity between student performance on state assessments and NAEP since the introduction of NCLB and conclude that “it is important to focus on the historical patterns informed by the NAEP.” Using NAEP data on 4th graders, they conclude that the growth in student achievement has actually slowed since the introduction of NCLB.

Turning to the broader literature on school accountability, several researchers have evaluated the achievement consequences of the accountability systems states developed during the 1990s. One study by Martin Carnoy and Susanna Loeb of Stanford, which was based on state-level NAEP data, found that the within-state growth in math performance between 1996 and 2000 was larger in states with higher values on an accountability index, particularly for African American and Hispanic students in 8th grade. Another study, by Eric Hanushek and Margaret Raymond, both also at Stanford, evaluated the impact of school-accountability policies on state-level NAEP math and reading achievement measured by the difference between the performance of a state’s 8th graders and that of 4th graders in the same state four years earlier. They classified states as having either “report-card accountability” or “consequential accountability.” Report-card states provided a public report of school-level test performance. States with consequential accountability both publicized school-level performance and attached consequences to that performance. Hanushek and Raymond found that the introduction of consequential accountability within a state was associated with increases in NAEP scores.

Both of these studies suggest that NCLB-style accountability provisions may increase student achievement and also demonstrate how state-level NAEP data can be used to evaluate accountability systems. The analysis described below effectively extends this important work to cover the more recent state accountability reforms that were compelled by NCLB.

Research Design

Given the various social, economic, and educational factors at work before and after NCLB was implemented, it is difficult to draw strong conclusions about the policy’s impact from a simple comparison of achievement trends before and after enactment of the law. For example, the nation was suffering from a recession around the time NCLB was implemented, which one might expect would have reduced student achievement in the absence of other forces. At the same time, other national education policies and programs were in place that may also have influenced student achievement.

Perhaps the central challenge in evaluation research is to identify a plausible comparison group that was unaffected by the intervention under study. In the case of NCLB, this is particularly difficult, as the policy simultaneously applied to all public schools in the United States.

We address this issue by comparing trends in student achievement across states that had varying degrees of prior experience with state school-accountability policies similar to those brought about by NCLB. The intuition behind this approach is that NCLB represented less of a “treatment” in states that had already adopted NCLB-like school-accountability policies prior to 2002. To the extent that NCLB-like accountability had either positive or negative effects on measured student achievement, we would expect, once NCLB had been implemented, to observe those effects most distinctly in states that had not previously introduced similar policies.

This strategy relies on the assertion that pre-NCLB school-accountability policies were comparable to NCLB—that is, that the two types of accountability regimes are similar in the most relevant respects. The fact that many state officials criticized NCLB, arguing that it duplicated their prior accountability systems, suggests the functional equivalence of the two sets of policies. To ensure that this is the case and relying on a number of different sources, we evaluated the comparison states according to whether the features of their pre-NCLB accountability policies closely resembled the key aspects of NCLB. We found that they were in fact quite similar.

As an additional check on the validity of our treatment and comparison groups, we used our research design to estimate the impact of NCLB accountability on outcomes that we would not expect to be affected, such as the state-level average poverty rate and median household income. The fact that our method does not find any “effect” of NCLB on such outcomes suggests that these states can serve as a plausible comparison group for isolating the impact of NCLB accountability.

We implement our research design in a more fine-grained manner than simply comparing achievement trends in the treatment and comparison states. We define the treatment as the number of years without prior school accountability between the 1991–92 academic year and the onset of NCLB. Hence, states with no prior accountability have a value of 11. Illinois, which adopted its policy in the 1992–93 school year, would have a value of 2. Texas would have a value of 4 since its policy started in 1994–95, and Vermont would have a value of 9 since its program began in 1999–2000. This method implies that the larger the value of this treatment variable, the greater potential impact of NCLB. The total effect we report is the impact of NCLB accountability in 2007 for states with no prior accountability relative to states that adopted school accountability in 1997 (the mean adoption year among states that adopted accountability prior to NCLB).

It is important to note that this research design will capture the impact of the accountability provisions of NCLB, but not the impact of other NCLB components such as the Reading First program or its Highly Qualified Teacher provisions. Additionally, our estimates will identify the impact of NCLB-induced school-accountability provisions on states without prior accountability policies. To the extent that one believes that states that expected to gain the most from accountability policies adopted them prior to NCLB, one might view the results we present as an underestimate of the average effect of school accountability.

Data

This analysis uses data on math and reading achievement from the state NAEP, which offers a representative sample of student achievement in each state at regular intervals. Participation in the state NAEP was voluntary prior to NCLB, although roughly 40 states did participate. NCLB made participation mandatory. The main advantage to using NAEP data for our analysis is that it is a low-stakes exam that is not directly tied to any state’s standards or assessments. Instead, NAEP aims to assess a broad range of skills and knowledge within each subject area. Consequently, NAEP data should be relatively immune to concerns about accountability-driven test-score inflation, such as may result from “teaching to the test.”

Because our research design depends on measuring achievement trends prior to NCLB, we limit our sample to states that administered the state NAEP at least twice prior to the implementation of NCLB. We include 2002 as a pre-NCLB data point in our analysis because, given the timing of the passage and implementation of the law, it seems unlikely that spring 2002 scores could have been substantially influenced by NCLB (see sidebar). All states administered NAEP in 2003, 2005, and 2007.

Our sample includes 39 states for 4th-grade math, 38 states for 8th-grade math, 37 states for 4th-grade reading, and 34 states for 8th-grade reading (see Figure 1). With a few exceptions, our analysis sample closely resembles the nation in terms of student demographics (e.g., percentage African American and percentage Hispanic), observed socioeconomic traits (e.g., the poverty rate), and measures of the levels and pre-NCLB trends in NAEP test scores.

Results

We find that the accountability provisions of NCLB increased 4th-grade math achievement by roughly 7.2 scale points (0.23 standard deviations) by 2007 in states with no prior accountability policies relative to states that adopted accountability systems in 1997. How large is this effect? As one point of reference, consider that the difference between the average scores of 4th and 8th graders in our sample suggests that students gain roughly 12 scale points per year. By this measure, the NCLB impact is equivalent to roughly two-thirds of the average annual gain in scale points. Consider also that the achievement gap between black and white 4th graders on the NAEP math exam is roughly 30 scale points (1 standard deviation), which means that the impact of NCLB is equivalent to about one-quarter of this difference. The effect for 8th-grade math is smaller (0.10 standard deviations) and falls just shy of achieving conventional levels of statistical significance. We find no effects for 4th- and 8th-grade reading.

The design of NCLB necessarily focused the attention of schools on helping students attain proficiency. Figure 2 presents our estimates of the effects of NCLB accountability on the percentage of students achieving at or above the basic and proficient performance levels on NAEP. Although states’ definitions of proficient vary widely, very few set the proficiency bar as high as NAEP and most correspond more closely to NAEP’s basic performance level. We find that NCLB accountability increased the share of students performing at or above basic in math by 10 percentage points among 4th graders and 6 percentage points among 8th graders. Math proficiency rates among 4th graders also increased by 6 percentage points. Again, however, we do not find consistent evidence that NCLB increased reading performance at either grade level.

Given NCLB’s focus on proficiency, one would expect the law to disproportionately influence achievement among previously low-achieving students. Our results showing larger increases in the percentage of students reaching the performance level of basic on the NAEP are broadly consistent with this theory. However, in contrast with some previous research and commonly voiced concerns, we do not find that the introduction of NCLB harmed students at higher points on the achievement distribution. Indeed, NCLB accountability seemed to increase achievement among higher-achieving students, if by a smaller amount than it did among their low-achieving peers. For example, in 4th-grade math, we find that NCLB increased scores at the 10th percentile by roughly 0.29 standard deviations compared with an increase of only 0.17 standard deviations at the 90th percentile (see Figure 3).

One of the primary objectives of NCLB was to reduce inequities in student performance by race and socioeconomic status. Indeed, this concern drove the requirement that, under the statute, accountability ratings be determined by subgroup performance in addition to aggregate school performance. Hence, it is of particular interest to understand the effect of NCLB accountability on specific student subgroups.

In 4th-grade math, these estimated effects are somewhat larger for Hispanic students relative to white students. Similarly, the effects were substantially larger among students who were eligible for subsidized lunch (regardless of race) relative to students who were not eligible. We also found relatively large effects for black students but only when our analysis weighted the state-year NAEP data by the corresponding enrollments of black students. This pattern suggests that NCLB generated more meaningful improvements in the achievement of black students in states where public schools served larger numbers of black students. The effects were roughly comparable for boys and girls.

In 8th-grade math, we find extremely large positive effects for Hispanic students and small, only marginally significant effects for white students. Unfortunately, the results for black students are too imprecisely estimated to warrant interpretation. The effects for students eligible for subsidized lunch are large and statistically significant. Interestingly, for 8th graders the effects are substantially larger for girls, with boys experiencing little if any benefit of accountability.

Unintended Consequences?

One concern about NCLB and most other test-based school-accountability policies is that they may cause schools to neglect subjects other than math and reading. NAEP data offer some opportunity to test this hypothesis in the context of NCLB. A sizable number of states administered state-representative NAEP tests in science. Unfortunately, during our analysis period, the 4th-grade science exam was only administered in 2000 and 2005 and the 8th-grade science exam was administered in 1996, 2000 and 2005. The lack of multiple pre- and post-NCLB measures of student achievement limit the power of our research design. Nonetheless, when we apply our research design to these data, we find no statistically significant effects at either grade level at any point on the achievement distribution. Moreover, we are able to rule out effects larger than roughly 0.10 standard deviations. While these results should be taken with a grain of salt, they cast doubt on some claims that NCLB accountability has had an adverse impact on student performance in science.

Another major concern with test-based accountability, including NCLB, is that it provides teachers an incentive to direct energy toward the types of questions that appear most commonly on the high-stakes test and away from other topics within the tested domain. As noted above, one of the benefits of the analysis presented here is that it relies on student performance on NAEP, which should be relatively immune from such test-score “inflation” since it is not used as a high-stakes test under NCLB or any other accountability system. It is nonetheless interesting to examine whether NCLB accountability has improved student achievement in any particular topic within math or reading. The NAEP math exam measures student performance in five specific topic areas: algebra, geometry, measurement, number properties and operations, and data analysis, statistics, and probability. Our results suggest that NCLB had a positive impact in all math topic areas for the 4th-grade sample. Among 8th graders, NCLB had a moderately large and statistically significant impact in data analysis and marginally significant effects in number properties and geometry.

The NAEP reading exam measures student competency in several skills related to comprehension: reading for information (i.e., primarily nonfiction reading), reading for literary experience (i.e., primarily fiction reading), and (for 8th grade only) the ability to perform a task (e.g., students apply knowledge from reading bus schedules or directions for repairing something). We find no significant differences in student achievement effects by topic area in reading; that is, NCLB accountability did not appear to have significant effects on student achievement in any of the three reading competencies. Keep in mind, however, that our research design does not allow us to comment on the effects of other aspects of the law, such as the Reading First program, that were explicitly designed to boost reading performance.

Summing Up

So how has NCLB accountability affected student achievement? Our results suggest that its consequences have been mixed. Specifically, we find that the accountability provisions of NCLB generated large and broad gains in the math achievement of 4th graders and somewhat smaller gains for 8th graders. Our results suggest that NCLB accountability had no impact on reading achievement for either group.

The mixed results presented here pose difficult but important questions for policymakers considering whether to “end” or “mend” NCLB. The evidence of substantial and almost universal gains in math is undoubtedly good news for advocates of NCLB. But the lack of any effect in reading, and the fact that the policy appears to have generated only modestly larger impacts among disadvantaged subgroups in math (and thus made only minimal headway in closing achievement gaps), suggests that the impact of NCLB has fallen short of its extraordinarily ambitious goals. Some commentators have argued that the failure of NCLB and earlier accountability reforms to close achievement gaps reflects a flawed, implicit assumption that schools alone can overcome the achievement consequences of dramatic socioeconomic disparities.

An effective redesign of accountability policies like NCLB may need to pay more specific attention to the processes and practices operating within schools. Along those lines, it is interesting to note that our evidence of differential effects by grade and subject is broadly similar to the results from evaluations of earlier state-level school-accountability policies. Understanding the sources of these differences is likely to be particularly useful as policymakers discuss the future design and implementation of school-accountability systems. For example, the unique effectiveness of NCLB in improving the math skills of younger students could be related to the biological evidence that cognitive skills are more malleable at early ages. These outcomes may also result from the specific ways in which schools and teachers have adjusted their instructional practices, perhaps differently for mathematics and reading. Much evidence suggests that school decisions about curricula (e.g., textbooks, instructional software, and the corresponding pedagogy) can have comparatively large effects on student achievement. Further research that can credibly and specifically examine how school and teacher responses have contributed to the achievement effects documented here would be a useful next step in identifying effective policies and practices that can reliably improve student outcomes.

Thomas Dee is associate professor of economics at Swarthmore College. Brian Jacob is professor of education policy and economics at the University of Michigan.

The constitutionality of state school-finance systems has been under attack for nearly 40 years. Since the California Supreme Court’s 1971 ruling in Serrano v. Priest, finance-reform advocates have filed 139 separate lawsuits in 45 states. The specific language varies from state to state, but virtually all state constitutions contain education clauses that require the state legislature to provide an “adequate,” “basic,” or “thorough and efficient” education for all children. Plaintiffs have relied on these provisions to seek increases in the financial resources devoted to public schools, especially those serving disadvantaged students. Courts have in turn deemed school-finance systems unconstitutional in 28 states.

While school-finance lawsuits have attracted significant attention in the legal community and generated numerous state-specific case studies, nationwide analyses of the effects of school-finance judgments (SFJs) have been relatively few. This small pool of studies has produced some common conclusions, namely, that such judgments reduce funding inequality between districts by increasing spending in the poorest districts and that they do so by transferring responsibility for education funding from local to state governments. Some questions remain unanswered, however, such as why SFJs have substantially different effects in different states.

A court’s ruling that an existing school-finance system is unconstitutional is only the first step toward funding reform. Some court orders provide instruction for how the legislature should fix the system, but most simply instruct state politicians to redesign the finance system themselves. In either case, the new finance system must garner the approval of the state legislature and governor. In other words, after the court ruling, the reform must pass through the state’s usual lawmaking process. States with similar court rulings may end up with very different reforms, depending on how the legislature and governor respond.

With this political process in mind, we decided to investigate how politics might influence the way an SFJ alters a state’s school-finance system. Our starting point was estimating the change in per-pupil funding that could be confidently attributed to an SFJ. We did this by comparing changes in funding in school districts where the state’s school-finance system has been ruled unconstitutional in a court challenge to funding changes in comparable districts in states where no SFJ has been issued. We studied district-level changes in school funding following 23 school-finance judgments issued between 1988 and 2005. The lawsuits were all related to general education funding, and each was the first SFJ in a state during our period of study. In total, we studied funding outcomes in more than 13,000 districts over 18 years.

What we were most interested to know is whether the change in funding differs if a state has unified Democratic control of the state legislature and the governorship at the time of the court decision, unified Republican control, or when control is divided between the two parties as, for example, when the governor is a Republican and the Democrats control one or both of the houses of the legislature. To find out, we compared the outcomes of SFJs issued in each of these circumstances.

We found that court-ordered finance reform alters district funding levels under each type of partisan regime. On balance, Democratic control results in across-the-board increases in state funding to local school districts, while Republican and divided-government regimes tend to produce funding increases targeted to poorer districts. SFJs in all three types of political environments lead to a shift in funding responsibility from local to state governments, although to differing degrees.

Which Party is Responsible?

As we began our study, we had to decide how to assign responsibility for school funding changes produced by an SFJ in the years following the judgment, especially when the party that controls the state government changed. We decided to focus on partisan control at the time of the court decision because the government at the time of the ruling is obligated to craft the policy response. Our approach, then, attributes the effect of the SFJ in subsequent years to the party in power when the judgment is made, even if there is a subsequent change in partisan control. We checked the validity of this decision by rerunning our analysis, attributing the funding associated with an SFJ in any given year to the party in control of the state government in that same year. With this method, our estimates of the relationship between partisan control and the effect of an SFJ, in dollars, were much less precise than when we used our preferred approach, although the substantive conclusions of our analysis remained the same. The better estimates lead us to conclude that party control at the time of the court decision has, on average, the most important role in determining the political response to an SFJ.

Table 1 lists the cases used in our analysis and the configuration of partisan control of the state government at the time of the court decision. Only three SFJs were issued during periods of unified Republican government: in New Hampshire, Ohio, and Wyoming. This suggests the need for caution in interpreting our results, especially about the patterns in school finance we see under Republican governments. There were seven judgments handed down during unified Democratic government (in Alabama, Kentucky, Maryland, Missouri, Tennessee, Vermont, and West Virginia) and 11 delivered when government was divided (in Connecticut, Idaho, Kansas, Massachusetts, Minnesota, Montana, North Carolina, New Jersey, New York, South Caroline, and Texas).

State, Local, and Federal Funding

While SFJs require a policy response from the state government, and therefore are expected to have a direct impact on state funding, they may also have an indirect effect on funding from local sources. Indeed, one concern over the efficacy of court-induced reforms is that local districts may reduce their own contribution to the schools in response to increases in state aid, thereby undermining efforts to increase total school spending. To provide a more comprehensive picture of the effect of SFJs, we look at the impact on both state and local funding.

Of course, because spending on schools also includes a small amount of federal aid, total funding is not simply the sum of state and local funding. Federal funds, which make up about 10 percent of total education funding, have until recently been limited to specific programs, such as the National School Lunch Program and special education. Thus, we would not expect a state court decision  to influence federal funding, an assumption that is borne out in the data.

Gauging the Effects

Our basic strategy was to compare changes in funding levels in districts where the state’s school-finance system has been ruled unconstitutional to funding changes in comparable districts in states where an SFJ has not been issued. We make these comparisons with groups of districts that had Democratic, Republican, or split-party control of the state government at the time the SFJ was issued. We allow for a one-year delay for the judgment to take effect because we assume that any changes in policy made as a result of the decision will be reflected in the next year’s budget, at the earliest.

Because most school-finance lawsuits are aimed at increasing funding for poor districts specifically, we designed our analysis to measure how the effects of SFJs, and of the party in control of the state government at the time of the decision, might be different for school districts with high rates of students in poverty and for districts where the students are better-off financially. To look for these differences, we divided each state’s districts into four quartiles based on the proportion of students living in poverty and allowed for the possibility that the effect of an SFJ, and of one under Democratic, Republican, or divided government, could be different in each quartile.

To isolate the effects of an SFJ on districts within each poverty quartile, we focus on changes in spending over time within specific school districts after taking into account changes from year to year in average education spending across all of the nation’s school districts. Thus we effectively control for unmeasured attributes of each school district that are constant over time and for national trends that affect all districts, such as economic conditions or changes in federal education policy that could have an impact on funding even in the absence of an SFJ. We adjust for inflation by converting all per-pupil funding figures to constant 2007 dollars.

Of course, there are other factors that likely influence changes over time in the level of per-pupil funding in a school district, including characteristics that change over time and influence either their receipt of state funding or the propensity of school districts to raise their own local revenue. We account for the variation in funding that should be directly attributed to the percentage of the student population living in poverty, independent of any change produced by an SFJ. We also include the total number of students in the district, to allow for the possibility that large districts operate differently from small districts. And we estimate the impact on per-pupil expenditure of the proportion of students in a district with Individualized Education Plans (IEPs), as students with IEPs generally have special needs that result in higher spending. Finally, we include the proportion of the student population that is African American and the proportion Hispanic. Although we have no reason to believe that these two variables directly cause changes in education funding, they may be correlated with other relevant factors, such as property values or population growth, for which we lack direct information.

In addition to district-specific characteristics, we take into account state-level characteristics that could influence state funding of education. In particular, we control for the fraction of the state’s population over age 65 to account for the possibility that the elderly oppose increases in school spending. We also control for the fraction of the population that is of school age, which captures aggregate demand for educational services. The final control variable in our analysis is per-capita income in the state, as the demand for government services may increase with income.

Annual district-level financial and demographic information comes from the Common Core of Data (CCD), available from the National Center for Education Statistics (NCES). For years in which CCD data are not available (1988–1992 and 2005), we use data from U.S. Census Bureau Elementary-Secondary Education Finance Survey (F-33). Our analysis considers only local school districts and parts of local supervisory unions with at least 100 students, as identified by the CCD. We exclude Hawaii and Washington, D.C., because each has only one school district.

Additional district demographic information, including the proportion of the population aged 5 to 17 and the proportion of school-aged children living in poverty, comes from the U.S. Census Small Area Income and Poverty Estimates for most years. For 1989 and 2005, the district demographic information comes from the School District Demographics System. Because district poverty information is not available for every year, we use the poverty estimates from the closest available survey year. For example, the district poverty estimates for 1996, 1997, and 1998 all use the data from 1997.

Partisan Patterns

A new and very clear picture about the impact of politics on SFJs emerged from our findings. The school-finance reforms implemented by Democratic state governments have substantially different effects on district funding than reforms produced by Republican or divided governments. When a Democratic state government implements an SFJ-induced reform, all districts, poor and non-poor alike, see increases in total funding. Under Republican and divided governments, districts with different levels of poverty fare quite differently.

Figure 1 represents our findings graphically. Each bar in the graph represents the effect of an SFJ, that is, the within-district change in spending after the decision, for each category of partisan control and district poverty. We present separate estimates for the change in total funding, in funding from state sources, and in funding from local revenues.

In Democrat-led reforms, our estimates show, districts in every poverty quartile see a shift from local to state funding after an SFJ. Local funding decreases, while state funding increases. This pattern of centralization of school funding is consistent with evidence from earlier studies, which also shows that localities partially offset state efforts to increase overall education spending after SFJs.

The upshot is a net increase in total funding ranging from roughly $750 to $1,000 per pupil—a sizable impact, given that total per-pupil funding in our sample is a little over $9,750 on average. While a few of the differences between quartiles are statistically significant, they are substantively small relative to the overall level of the funding increases. Indeed, if anything, the results indicate that the most affluent districts fare better than the poorest districts, in terms of total funding, when Democrats are in power, although this difference is not statistically significant. We should note here that high poverty does not necessarily imply low spending (in many states, high-poverty districts have the highest spending levels), so our findings here do not bear directly on spending inequality.

School-finance rulings handed down to divided governments produce decidedly different results. State funding increases across the board, but the changes in state funding differ markedly across the levels of district poverty: The poorer the district, the larger the increase in state funding. But, as in states with Democrat-led reforms, SFJs are not unmitigated wins for school-district budgets. All four quartiles see sizable reductions in funding from local sources. These reductions are large enough that the poorest quartile is the only one to see positive net changes in total funding. Overall, divided government reforms appear to represent a more or less straightforward redistribution of funding toward the poorest districts. The net effects on state education funding appear to be budget-neutral, as we estimate that there is little change in total education funding after an SFJ under a divided government. That said, the net increase of roughly $175 per student in total funding for poor districts is fairly modest when compared to total per-pupil funding.

Republican-controlled reforms present yet a third pattern of funding changes. Under Republican governments, funding shifts from local to state only for the poorest districts. Districts in the most affluent quartile face cuts in state funding, but they are able to more than compensate for these reductions by increasing local funding. In other words, Republican-led reforms involve centralization of funding for the poorest districts and decentralization of funding for the richest districts. The middle two quartiles are essentially unaffected. On net, both the poorest and the richest districts see increases in total funding, the former courtesy of state aid and the latter financed from their own tax base. Indeed, the richest half of districts in Republican states are the only group under any partisan regime to experience an increase in local funding following an SFJ.

Alternative Explanations

A lingering concern with our results may be that party control of the state government is related to the decision to file a school-reform lawsuit. Finance reform advocates may time the filing of their lawsuits to take advantage of what they view as particularly favorable political conditions. Another possibility is that advocates might resort to litigation only when the legislative and political process fails to provide reform. Either of these possibilities means that SFJs might have effects that appear to be associated with party control but are not actually caused by the response of the party in power.

We answer by first noting that because nearly all states—45 of 50—were subject to at least one education-finance lawsuit, the central issue is not whether a state would face a suit but when. Beyond that point, we believe that this is not a major concern for three reasons: 1) the amount of time between lawsuit filing and the court decision is often long and always unpredictable; 2) the party in control often changes between the lawsuit filing and decision; and 3) lawsuits do not appear to be precipitated by changes in political regime. Among the 23 cases included in our study, the length of time from the initial filing through the final appellate court decision ranged from less than a year to nine years. On average, the process took four years. Due to the length of time the suits take and the variability of the speed of the adjudication process, advocates could not effectively time their lawsuits to specific political circumstances. In almost half of the cases (11 out of 23), the party in control changed between the time of filing and the time of decision. Further, school-finance lawsuits do not appear to be triggered by changes in party control. On average, the party in control in the state was stable for six years prior to the filing of a case. In only three cases did the party in control change in the year of the lawsuit filing, and for each of those three cases, the party in control changed again before the lawsuit was decided.

Conclusion

Which partisan arrangement leads to the best results for poor districts after a school-finance judgment? That question requires stepping into the debate about the relationship between student outcomes and school funding and goes beyond the evidence we present here. What our study does show is one of the many possible ways that politics can influence the implementation of court-ordered school-finance reform. Clearly, reforms implemented by Democrats produce the largest net increases in funding for all students. However, by delivering roughly equivalent funding increases to districts at all income levels, Democrat-led reforms do not target new resources to districts serving poor students. Reforms implemented by divided or Republican governments deliver concentrated benefits to districts serving poor students. In these instances, however, the actual flow of new dollars into poor districts is more meager than when Democrats are in control.

Christopher Berry is assistant professor at the Harris School of Public Policy at the University of Chicago. Charles Wysong is a student at Stanford Law School.

It is unclear whether and how public funds should be mobilized to close those gaps. Some advocate expanding existing programs that target disadvantaged children on the grounds that limited public resources should be directed toward the families and children most in need. Others consider the perennial underfunding of targeted programs like Head Start as evidence of a lack of political support for this approach, and argue that providing universal access is needed to ensure adequate public funding over the long run. In other words, any new funding for preschool education must benefit middle-class children if it is to gain their parents’ political backing. Or so it is argued.

Existing research provides little insight into the relative merits of universal programs and those targeted to specific groups. While there have been several recent studies of the short-term effects of universal preschool programs in the U.S., there is no evidence to date on long-term consequences. Some studies suggest that Head Start has lasting effects in reducing criminal behavior and increasing educational attainment, but this program is much more intensive than any universal program is likely to be and serves a very disadvantaged population.

In the absence of direct evidence on the types of preschool programs now under consideration, this study attempts to shed light on the likely consequences of a new universal program by estimating the impact of earlier state interventions to introduce kindergarten into public schools. In the 1960s and 1970s, many states, particularly in the southern and western parts of the country, for the first time began offering grants to school districts operating kindergarten programs. Districts were quick to respond. The average state experienced a 30 percentage point increase in its kindergarten enrollment rate within two years after an initiative, contributing to dramatic increases in kindergarten enrollment (see Figure 1). These interventions present an unusual opportunity to study the long-term effects of large state investments in universal preschool education.

My results indicate that state funding of universal kindergarten had no discernible impact on many of the long-term outcomes desired by policymakers, including grade retention, public assistance receipt, employment, and earnings. White children were 2.5 percent less likely to be high school dropouts and 22 percent less likely to be incarcerated or otherwise institutionalized as adults following state funding initiatives, but no other effects could be discerned. Also, I find no positive effects for African Americans, despite comparable increases in their enrollment in public kindergartens after implementation of the initiatives. These findings suggest that even large investments in universal early-childhood education programs do not necessarily yield clear benefits, especially for more disadvantaged students.

Kindergarten in the U.S.

Many state governments have only recently introduced state grants for school districts that operate kindergarten programs. Kindergartens began outside of the public school system, funded largely through philanthropic organizations or private tuition. Over the first half of the 20th century, kindergartens slowly became incorporated into urban schools, at the same time gaining partial funding through local taxes. As late as the mid-1960s, however, such programs continued to rely heavily on local resources, as only 26 states and the District of Columbia helped fund kindergarten costs. There were remarkable changes over the next decade, however: Between 1966 and 1975, 19 states began funding kindergarten for the first time. The majority of these states were in the South, but the West was also well represented. By the late 1970s, only two states—Mississippi and North Dakota—did not fund kindergarten programs (see Figure 2).

Initially, states channeled their funding to districts in one of two ways. Some states revised existing funding formulas to include financial support for kindergartens on a basis equivalent to support for all other grades in a state’s public school system. Other states appropriated separate monies for kindergarten, an approach that made kindergarten funding more vulnerable to budget cuts. Eventually, however, all states made kindergarten a part of the basic state school program.

The initiatives were introduced during a period of rising labor-force participation among women with young children, so kindergarten’s popularity may have been due to the fact that it provided families with subsidized child care. The stated purpose, however, was to improve children’s educational outcomes. In particular, it was claimed that kindergarten would provide the preparation children need to succeed in the elementary school years. Greater success in school would, in turn, reduce state spending not only on special education and “re-education” of children who failed, but also on public assistance and incarceration over the long term.

Whether state funding of kindergarten was capable of achieving these goals is open to question. Kindergartens have historically maintained a curriculum focused more on children’s social development and less on academic training. While a focus on socialization does not preclude long-term effects, kindergarten programs lacked features of some targeted interventions—such as parental involvement and health services—that may be critical to their success. State-funded kindergartens for five-year-olds may also have reduced enrollment in private kindergartens and in education programs funded through Head Start and Title I. My study seeks to shed light on these important policy questions of relevance to the current conversations concerning early childhood education.

Method and Data

To find out the long-term impacts of the introduction of universal kindergarten, I take advantage of the staggered introduction of state funding for kindergarten from the 1960s forward, combined with the fact that children generally attend kindergarten at age five. More specifically, I calculate the average difference in outcomes between individuals who were age five before the introduction of kindergarten funding and children born in the same state who were five years old after the initiative was introduced. I further adjust these comparisons to take into account the fact that kindergarten enrollment was increasing gradually in many states prior to the adoption of state funding. The remaining differences should reflect the long-run effects of the typical state-funded kindergarten program.

I restrict my analysis to the 24 states that introduced state funding for universal kindergarten after 1960 because the data needed for the analysis are not available for earlier years. I also limit attention to the 1954 to 1978 birth cohorts because they span the period over which most of these funding initiatives were passed, and doing so provides me with data both before and after the introduction of these initiatives necessary to estimate the effects of kindergarten funding on long-term outcomes.

I combine data from several sources. I measure the kindergarten enrollment rate with the state kindergarten-to-first-grade enrollment ratio, calculated from the federal Common Core of Data and earlier published data. Data for the analysis of the initiatives’ long-term effects were drawn from Public Use Microdata Samples (PUMS) of the Decennial Census. In particular, I examine 1) whether a child was below grade for age while still of school age (a proxy for grade retention); 2) three indicators of adult educational attainment (high school dropout, high school degree only, and some college); 3) adult wage and salary earnings and indicators of employment and receipt of public assistance income; and 4) an indicator for residence in institutionalized group quarters, a widely used proxy for incarceration.

Limited Impact

I begin the empirical analysis by examining how the funding initiatives affected kindergarten enrollment. The results confirm that funding had a large, immediate impact on kindergarten participation. In the first year in which funding was available, the kindergarten enrollment rate in the typical state was about 15 percentage points higher than would have been the case in the absence of state funding. Two years out, it was 33 percentage points higher, and the lion’s share of gains in kindergarten enrollment from the funding initiative had been achieved. Anecdotal evidence suggests that the take-up of kindergarten was not completely immediate because of shortages of classrooms and teachers rather than because of a gradual increase in local demand. On net, the public school kindergarten enrollment rate of children turning five after an initiative was about 30 percentage points higher than it would otherwise have been.

I next investigate whether these developments were matched by changes in child well-being. Because grade retention and educational attainment were arguably the prime targets of policymakers, I first consider the effects of kindergarten funding on those indicators. Whites had more education as adults as a result of the initiatives, but the effect was quite small: only a 2.5 percent reduction in the dropout rate (see Figure 3). Because the dropout rate among whites prior to kindergarten funding was roughly 15 percent, this reduction amounts to less than half of 1 percentage point, which is a small effect even if we take into account that kindergarten enrollment rose 30 percentage points (not a full 100 percentage points) as a result of the initiatives. College attendance also increased among whites, but by an even smaller amount. The analogous estimates for African Americans suggest that affected children attained lower levels of education. While not statistically significant, the estimates are sufficiently precise to rule out the possibility that African Americans experienced even the small positive gains in educational attainment evident among whites. The apparent gains in educational attainment for whites occurred without significant reductions in grade repetition, either in absolute terms or relative to African Americans.

I then turn to an investigation of the impacts on earnings, employment, public assistance receipt, and the proxy for incarceration described earlier. I again find little evidence that kindergarten funding affected these outcomes. The most notable exception is that whites of kindergarten age after passage of a funding initiative were less likely to reside in prisons or institutionalized group quarters as adults. The effect is relatively large, at 22 percent. Once again, however, no such effects were observed for African Americans. Moreover, I find no evidence of an impact of state kindergarten funding on earnings for individuals of either race. The estimated effects on earnings are imprecise, however, and leave open the possibility that kindergarten attendance had effects on earnings comparable to any other year of education for African Americans and whites alike. In general, the earnings estimates should be viewed with caution, as they could be distorted by the fact that the sample includes some individuals who are young and could still be enrolled in school.

These results remain essentially unchanged when estimated using to a series of alternative approaches, including adding controls for state demographic and labor market conditions. I also perform the analysis separately by gender, which reveals that the effect of kindergarten funding on institutionalization for whites is primarily due to its effect on men, for whom the institutionalization rate is much higher. The magnitude of the effect for white men is similar to that observed for whites overall (a reduction of 23 percent). Among African Americans, there are no effects on institutionalization rates for men or women. The gender-specific results also reveal that kindergarten funding was associated with significantly lower earnings for African American women. To the extent that kindergarten funding displaced African American enrollment in more intensive early education, a possibility that I explore below, these findings would be consistent with recent findings that girls are more responsive to intensive preschool interventions.

Why Did African Americans Not Benefit?

My main results imply that there were some positive impacts of state subsidization of kindergarten, particularly on incarceration rates. What is potentially unexpected, however, is that the funding initiatives appear to have had positive effects only for whites. What might explain these findings? I explore three broad hypotheses for why African Americans might not have benefited as much as whites from the funding initiatives: 1) kindergarten funding disproportionately drew African Americans out of higher-quality education settings; 2) instead of raising additional revenue to fund local kindergarten programs fully, school districts offered lower-quality kindergarten programs to African Americans or moved funds from existing school programs from which African Americans may have disproportionately benefited; and 3) African Americans were more adversely affected by any subsequent “upgrading” of school curricula as more students entered elementary grades having attended kindergarten. The first of these hypotheses receives the most support in the available data.

Data from the Panel Study of Income Dynamics suggest that the introduction of state funding for kindergarten prompted a reduction in Head Start participation among African Americans. The existence of kindergarten funding among all states in a region (relative to none) was associated with a statistically significant 25-percentage-point-reduction in the likelihood that an African American child attended Head Start at age five. Given an enrollment rate of 26 percentage points across the observed cohorts, this estimate implies that state funding for kindergartens essentially eliminated enrollment of African American five-year-olds in Head Start (see Figure 4). By comparison, enrollment of whites in Head Start at age five was much lower (2 percent), and the change in enrollment after the average funding initiative close to zero.

Together with historical accounts of the importance of Head Start in providing education for five-year-olds in the absence of state-funded kindergartens, these estimates strengthen support for the hypothesis that state funding for kindergartens decreased enrollment of African American five-year-olds in federally funded early education for the poor. It is difficult to gauge the extent to which the movement of African American five-year-olds from Head Start to kindergarten might have offset positive impacts of kindergarten attendance elsewhere in the African American population. However, a back-of-the-envelope calculation suggests that the reduction in Head Start attendance among African Americans may account for at least 16 percent of the 1-percentage-point increase in the African American-white gap in high school dropout rates after the initiatives were passed. Head Start has also been found to reduce criminal behavior among African-American males.

I uncover no support for the hypothesis that school districts failing to supplement the state grants placed African American students in lower-quality programs, either in kindergarten or in later grades. I also detect no evidence that the establishment of kindergarten programs as a result of the funding initiatives prompted an increase in academic expectations of students in the early grades, which would have adversely affected children with low levels of achievement. Because the data available to test these alternative hypotheses are not ideal, however, these conclusions must be viewed with caution.

Back to the Future

Although there is great interest among policymakers in extending free early education to disadvantaged children, evidence to date on long-run effects of preschool has been limited to experimental evaluations of model preschools and nonexperimental studies of Head Start. This study has attempted to expand this literature by measuring the long-term effects of a historical episode of public investment in universal early education—the introduction of state funding for public school kindergarten in the 1960s and 1970s. I find evidence that state funding of universal kindergarten lowered high-school dropout and institutionalization rates among whites, but not among African Americans, and detect no impact of state funding for children of either race on grade retention, public assistance receipt, employment or earnings. Why the positive effects for whites occur for dropout and incarceration only is not entirely clear and should be grounds for future research.

These findings complement those of existing research on the long-term effects of targeted programs. First, they suggest that, in the absence of higher-quality alternatives, participation in a low-intensity preschool program may have some limited positive long-term effects. In other words, even a weak program may be better than no program at all, as can be seen in the results for whites. Second, when alternatives already exist for many disadvantaged children, universal programs may not yield additional benefits for that group.

Though there are clear limits to the generalizability of these findings, they do provide some tentative lessons for policymakers. On one hand, the higher rates of preschool participation among children today suggest that any positive long-term effects of extending universal public schooling to four-year-olds may be even smaller than those estimated here for kindergarten. On the other hand, the universal preschool programs being proposed today have a more academic orientation than kindergarten has had, and may therefore have larger impacts on long-term well-being despite significantly “crowding-out” enrollment in other programs. The truth will only be discovered in the years to come.

Elizabeth U. Cascio is assistant professor of economics at Dartmouth College.

Charter schools have become a popular alternative to traditional public schools, with some 5,000 schools now serving more than 1.5 million students, and they have received considerable attention among researchers as a result.

Most studies focus on the effects of charter attendance on short-term student achievement (test scores), using either data sets that follow students over time (see “Results from the Tar Heel State,” research, Fall 2005) or random assignment via school admission lotteries (see “New York City Charter Schools,” research, Summer 2008) to control for differences between students in charter and traditional public schools. Beyond measuring achievement effects, however, there has been only limited analysis of the impacts of charters on the students who attend them. Even less research has been conducted on the effects of charter high schools specifically, though a large portion of all charter schools in the U.S. serve some or all of the high school grades.

Developing a high school model suited to the 21st-century student has been the Holy Grail of education reform in recent years, absorbing governors, task forces, and vast sums spent on small schools, university-based schools, and concept schools (see “High School 2.0,” features). With roughly 30 percent of American students dropping out before receiving a diploma—a rate that has been stable for several decades—assessing existing alternatives to the traditional high school is an urgent task.

In this study we use data from Chicago and Florida to estimate the effects of attending a charter high school on the likelihood that a student will complete high school and attend college. Given the impact of educational attainment on a variety of economic and social outcomes, a positive result could have significant implications for the value of school-choice programs that include charter high schools. We find evidence that charter high schools in both locations have substantial positive effects on both high school completion and college attendance. Controlling for key student characteristics (including demographics, prior test scores, and the prior choice to enroll in a charter middle school), students who attend a charter high school are 7 to 15 percentage points more likely to earn a standard diploma than students who attend a traditional public high school. Similarly, those attending a charter high school are 8 to 10 percentage points more likely to attend college (see Figure 1). Results using an alternative method designed to address concerns about unmeasured differences between students attending charter and traditional public high schools suggest even larger positive effects. Our main results are comparable to those of some studies which find that attending a Catholic high school boosts the likelihood of high school graduation and college attendance by 10 to 18 percentage points.

Methods

Determining the influence of charter school attendance on educational attainment is difficult because students who choose to attend charter high schools may be different from students who choose to attend traditional public high schools in ways that are not readily observable. The fact that the charter students and their parents actively sought out an alternative to traditional public schools suggests the students may be more motivated or their parents more involved in their child’s education than is the case for students attending traditional public schools. Since these traits are not easily measured, the estimated impact of charter high schools on educational attainment could be biased.

Our main analysis uses two methods to address students’ self selection into charter schools. First, we control for any observable differences between charter and non-charter high school students prior to high school entry. These include factors such as race/ethnicity, gender, disability status, and family income. The most important characteristic included among our statistical controls is 8th-grade test score, which aims to capture differences in student ability and students’ educational experiences prior to high school.

Second, we limit our analysis to students who attended a charter school in 8th grade, just prior to beginning high school. That is, we compare high school and postsecondary outcomes for 8th-grade charter students who entered charter high schools (the treatment group) with outcomes for 8th-grade charter students who entered conventional public high schools (the comparison group). If there are unmeasured student or family characteristics that lead to the selection of charter schools in general, these unmeasured characteristics should be relatively constant among students and families who choose charter middle schools. Unlike other nonexperimental studies of charter school impacts, our study therefore addresses student self-selection into charter schools directly by ensuring that the comparison students as well as the treatment students were once charter choosers.

Charter school 8th graders who went on to attend a charter high school differed from their peers who subsequently attended a traditional public high school in several respects, particularly in Florida, which suggests the importance of taking such differences into account when assessing the effects of charter attendance (see Figure 2). However, there may still be unmeasured differences that explain why one charter 8th grader attends a charter high school while another charter 8th grader attends a traditional public high school. For this reason, we estimate charter school effects by comparing students who are more likely to attend a charter school because they live closer to one to those less likely to attend a charter school because it is less convenient. For many charter middle-school students, attending a charter high school may be infeasible due to the lack of a charter high school within a reasonable distance. Such students make different choices not because of unmeasured characteristics, but because of a factor out of their control: the distance from home to the nearest charter school.

Data

The data required to analyze the impact of charter high schools on educational attainment are substantial. One must have data on school type (charter or public) and test scores of individual students prior to high school, individual-level high school attendance records and exit information, and college attendance after high school. Finally, the jurisdiction studied must have a sufficient enrollment of students in charter high schools to provide reliable results. The areas we analyze, the state of Florida and the city of Chicago, are two of just a handful of places where all of the necessary data elements are currently in place.

The Florida data, which cover the four cohorts of 8th-grade students from the school years 1997–98 to 2000–01, come from a variety of sources. The primary source for student-level information is the Florida Department of Education’s K-20 Education Data Warehouse (K-20 EDW), an integrated longitudinal database covering all public school students in the state of Florida. The K-20 EDW includes detailed enrollment, demographic, and program participation information for each student, as well as reading and math achievement test scores.

As the name implies, the K-20 EDW includes student records for both K–12 public school students and students enrolled in community colleges or four-year public universities in Florida. The K-20 EDW also contains information that allows us to follow students who attend private institutions of higher education within Florida. Data from the National Student Clearinghouse, a national database that includes enrollment data on 3,300 colleges from throughout the United States, is used to track college attendance outside the state of Florida. Any individual who does not show up as enrolled in a two- or four-year college or university is classified as a non-attendee.

High school graduation is measured using withdrawal information and student award data from the K-20 EDW. Only students who receive a standard high school diploma are considered to be high school graduates. Students earning a GED or special education diploma are counted as not graduating. Similarly, students who withdrew with no intention of returning or left for other reasons, such as nonattendance, court action, joining the military, marriage, pregnancy, and medical problems, but did not later graduate, are counted as not graduating.

The Chicago data, which cover the five cohorts of students who were in 8th grade during the school years 1997–98 to 2001–02, were obtained from the Chicago Public Schools. The data include 8th-grade math and reading test scores and information on student gender, race/ethnicity, bilingual status, free or reduced-price lunch status, and special education status. This data set is also linked to the National Student Clearinghouse. High school graduation is determined by withdrawal information from the Chicago Public Schools data. As in Florida, only students who receive a standard high school diploma are considered to be high school graduates.

Results

The raw data on our study population of students who were in charter schools in 8th grade reveal substantial differences in educational attainment between attendees of charter high schools and those of traditional public high schools. In Florida, 57 percent of students who went from a charter school in 8th grade to a traditional public school in 9th grade received a standard high school diploma within four years, compared to 77 percent of charter 8th graders who attended a charter high school. In Chicago, the corresponding high school graduation rates were 68 and 75 percent. Similar differences are found for college attendance. In Florida, among the study population of charter 8th graders, 57 percent of students attending a charter school in 9th grade went to either a two- or four-year college within five years of starting high school, whereas among students who started high school in a traditional public school the college attendance rate was only 40 percent. In Chicago, the gap in college attendance is smaller but still sizable: among the study population of charter 8th graders, 49 percent of students at charter high schools attended college, compared to 38 percent of students at traditional public high schools.

Controlling for student demographics, 8th-grade test scores, English language skills, special education program participation, free or reduced-price lunch status (a measure of family income), and mobility during middle school does not alter the basic patterns of graduation and college attendance seen in the descriptive comparisons. The estimated impact of attending a charter high school on the probability of obtaining a high school diploma is positive in both Florida and Chicago. In Chicago, students who attended a charter high school were 7 percentage points more likely to earn a regular high school diploma than their counterparts with similar characteristics who attended a traditional public high school. The graduation differential for Florida charter schools was even larger, at 15 percentage points. The findings for college attendance are remarkably similar in Florida and Chicago. Among the study population of charter 8th graders, students who attended a charter high school in 9th grade are 8 to 10 percentage points more likely to attend college than similar students who attended a traditional public high school (see Figure 1).

As discussed above, there remains the possibility that unobserved changes occur between 8th and 9th grade that influence both high school choice and subsequent educational attainment. For example, dissatisfaction with performance in a charter middle school that is not captured by test scores (such as discipline issues or a poor fit between the student’s interests or ability and the curriculum being offered) could lead parents to choose to send their child to a traditional public high school. When we correct for this potential bias by examining students who attended charter or traditional public school based on proximity, we continue to find highly significant positive effects of attending a charter high school on both receipt of a high school diploma and college enrollment. The magnitude of the effects is large, roughly double the size of our main results.

This pattern suggests that, among students enrolled in charter schools as 8th graders, it is those who are less likely to graduate who are choosing to attend charter high schools. We can only speculate as to why this is so. It is possible that parents whose children are at risk of dropping out are more likely to choose charter high schools in a belief that the traditional public school environment would make it more likely that their child leaves school early. Alternatively, although we control for free or reduced-price lunch eligibility, it may be the case that low-income families have a stronger preference for charter schools. If so, families with children in charter high schools would be less likely to be able to afford to send their children to college.

Possible Mechanisms

The analyses reported above cannot explain how or why charter high schools appear to produce positive effects on their students’ educational attainment. Our study lacks data on operations and instruction in the charter schools, so we have little opportunity to explore the mechanisms contributing to their success. Nonetheless, we have a few pieces of information that permit exploratory analyses of factors that might play a role.

First, it is worth considering that charter high schools may raise rates of high school graduation and college enrollment directly, or indirectly through improved academic achievement. We attempt to distinguish between these explanations by controlling in the analysis for math and reading achievement as measured in the 10th grade. Controlling for 10th-grade test scores explains about half the graduation differential for charter high schools in Florida but less than 20 percent of the difference in Chicago. And it has an even smaller effect on the results for college enrollment, reducing the estimated effect of charter school attendance by only about 10 percent in both locations. These patterns suggest that the positive effects of charter school attendance on educational attainment are not due solely to measured differences in the achievement of students in charter and traditional public high schools. This result is similar to those found in some studies of Catholic high schools, which suggest larger benefits for attainment than for test scores.

Second, given that charter high schools tend to be much smaller than traditional public high schools, charter school effects might simply be attributable to their smaller size. In order to assess this possibility, we ran the analyses for high school graduation and college attendance again with an additional control for the total number of students attending the school. The results are comparable to those reported above, indicating that the estimated effects of charter high schools are not due to differences in school size.

Third, we consider the possibility that the charters’ success might be related to grade configurations that often differ from those of traditional public schools. In the traditional public school sector in both Chicago and Florida, high schools are almost always separate from middle schools. This is not the case for charter schools. In 2001–02, about 22 percent of charter schools in Florida offering middle-school grades also offered some or all high-school grades. As a result, about 30 percent of Florida charter 8th-grade students attended schools that also offered at least some high-school grades. In Chicago, 40 percent of charter middle schools offered both middle- and high-school grades, and nearly half of the 8th-grade charter students could attend at least some high-school grades without changing schools. This raises the possibility that the measured effects of attending a charter high school on educational attainment could simply reflect advantages of grouping middle and high school grades together, thereby creating greater continuity for students and eliminating the disruption often associated with changing schools.

In order to examine whether charter-school effects might be attributable to eliminating the transition between middle and high school, we restricted the Florida analysis to those students whose 8th-grade charter school did not offer 9th grade and ran our analyses again. For high school graduation, restricting the sample produces estimates that are nearly identical to the original estimates from our main method. Using the restricted sample and our alternative method, the estimates are about 30 percent smaller than when the full sample is employed, but still large. Meanwhile, estimates of the effect of attending a charter high school on college enrollment are even larger using the restricted sample than with the original sample that includes schools offering both 8th and 9th grade. In Florida, grade configuration is not a primary driver of the estimated positive effects of charter high schools on attainment. In Chicago, however, we could not run similar analyses because grade configuration is too strongly correlated with charter status; we therefore cannot rule out the possibility that positive results in Chicago could be partly attributable to eliminating the transition from middle school to high school.

Finally, we examined an interpretive concern arising from the fact that some charter schools in Florida are former traditional public schools that converted to charter status. If conversion schools were better-than-average traditional public schools to begin with, they may be distorting the estimated impact of charters on educational attainment. We calculated separate effects for Florida conversion and non-conversion (“de novo”) charters in Florida. (In Chicago, virtually all of the charter high schools in our sample were de novo charters). We found that although Florida’s conversion charters have significantly greater effects on high school graduation than do de novo charters, the impact of non-conversion charters is still sizable (nearly equal to the estimate in Chicago). For college attendance, the estimated positive impacts of Florida’s de novo charters are statistically indistinguishable from the estimated positive impacts of Florida’s conversion charters.

Conclusions

Although a number of recent studies analyze the relationship between charter school attendance and student achievement, this is the first analysis of the impacts of charter school attendance on educational attainment. We find that charter schools are associated with an increased likelihood of successful high-school completion and an increased likelihood of enrollment at a two- or four-year college in two disparate jurisdictions, Florida and Chicago. The reasons for these large charter-school effects are not clear. There is certainly room for future work to explore how differences in curricula, expectations, peer characteristics, and other factors may cause charter schools to diminish the high-school dropout rate and ease the transition to postsecondary schooling.

Our findings are consistent with some research on the efficacy of Catholic schools, which finds substantial positive effects of attending a Catholic high school on educational attainment. While just a first step, the results presented here and in the Catholic-school literature suggest that school-choice programs that include alternatives to traditional public high schools may reduce high-school dropout rates and promote college attendance.

Kevin Booker is researcher at Mathematica Policy Research, Inc. Tim R. Sass is professor of economics at Florida State University. Brian Gill is senior social scientist at Mathematica Policy Research, Inc. Ron Zimmer is associate professor at Michigan State University. This article is adapted from research reported in Charter Schools in Eight States (RAND Corporation, 2009).

Researchers have recently begun to learn more about the effects of time spent on learning from natural experiments around the country. This new body of evidence, to which we have separately contributed, suggests that extending time in school would in fact likely raise student achievement. Below we review past research on this issue and then describe the new evidence and the additional insights it provides into the wisdom of increasing instructional time for American students.

We also discuss the importance of recognizing the role of instructional time, explicitly, in accountability systems. Whether or not policymakers change the length of the school year for the average American student, differences in instructional time can and do affect school performance as measured by No Child Left Behind. Ignoring this fact results in less-informative accountability systems and lost opportunities for improving learning outcomes.

Emerging Evidence

More than a century ago, William T. Harris in his 1894 Report of the Commissioner [of the U.S. Bureau of Education] lamented,

The boy of today must attend school 11.1 years in order to receive as much instruction, quantitatively, as the boy of fifty years ago received in 8 years…. It is scarcely necessary to look further than this for the explanation for the greater amount of work accomplished…in the German and French than in the American schools.

The National Education Commission on Time and Learning would echo his complaint one hundred years later. But the research summary issued by that same commission in 1994 included not one study on the impact of additional instruction on learning. Researchers at that time simply had little direct evidence to offer.

The general problem researchers confront here is that length of the school year is a choice variable. Because longer school years require greater resources, comparing a district with a long school year to one with a shorter year historically often amounted to comparing a rich school district to a poor one, thereby introducing many confounding factors. A further problem in the American context is that there is little recent variation in the length of school year. Nationwide, districts generally adhere to (and seldom exceed) a school calendar of 180 instructional days. And while there was some variation in the first half of the 20th century, other policies and practices changed simultaneously, making it difficult to uncover the separate effect of changes in instructional time.

Among the first researchers to try to identify the impact of variation in instructional time were economists studying the effect of schooling on labor market outcomes such as earnings. Robert Margo in 1994 found evidence suggesting that historical differences in school-year length accounted for a large fraction of differences in earnings between black workers and white workers.

Using differences in the length of the school year across countries, researchers Jong-Wha Lee and Robert Barro reported in 2001 that more time in school improves math and science test scores. Oddly, though, their results also suggested that it lowers reading scores. In 2007, Ozkan Eren and Daniel Millimet examined the limited variation that does exist across American states and found weak evidence that longer school years improve math and reading test scores.

Work we conducted separately in 2007 and 2008 provides much stronger evidence of effects on test scores from year-to-year changes in the length of the school year due to bad weather. In a nutshell, we compared how specific Maryland and Colorado schools fared on state assessments in years when there were frequent cancellations due to snowfall to the performance of the very same schools in relatively mild winters. Because the severity of winter weather is inarguably outside the control of schools, this research design addresses the concern that schools with longer school years differ from those with shorter years (see research design sidebar).

While our studies use data from different states and years, and employ somewhat different statistical methods, they yield very similar results on the value of additional instructional days for student performance. We estimate that an additional 10 days of instruction results in an increase in student performance on state math assessments of just under 0.2 standard deviations. To put that in perspective, the percentage of students passing math assessments falls by about one-third to one-half a percentage point for each day school is closed.

Other researchers have examined impacts of instructional time on learning outcomes in other states, with similar results. For example, University of Virginia researcher Sarah Hastedt has shown that closures that eliminated 10 school days reduced math and reading performance on the Virginia Standards of Learning exams by 0.2 standard deviations, the same magnitude we estimate for the neighboring state of Maryland. Economist David Sims of Brigham Young University in 2008 took advantage of a 2001 law change in Wisconsin that required all school districts in that state to start after September 1. Because some districts were affected while others were not, he was also able to provide unusually convincing evidence on the effect of changes in the number of instructional days. He found additional instruction days to be associated with increased scores in math for 4th-grade students, though not in reading.

Collectively, this emerging body of research suggests that expanding instructional time is as effective as other commonly discussed educational interventions intended to boost learning. Figure 1 compares the magnitude of the effect of instructional days on standardized math scores to estimates drawn from other high-quality studies of the impact of changing class size, teacher quality, and retaining students in grade. The effect of additional instructional days is quite similar to that of increasing teacher quality and reducing class size. The impact of grade retention is comparable, too, though that intervention is pertinent only for low-achieving students.

Although the evidence is mounting that expanding instructional time will result in real learning gains, evidence on the costs of extending the school year is much scarcer and involves a good deal of conjecture. Perhaps the best evidence comes from a recent study in Minnesota, which estimated that increasing the number of instructional days from 175 to 200 would cost close to $1,000 per student, in a state where the median per-pupil expenditure is about $9,000. The total annual cost was estimated at $750 million, an expense that proved politically and financially infeasible when the proposal was recently considered in that state. Comparing costs of expanding instructional days with the costs of other policy interventions will be an analytic and policy exercise of real importance if the call for expanded instructional time is to result in real change.

Complicating this analytic task are differences in costs that exist across schools and states. Utilities, transportation, and teacher summer-labor markets vary widely across geographic areas, and all affect the cost of extending the school year. So, while the benefits of extending the school year may exceed the costs in some states or school districts, they may not in others. A further complication is the possibility of diminishing returns to additional instructional time. Our research has studied the effect of additional instructional days prior to testing, typically after approximately 120 school days. The effect of extending instructional time into the summer is unknown. Also, our research has focused on the variation in instructional days prior to exams, or accountable days. The effect of adding days after exams could be quite different.

Costs of extending school years are as much political as economic. Teachers have come to expect time off in the summer and have been among the most vocal opponents of extending school years in several locations. Additional compensation could likely overcome this obstacle, but how much is an unresolved and difficult question.

Teachers are not the only ones who have grown accustomed to a summer lasting from June through August. Students and families have camps, vacations, and work schedules set up around summer vacation. “Save Our Summers” movements have for years decried the benefits of additional instructional days and proclaimed the benefits of summer vacation, and the movements have grown as states have considered extending the school year and individual school districts have moved up their start dates. Longer school years might reduce tourism and its accompanying tax revenue. These additional costs likely vary by state and district, but are clearly part of the analytic and political calculus.

Time and Accountability

As education policymakers consider lengthening the school year and face trade-offs and uncertainties, it is important to recognize that expanding instructional time offers both opportunities and hazards for another reform that is well established, the accountability movement. Educators, policymakers, parents, and economists are sure to agree that if students in one school learn content in half the time it takes comparable students at another school to learn the same content, the first school is doing a better job. How students would rank these schools is equally obvious. Yet state and federal accountability systems do not account for the time students actually spent in school when measuring gains, and so far have no way of determining how efficiently schools educate their students.

One implication of this oversight is that accountability systems are ignoring information relevant to understanding schools’ performance. Year-to-year improvements in the share of students performing well on state assessments can be accomplished by changes in school practices, or by increases in students’ exposure to school. Depending on the financial or political costs of extending school years, those with a stake in education might think differently about gains attributable to the quality of instruction provided and gains attributable to the quantity.

To see how the contributions of these inputs might be separated, consider data from Minnesota. Between 2002 and 2005, 3rd graders in that state exhibited substantial improvements in performance on math assessments, a fact clearly reflected by Minnesota’s accountability system. But during that period, there was substantial year-to-year variation in the number of instructional days students had prior to the test date. In Figure 2, we plot both the reported test scores for Minnesota 3rd graders (the solid line) and the number of days of instruction those students received (the bars). Useful, and readily calculated, is the time series of test scores, adjusting for differences in the number of instructional days (the dotted line).

Comparing the reported and adjusted scores is useful for at least two reasons. First, it illustrates the role of time as a component of test gains. Overall, scale scores increased by 0.4 standard deviations from 2001–02 to 2004–05. Of this increase, a large portion was attributable to expansion in instructional time prior to the test date. Adjusting for the effect of instructional days, we estimate that scores increased by roughly 0.25 standard deviations, nearly 40 percent less than the reported gains.

Second, the comparatively steady gain in adjusted scores over the period provides evidence of improvements in instructional quality, independent of changes in the amount of time students were in class. The fast year-to-year increases in the first and last periods result in large part from increases in the amount of time in school, while the negligible change in overall scores between 2003 and 2004 does not pick up real gains made despite a shortened school year. Adjusted scores pick up increases in learning gains attributable to how schools used instructional time, such as through changing personnel, curricula, or leadership. The point here is that time-adjusted scores provide information that is just as important as the overall reported scores for understanding school improvements. A robust accountability system would recognize that more instructional time can be used to meet goals, but that more time is neither a perfect substitute for, nor the same thing as, better use of time.

The Hazards of Ignoring Time

Failing to account for the role of time in student learning not only means missed opportunity, it also creates potential problems. First, it can allow districts to game accountability systems by rearranging school calendars so that students have more time in school prior to the exam, even as the overall length of the school year remains constant. Beginning in the 1990s, districts in a number of states began moving start dates earlier, with many starting just after the first of August. The question arose whether these changes might be linked to pressures on districts to improve performance on state assessments. David Sims showed that Wisconsin schools with low test scores in one year acted strategically by starting the next school year a bit earlier to raise scores. Evidence of gaming soon emerged in other states as well. Wisconsin passed its 2001 law requiring schools to begin after September 1 to prevent such gaming; similar laws were recently passed in Texas and Florida.

The motives driving earlier start dates could spill over into other instructional policies. Minnesota moved its testing regimen from February to April in the wake of accountability standards, while Colorado legislators have proposed moving their testing window from March into April, with advocates suggesting that the increased time for instruction would make meeting performance requirements under No Child Left Behind more feasible for struggling schools. While administering the test later in the year has potential benefits in measured performance, grading the tests over a shorter time frame costs more, estimated at some $3.9 million annually in Colorado. Schools thus sacrifice educational inputs (such as smaller classes or higher teacher salaries) to pay for the later test date.

A second hazard involves fairness to schools at risk of being sanctioned for poor performance: these schools can face longer odds if weather or other schedule disruptions limit school days. The impact of instructional time on learning means that one factor determining the ability of schools to meet performance goals is not under the control of administrators and teachers. We illustrate the effects of time on making adequate yearly progress (AYP) as defined by No Child Left Behind by comparing the performance of Maryland schools the law identified as underperforming to estimates of what the performance would have been had the schools been given a few more days for instruction.

We begin with data from all elementary schools in Maryland that did not make AYP in math and reading during the 2002–03 to 2004–05 school years. We adjust actual performance by the number of days lost in a given year multiplied by the marginal effect of an additional day on test performance as reported in Marcotte and Hemelt’s study of Maryland schools. This allows us to estimate what the proficiency rates in each subject would have been had those schools been open for all scheduled instructional days prior to the assessment. We then compare the predicted proficiency rate to the AYP threshold.

We summarize the results of this exercise in Figure 3. The light bars represent the number of schools failing to make AYP in math and reading in various years. The dark bars are the number of those schools that we predict would have failed to make AYP if the schools had been able to meet on all scheduled days. We make these estimates assuming that low-performing schools would have made average gains with each additional day of instruction.

The average number of days lost to unscheduled school closings varied substantially over the period, from more than 10 to fewer than four and a half. Many schools that did not make AYP likely would have had they not lost so many school days. For example, we estimate that 35 of the 56 elementary schools that did not make AYP in math in 2002–03 would have met the AYP criterion if they had been open during all scheduled school days. Even if these schools were only half as productive as the typical school, 24 of the 56 flagged schools would likely have made AYP if they had been open for all scheduled days.

There is, however, a way to reduce risks like these for schools and to limit incentives for administrators to move start or test dates at the same time: that is to recognize and report time as an input in education. A simple and transparent way to do this is for state report cards, which inform parents about school outcomes and summarize the information on AYP status, to include information about the number of instructional days at test date as well as the total number of instructional days for the year. This information is readily available and already monitored by schools, districts, and states. Local and state education authorities could use it when assessing performance, for example, in hearing an appeal from a school that failed to meet its AYP goals. Further, this information could be used to estimate test scores adjusted for instructional days, to be used alongside unadjusted changes in performance. Distinguishing between gains due to expanded instruction time and better use of that time can enrich accountability systems and provide more and better information to analysts and the public alike.

Looking Ahead

There can be no doubt that expanding the amount of time American students spend in school is an idea popular with many education policymakers and has long been so. What makes the present different is that we now have solid evidence that anticipated improvements in learning will materialize.

Practical obstacles to the extension of the school year include substantial expense and stakeholder attachment to the current school year and summer schedule. The benefits of additional instructional days could diminish as school years are lengthened. Further, it is unknown how teachers would use additional instructional days if they are provided after annual testing is already finished. Simply extending the year well after assessments are given might mean that students and teachers spend more days filling (or killing) time before the end of the year. This would make improvements in learning unlikely, and presumably make students unhappy for no good reason.

Though the issue has seen little movement in the past and faces real opposition going forward, the policy climate appears likely to be favorable once the fiscal challenges now facing public school systems recede. It is our hope that policymakers and administrators who try to take advantage of this window of opportunity don’t harm reforms that have succeeded in improving learning outcomes and don’t implement reforms in a manner that would fail to do the same. Advocates for extended school years have so far said virtually nothing about whether or how accountability systems should accommodate longer school years.

Across the country, a small number of schools and districts are modifying or extending the academic year. The Massachusetts 2020 initiative has provided resources for several dozen schools to increase the number of instructional days they offer from 180 to about 200. Other examples include low-performing schools that have lengthened their school day in an effort to improve, and the longer school days, weeks, and years in some charter schools. However, such initiatives remain rare, with no systemic change in the instructional time provided to American students. Our work confirms that increasing instructional time could have large positive effects on learning gains. Encouraging schools and districts to view the school calendar as a tool in the effort to improve learning outcomes should be encouraged in both word and policy.

Dave E. Marcotte is professor of public policy at the University of Maryland, Baltimore County. Benjamin Hansen is a research associate at IMPAQ International, LLC.

Debates over tracking reached their high point in the United States in the 1990s. An influential report published in 1998 by the Thomas B. Fordham Foundation argued that the available research did not support the contention that tracking doomed impoverished students to inferior schooling, nor did it support universal adoption of the practice. Over the last decade, patterns in grouping students have changed markedly in the U.S.; high school students are no longer placed in rigidly defined general-education or noncollege tracks but have the flexibility to move between course levels for different subjects. These changes may have assuaged some critics, but the broader debate over tracking remains unsettled.

The central challenge in measuring the effect of tracking on performance is that schools that track students may be different in many respects from schools that do not. For example, they may attract a different pool of students and possibly a different pool of teachers. The ideal situation to assess the impact of tracking on test scores of different groups of students would be one in which students were assigned to tracking or nontracking schools randomly, and the performance of students could be compared across school types.

We shed light on these issues using data from Kenya. In 2005, each of 140 primary schools in western Kenya received funds from the nongovernmental organization International Child Support (ICS) Africa to hire an extra teacher. One hundred twenty-one of these schools had a single 1st-grade class and used the new teacher to split the students into two classes. In 61 randomly selected schools, students were assigned to classes based on prior achievement as measured by test scores. In the remaining 60 schools, students were randomly assigned to one of the two classes, without regard to their prior academic performance.

The results showed that all students benefited from tracking, including those who started out with low, average, and high achievement. At the tracking schools, the test scores of students who started out in the middle of their class do not seem to be affected by which section (top or bottom) the students were later assigned to. In other words, any negative effects of being with lower-achieving peers were more than offset in tracked settings by the benefit of the teacher being able to better tailor instruction to students’ needs.

Primary Education in Kenya

The Kenyan education system includes eight years of primary school and four years of secondary school. Like many other developing countries, Kenya has recently made rapid progress toward the goal of universal primary education. After the elimination of school fees in 2003, primary school enrollment rose nearly 30 percent, from 5.9 million in 2002 to 7.6 million in 2005. This is typical of what is happening in sub-Saharan Africa overall, where the number of new entrants to primary school increased by more than 30 percent between 1999 and 2004.

This progress creates its own new challenges, however. Pupil-teacher ratios have grown dramatically, particularly in lower grades. In our sample of schools in western Kenya, the median 1st-grade class in 2005 (after the introduction of free primary education, but before the class-size-reduction program we study here) had 74 students and the average class size was 83. These classes are heterogeneous in a number of ways: Students differ vastly in age, school readiness, and support at home. Many of the new students are first-generation learners and have not attended preschools, which are neither free nor compulsory in Kenya. These challenges are not unique to Kenya; they confront many developing countries where school enrollment has risen sharply in recent years. Understanding the roles of tracking and peer effects in this type of environment is thus critically important.

Our results are most likely to be directly applicable to settings where classes are large, the student population is heterogeneous, and few additional resources are available to teachers. It is unclear whether similar results would be obtained in different contexts, such as developed countries, where smaller class sizes may allow more tailored instruction even without tracking, and extra resources, such as remedial education, computer-assisted learning, and special education programs, may already provide tools to help teachers deal with different types of students.

Design of the Experiment

This study takes advantage of a class-size-reduction program and evaluation that involved primary schools in Bungoma and Butere-Mumias in Western Province, Kenya. Of 210 primary schools in these districts, 140 schools were randomly selected to participate in the Extra-Teacher Program. With funding from the World Bank, ICS Africa provided each of the 140 selected schools with funds to hire an additional 1st-grade teacher on a contractual basis starting in May 2005, the beginning of the second term of that school year. Most of the schools (121) had only one 1st-grade class, which was split into two classes when the new teacher was hired. The 19 schools that already had two or more 1st-grade classes added another class.

It is important to note that the incentives facing the newly hired teachers differed from those facing civil-service teachers already working in program schools. The new teachers had clear incentives to work hard to increase their chances of having their short-term contracts renewed and of eventually being hired as civil-service teachers—a desirable outcome in a society where government jobs are highly valued. In contrast, the difficulty of firing civil-service teachers implies that they had weak extrinsic incentives and may be more sensitive to factors affecting their intrinsic motivation.

Average class size was reduced from 84 to 46 students in the 140 schools that received funds for a new teacher. The program continued for 18 months, which included the last two terms of 2005 and the entire 2006 school year, and the same cohort of students remained enrolled in the program.

From the 121 schools that had originally only one 1st-grade class, 60 schools were randomly selected to assign students to one of the two classes by chance. We call these schools the “nontracking schools.” In the remaining 61 schools (the “tracking schools”), the children were divided into two sections according to their scores on exams administered by the school during the first term of the 2005 school year. The 50 percent of the class with the lowest exam scores were assigned to one section (the “bottom class”) and the rest were assigned to the other (the “top class”).

After students were assigned to classes, the contract teacher and the civil-service teacher were also randomly assigned to classes. In the second year of the program, all children not repeating the grade remained assigned to the same group of peers and the same teacher.

Data

Our initial sample consists of approximately 10,000 students enrolled in 1st grade in March 2005 in one of the 121 primary schools participating in the study. The outcome of interest is student academic achievement, as measured by scores on a standardized math and language test first administered in all schools 18 months after the start of the program. Trained proctors administered the test, which was then graded blindly by data processors. In each school, 60 students (30 per class) were drawn from the initial sample to participate in the tests. If a class had more than 30 students, students were randomly sampled.

The test was designed by a cognitive psychologist to measure a range of skills students may master by the end of 2nd grade. One part of the test was written and the other part oral, administered one-on-one. Students answered math and literacy questions ranging from counting and identifying letters to subtracting three-digit numbers and reading and understanding sentences.

To limit attrition from the experiment, proctors were instructed to go to the homes of sampled students who had dropped out or were absent on the day of the test and to bring them to school for the test. It was not always possible to find the child, however, and the resulting attrition rate on the test was 18 percent. However, there was no difference between tracking and nontracking schools in overall attrition rates. In total, we have postintervention test-score data for 5,796 students.

In addition, each school received unannounced visits several times during the course of the study. During these visits enumerators checked, upon arrival, whether teachers were present in school and whether they were in class and teaching, and then took a roll call of the students.

To measure whether the effects of the program persisted, the children who had been sampled for the first postintervention test were tested again in November 2007, one year after the program ended. During the 2007 school year, these students were overwhelmingly enrolled in grades for which their school had a single class, so tracking was no longer an option. Most of these students had reached 3rd grade by that time, but those repeating an earlier grade were also tested. The attrition rate for this portion of the experiment was 22 percent. Neither the proportion nor the characteristics of children who could not be tested differed between the tracking and nontracking schools.

The Impact of Tracking

We estimate the impact of tracking on student achievement by comparing the postintervention (18 months after the experiment began) test scores of students in the tracking and nontracking schools. Taking the average of students’ scores on math and literacy exams, we find that students in tracking schools scored 0.14 standard deviations higher than students in nontracking schools overall. When we adjust the comparison to take into account minor differences in student characteristics across the two groups of schools, the effect increases to 0.18 standard deviations. There was no significant difference between the impact of the program on math and literacy scores when we examined the subjects separately.

How large were these effects? A typical student with a literacy score one standard deviation above that of the average student could correctly spell 5.5 of 10 words included on the exam, while the average student could spell only two. Similarly, students with a math score one standard deviation above the average were able to perform single-digit multiplications, whereas those at the mean could not. The average effect of tracking was roughly one-fifth the size of these performance differences.

These gains persisted beyond the duration of the program (see Figure 1). When the program ended, most students had reached 3rd grade, and all but five schools had only one 3rd-grade class. The remaining students had repeated and were in 2nd grade where, once again, most schools had only one large class, since after the program ended they did not have funds for additional teachers. Even so, the test scores of students in tracking schools remained 0.16 standard deviations higher than those of students in nontracking schools overall (and 0.18 standard deviations higher with control variables). The persistence of the benefits of tracking is striking, as many evaluations find that the test-score effects of successful interventions fade over time. It seems that tracking helped students master core skills in 1st and 2nd grade that in turn helped improve their learning later on.

We also examine whether the effect of tracking differs between initially high-scoring students (who are grouped with other strong students in tracking schools) and initially low-scoring students (who are grouped with other low-scoring students in tracking schools). We find that both groups of students benefited from tracking, and by approximately the same amount. A year after the intervention ended, the effect persisted for both the top and bottom classes.

Tracking increases test scores for students taught by contract teachers. In fact, students initially scoring low who were assigned to contract teachers benefited even more from tracking than students who initially scored high. But students who initially scored low showed only a small and statistically insignificant benefit if assigned to a civil-service teacher. In contrast, tracking substantially increased scores for students who initially scored high and were assigned to a civil-service teacher. Below we discuss other evidence that tracking led civil-service teachers to increase effort when they were assigned to high-scoring students but not when assigned to low-scoring students.

Changes in Peer Achievement

Data from the tracking schools allow us to estimate the effect of being taught with a higher-achieving vs. lower-achieving peer group by comparing students with baseline test scores in the middle of the distribution. Because of the way tracking was done (splitting the grade into two classes at the median baseline test score), the two students closest to the median within each school were assigned to classes where the average prior achievement of their classmates was very different.

By comparing pairs of students right around the cutoff, we can estimate the effect of being the lowest-achieving child in the class compared to being the highest-achieving student in the class. We find that, despite the large gap in average peer achievement (1.6 standard deviations in baseline test scores) between the top and bottom classes, the students just below the cutoff have postintervention test scores similar to students just above the cutoff. Moreover, when we compare students around the cutoff at the tracking schools with students of similar ability at the nontracking schools, we find that students at the tracking schools score higher at the end of the intervention than the comparable students in the nontracking schools. These results imply that being the best student in a class of relatively weak students and being the worst student in a class of relatively strong students are both better than being the middle student in a heterogeneous class. This evidence suggests that students benefit from homogeneity because the teacher does not need to spend time addressing the needs of students performing at widely varying levels.

Learning from Peers vs. Learning from Teachers

We took a separate look at students in schools where students were not tracked but instead assigned to classes randomly. The random assignment of students and teachers within these schools made it possible to see whether and how peer achievement affected the performance of individual students when education took place in an untracked setting. We found that it did. If peer achievement was higher—0.10 standard deviations higher, to be exact—students learned 0.04 standard deviations more than they would have otherwise.

These results, taken together with those reported earlier, indicate that peer influence depends on whether or not classes are tracked. In untracked classes, where there is considerable heterogeneity of performance, students learn less if their peers are lower performing. At least in this particular setting, however, the homogeneous classes that are created by tracking seem to allow the teacher to deliver instruction at a level that reaches all students, thus offsetting the effect of having lower-performing peers. Interestingly, combining the direct effect of peer achievement with the fact that the median children in each school did not suffer from being assigned to the bottom track suggests that teachers focus their attention not on the median student in the class, but at students considerably above the median.

Why Did Tracking Work?

Two additional pieces of evidence shed light on the question of why tracking had such clear benefits. First, we look at teacher presence and effort. Do they spend more time in class and teaching? Then, we examine whether the test-score gains in tracking schools were concentrated among simpler or more complex tasks and whether this varied by students’ initial achievement levels. Our results confirm that students in tracked classes seem to have benefited from more-focused teaching and perhaps also from greater teacher effort.

Teacher absence is a major problem in Kenya, as in many developing countries. Only 59 percent of teachers were in class and teaching during unannounced visits to a comparable sample of schools that did not receive an additional teacher. Overall, teachers in tracking schools were 9.6 percentage points more likely to be found in school and teaching during random spot checks than their counterparts in nontracking schools, who were present and teaching only about half of the time. There were, however, large differences across teachers. The contract teachers were much more likely to be found in school and teaching (74 percent versus 45 percent for the civil-service teachers), and their absence rate was unaffected by tracking (see Figure 2). The civil-service teachers were 10 percentage points more likely to be in schools and teaching in tracking schools than in nontracking schools when they were assigned to the top class. This difference is statistically significant and amounts to a 25 percent increase in teaching time. However, the difference between tracking and nontracking school types was smaller and statistically insignificant for civil-service teachers assigned to the bottom classes.

These results suggest that teachers may be more motivated to teach a group of students with high initial scores than a group with low initial scores or a heterogeneous group. Recall that students assigned to the top class with a civil-service teacher benefited more from tracking than those assigned to the bottom class with a civil-service teacher. Increased teacher effort may help explain this pattern.

Another hypothesis consistent with both the tracking results and the effects from random peer assignment is that tracking by initial achievement improves student learning because it allows teachers to focus instruction. Teaching a more homogeneous group of students might allow teachers to adjust the material covered and the pace of instruction to students’ needs. For example, a teacher might begin with more basic material and instruct at a slower pace, providing more repetition and reinforcement, when students are initially less prepared. With a group of initially higher-achieving students, the teacher can increase the complexity of the tasks and pupils can learn at a faster pace. With a heterogeneous group, they may be compelled to cover both simple and advanced material, spending less time on each, which would hurt all students.

One way to examine this is to see whether children with different initial achievement levels gained from tracking differentially in terms of the difficulty of the material that they learned. While the results for language are mixed, the estimates for math suggest that, although the total effect of tracking on children in the bottom class is significantly positive for all levels of difficulty, these children gained from tracking more than other students on the easier questions and less on the more-difficult questions. Conversely, students assigned to the top class benefited less on the easier questions, and more on the more-difficult questions. In fact, they did not significantly benefit from tracking for the easier questions, but they did significantly benefit from it for the more-difficult questions. These results suggest that tracking helped by giving teachers the opportunity to focus on the competencies that children were not mastering.

Conclusion

A central challenge of education systems in developing countries—the context for which our results are most relevant—is that students in the same grades and classrooms are extremely diverse. Our results show that grouping students by preparedness or prior achievement and focusing the teaching material at the most appropriate level could potentially have large positive effects with little or no additional resource cost. One could also target more resources to the weaker group, further helping them to catch up with their more-advanced counterparts. It is often suggested that there is a trade-off between the value of targeting resources to weaker students, and the costs imposed on them by separating them from stronger students. We find no evidence for such a trade-off in this context.

Our results may also have implications for debates over school choice and voucher systems. A common criticism of such programs is that they may hurt some students if they lead to increased sorting of students by initial achievement and if all students benefit from having peers with higher initial achievement. If tracking is indeed beneficial, this is less of a concern.

Esther Duflo is professor of economics at the Massachusetts Institute of Technology. Pascaline Dupas is assistant professor of economics at University of California, Los Angeles. Michael Kremer is professor of economics at Harvard University.

An interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future is available here.

School choice supporters, including hundreds of private school students in crisp uniforms, filled Washington, D.C.’s Freedom Plaza last May to protest a congressional decision to eliminate the city’s federally funded school voucher program after the next school year (to see additional images of this event please click here). That afternoon, President Obama announced a compromise proposal to grandfather the more than 1,700 students currently in the District of Columbia Opportunity Scholarship Program, funding their vouchers through high school graduation, but denying entry to additional children. Both program supporters and opponents cite evidence from an ongoing congressionally mandated Institute of Education Sciences (IES) evaluation of the program, for which I am principal investigator, to buttress their positions, rendering the evaluation a Rorschach test for one’s ideological position on this fiercely debated issue.

School vouchers provide funds to parents to enable them to enroll their children in private schools and, as a result, are one of the most controversial education reforms in the United States (to see an interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future please click here). Among the many points of contention is whether voucher programs in fact improve student achievement. Most evaluations of such programs have found at least some positive achievement effects, but not always for all types of participants and not always in both reading and math. This pattern of results has so far failed to generate a scholarly consensus regarding the beneficial effects of school vouchers on student achievement. The policy and academic communities seek more definitive guidance.

The IES released the third-year impact evaluation of the Opportunity Scholarship Program (OSP) in April 2009. The results showed that students who participated in the program performed at significantly higher levels in reading than the students in an experimental control group. Here are the study findings and my own interpretation of what they mean.

Opportunity Scholarships
Currently, 13 directly funded voucher programs operate in four U.S. cities and six states, serving approximately 65,000 students. Another seven programs indirectly fund private K—12 scholarship organizations through government tax credits to individuals or corporations. About 100,000 students receive school vouchers funded through tax credits. All of the directly funded voucher programs are targeted to students with some educational disadvantage, such as low family income, disability, or status as a foster child.

Nineteen of the 20 school voucher programs in the U.S. are funded by state and local governments. The OSP is the only federal voucher initiative. Established in 2004 as part of compromise legislation that also included new spending on charter and traditional public schools in the District of Columbia, the OSP is a means-tested program. Initial eligibility is limited to K—12 students in D.C. with family incomes at or below 185 percent of the poverty line. Congress has appropriated $14 million annually to the program, enough to support about 1,700 students at the maximum voucher amount of $7,500. The voucher covers most or all of the costs of tuition, transportation, and educational fees at any of the 66 D.C. private schools that have participated in the program. By the spring of 2008, a total of 5,331 eligible students had applied for the limited number of Opportunity Scholarships. Recipients are selected by lottery, with priority given to students applying to the program from public schools deemed in need of improvement (SINI) under No Child Left Behind. Scholars and policymakers have since questioned the extent to which SINI designations accurately signal school quality because they are based on levels of achievement instead of the more informative measure of achievement gains over time.

The third-year impact evaluation tracked the experiences of two cohorts of students. All of the students were attending public schools or were rising kindergartners at the time of application to the program. Cohort 1 consisted of 492 students entering grades 6—12 in 2004. Cohort 2 consisted of 1,816 students entering grades K—12 in 2005. The 2,308 students in the study make it the largest school voucher evaluation in the U.S. to employ the “gold standard” method of random assignment.

Voucher Effects
Researchers over the past decade have focused on evaluating voucher programs using experimental research designs called randomized control trials (RCTs). Such experimental designs are widely used to evaluate the efficacy of medical drugs prior to making such treatments available to the public. With an RCT design, a group of students who all qualify for a voucher program and whose parents are equally motivated to exercise private school choice, participate in a lottery. The students who win the lottery become the “treatment” group. The students who lose the lottery become the “control” group. Since only a voucher offer and mere chance distinguish the treatment students from their control group counterparts, any significant difference in student outcomes for the treatment students can be attributed to the program. Although not all students offered a voucher will use it to enroll in a private school, the data from an RCT can also be used to generate a separate estimate of the effect of voucher use (see sidebar, page 50).

Using an RCT research design, the ongoing IES evaluation found no impacts on student math performance but a statistically significant positive impact of the scholarship program on student reading performance, as measured by the Stanford Achievement Test (SAT 9). The estimated impact of using a scholarship to attend a private school for any length of time during the three-year evaluation period was a gain of 5.3 scale points in reading. That estimate provides the impact on all those who ever attended a private school, whether for one month, three years, or any length of time in between (see Figure 1). Consequently, the estimate should be interpreted as a lower-bound estimate of the three-year impact of attending a private school, because many students who used a scholarship during the three-year period did not remain in private school throughout the entire period. The data indicate that members of the treatment group who were attending private schools in the third year of the evaluation gained an average of 7.1 scale score points in reading from the program.

What do these gains mean for students? They mean that the students in the control group would need to remain in school an extra 3.7 months on average to catch up to the level of reading achievement attained by those who used the scholarship opportunity to attend a private school for any period of time. The catch-up time would have been around 5 months for those in the control group as compared to those who were attending a private school in the third year of the evaluation.

Over time, in my opinion, the effects of the program show a trend toward larger reading gains cumulating for students. Especially when one considers that students who used their scholarship in year 1 needed to adjust to a new and different school environment, the reading impacts of using a scholarship of 1.4 scale score points (not significant) in year 1, 4.0 scale score points (not significant) in year 2, and 5.3 scale score points (significant) in year 3 suggest that students are steadily gaining in reading performance relative to their peers in the control group the longer they make use of the scholarship. No trend in program impacts is evident in math.

What explains the fact that positive impacts have been observed as a result of the OSP in reading but not in math? Paul Peterson and Elena Llaudet of Harvard University, in a nonexperimental evaluation of the effects of school sector on student achievement, suggest that private schools may boost reading scores more than math scores for a number of reasons, including a greater content emphasis on reading, the use of phonics instead of whole-language instruction, and the greater availability of well-trained education content specialists in reading than in math. Any or all of these explanations for a voucher advantage in reading but not in math are plausible and could be behind the pattern of results observed for the D.C. Opportunity Scholarships. The experimental design of the D.C. evaluation, while a methodological strength in many ways, makes it difficult to connect the context of students’ educational experiences with specific outcomes in any reliable way. As a result, one can only speculate as to why voucher gains are clear in reading but not observed in math.

Student Characteristics
The OSP serves a highly disadvantaged group of D.C. students. Descriptive information from the first two annual reports indicates that more than 90 percent of students are African American and 9 percent are Hispanic. Their family incomes averaged less than $20,000 in the year in which they applied for the scholarship.

Overall, participating students were performing well below national norms in reading and math when they applied to the program. For example, the Cohort 1 students had initial reading scores on the SAT-9 that averaged below the 24th National Percentile Rank, meaning that 75 percent of students in their respective grades nationally were performing higher than Chart 1 in reading. In my view, these descriptive data show how means tests and other provisions to target school voucher programs to disadvantaged students serve to minimize the threat of cream-skimming. The OSP reached a population of highly disadvantaged students because it was designed by policymakers to do so.

Did Only Some Students Benefit?
Several commentators have sought to minimize the positive findings of the OSP evaluation by suggesting that only certain subgroups of participants benefited from the program. Martin Carnoy states that “the treated students in Cohort 1 were concentrated in middle schools and the effect on their reading score was significantly higher than for treated students in Cohort 2.” Henry Levin likewise asserts that “the evaluators found that receiving a voucher resulted in no advantage in math or reading test scores for either [low achievers or students from SINI schools].”

The actual results of the evaluation provide no scientific basis for claims that some subgroups of students benefited more in reading from the voucher program than other subgroups. The impact of the program on the reading achievement of Cohort 1 students did not differ by a statistically significant amount from the impact of the program on the reading achievement of Cohort 2 students, Carnoy’s claim notwithstanding. Nor did students with low initial levels of achievement and applicants from SINI schools experience significantly different reading gains from the program than high achievers and non-SINI applicants. The mere fact that statistically significant impacts were observed for a particular subgroup does not mean that impacts for that group are significantly different from those not in the subgroup. For example, Group A and Group B may have experienced roughly similar impacts, but the impact for Group A might have been just large enough for it to be significantly different from zero (or no impact at all), while Group B’s quite similar scores fell just below that threshold.

From a scientific standpoint, three conclusions are valid about the achievement results in reading from the year 3 impact evaluation of the OSP:

• The program improved the reading achievement of the treatment group students overall.
• Overall reading gains from the program were not significantly different across the various subgroups examined.
• Three distinct subgroups of students—those who were not from SINI schools, students scheduled to enter grades K-8 in the fall after application to the program, and students in the higher two-thirds of the performance distribution (whose average reading test scores at baseline were at the 37th percentile nationally)—experienced statistically significant reading impacts from the program when their performance was examined separately. Female students and students in Cohort 1 saw reading gains that were statistically significant with reservations due to the possibility of obtaining false positive results when making comparisons across numerous subgroups.
  Why examine and report achievement impacts at the subgroup level, if the evidence indicates only an overall reading gain for the entire sample? The reasons are that Congress mandated an analysis of subgroup impacts, at least for SINI and non-SINI students, and because analyses at the subgroup level might have yielded more conclusive information about disproportionate impacts for certain types of students.

Expanding Choice
The OSP facilitates the enrollment of low-income D.C. students in private schools of their parents’ choosing. It does not guarantee enrollment in a private school, but the $7,500 voucher should make such enrollments relatively common among the students who won the scholarship lottery. The eligible students who lost the scholarship lottery and were assigned to the control group still might attend a private school but they would have to do so by drawing on resources outside of the OSP. At the same time, students in both groups have access to a large number of public charter schools.

The implication is that, for this evaluation of the OSP, winning the lottery does not necessarily mean private schooling, and losing the lottery does not necessarily mean education in a traditional public school. Members of both groups attended all three types of schools—private, public charter, and traditional public—in year 3 of the voucher experiment, although the proportions that attended each type differed markedly based on whether or not they won the scholarship lottery (see Figure 2). In total, about 81 percent of parents placed their child in a private or public school of choice three years after winning the scholarship lottery, as did 46 percent of those who lost the lottery. The desire for an alternative to a neighborhood public school was strong for the families who applied to the OSP in 2004 and 2005.

These enrollment patterns highlight the fact that the effects of voucher use reported above do not amount to a comparison between “school choice” and “no school choice.” Rather, voucher users are exercising private school choice, while control group members are exercising a small amount of private school choice and a substantial amount of public school choice. The positive impacts on reading achievement observed for voucher users therefore reflect the incremental effect of adding private school choice through the OSP to the existing schooling options for low-income D.C. families.

Parent Satisfaction
Another key measure of school reform initiatives is the perception among parents, who see firsthand the effects of changes in their child’s educational environment. Whenever school choice researchers have asked parents about their satisfaction with schools, those who have been given the chance to select their child’s school have reported much higher levels of satisfaction. The OSP study findings fit this pattern. The proportion of parents who assigned a high grade of A or B to their child’s school was 11 percentile points higher if they were offered a voucher, 12 percentile points higher if their child actually used a scholarship, and 21 points higher if their child was attending a private school in year 3, regardless of whether they were in the treatment group. Parents whose children used an Opportunity Scholarship also expressed greater confidence in their children’s safety in school than parents in the control group.

Additional evidence of parental satisfaction with the OSP comes from the series of focus groups conducted independently of the congressionally mandated evaluation. One parent emphasized the expanded freedom inherent in school choice:

“[The OSP] gives me the choice to, freedom to attend other schools than D.C. public schools….I just didn’t feel that I wanted to put him in D.C. public school and I had the opportunity to take one of the scholarships, so, therefore, I can afford it and I’m glad that I did do that.” (Cohort 1 Elementary School Parent, Spring 2008)

Another parent with two children in the OSP may have hinted at a reason achievement impacts were observed specifically in reading:

“They really excel at this program, `cause I know for a fact they would never have received this kind of education at a public school….I listen to them when they talk, and what they are saying, and they articulate better than I do, and I know it’s because of the school, and I like that about them, and I’m proud of them.” (Cohort 1 Elementary School Parent, Spring 2008)

These parents of OSP students clearly see their families as having benefited from this program.

Previous Voucher Research
The IES evaluation of the DC OSP adds to a growing body of research on means-tested school voucher programs in urban districts across the nation. Experimental evaluations of the achievement impacts of publicly funded voucher and privately funded K—12 scholarship programs have been conducted in Milwaukee, New York City, the District of Columbia, Charlotte, North Carolina, and Dayton, Ohio. Different research teams analyzed the data from New York City (three different teams), Milwaukee (two teams), and Charlotte (two teams). The four studies of Milwaukee’s and Charlotte’s programs reported statistically significant achievement gains overall for the members of the treatment group. The individual studies of the privately funded K—12 scholarship programs in the District of Columbia and Dayton reported overall achievement gains only for the large subgroup of African American students in the program. The three different evaluators of the New York City privately funded scholarship program were split in their assessment of achievement impacts, as two research teams reported no overall test-score effects, but did report achievement gains for African Americans; the third team claimed there were no statistically significant test-score impacts overall or for any subgroup of participants.

The specific patterns of achievement impacts vary across these studies, with some gains emerging quickly, but others, like those in the OSP evaluation, taking at least three years to reach a standard level of statistical significance. Earlier experimental evaluations of voucher programs were somewhat more likely to report achievement gains from the programs in math than in reading—the opposite of what was observed for the OSP. Despite these differences, the bulk of the available, high-quality evidence on school voucher programs suggests that they do yield positive achievement effects for participating students.

Conclusions
School voucher initiatives such as the District of Columbia Opportunity Scholarship Program will remain politically controversial in spite of rigorous evaluations such as this one, showing that parents and students benefited in some ways from the program. Critics will continue to point to the fact that no impacts of the program have been observed in math, or that applicants from SINI schools, who were a service priority, have not demonstrated statistically significant achievement gains at the subgroup level, as reasons to characterize these findings as disappointing. Certainly the results would have been even more encouraging if the high-priority SINI students had shown significant reading gains as a distinct subgroup. Still, in my opinion, the bottom line is that the OSP lottery paid off for those students who won it. On average, participating low-income students are performing better in reading because the federal government decided to launch an experimental school choice program in our nation’s capital.

The achievement results from the D.C. voucher evaluation are also striking when compared to the results from other experimental evaluations of education policies. The National Center for Education Evaluation and Regional Assistance (NCEE) at the IES has sponsored and overseen 11 studies that are RCTs, including the OSP evaluation. Only 3 of the 11 education interventions tested, when subjected to such a rigorous evaluation, have demonstrated statistically significant achievement impacts overall in either reading or math. The reading impact of the D.C. voucher program is the largest achievement impact yet reported in an RCT evaluation overseen by the NCEE. A second program was found to increase reading outcomes by about 40 percent less than the reading gain from the DC OSP. The third intervention was reported to have boosted math achievement by less than half the amount of the reading gain from the D.C. voucher program. Of the remaining eight NCEE-sponsored RCTs, six of them found no statistically significant achievement impacts overall and the other two showed a mix of no impacts and actual achievement losses from their programs. Many of these studies are in their early stages and might report more impressive achievement results in the future. Still, the D.C. voucher program has proven to be the most effective education policy evaluated by the federal government’s official education research arm so far.

The experimental evaluation of the District of Columbia Opportunity Scholarship Program is continuing into its fourth and final year of studying the impacts on students and parents. The final evidence collected from the participants may confirm the accumulation of achievement gains in reading and higher levels of parental satisfaction from the program that were evident after three years, or show that those gains have faded. Uncertainty also surrounds the program itself, as the students who gathered on Freedom Plaza in May currently are only guaranteed one final year in their chosen private schools. What will policymakers see as they continue to consider the results of this evaluation? The educational futures of a group of low-income D.C. schoolchildren hinge on the answer.

Patrick J. Wolf is professor of education reform at the University of Arkansas and principal investigator of the D.C. Opportunity Scholarship Program Impact Evaluation. The opinions expressed in this article are his own.

An unabridged version of this article is available here.

The ongoing global financial crisis is forcing many employers, from General Motors to local general stores, to take a hard look at the costs of the compensation packages they offer employees. For public school systems, this will entail a consideration of fringe benefit costs, which in recent years have become an increasingly important component of teacher compensation. During the 2005–06 school year, the most recent year for which U.S. Department of Education data are available, the nation’s public schools spent $187 billion in salaries and $59 billion in benefits for instructional personnel. Total benefits added about 32 percent to salaries, up from 25 percent in 1999–2000. The increase reflects the well-known rise in health insurance costs, but it also appears to include growing costs of retirement benefits, which have received much less attention.

Conventional wisdom holds that teacher pensions (along with other public pensions) are more costly than private retirement benefits, for reasons dating to an earlier era of low teacher salaries over lifelong careers. In spite of dissent from this view by some researchers (see sidebar), in this case we find that conventional wisdom is right: the cost of retirement benefits for teachers is higher than for private-sector professionals.

To track changes in retirement costs and compare employer contributions to retirement for public school teachers with those for private-sector professionals, we draw on recent data from a major employer survey conducted by the U.S. Department of Labor. These data show that the rate of employer contributions to retirement benefits for public school teachers in 2008 is substantially higher than for private professionals: 14.6 percent of earnings for teachers vs. 10.4 percent for private professionals. Moreover, the gap has widened over the four years the data have been available. Between March 2004 and September 2008, the difference more than doubled, rising from 1.9 to 4.2 percentage points (see Figure 1).

There are several reasons one might expect employer contributions to retirement to be higher for teachers. First, nearly all teachers are covered by traditional defined benefit (DB) pension plans, in which employees receive a regular retirement check based on a legislatively determined formula. These plans have, over the years, come to offer retirement at relatively young ages, at a rate that replaces a substantial portion of final salary. U.S. Department of Education data show a median retirement age for public school teachers of 58 years, compared to about 62 for the labor force as a whole. A teacher in her mid-50s who has worked for 30 years under a typical teacher pension plan will be entitled to an annuity at retirement of between 60 and 75 percent of her final salary. In nearly all plans this annuity has some sort of cost-of-living adjustment. One does not generally observe comparable retirement plans for professionals and lower-tier managers in the private sector, since most employers have replaced traditional DB plans with defined contribution (DC) or similar 401(k)-type plans, in which the employer and employee contribute to a retirement account that belongs to the employee. Nor do those traditional DB plans that remain typically reward retirement at such early ages; they more nearly resemble Social Security, where eligibility is age 62 for early retirement, and 66 and rising for normal retirement.

The Survey Data

Our analysis draws on data from the National Compensation Survey (NCS), an employer survey developed by the Bureau of Labor Statistics (BLS). The NCS survey is designed to measure employer costs for wages and salaries and fringe benefits across a wide range of occupations and industries in the public and private sectors. Although the BLS has been reporting quarterly fringe-benefit cost data for various public and private employee groups for more than a decade, only since March 2004 has the bureau broken out these fringe-benefit cost data for public school K–12 teachers. In this article we use those data to compare retirement benefit costs for public K–12 teachers with costs for private-sector professionals. We use the most detailed available private-sector comparison group, “management, professional, and related,” a category that includes business and financial managers, operations specialists, accountants and auditors, computer programmers and analysts, engineers, lawyers, physicians, and nurses.

We measure the cost of retirement benefits as a percentage of earnings. Virtually all states specify in law that the employer will contribute a certain percentage of teacher salaries to a DB pension fund (employee contributions are similarly specified), and it is commonplace to compare such contribution rates among the states. Similarly, private-sector employers offering DC plans will typically specify their contribution as a percentage of salary (often as a match to employee contributions). Unlike some other benefits (e.g., health insurance), if salaries change, the dollar costs for retirement benefits move proportionally. On the benefit side, the DB formula ties one’s starting annuity to final average salary, while the adequacy of a DC plan is commonly thought of in terms of the salary replacement rate. Thus it is natural to specify retirement costs as a percentage of salary, both for teachers and for private-sector professionals.

In making this comparison, we must account for the fact that, while all of the private-sector professionals are covered by Social Security, a number of public school teachers are not. Some of the higher cost of employer retirement plans for teachers is offset by lower employer contributions for Social Security benefits. Thus, we should compare the contribution rates for employer-provided retirement benefits and Social Security for both groups of workers. While the BLS reports the Social Security contribution rate for private professionals, it does not report a similar rate for teachers. However, we are able to make such an adjustment by multiplying the share of teachers covered by Social Security, which the BLS estimates to be 73 percent, times the employer contribution rate (6.2 percent). This assumes that the vast majority of teachers are below the Social Security earnings cap (currently $102,000) and that the share of teachers in Social Security has been steady over the four years for which we make the comparison.

A time series with quarterly data for these benefit percentages is reported in Figure 1. Two patterns are visible. First, the contribution rate is considerably higher for public school teachers than for private professionals. In the most recent quarter for which data are reported, ending September 2008, the employer contribution rate for public K–12 teachers (14.6 percent) was 4.2 points higher than that for private-sector professionals (10.4 percent). Second, the gap is widening. While the private sector contribution rate has been relatively flat over the four years, the rate for public school teachers has markedly increased, doubling the gap between them from one-fifth to two-fifths.

In one important respect, it is likely that the BLS data underestimate the cost of retirement benefits for public school teachers. Many public school districts (and some states) provide health insurance benefits for retired public school teachers. In the course of this research we were surprised to learn that retiree health insurance benefits are not included in the BLS employment cost estimates. Since private employers have largely eliminated this benefit, this means that our estimate of the gap in retirement benefits favoring public school teachers is low, although we cannot be sure of the extent of the underestimate.

Social Security and Teachers

While the overall employer contribution rate for public school teachers is higher than for private-sector professionals, the group average may mask differences between teachers who are and are not covered by Social Security. In order to assess this point empirically, we examined directly the data on employer contributions for teacher pension funds. We find that total employer contributions for both groups of public school teachers are higher than for private-sector professionals.

Most teachers are in statewide pension funds, with a relatively small number in district funds (e.g., New York City, Denver, St. Louis). Data on employer contributions for these plans are available in annual financial reports for each fund, which are surveyed by the National Association of State Retirement Administrators (NASRA).

Using data on contributions from NASRA and pension fund annual reports where necessary, and using weights based on the number of teachers employed in each state or district as reported in the NCES Common Core of Data, it is possible to compute average employer contribution rates for teachers. First we consider teachers who are in states and districts covered by Social Security. For these teachers, we calculate the weighted average employer contribution to be 9 percent of earnings. This can be compared to the estimate of employer contributions to retirement for private-sector professionals and managers, calculated from the BLS data as 4.7 percent for the comparable period (FY07). This is a 4.3 percent difference favoring public school teachers, almost double, in those states and districts where teachers are enrolled in Social Security, so the comparison is on an equal footing.

For states and districts where teachers are not in Social Security, we calculate the average employer contribution at 11.1 percent of earnings. Of course, this is considerably higher than the 4.7 percent retirement contributions for private-sector professionals, but, perhaps surprisingly, it even exceeds their employers’ combined contributions to retirement and Social Security, which averaged 10.3 percent for FY07. Thus, as Figure 2 shows, comparing teachers with professionals in private-sector employment, total employer contributions are higher for teachers whether or not they are also covered by Social Security.

Our analysis of evidence from the BLS National Compensation Survey and the NASRA Public Fund Survey shows that the employer contribution rates for public school teachers are a larger percentage of earnings than for private-sector professionals and managers, whether or not we take account of teacher coverage under Social Security. In addition, the BLS data show that the contribution rate for teachers is clearly trending upward.

Looking Ahead

What are the likely trends going forward for the cost of teacher retirement benefits? No one knows for sure, but we can identify the two key factors that will drive these costs: future developments in the benefits themselves and in their funding. The trend through much of the postwar period was to enhance the retirement formulas in various ways, including reducing the age or service requirement for full benefits. For example, just last year New York City agreed to enhance its pension formula for younger teachers. But there is evidence that benefit enhancement has generally abated in recent years. There are even a few states, including Texas, that have moved to reduce benefits for newly hired teachers. However, this is unlikely to reduce costs in the near future, since benefits for incumbent teachers are protected by law in most states.

The other factor to consider is the funding status of teacher pension plans. The vast majority of teacher pension plans are not fully funded. This means that contributions include both the “normal cost” of pension liabilities accruing to current employees and the legacy costs of amortizing unfunded liabilities accrued previously (due to a variety of reasons, including the original pay-as-you go nature of most plans, as well as unfunded benefit enhancements over the years). In theory, if the actuarial assumptions hold true going forward and no new benefits are enacted, the amortization costs will eventually disappear (after 30 years, under a typical funding schedule), in much the same way that a homeowner’s monthly expenses decline when the mortgage gets paid off.

However, the near-term prospects may be very different. For one thing, public pension funds face the possibility of important accounting changes. Unlike private pension funds, public fund actuaries have been allowed to discount future liabilities at a rate of about 8 percent, the assumed long-run market return on fund assets. Finance economists have argued that such a high discount rate is imprudent, however, and there have been signs that public accounting standards might move toward the private-sector rules, based on corporate bond and Treasury rates, which could reduce the discount rate to about 5 percent. This would dramatically raise the required amortization payments.

Finally, it bears noting that the market value of pension funds has fallen precipitously as of this writing (December 2008). Barring a major market recovery, pension funds across the country will have new, large unfunded liabilities. Under actuarial smoothing methods, these losses will be phased in, raising required amortization payments over the next few years. If the accounting rules for public funds also change, reducing the discount rate on liabilities, the employers of public school teachers, along with other public employers, will face a double hit, requiring sharp increases in contributions. By contrast, those private employers who have switched over to defined contribution plans in recent decades will be unaffected. In short, there are good reasons to believe that the contribution gap we have documented will continue to widen in coming years.

Robert M. Costrell is professor of education reform and economics at the University of Arkansas. Michael Podgursky is professor of economics at the University of Missouri–Columbia.

Refusing to consent to the underlying ideology, Swan failed repeatedly. The college threatened to expel him from the teacher training program unless he signed a contract agreeing to undergo diversity training and accept extra scrutiny of his student teaching. After a national civil-liberties group intervened on his behalf, Swan was allowed to continue in the program, and WSU has since revised its evaluation form. The new version requires professors to evaluate students’ “willingness to consider multiple perspectives on social and institutional factors that can impede or enhance students’ learning.” Dean of Education Judy Mitchell explained, “We’ve changed the format and clarified the words, but we haven’t changed the standards.”

Advocates of dispositions assessments of the kind in place at WSU defend the screening of pre-service teachers, whether at program entry or later on in the certification process, as standard practice and argue that “dispositions” are merely those attitudes and behaviors necessary to successful teaching. Critics see the combination of program accreditation standards, revised by the National Council for Accreditation of Teacher Education (NCATE) in 2000; a growing curricular emphasis on “social justice” issues; and a left-leaning education professoriate as yielding a one-sided approach to teacher education and the certification of teachers based on ideology, rather than teaching skills or mastery of content knowledge.

As a historian, I am most struck by the parallels between the dispositions assessments of today’s aspiring teachers and the evaluations of teachers’ mental hygiene and personality that began in the 1940s and continued for two decades. As is the case today, from 1940 to 1960 teacher educators sought to protect the interests of schoolchildren by socially engineering “desirable” characteristics in their teachers. What have changed are the personal qualities deemed most important for success in the classroom.

Assessing Teacher Dispositions

What is the purpose of dispositions assessment? What entity or body is in the best position to make this assessment? If the purpose is to ensure that access to children is denied to those who are truly deviant (sexual predators) or those who could harm children (drug dealers, felony offenders, child abusers), then it seems the assessment is best made by the government, which has the resources and responsibility to identify these people. If the purpose is to ensure that potential teachers have basic characteristics like honesty or fairness, existing standards such as university honor codes in higher education should suffice. If the purpose is to see how a teacher acts in a certain environment (be it an urban, suburban, or rural school, with a diverse or homogeneous student body), then perhaps those in that environment can best perform that assessment, taking into account the standards, mores, and preferences of the community. The ultimate employers of teachers, local school districts, can and do screen for the characteristics they want in their employees. Why, then, is it also necessary for teacher educators to assess the personal and political beliefs of aspiring teachers? Perhaps the policing of teacher personality and dispositions is just a way for teacher educators to extend their control even further into the public school classroom.

The harshest critics of dispositions assessment accuse education schools of acting as ideological gatekeepers to employment in public schools. Indeed, web site after web site shows schools of education that list among their teacher-education program goals the inculcation of political views alongside intellectual curiosity and such work habits as punctuality. The University of Alabama’s College of Education is “committed to preparing individuals to promote social justice, to be change agents, and to recognize individual and institutional racism, sexism, homophobia, and classism.…” In the teacher education program at the Harvard Graduate School of Education, students are asked to  “act as leaders and agents for organizational change in their classrooms, schools, and society…continually examine their own identities, biases, and social locations, seeking knowledge of students’ cultures and communities, and pursuing a complex understanding of societal inequities as mediated through classism, heterosexism, racism, and other systems of advantage.” Some program descriptions explain that requiring awareness of these issues and a commitment to addressing them ensures teachers will teach all children. In an October 2006 letter defending the conceptual framework of Teachers College, Columbia University, against accusations of political screening, President Susan H. Furhman wrote, “We believe that responsiveness to the diversity of students’ backgrounds and previous experiences are [sic] essential for effective teaching” (see Figure 1).

Not all universities make the leap from classroom behavior to ideology: The “Teacher Education Professional Dispositions and Skills Criteria” at Winthrop University in South Carolina are only basic indicators of professional commitment, communication skills, interpersonal skills (among them, “Shows sensitivity to all students and is committed to teaching all students”), emotional maturity, and academic integrity; acknowledging social inequities is not mentioned. The difficulty, however, in assessing dispositions, whether they espouse social justice or are seemingly harmless as at Winthrop, arises when the assessors make value judgments rather than encourage academic freedom and respect freedom of conscience. As the Swan case at Washington State University shows, some teacher education programs clearly demand allegiance to a particular perspective on the politics of education.

If schools encourage students to respond honestly to teacher education assignments, and then use any responses that differ from accepted beliefs as grounds for dismissal, that is political screening and a clear denial of academic freedom. A student accused Le Moyne College, a private, Jesuit-run school, of doing just that. In 2004, administrators dismissed the politically conservative graduate student after he wrote a paper on classroom management that questioned the value of multicultural education and expressed limited support for the use of corporal punishment in the classroom. At the Brooklyn College School of Education, some students complained after a teacher showed the Michael Moore film Fahrenheit 9/11 on the day before the 2004 presidential election. The university asked one student to leave, accused two others of plagiarism, and then denied the two students the right to bring a witness or an attorney to their hearing. K. C. Johnson, a faculty member who questioned the accusation of plagiarism and defended the students in Inside Higher Ed, then faced possible investigation by the university. The hallmarks of a professional program of teacher preparation within a university should be the free exploration of ideas. Yet it seems some teacher preparation programs substitute professional socialization, and the political conformity it requires, for a commitment to academic freedom.

The controversy over political screening of prospective teachers by teacher educators came to a head at the June 2006 reauthorization hearing for the National Council for Accreditation of Teacher Education (NCATE) with the U.S. Department of Education. Within the list of dispositions aspiring teachers might be required to possess, the agency had included “social justice,” a phrase that, to many, signals a value-laden ideology. Under pressure from a number of groups, NCATE president Arthur Wise announced that the agency would drop “social justice” from its accreditation standards; he maintains that social justice was never a required disposition.

NCATE’s definition of “dispositions” and its inclusion of social justice as part of that definition had caused considerable consternation. Among the groups represented at the hearing were the National Association of Scholars, which had filed the complaint, and the Foundation for Individual Rights in Education (FIRE), founded and headed by civil libertarians Alan Charles Kors, professor of history at the University of Pennsylvania, and Harvey Silverglate, a criminal defense attorney. FIRE, an organization dedicated to the preservation of free speech, has accused a number of universities, including Washington State University on behalf of Edward Swan, of evaluating students on the basis of their political views and thereby violating their First Amendment rights.

Arthur Wise has staked out NCATE’s position that dispositions are only “commonsense expectations” for teacher behavior and insists that the accrediting agency does not condone the evaluation of attitudes. Whether or not that is the case, most teacher education programs in this country receive accreditation from NCATE and follow its lead. Even though NCATE has now dropped “social justice” as a disposition, the agency stands behind dispositions assessment and institutions’ use of “social justice” as a curricular theme. The phrase appears in countless teacher-preparation program and course descriptions. Critics are not hopeful that NCATE’s action will curb abuses. In her testimony at the NCATE hearing, American Council of Trustees and Alumni president Anne D. Neal asked that the agency’s reauthorization be denied “until it affirmatively makes clear that teacher preparation programs are not expected to judge the values and political beliefs of teacher candidates and asks that its members review and revise their standards accordingly.”

Judging Fitness Is Nothing New

Society has long been concerned with the behavior, both inside and outside of the classroom, and the character of public school teachers. A century ago, local school boards carefully selected school teachers they deemed “fit to teach,” whose behavior comported with community values. They could not smoke or drink. Female teachers could not socialize with men while unchaperoned. They could not marry. They were not to display or engage in behaviors considered deviant, such as lesbianism. They were to dress conservatively and attend church. Violation could cost a teacher her job.

School officials and boards also scrutinized teachers’ political views. During World War I, the superintendent of the Cleveland public schools suggested firing those teachers sympathetic to Germany, and anti-war teachers did lose their jobs in New York City. In the 1920s and 1930s, more than a dozen states, typically those in which there were anti-communist crusades, required teachers to take loyalty oaths.

In public-school classrooms, as educational progressivism steadily gained influence during the first half of the 20th century, the focus in classrooms gradually shifted from rigorous academic study and discipline to children’s personality development and mental health. Education historian Sol Cohen describes the “medicalization” of education as the “infiltration of psychiatric norms, concepts and categories of discourse” into American education. Cohen reports that by 1950, there was “a national consensus on the role of personality development in American education” and that this included the view that “the school is basically an institution to develop children’s personality and that personality development of children should take priority over any other school objective.”

Attention turned as well toward the “mental hygiene” of the teacher, whose actions and attitudes would no doubt influence the children in her charge. As Douglas Spencer, instructor of psychological counseling at Teachers College, Columbia University, wrote in 1938, the teacher was to “demonstrate in her own personality adjustment sound mental health and emotional maturity.” As the 1940s began, a growing chorus of educators called for teacher qualification and selection to be based on mental health, first and foremost, and many expected this to be achieved through the teacher education process. However, market pressures on teacher education institutions made this problematic. Government policies provided tax funds for training teachers through the publicly supported teachers colleges, which did not have selective admissions requirements. Meanwhile, the number of both school-age children and college attendees grew steadily, with more than one-quarter of college degrees being granted in the field of education.

The rapid expansion of the teaching workforce hindered efforts to select teachers on mental hygienic grounds, even before the teacher shortage that developed in the 1950s. Reports of teachers with mental disturbances and even mental illnesses made professional and public headlines throughout the 1940s and 1950s. Public concern grew about maladjusted or neurotic teachers and their inability to ensure the proper psychological development of the children under their tutelage. Some feared that, as with contagious disease, psychological disorders would spread from teacher to child. Various personality traits of the maladjusted teacher emerged in the literature of the time. Shy, nervous, timid, easily excitable, disorganized, irresponsible, introverted, sexually repressed, or hot-tempered teachers were considered unfit for the classroom. A 1961 text, The Mentally Disturbed Teacher, documented purportedly true incidents about such teachers, suggesting that teachers who used corporal punishment could be mentally ill or that irritability in a teacher may be a sign of alcoholism, to take two examples. One suggestion for improving the mental health of the teaching body was for schools to keep a record of the teacher’s “attainments and attitudes,” including her cultural background and her community leadership.

As early as the 1940s, teacher education institutions began to use rating scales, placement tests, and personal interviews as screening devices for measuring mental hygiene and teacher personality. For some assessments, candidates filled out questionnaires; for others, faculty, administrators, or psychologists observed the teacher and made judgments. The University of Utah required teacher candidates to take the Minnesota Multiphasic Personality Inventory and the Strong Vocational Interest Blank. The College of Education at the State College of Washington used the Minnesota Teacher Attitude Inventory. Still other institutions employed a variety of assessment measures, such as the Rorschach test, James Cattell’s 16 Factor Personality test, the Guilford-Zimmerman Temperament Survey, the Thurstone Temperament Schedule, and a host of other batteries designed to explore the teacher’s behavior, personality, and attitude.

In 1953, Ruth A. Stout, director of field programs at Kansas State Teachers Association and later professor of education at Teachers College, Columbia University, completed a comprehensive study of admission practices in teacher education institutions. Stout surveyed 785 of 865 accredited teacher-training schools and found that a majority identified emotional stability as being of primary importance and that approximately 45 percent actually assessed students’ emotional stability, identifying it as the second most important criterion for determining fitness for teaching, behind academic credentials. Assessment of emotional stability became more important, Stout reported, as students progressed through their teaching preparation, with more institutions using it to determine admission to student teaching than to the teacher education program.

Research on Teacher Personality

Experimental and statistical research on personality development exploded onto the 1940s education scene, replacing earlier anecdotal surveys. The Journal of Experimental Education and the Journal of Educational Research published much of this research, which used psychological or personality indexes to “scientifically” determine the relationship between personality and “good teaching.” The ultimate goal was to connect personal traits with teaching effectiveness, thus enabling better selection of teacher candidates. Sometimes, researchers measured teacher success based on the observation of classroom supervisors. At other times, they used data on students’ class rank, college grades, or other measures of student performance.

The results of the research were as diverse as the assessment instruments used. Some found good teachers were more gregarious, adventurous, frivolous, artistic, polished, cheerful, kind, and interested in the opposite sex than teachers rated poorer in performance. Others found good teachers to be those whose attitudes were positive toward children and administrators. A few studies that tried to correlate teacher factors (both intelligence and personality) with effectiveness found teaching too complex to be influenced by any one or two factors. Nonetheless, institutions pushed forward with the use of personality tests to select among teacher candidates, often using multiple indexes, even as critics warned that some instruments had low predictive validity, that there was inconsistency in results, or that the lack of replication warranted cautious use.

In a 1956 review of the research on “School Personnel and Mental Health,” J. T. Hunt, a professor at the University of North Carolina, noted that “efforts to identify personality differences between superior and inferior school personnel, to isolate a ‘teacher personality,’ or to predict either competence or effectiveness of student teachers by means of psychometric or projective instruments, led to limited results.” Unlike most of the research he reviewed, Hunt recognized that personality was not a monolithic attribute, as there were many kinds and types of teacher personalities and roles. More presciently, Hunt called for research that would consider the “varying value standards of judges.” “Very little attention seems to have been paid,” he concluded, “to the actual attitudes and expectations of persons” who assess teachers. He called for research that placed university administrators under the personality microscope.

University of Chicago professors Jacob Warren Getzels and P. W. Jackson in 1960 followed Berkeley professor Fred Tyler’s lead in arguing that no consensus existed among researchers, and presumably educators generally, as to what was considered good teaching. Getzels and Jackson pointed out that the authoritarian teaching style considered “good” at the end of the 19th century had given way to the personal style brought into vogue with progressive education, which would in time give way to another. Without definitive criteria for good teaching, the personality indexes used in teacher personality research had no validity. The tests, they claimed, were chosen for “irrelevant reason” or for “no apparent reason at all.” Thus, the entire design of research on teacher personality was flawed.

Mental Hygiene as Curriculum

The mental hygiene perspective nonetheless held sway throughout the 1950s, and the concept of personality became as important in teacher education as academic and technical preparation for the classroom—as important as content knowledge and skills. The separation of the teacher into “technician” and “personality,” a distinction noted by mental hygienist Harry Rivlin in 1955, required that teacher education prepare students along both lines. Teacher educators often prioritized personality over other aspects of a teacher’s abilities.

In 1955, Percival Symonds, professor at Teachers College, Columbia University, where he taught mental hygiene, headed a study publicized by the Council for Research in the Social Sciences of Columbia University. His study recommended “a change in emphasis in teacher-training from intellectual courses to experiences for the better personal adjustment of teachers.” For a decade, Symonds had promoted the inclusion of psychotherapeutic principles and methods in the mental hygienic treatment of teacher maladjustment. He used psychotherapy in his classes; his students wrote autobiographies and he analyzed them. As he noted in a Newsweek article, he “first gained the pupil’s confidence to a point where they would feel free enough to drag all the family skeletons out of the closet”; of course, he found maladjustment everywhere. According to a 1955 Education article by Leon Mones, then an assistant superintendent in Newark, New Jersey, and a former principal, Symonds and others were openly advocating that the emotional life of the teacher become the focus of teacher preparation, since “it is the teacher’s personality that is the tool with which he works rather than the content in which he gives instruction.”

Educational psychology courses aimed at understanding children were standard fare for teacher preparation in the 1920s. But even by the mid-1940s, the goal of psychology coursework had become the teacher’s own mental health. Bank Street College of Education in New York, San Francisco State College, the University of Texas, and the University of Wisconsin incorporated lectures on mental health with “psychiatrically supervised individual guidance” of pre-service teachers. The experimental use of psychoanalysis in teacher education even received funding from the National Institute of Mental Health.

At Bank Street College, teacher educator and director of research Barbara Biber extolled the virtues of a program that applied “the concept of the unified nature of cognitive and affective development…on the teacher-training level” and was based on “a process of integrating new knowledge with an old self.” Bank Street faculty members looked for certain dispositions in their candidates: relatedness to children, an orientation to the psychology of growth, their relation to authority, their emotional strength, and their motivation. The Institute for Child Study at the University of Maryland emphasized the ideal of the “self-actualized individual” in its graduate-level instruction. A human relations seminar at the Merrill-Palmer School aimed “to help the individual teacher express and explore the values, meanings, and dynamics of personal and professional experiences, to achieve self-awareness, and to develop sensitive, understanding, responsive attitudes.”

Still, psychiatrists reported that teachers, especially novices, did not know how to handle their negative feelings. I. N. Berlin, a professor of psychiatry and psychiatric consultant to school districts in San Francisco, San Joaquin County, and Stockton, California, argued that some mental pathologies that were causal factors in teacher maladjustment and ineffectiveness in the classroom were, unfortunately, exacerbated rather than alleviated by teacher education. Berlin’s criticism of teacher training reflected the belief of some psychiatrists that there were limits to teacher education’s ability to ensure mentally healthy teachers.

Learning from History

The screening of prospective teachers for maladjustment 50 years ago and the dispositions assessments going on today have remarkable similarities. As William Damon of Stanford has noted, dispositions assessment “opens virtually all of a candidate’s thoughts and actions to scrutiny…[and] brings under the examiner’s purview a key element of the candidate’s very personality.” The same underlying assumption—that scientific means of selection and training could guarantee good teachers—held sway at mid-century with respect to mental hygiene. Teacher educators who guarded entry to the profession used the techniques of science to study, measure, and evaluate the teacher candidate as do those who guard entry today. Only the specific values and attitudes they appraise have changed. Advocates of dispositions assessment claim that their methods are “standards-based” and provide “accountability” —scientific-sounding catchwords that hold considerable weight in the current political climate. Both sets of desirable characteristics—summed up in the terms mental hygiene and social justice—are tied to progressivism and appear as core components of the teacher preparation curriculum, with the effect of deemphasizing academic knowledge, or at least requiring subject-matter learning and even pedagogy to make room for them. And hard evidence was and still is lacking. Researchers could never link with any certainty particular personality traits with effective teaching. Nor, as Frederick Hess explains, is there any scientific evidence that requiring teachers to have certain views about “sexuality or social class” ensures that they teach all students: “Screening on ‘dispositions’ serves primarily to cloak academia’s biases in the garb of professional necessity.”

The history of teacher screening reveals how deeply rooted such practices are in American teacher education. Whether the standard is mental hygiene or possessing the proper political and ideological disposition, the elimination of candidates who do not pass muster gives teacher educators the power to determine who gains access to a classroom based on the values the teacher educators prefer. While the courts have permitted certifying agencies to require “good moral character” of teacher applicants, as legal scholars Martha McCarthy and Nelda Cambron-McCabe note, they “will intervene…if statutory or constitutional rights are abridged.” Thus, while pledging loyalty to federal and state constitutions is a permissible condition for obtaining a teacher license, swearing an oath to progressivism is not. Given the evidence and the history, there should be real concern, as teacher educator Gary Galluzzo has said, that “students’ views and personalities are being used against them” whenever dispositions are assessed. Those committed to academic freedom within higher education should be concerned when professional socialization trumps freedom of conscience in teacher education programs.

Laurie Moses Hines is assistant professor at Kent State University Trumbull campus, where she teaches in the Cultural Foundations of Education program and in the history department.

In 2000, public elementary and secondary schools spent roughly $180 billion on teachers’ salaries and benefits, about half of their total expenditures; most of it was distributed according to fixed salary schedules that considered only a teacher’s education and years of experience. This system has its origins in the first half of the 20th century and was partly a response to the racial and gender discrimination that existed under more discretionary systems at that time.

However, over the past 20 years more educators have wondered whether such pay packages can attract, motivate, and retain high-quality teachers in a highly competitive professional world (see Forum). In response to such concerns, there was a flurry of merit pay activity in the early 1980s. Twenty-nine states had initiated some sort of merit pay program for teachers by 1986. Since then, however, almost all of them have been diluted or discontinued. A 1997 study by economists Dale Ballou and Michael Podgursky reported that 12 percent of school districts were using merit pay in some way, but the amount of incentive in these districts averaged only two percent of base pay.

Critics of merit pay argue that the falloff in such programs was due to the fundamental technical difficulties of accurately identifying effective teachers and rewarding good teaching practices. Proponents of performance-based pay insist that these experiments were too limited in scope and were destined to fail in the face of stiff opposition from teachers and unions.

Despite widespread pessimism among educators about whether merit pay systems can effectively reward good teachers, most of the limited empirical evidence has been surprisingly positive. For example, two studies (in 1992 and 1997) found that the math and reading test scores of students in South Carolina improved significantly when the students were taught by teachers receiving merit pay. Similarly, related and more recent literature suggests that mathematics students learn more when their teachers have certification in mathematics.

Policymakers should be cautious in interpreting this sort of evidence, however, as the apparent benefits of certification could merely reflect differences in the students placed in their classrooms. For example, teachers who receive merit pay may tend to select schools and classes whose students are high achievers for other reasons. Likewise, parents especially engaged in their children’s education may work to ensure their assignment to teachers with strong credentials. Such subtle differences may not be visible in the data typically available to researchers.

New Evidence

The effectiveness of these short-lived merit pay programs is exceptionally difficult to measure because of these selection effects. However, the fortuitous overlap of two Tennessee programs from the mid-1980s and 1990s provides an unusual opportunity to circumvent this problem. Project STAR (Student Teacher Achievement Ratio) was a large-scale class-size experiment that began with kindergarten students in the fall of 1985. At roughly the same time, the Volunteer State began directing pay increases to teachers deemed meritorious under a Career Ladder Evaluation System.

The fact that both teachers and students in schools participating in Project STAR were assigned randomly to classrooms allows for an especially rigorous test of whether a merit pay system can effectively reward good teachers.

Before describing the Tennessee programs in detail, however, we need to take a closer look at some of the objections to merit pay for teachers. One concerns the problem of designing valid evaluation procedures for measuring teacher performance. Under an efficient merit pay plan in any industry, employers should be able to explain clearly why an employee did not receive merit pay and what he or she would need to do to get it. Whether these conditions can be met in the teaching profession, where there is no single blueprint for effective practice, has been the most contentious issue surrounding merit pay. This evaluation problem is further complicated by the fact that schools have goals other than cognitive achievement (for instance, promoting citizenship, fostering individual development, and reducing drug use and violence) that are difficult to measure and are often achieved only with teachers’ cooperation.

These concerns raise the possibility that attempts to reward meritorious teachers could even have perverse consequences. For example, merit pay systems may discourage cooperation among teachers or otherwise foster a demoralizing and unproductive work environment.

While these problems may explain why merit pay plans have often been dismantled, some researchers suggest that they are excuses, not reasons. Dale Ballou, an economist at Vanderbilt University, has argued that merit pay is widely and successfully used in private schools, which suggests that there is nothing unique about education that makes merit pay infeasible or unattractive. Ballou notes that the amount of merit pay in private schools is quite large and that the teachers who report receiving it have earnings that are nearly 10 percent higher than their nonmerit counterparts. In contrast, the earnings of merit pay teachers in public schools are only 2 percent higher than their nonmerit colleagues. Ballou attributes the frequent dismantling of alternative compensation for public school teachers to union opposition.

The Career Ladder

Can we devise a merit pay system that overcomes the challenges of definitional clarity and valid measurement? Can we do so without directly incorporating measures of students’ progress on standardized tests?

The Tennessee programs initiated by Governor Lamar Alexander in 1984 offer some reason to believe that we can. At the same time, they underscore the considerable difficulty of doing so in a fair, equitable, and effective manner.

Part of Tennessee’s Comprehensive Educational Reform Act, the Career Ladder Evaluation System was both well funded and sophisticated in its approach to teacher evaluation. As Richard M. Brandt, a professor at the University of Virginia, wrote in 1995, the program was “perhaps the country’s most comprehensive experiment in summative evaluation.”

Governor Alexander, who would go on to become secretary of education under President George H. W. Bush and is now a U.S. senator, was more colloquial in his description, calling the program “an old-fashioned horse trade with teachers. Taxpayers said to teachers, ‘The state will pay you up to 70 percent more based on your performance if you’ll promise to be evaluated every five years.’”

Rung by Rung

While it lasted–for 13 years–the now-defunct Career Ladder had many of the elements that merit pay backers believed a good program should have, including multidimensional evaluations and a hierarchy of professional development (in other words, a career ladder) that was coordinated with significant financial and professional rewards.

The ladder had five distinct stages, ranging from probationary to master. Fast-track options allowed those who had been teaching before 1984 to advance immediately, subject to successful evaluations, to a career level matching their experience.

For new teachers, however, the first rung of the career ladder was a one-year probation supervised by two tenured teachers from their school. Subject to a favorable review by the school district, using state-approved criteria, these teachers were then placed on apprentice status for three years. At the end of those three years, the school district could recommend that the teacher be granted a five-year certification for professional, or Career Level I, status, which included a $1,000 salary supplement from the state.

Then, at the end of the five-year Level I stage, a teacher could either apply for another five-year Level I certification or seek a five-year certification as a Level II teacher. Advancement required evidence of superior performance, as defined by a state commission and the state board of education, but it also came with a $2,000 state supplement for those who chose a 10-month contract and $4,000 for those choosing an 11-month contract, a significant bonus to teachers’ salaries at the time.

At the end of the Level II certification period, the same kind of option was available: a teacher could seek recertification at Level II or pass more rigorous evaluations to receive a Level III certification and a salary supplement of as much as $7,000.

The evaluations that occurred at each stage of the career ladder assessed teachers on multiple “domains of competence” using several distinct data sources (such as student and principal questionnaires, peer evaluations, a teacher’s portfolio, and a written test). On the first three rungs of the ladder (probation, apprentice, Level I), the local school districts were responsible for evaluating and certifying performance. The key evaluator at these stages–typically the principal–received three to five days of state training on evaluation instruments and procedures. In contrast, the evaluations for certifications at Levels II and III were conducted largely by a three-member team of peers from outside the teacher’s district. These evaluators received three to four weeks of training and were often Level III teachers from other districts who had been borrowed for a year by the state certification commission. The extensive training provided to the Level II and Level III evaluators was considered appropriate since they fielded more complex evaluation instruments intended to discriminate among “good, superior, and outstanding” teachers.

Under the original formulation of the career ladder, participation was optional for veteran teachers and mandatory for new teachers. It was initially expected that new teachers who failed to advance to Level I status after their apprenticeship would be fired, since they would no longer be eligible for the state portion of their salary. However, in 1987, the career ladder was revised to make it optional for all teachers. The major consequence of failing to advance to Level I status was essentially the lost opportunity for the salary supplement.

Interestingly, it appears that relatively few teachers faced this cost. Nearly all of the state’s teachers (94 percent of them, according to one report) chose to enter the career-ladder program. A state audit in 1991 revealed that 95 percent of eligible teachers had achieved Level I certification, prompting criticism that the standards for this designation had been severely diluted. However, among teachers applying for certification at Levels II and III, the success rate was only 79 percent.

Though most teachers chose to participate, and the success rates for certification were quite high, some expressed criticisms that echoed issues often raised by merit pay critics: for example, that three classroom visits (some of them prearranged) were inadequate for evaluating teaching performance objectively and that separating the staff into levels strained relations among teachers and hurt morale. Even the application process was criticized for emphasizing, as the Christian Science Monitor reported, “cunning and endurance . . . rather than merit.” The criticisms suggest that, despite the relative sophistication of the career ladder, its efficacy in rewarding high-quality teachers remains an open question.

Project STAR

Coincidentally, a compelling way to evaluate the success of the career ladder system comes via data from Governor Alexander’s Student Teacher Achievement Ratio program. Project STAR was an experimental study of class-size reduction that also began in the fall of 1985. That year, it included 6,325 kindergarten students from 79 participating schools. The experiment lasted for three more years, following students through the 3rd grade. Overall, roughly 11,600 students participated, with additional students entering the participating schools in the 1st, 2nd, and 3rd grades. Participating schools were drawn from around the state and, by legislative mandate, included inner-city and suburban schools from larger metropolitan areas (Knoxville, Nashville, Memphis, and Chattanooga) as well as rural schools and those from smaller towns. All students in classrooms included in the experiment were given the Stanford Achievement Tests in math and reading in the spring of each year.

Pooling the information from the experiment’s four years yields a single data set with roughly 24,000 student observations for each subject. Roughly one-third of these observations are for black students, and nearly half were for students eligible for the free-lunch program. Fully 91 percent of the student observations in the dataset come from classrooms taught by teachers participating in the career ladder: 15 percent had teachers with probationary or apprentice status, 69 percent had teachers at Level I, while just seven percent had teachers who had reached Level II or III.

The key feature of the experimental design of Project STAR was that students and teachers within participating schools and grades were randomly assigned to one of three class types: small classes, regular-sized classes, or regular-sized classes with teacher aides. These random assignments allow us to use the STAR data to compare the performance of students assigned to career-ladder teachers with the performance of students in the same school and grade who were assigned to nonparticipating teachers.

Restricting the comparison to students attending the same school is essential because student-teacher pairings were random only within a given school. That is, the experiment did not move students and teachers to schools they would not otherwise have attended or staffed. This unfortunately means that some schools in the data set–those with classrooms taught by teachers with the same career-ladder status–do not offer useful information for looking at the effects of career-ladder status.

It should also be noted that student attrition from schools participating in the experiment was high, ranging from 20 to 30 percent each year, and that roughly 10 percent of students moved between small and regular classes. While most of the movement between classes was due to parental complaints or behavioral problems, the attrition figures could also reflect other factors unrelated to the study, such as students’ moving out of a school’s geographic zone or having to repeat a grade. However, if parents of students with unobserved propensities for high achievement sought out master teachers by class reassignment or by moving to another school altogether, our results would overstate the quality of career-ladder teachers.

Fortunately, we expect that these problems are less important for a study of the career ladder than for one about class size. Unlike a multiyear assignment to a particular class size, a one-year assignment to a particular teacher does not provide a strong incentive for attrition or reassignment. Students would be assigned a new teacher in the next academic year. By contrast, students placed in a large class were expected to remain in large classes through the 3rd grade.

Still, to evaluate whether the experiment successfully matched students and teachers randomly within schools, we examined the association between students’ traits and their assignment to a teacher of their own race. If the pairings of students and teachers were indeed random and remained so over time, we should find no within-school association between observed student traits and exposure to teachers in the career ladder. As expected, students’ race, gender, age, eligibility for the free-lunch program, and class-size assignment all exhibit small and statistically weak within-school relationships with assignment to a career-ladder participant.

Finally, because the student-teacher pairings were initially random, any statistically significant difference in performance between students with and without career-ladder teachers should be attributable to true differences in the quality of the teachers. The most conventional interpretation of such performance differences would be that the program provided effective incentives for teachers and that the evaluations carefully discriminated among teachers of high and low quality. However, the high pass rates on career-ladder evaluations suggest that these assessments were not particularly discriminating (at least through Level I). This raises the possibility that, if career-ladder teachers were more effective, it was simply because better teachers were more willing to negotiate the bureaucratic impediments to advancing on the career ladder. Nonetheless, even if the career ladder led only to self-sorting of teachers by quality, it would indicate that the program successfully directed its financial and professional rewards to meritorious teachers.

Results

To see what these Tennessee programs tell us about merit pay, let’s first look at the effects simply of having a teacher in the career-ladder program, ignoring for the moment the teacher’s specific level of accomplishment. To eliminate the effects of any chance differences in performance caused by other observable characteristics, our analysis takes into account students’ age, gender, race, and eligibility for the free lunch program; whether they had been assigned to a small class; and whether they were assigned to a teacher of the same race–which earlier research using these same data found to have a large positive effect on student performance (see “The Race Connection,” Spring 2004). We also include as control variables two conventional indicators of teacher quality: experience and possession of a graduate degree.

Our main results indicate that students with career-ladder teachers scored nearly 3 percentile points higher in mathematics than students with other teachers. They also suggest that reading scores were nearly 2 percentile points higher among these students, though the results for reading fall just short of conventional levels of statistical significance (see Figure 1).

The estimated effects on reading scores are statistically indistinguishable from zero primarily because they are less precise. If the effect on reading performance of having a career-ladder teacher were as precisely estimated as the effect of being in a smaller class, it would also be statistically significant. That it is not may reflect the fact that the experiment was designed to evaluate the effects of differences in class size, not the career-ladder program.

Regardless, our best guess is that having a career-ladder teacher in either subject had a quite large effect. The estimated gains associated with assignment to a career-ladder teacher equal 40 to 60 percent of the gains associated with assignment to a class with roughly 15 students rather than 22. Furthermore, the gains are approximately equivalent to a third of the black-white gap in test scores among students in the experiment.

When evaluating these results, it is important to keep in mind that 91 percent of the student observations in the data set came from classrooms with teachers certified by the career ladder. The benefits of having a career-ladder teacher are measured relative to a somewhat atypical base–namely, the small group of students whose teachers chose not to apply for the program or were unsuccessful in their application.

Our second analysis, therefore, considered not only the teacher’s participation in a career ladder, but also the teacher’s status within the program. That is, we looked separately at the effects of having a teacher at the probationary or apprentice level, at Level I, and at Level II or III.

In math, the career-ladder teachers at the probationary/ apprentice level and at Level I were the most successful at promoting achievement. In contrast, career-ladder teachers at the master level did not have a statistically significant effect on math scores (see Figure 2).

This surprising pattern could in theory reflect the success of the career ladder in attracting (and retaining) new, high-ability math teachers and in providing these new teachers with early mentoring and professional development. However, an alternative explanation is that novice teachers, many of whom quickly leave teaching, happen to be particularly adept at teaching math. The fact that we have already controlled for differences in teachers’ experience makes this explanation unlikely. Moreover, a similar pattern emerges when we look only at students with teachers having five or more years of experience, a good number of whom remained at the probationary/apprentice level (perhaps because fast-track options were not available in their area). In short, it appears that the career ladder simply was not very effective at distinguishing superior or outstanding math teachers from those who were merely competent.

In reading, by contrast, assignment to a Level II or Level III teacher was associated with a large and statistically significant increase in reading achievement, while estimates of the effects of having a teacher from both of the other two groups remained positive but statistically insignificant. This suggests the career ladder may have been modestly successful in identifying the most outstanding teachers in reading.

Conclusions

Overall, our results suggest that Tennessee’s Career Ladder Evaluation System was at least partially successful at rewarding teachers who were relatively effective at promoting student achievement. Though the program was voluntary for veteran teachers, the combination of large bonuses and relatively undemanding evaluations–at least at the lower levels–led the vast majority of teachers to enter. Nonetheless, assignment to a teacher who had been certified by the career-ladder evaluations led to large and statistically significant increases in mathematics scores and sizable, though statistically insignificant, increases in reading scores.

But our findings also suggest that the teachers who were on the highest rungs of the career ladder (and received the largest pay increases) were not consistently better at promoting student achievement. In reading, only students with a teacher at the highest levels of the career ladder made statistically significant gains. In contrast, the math-score gains associated with having a career-ladder teacher actually appear to have been concentrated among students with teachers on the lowest rungs of the career ladder. These mixed findings underscore the challenge of designing a system of teachers’ compensation that rewards quality in a fair and equitable manner–a political challenge as much as a technical one.

Despite some success in rewarding teachers for producing better student outcomes, the career ladder was a target of the same criticisms that challenge virtually all attempts to tinker with systems of teachers’ compensation. A few years of budgetary constraints helped kill the will to keep it all together. Thus, having made participation in the career ladder voluntary for teachers in 1987, it was perhaps inevitable that the Tennessee legislature in 1997 voted to prevent additional teachers from entering the program and becoming eligible for merit bonuses. Teachers already in the program, though no longer subject to regular evaluations, were allowed to keep their bonuses for the duration of their careers.

As Lamar Alexander lamented at the time, “Those who questioned the Model-T Ford didn’t try to kill it. They replaced it with something better.” Continuing debates over merit pay programs in districts in Tennessee and beyond indicate that efforts to find such a replacement are under way. But it may still be too early to tell whether the future for merit pay for teachers will resemble that of the Edsel or the Mustang.

-Thomas S. Dee is an assistant professor in the Department of Economics at Swarthmore College. Benjamin J. Keys is a graduate student in the Department of Economics at the University of Michigan.

Podcast: Robert Costrell and Michael Podgursky talk with Education Next.

An unabridged version of this article is available here.

Teacher pensions consume a substantial portion of school budgets. If relatively generous pensions help attract effective teachers, the expense might be justified. But new evidence suggests that current pension systems, by concentrating benefits on teachers who spend their entire careers in a single state and penalizing mobile teachers, may exacerbate the challenge of attracting to teaching young workers, who change jobs and move more often than did previous generations.

The design of teacher pension plans is a timely concern: like other public pension plans, those for teachers are becoming more costly. Employer contributions to pension funds tack on a larger percentage of earnings for public school teachers than for private-sector managers and professionals, and this gap is widening (see “Teacher Retirement Benefits,” research, Spring 2009, Figure 1). Those data do not yet reflect the impact of the stock market decline since 2007: the drop in the value of pension funds means further increases in employer contributions will be required to fund promised benefits. As fiscal concerns force states to reevaluate the costs of teacher pension plans, officials might also consider the plans’ consequences for teacher quality.

In earlier work we highlighted the peculiar incentives for retirement built into these plans (see “Peaks, Cliffs, and Valleys,” features, Winter 2008). Most plans create large spikes in pension wealth accumulation for teachers in their 50s. These spikes act as an incentive for teachers to stay in the classroom until their pension wealth reaches its peak and then push them into retirement shortly thereafter, as pension wealth accumulation turns negative.

We now extend this line of research by focusing on the distribution of pension benefits among teachers of varying career lengths and the penalties for those who switch systems. We examine pension formulas in six state plans and develop measures of the redistribution of pension wealth from teachers who separate early to those who separate later. We compare existing defined benefit (DB) teacher pension systems to fiscally equivalent systems that treat all teachers equally and find that the former often redistribute about half the pension wealth of an entering cohort of teachers to those who separate in their mid-50s from those who leave the system earlier. We then show that this back loading produces very large losses in pension wealth for mobile teachers. Compared to a teacher who has worked 30 years in a single state system, a teacher who has put in the same years but split them between two systems will often lose well over one-half of her pension wealth. It is difficult to justify such a system of rewards and penalties on grounds related to fairness or teacher quality.

Teacher Pensions 101

Public school teachers are almost universally covered by traditional defined benefit pension systems. In such a system, the employer has an obligation to provide a regular retirement check to employees upon their retirement. Typically, a DB teacher pension plan requires that both teachers and employers make a contribution each year to a pension trust fund. The salient characteristic of a traditional DB system is that for any individual, benefits are not tied to contributions.

More specifically, once a teacher is “vested” (usually after 5 or 10 years), she becomes eligible to receive a pension upon reaching a certain age or length of service. These eligibility rules vary across states, but they typically allow a teacher to draw a pension well before age 65, especially if she has been working since her mid-20s. Benefits at retirement are usually determined by a formula that takes into account years of service and the final average salary (FAS), which is an average of the last few years of salary (typically three). In Missouri, for example, teachers eligible for normal retirement earn 2.5 percent (the “multiplier”) for each year of teaching service. Thus, a teacher with 30 years of service would earn 75 percent of the final average salary. So if the FAS were $60,000, she would receive $45,000 every year for the rest of her life. If the teacher were to separate from service prior to being eligible to receive the pension, the first payment would be deferred and the amount of the pension would be frozen until that time. Once the pension payments begin, there is typically some form of inflation adjustment, although the specifics again vary from state to state.

We examined teacher pension plans in six states. While the states were not randomly chosen (we inhabit two of them), their plans are indicative of many teacher pension plans. Because the composite effect of each system is hard to discern by simply looking at the benefit formula, we examine patterns of pension wealth accumulation by age of separation.

Calculating Pension Wealth

We use the benefit formulas of pension plans to estimate the pension wealth of individual teachers. When an individual retires under a DB plan, she is entitled to a stream of payments that has a lump-sum value that we calculate using standard actuarial methods (which take into account expected mortality patterns and adjust the sum of payments to reflect the fact that they are received over many years rather than at a single point in time).

The heavy S-shaped curve in Figure 1 depicts pension wealth (net of employee contributions) for 25-year-old entrants to the Missouri teaching force who work continuously until they leave teaching at various ages. The salary schedule assumed is that of the state capital (Jefferson City), under which teachers receive experience-based salary increases and are also paid more if they have a master’s degree. The accumulation of pension wealth is not smooth and steady, but rises with fits and starts, due to rules of eligibility for early retirement and the like. In Missouri, after vesting at five years, a teacher is eligible for a pension at age 60. Her pension wealth—the current value of those deferred benefits—grows fairly steadily until age 45. The curve becomes steeper at age 46 because of a provision that allows teachers to begin collecting a pension when their age and years of service sum to 80, which brings her pension forward to age 59 and earlier. Then there is a big jump at age 50, because the 25th year of service makes a teacher eligible for an immediate pension (albeit with a reduced multiplier). Growth in pension wealth continues to be rapid in subsequent years as the multiplier is increased to its “normal” rate of 2.5 percent. Then, following a final bump in the benefit formula’s generosity at 31 years of service (age 56), net pension wealth starts shrinking. As is evident, complex pension rules lead to pension wealth curves that are irregularly shaped and bear no resemblance to the smoothly growing cumulative value of contributions.

(Pension) Wealth Redistribution

The result of these complex pension rules is that teachers who leave the profession in their 50s receive more pension wealth (as a percentage of cumulative earnings) than those who separate earlier. To develop a measure of the resulting redistribution, we compare existing DB systems to a fiscally equivalent plan where pension wealth is neutrally distributed: a cash balance (CB) system. CB systems calculate employee retirement benefits based on the cumulative contributions, with a guaranteed rate of return. Thus, pension wealth is a fixed percentage of cumulative earnings, regardless of retirement age.

In dollar terms, pension wealth grows smoothly under a CB system. Figure 1 compares the accrual of pension wealth under Missouri’s DB plan (the S-shaped curve) with the smooth accrual under a hypothetical CB plan. This diagram readily illustrates the redistribution of pension wealth toward those who retire in their 50s from those who leave teaching earlier. Teachers who retire before age 49 in Missouri receive less pension wealth than they would under a CB plan, while teachers who retire later receive considerably more.

We have developed a numerical measurement of this redistribution. Specifically, to compare net pension wealth across different ages of separation, we measure it at a fixed point in time, and we also estimate the frequency of separations at different ages. In this fashion, we can calculate weighted averages of net pension wealth for winners, losers, and the whole cohort of 25-year-old entrants. When we compare the Missouri plan to the fiscally equivalent CB plan, we find that 46 percent of pension wealth is redistributed from those leaving teaching at an average age of 36.6 to those separating at an average age of 54.2.

We made the same calculations of the distributional impact of the DB plans in the other states. In all states, the degree of redistribution is substantial. In Massachusetts, for example, average pension wealth is low, but 61 percent of it is redistributed. The degree of redistribution is also relatively high in Ohio (49 percent) and Texas (47 percent, for new hires), while it is somewhat lower in Arkansas (39 percent) and California (36 percent). As in Missouri, the redistributive gains are concentrated among those who retire in their 50s, while the losses are dispersed among all early leavers. This pattern holds particularly true for Massachusetts, where the gains are concentrated among just one-fifth of the cohort.

To summarize, there is significant variation among states in the magnitude of the gains and losses compared to a simple CB system, but all states redistribute net pension wealth to a substantial degree to those who retire in their 50s (after about 30 years of service) from those who leave a teaching position after shorter periods. In addition to the issue of equity, this has serious implications for teacher mobility, to which we now turn.

Moving Costs

It is well understood that DB pension plans penalize mobility, yet the sources of these costs are rarely delineated or quantified in a systematic way. There are several factors that reduce pension wealth when a teacher moves. First, teachers who leave a system before they are vested have no claim on a pension. Upon termination, or shortly thereafter, any teacher contributions are returned with interest (the rate varies, and can be well below market), but the teacher does not receive employer contributions. This is a major source of loss for many young teachers, since most teacher pension systems have a vesting period of five years or longer and the vast majority of early-career teacher turnover occurs in the first five years on the job. In fact, nine states have a 10-year vesting period for teachers. With such long vesting windows, many teachers will receive no employer contributions toward retirement as a result of their work in the classroom.

Although the effects of these vesting windows are large, they are at least fairly transparent for young teachers. This information is routinely provided to those newly hired. Even for teachers who are vested, however, there remain potentially large costs from mobility, and these are less obvious. One cost comes from the fact that teacher DB pensions are all based on final average salary. When a teacher leaves the profession before normal retirement age, the value of her annuity is tied to her salary at the time of her separation. No adjustment is made for ensuing salary growth or inflation.

Other costs to mobility arise from the service eligibility rules for normal and early retirement. Teachers who separate from a plan with, say, fewer than 20 years of service will often not be able to begin collecting their pensions until much later than teachers who remain in the plan until they meet eligibility requirements. At any given age, pension wealth is therefore lower for the mobile teacher—who has left one system early and entered another system late—simply because she can expect to collect fewer pension checks. Alternatively, she may be able to draw her pension at the same time as the teacher who stays in one system, but with a penalty. Either way, the costs are substantial.

Switching Systems

Pension wealth calculations similar to those above provide a comprehensive method for evaluating the costs of mobility. Specifically, let us continue to examine the pension wealth of a hypothetical teacher who enters at age 25 and works continuously. However, now, rather than working continuously in the same system, at age 40, after 15 years in state A, she moves to state B, which has the same pension formula and same pay grid, and ultimately retires. We assume that she collects two pensions, one in each of the states in which she worked. The pure mobility cost can be thought of as the loss from moving at age 40 to an identical state, but with zero creditable service.

The hypothetical wealth trajectory described above is depicted as the dotted curve in Figure 1 for Missouri. As discussed above, the heavy solid curve illustrates net pension wealth for continuous service under the DB plan, evaluated at date of separation. The dotted segment represents the wealth trajectory for a teacher who moves after 15 years, at age 40, diverging at that point from the solid curve for the teacher who stays. For the first five years, the dotted curve is flat since the teacher must get vested in the new system. After vesting, the teacher is entitled to two pensions, one from the old job and one from the new one. However, the loss from mobility continues to widen in the following years, as the teacher who stays becomes eligible for earlier and earlier retirement, while the teacher who moves does not earn enough service credit to advance the pension from age 60.

Under a continuous career, our hypothetical teacher would obtain 30 years of service by age 55, qualifying her for “normal” retirement benefits immediately at 75 percent of final average salary. This is worth $626,088 at age 55. The split career of the mobile teacher means that she receives two annuities, each of which is for 37.5 percent of final average salary, but the FAS for the first pension is of course much lower. In addition, neither the first nor the second pension would be drawn until “normal” retirement at age 60. This means that five years of pension payments are lost. These two factors together reduce the net pension wealth to $219,163, a loss from mobility of $406,925. This is the gap between the dotted and solid curves in Figure 1 at age 55. The cost of mobility is 65 percent of pension wealth.

By contrast, under the hypothetical cash balance system, also depicted in Figure 1, there is no loss from mobility. Net pension wealth, the cumulative value of employer contributions, is a constant percentage of cumulative earnings, regardless of whether they accrue in one job or two.

Table 1 provides summary calculations of these mobility losses for all six states. A glance down the first column shows substantial mobility costs in all six states, ranging from approximately $200,000 to more than $500,000. As the table also shows, these losses are large in relative terms as well, ranging from 41 percent to 74 percent of net pension wealth for teachers who stay.

Figure 2 depicts the sources of these losses, as well as the variation across states. For each state, the full bar gives the net pension wealth of a teacher who stays in the system to age 55, and the bottom portion, in black, is that of the mobile teacher. The middle portion gives the loss from mobility due to freezing FAS on her first job. The top portion gives the mobility cost imposed by service eligibility rules. Specifically, splitting 30 years of service credit between two jobs delays the first pension draw and can also affect the replacement rate (the annual pension as a percentage of FAS).

The costs from the split in service credit are generally large and vary across states. In Missouri, Arkansas, and Ohio, these rules lead to a delay of first pension draw from age 55 to 60, while in California, the first draw is delayed to age 57. In Texas, the mobile teacher delays first draw to 63, but she gains a higher replacement rate as a result. In Massachusetts, there is no delay for first draw, but the mobile teacher sacrifices a large increase in the replacement rate that is awarded to 30-year veterans. All in all, the service eligibility rules for early retirement, pension bumps, and the like—little known to the general public (and, we suspect, to many young teachers)—can impose large costs on teachers who move.

Final Considerations

Our work offers the first detailed analysis of the distribution of net pension benefits among teachers of varying ages of separation and the corresponding costs that teacher pension systems impose on mobile teachers. We find that in a typical DB system, compared to a neutral system, half an entering cohort’s pension wealth is redistributed to teachers who separate in their 50s, from those who separate earlier. One of the main reasons is that teachers who teach into their 50s can start collecting a pension immediately, while teachers who leave earlier often must defer their pension until age 60 or later, so they collect fewer payments over their retirement.

This inequality in benefits produces very large losses in pension wealth for mobile teachers. We estimate that teachers who split a 30-year career between two pension plans often retire with less than half the pension wealth accrued by teachers who complete a similar career in a single system. Again, one of the main reasons is that teachers who split their career often cannot begin collecting pension payments as early as those who stay in one system.

Our discussion has focused on teachers. However, the problems we have identified extend to other professional staff in public schools. School administrators are always included in teacher retirement systems. The market for administrators in urban school districts is increasingly becoming national in scope, yet for mobile administrators retirement benefit systems with 5- to 10-year vesting systems can have a devastating effect on retirement savings.

The impediments to mobility—for both teachers and administrators—may be particularly problematic for charter schools. Many charter schools are part of organizations (e.g., Knowledge Is Power Program [KIPP], Edison Learning, Imagine Schools) that operate in more than one state. Edison Learning, for example, operates schools in 16 states. As these schools attempt to replicate their school models, it is valuable to them to move staff from one location to another, particularly when they start new schools, in much the same way business firms relocate managers. As we have shown, current educator retirement benefit systems make such mobility very costly in those states where charter school employees are required to participate in the state’s teacher pension plan.

Such a system of rewards and penalties is hard to justify. To appreciate the importance of mobility, consider the large differences in the growth of public school enrollment between states. The National Center for Education Statistics projects that states such as Nevada and Arizona will see enrollment growth in excess of 40 percent between 2005 and 2017. Louisiana, Vermont, and Rhode Island can expect enrollment declines of 10 percent or more over this same period. Heavily populated states such as Michigan and New York can anticipate declines of between 5 and 6 percent. In a well-functioning labor market, one would see considerable movement of workers from areas of contracting demand to areas in which demand is increasing. In the case of teaching, however, the pension systems impose large costs on those who move.

The barriers to reform are primarily political. First, states have a coordination problem. It is in no state’s individual interest to facilitate mobility out of the state; to the contrary, states are inclined to keep average pension costs down by skimping on benefits for those who depart. In addition, the distribution of benefits within states between short-term and career teachers will be governed by the relative influence of junior versus senior educators in educator groups and state politics. Influence generally increases with seniority for a variety of reasons, and these are enhanced in the case of pension politics, because the benefits of pensions are far more immediate and tangible for senior educators than for junior ones. The opaque nature of final-average-salary DB systems, with their complicated eligibility rules, only reinforces this imbalance.

All that said, these barriers are not insurmountable. Similar issues arise in higher education, and yet the benefits of academic mobility have led many state and private universities to offer more portable retirement plans. As states grapple with the pension difficulties they now face, they should consider systems with smooth wealth accrual, such as the CB plan described in this article. Another alternative to consider might be a hybrid such as TIAA-CREF, which has features of both CB and defined-contribution plans and has proven popular in higher education. Such systems are more transparent, tie benefits more closely to contributions, and do not penalize mobility or job shopping among young teachers. At a minimum, education policymakers should consider experiments that provide actuarially fair alternatives to traditional DB plans for new teaching recruits, and evaluate their utility for recruiting and retaining high-quality teachers.

Robert M. Costrell is professor of education reform and economics at the University of Arkansas. Michael Podgursky is professor of economics at the University of Missouri–Columbia.

Are test scores the educational outcomes that parents value most? We tackle this question by examining the types of teachers that parents request for their elementary school children. We find that, on average, parents strongly prefer teachers whom principals describe as best able to promote student satisfaction, though parents also value teacher ability to improve student academics. These aggregate effects, however, mask striking differences across schools. Parents in high-poverty schools strongly value a teacher’s ability to raise student achievement and appear indifferent to student satisfaction. In wealthier schools the results are reversed: parents most value a teacher’s ability to keep students happy.

Data

This study combines data on teacher requests (by parents) and teacher evaluations (by principals) from 12 elementary schools in a midsized school district that asked to remain anonymous, in the western United States. The students in the district are predominantly white (73 percent), but there is a reasonable degree of diversity in terms of ethnicity and socioeconomic status. Roughly 35 percent of the white students are eligible for free or reduced-price lunch. Latino students, 84 percent of whom are eligible for free or reduced-price lunch, comprise 21 percent of the student population. Achievement levels in the district nearly match the average of the nation (49th percentile on the Stanford Achievement Test).

There is no formal procedure for parents to request specific teachers in the district. Principals report that they assign students to classes with an eye toward balancing race, gender, and ability across classrooms within the same grade. Parents submit requests during the spring or summer, and principals make assignments over the summer. During our analysis period, roughly 22 percent of parents requested a teacher each year and 79 percent of teachers received at least one parental request. Parents are also able to request that their child not be placed with a particular teacher (a “negative request”). Only about 9 percent of teachers received any negative requests, and 92 percent of teachers with negative requests had at least one positive request as well. Principals report that they are generally able to honor almost all requests, giving parents an incentive to truthfully reveal their first preference.

Parents in the district appear to have strong and varied preferences for teachers. Among those teachers receiving at least one request, the average number of requests was 6.2. Whereas the teacher at the 25th percentile received only 2 requests, the teacher at the 75th percentile received 8 requests. Moreover, there are often large differences between the most-requested and least-requested teacher within the same school, grade, and year: The average difference is 7.4, and in 10 percent of grades, the difference is larger than 17.

Our data include information on requests made for the 2005–06 school year (the “request year”) in the summer of 2005 for kindergarten through 6th-grade teachers in all 12 schools in our sample, as well as information from an earlier year for two of the schools. We exclude from our analysis those teachers parents could not have plausibly requested—mainly new teachers (unless parents specifically requested the “new” teacher), who comprised about 17 percent of those teaching in the request year. Note that we include teachers who did not receive any requests, as long as they taught in the same grade and school in the request year and the prior year. Our final sample consists of 256 individual teachers. Parents who made requests chose, on average, from among approximately three different teachers.

With the assistance of the district, we linked the parental request data to administrative data on teachers and students. Because the administrative files provide only a very coarse measure of family socioeconomic status—eligibility for the federal free or reduced-price lunch program—we constructed an additional proxy for family income by matching each student’s residential address to U.S. Census data on the median household income in the student’s neighborhood.

Finally, to supplement our information on teachers, we administered a survey to all elementary school principals in February 2003 and March 2006. In these surveys, we asked principals to evaluate their teachers along a variety of dimensions, including dedication and work ethic, organization, classroom management, parent satisfaction, positive relationship with administrators, student satisfaction, role model value for students, and ability to raise math and reading achievement. The average rating was roughly 8 on a scale of 1 to 10, indicating that principals were quite lenient in their assessments. On the basis of these survey results, we created three measures: (1) the principal’s overall assessment of the teacher’s effectiveness, which is a single item from the survey; (2) the teacher’s ability to improve student academic performance, which is a simple average of the organization, classroom management, reading achievement, and math achievement survey items; and (3) the teacher’s ability to increase student satisfaction, which is a simple average of the role model and student satisfaction survey items. If a teacher was rated by the principal on both the 2003 and 2006 surveys, we use the average of the two ratings.

In previous research using the 2003 principal survey data (see “When Principals Rate Teachers,” research, Spring 2006), we found that principals in the district are usually able to identify the most and least effective teachers in their schools, as measured by their students’ academic progress. However, principals appear to be less successful in differentiating between teachers near the middle of the distribution of teacher effectiveness.

What kinds of parents make requests?

We begin by examining the characteristics of families who make requests. This is important for two reasons. First, our analysis of parent preferences will reflect only the views of those parents who actually made requests, so it is important to understand this group. Second, whether different types of families are more or less likely to make a request has important implications. If high-income parents are more likely to make a request, and such requests are for better teachers on average, then the availability of requests could exacerbate the achievement gap between students from low- and high-income families, even if all families equally value academic achievement.

In this district, families that are not eligible for the federal lunch program are about twice as likely to make a request as those that are eligible: 30 percent of families who are not eligible for free or reduced-price lunch make a request compared with only 13 percent of eligible families. Interestingly, these fractions are nearly identical across schools with very different poverty levels. Thus the socioeconomic makeup of the school does not appear to affect whether parents make a request, although the socioeconomic status of the family does.

We also conducted a more sophisticated analysis that measures the relationship between a family’s demographic characteristics (such as eligibility for free- or reduced-price lunch, median household income of the student’s residential neighborhood, race, and student prior achievement level), a school’s poverty level, and the likelihood that the parent makes a request. These results confirm that, conditional on the characteristics of the family and student, parents in high- and low-poverty schools are about equally likely to make a request. However, parents of low-income students are about 6 percentage points less likely to make a request than parents of high-income students (9 percent vs. 15 percent). Additionally, parents from high-income neighborhoods are about 4 percentage points more likely to make a request than parents from low-income neighborhoods (17 percent vs. 13 percent). Finally, Hispanic parents are significantly less likely to request a particular teacher for their child than are other families in the district.

After taking into account differences in socioeconomic status, we found that parents of higher-achieving students are more likely to make a request, which perhaps reflects greater sophistication or interest on the part of these families. The parents of a student whose performance is 1 standard deviation above the mean are about 8 percentage points more likely to make a request than the parents of an otherwise similar student whose performance is 1 standard deviation below the mean (19 percent vs. 11 percent).

What kinds of teachers do parents request?

In general, parents who make a request exhibit a strong preference for teachers who have received higher overall ratings by the school principal. However, recall that the principals’ survey responses allowed us to construct separate measures of two distinct aspects of teacher quality: the ability to improve student achievement and the ability to provide an enjoyable classroom experience for students. While positively correlated, these two factors appear to reflect distinct characteristics that vary across teachers. Overall, we find that parents value the teacher’s performance on both the student satisfaction and achievement measures, but give more weight to the satisfaction measure.

Even more interesting, however, we find stark differences across schools in the type of teachers that parents tend to request. We find that parents making requests in high-poverty schools place less value on student satisfaction than those in lower-poverty schools. Conversely, parents in high-poverty schools value a teacher’s ability to improve student achievement considerably more than parents in lower-poverty schools.

On the other hand, within a school, a family’s own socioeconomic status is uncorrelated with the type of teacher a parent requests. That is, both more- and less-advantaged parents in low-income schools tend to request teachers that are rated highly in terms of their ability to improve student achievement. In contrast, parents from all backgrounds in higher-income schools tend to request teachers who are rated more highly in terms of their ability to improve student satisfaction. When we control for the socioeconomic status of both the student and school, our findings are the same: student characteristics are not related to the type of teachers that parents prefer, while school characteristics are strongly related to parental preferences for teachers.

To quantify these differences, we used our results to simulate parent choices (see Figure 1). For the sake of simplicity, we first consider a situation in which a parent can choose between two teachers: one teacher has an average rating for both achievement and satisfaction; the other teacher has an average rating for achievement, but a high rating on the satisfaction measure (i.e., a rating 1 standard deviation above the mean). We calculate the percentage of parents with average background characteristics who would choose the high-satisfaction teacher. Next, we change one characteristic of either the parent or school and calculate how this change would affect the percent of parents who would choose the high-satisfaction teacher.

In a school where 80 percent of the children are eligible for free or reduced-price lunch, the parents of the average child would have a 48 percent chance of selecting the teacher with a high-satisfaction and average achievement rating over the teacher with average ratings on both satisfaction and achievement. In other words, these parents are no more likely to choose the high-satisfaction teacher than if they had randomly chosen which teacher to request. In contrast, if the child attends a school where only 20 percent of the students are eligible for free or reduced-price lunch, there would be a 65 percent probability that their parents would select the high-satisfaction teacher. The 17 percentage point difference is large and statistically significant.

We then consider the scenario where the choice is between two teachers who have the same satisfaction rating but different achievement ratings, and see the opposite result. Parents in the lower-poverty school are no more likely than they would be by chance to select the teacher with a high achievement rating (51 percent), whereas parents in the higher-poverty school would choose the teacher with a higher achievement rating 62 percent of the time. Again, the difference of 11 percentage points is statistically significant.

As one might expect, parents of kindergarten children appear to value satisfaction more and academics less than other parents, though this difference is small and bordering on statistical insignificance. Grade level is otherwise unrelated to preferences for teacher attributes.

Parent requests and classroom effectiveness

It is important to emphasize that the results presented above reflect both what parents observe and what they value. To the extent that parents have less information on a particular teacher characteristic, our findings may underestimate parent preferences for this characteristic. In particular, one might be concerned that parents do not have accurate information on teachers’ ability to raise student achievement. For this reason, we focus primarily on information from the principal survey, which likely reflects teacher behaviors or qualities that parents might learn from observing the teacher’s classroom or speaking with friends and neighbors who have had experience with the teacher in the past.

To test the sensitivity of our results to this methodological decision, we constructed a value-added indicator that measures a teacher’s contribution to student achievement (accounting for a wide variety of student and classroom characteristics that could affect achievement independent of the teacher’s ability). We find that teachers who perform better on our value-added measure also receive more parent requests, even after controlling for the student satisfaction measure from the principal surveys. However, when we also control for the principal-reported academic measure, this relationship is no longer significant, although the relationships between parent requests and both principal-reported measures remain positive and significant. These results suggest either that the academic considerations parents value are better captured by principal ratings or that parents have difficulty observing how much value a teacher adds to reading and math test scores.

An explanation?

The results presented above suggest that parents in low-income schools strongly value student achievement and are essentially indifferent to a teacher’s ability to promote student satisfaction. The results are reversed for families in higher-income schools. At the same time, we find that parent preferences within schools are identical across several measures of family socioeconomic status. How should we interpret these results?

One possible explanation emphasizes the role of school context in the educational process, particularly the interaction between parents, schools, and students. In this view, high- and low-income parents have similar preferences for student outcomes, but face constraints that are correlated with school demographics. Because academic resources are relatively scarce in higher-poverty schools (e.g., there are more disruptive peers, lower academic expectations, fewer financial resources, and less-competent teachers), parents in these schools seek teachers skilled at improving achievement even if this comes at the cost of student satisfaction.

If this explanation were true, we would expect to find a positive association between school-level income and school-level academic inputs, and a negative association between school-level income and the differences in the value-added by teachers within the same school. The second prediction is simply a consequence of diminishing returns to academic inputs. More specifically, if the average quality of teachers in a school is already high, being assigned to one of the better teachers will have only a limited effect on student achievement.

To what extent are these predictions borne out in the data? A comparison of observable teacher characteristics across schools provides some support for the first prediction. As in most other school districts, the teachers in higher-poverty schools in our sample have fewer years of experience than their counterparts in lower-poverty schools (11.8 years vs. 14.0 years). In comparison to their counterparts, teachers in higher-poverty schools are less likely to have credits beyond a bachelor’s degree (66 percent vs. 78 percent) and are less likely to have attended the most prestigious local university (75 percent vs. 80 percent) for their undergraduate degree. In addition, the variance of our value-added measure is significantly higher within higher-poverty schools than in lower-poverty schools, even after we control for the experience level and other observable characteristics of teachers within each school, which supports the second prediction. Hence, while certainly not conclusive, the available evidence is consistent with the explanation offered above.

Conclusions

Our findings suggest that what parents want from school depends on the educational context in which they find themselves. In particular, in low-income schools where academic resources are scarce, motivated parents are more likely to choose teachers based on their perceived ability to improve academic achievement. On the other hand, in higher-income schools these parents seem to respond to the relative abundance of academic resources by seeking out teachers who also increase student satisfaction. This may reflect a parental preference for their children to enjoy school, or it might reflect parental preferences for teachers who emphasize academic facets that increase student satisfaction but are not captured by standardized test scores, such as critical thinking or curiosity.

In considering the policy implications of this research, it is important to recognize that our analysis reflects parent decisions conditional on school choice. In principle, students in this district can attend any school, although in practice the vast majority of students simply attend their neighborhood school. Because the school choice decision is quite different from the teacher choice decision, our findings do not map directly onto the school choice debate. However, the results represented here do inform other policy issues. For example, they suggest that the parents of low-income, minority, and low-achieving children are much less likely to take advantage of informal opportunities to exercise choice from among teachers. This highlights the potential adverse impacts of honoring parental requests on the equitable distribution of education resources. Our results also suggest that different socioeconomic groups are likely to react quite differently to accountability policies, such as those embodied in No Child Left Behind. In more affluent schools, parents are likely to oppose measures that increase the focus on standardized test scores at the cost of student satisfaction. More generally, programs that increase the focus on basic skills or classroom management at the expense of student enjoyment or other academic facets not measured on standardized tests are likely to be unpopular in more affluent schools.

Brian Jacob is professor of education policy and economics at theGerald R. Ford School of Public Policy at the University of Michigan.Lars Lefgren is assistant professor of economics, Brigham YoungUniversity. This article summarizes research that will be publishedin a forthcoming article in the Quarterly Journal of Economics.

We address two main questions about charter schools in the city. First, who enrolls in New York City’s charter schools? And, second, how well are the schools educating students? What we found is that, compared with other students in the traditional public schools, charter school applicants are more likely to be black and poor but are otherwise fairly similar. We also found that charter school students benefit academically from their charter school education. Charter school students in grades 3 through 8 perform better than we would expect, based on the performance of comparable students in traditional public schools, on both the math and reading portions of New York’s statewide achievement tests. There is not yet a sufficient number of charter school students in grades 9 through 12 for us to report achievement effects for this group.

Data Collection

Forty-seven charter schools were operating in New York City in the 2005–06 school year, the most recent for which we have test-score results, and all but five are included in the analysis presented here. Two schools, Manhattan Charter School and South Bronx Charter School for International Cultures and the Arts, are participating in our ongoing study but are not included in the analysis because they do not yet have any students in test-taking grades. One school, ReadNet Bronx Charter School, was in the process of closing in 2005–06. The absence of ReadNet Bronx from our evaluation is likely to have only a small impact on our assessment of student achievement because the school had only two years of test-taking students before it closed. The New York Center for Autism Charter School is not included in the study because it serves a very special population and is not compatible with many elements of the study. The United Federation of Teachers Elementary Charter School has declined to participate in the study so far, but it does not yet have any students in test-taking grades.

Charter schools must advertise their availability to all students eligible to attend public schools and are not allowed to select their students from among applicants. Instead, if a charter school in New York receives more applicants than it has places, it must enroll students based on a random lottery. Each spring, charter schools that are oversubscribed hold admissions lotteries.

Our study data are collected as follows: First, the information from each charter school application is sent to the New York City Department of Education for inclusion in its administrative database. This database contains entries for all students who attend New York City’s traditional public schools and for all students who attend New York City’s charter schools. A contractor for the department uses the maximum amount of information possible—for example, the student’s name, birth date, and Social Security number, if available—to match each applicant to a corresponding existing entry in the department’s database. The contractor then extracts information on each student’s demographic characteristics, enrollment, test scores, and certification for and participation in various programs such as free and reduced-price lunch, special education, and English-language services. This information is gathered from both the years before and the years after the application to a charter school and sent to us with an encrypted student identification number.

We first obtained application data on the lottery conducted in the spring of 2005 for the 2005–06 school year, and we requested application data from earlier years as well. Not all schools had archived this information or had requested all of the elements that would prove helpful in matching up their applicants. The 2005–06 application data therefore have the most complete coverage of schools and the most information on which to match. In order to be as representative as possible, the analysis of the characteristics of charter school applicants described below is based on the data from that year. In our achievement analysis, however, we use data from all lotteries for which we have application data.

The Applicants

Who applies to New York City’s charter schools? In answering this question, it is important to recognize that the charter schools are located in neighborhoods that are substantially poorer than and almost twice as black as the average New York City neighborhood. Charter school neighborhoods contain only one-third as many whites and Asians as the average New York City neighborhood. In fact, it is no exaggeration to say that if the charter schools draw from their neighborhoods, they will draw students who are 90 to 95 percent black or Hispanic. The charter schools are thus in a situation that people sometimes find confusing. Normally, if we say that a traditional public school is “more black” or “more Hispanic,” we mean to imply that the school has fewer white students. However, for New York City’s charter schools, “more black” or “more Hispanic” cannot imply “less white” because there are hardly any whites (or Asians) to be displaced. Instead, when we say a New York City charter school is “more black” than surrounding schools, it is automatically “less Hispanic” (and vice versa). Any school that disproportionately serves black students will disproportionately not serve Hispanic students. These are not two independent comments: they are the same comment!

As one might predict based on their neighborhoods, applicants to New York City’s charter schools are twice as likely to be black (64 percent versus 32 percent) and much less likely to be white or Asian (7 percent versus 28 percent) than the average public school student in New York City. Because charter school students are disproportionately likely to be black, they are somewhat less likely to be Hispanic (27 percent versus 39 percent). About half of charter school applicants are female, just like students in the traditional public schools (see Figure 2).

There is no simple explanation for the disproportionate appeal of charter schools to blacks. While a couple of charter schools—Harriet Tubman and Sisulu-Walker—are named after a black person, most of the charter schools, not a few, disproportionately draw black students. Nor does the explanation seem to involve strong language barriers for Hispanics. Traditional public schools and charter schools located in areas with significant Hispanic populations provide the same level of Spanish-language translation for school materials. In both sets of schools, key materials, such as applications, school calendars, and school descriptions are usually available in Spanish. A more complex story is needed. For instance, black parents may feel more comfortable “disagreeing” with their regular school assignment than Hispanic parents do, particularly if the parents in question are recent immigrants.

A common proxy for poverty is a student being certified to receive a free or reduced-price lunch. (To get certified, a student’s household income must be less than 185 percent of the federal poverty line.) Using this proxy, we find that the applicants to charter schools are much more likely to be poor than is the average New York City student (93 percent versus 74 percent).

Unfortunately, charter schools and regular public schools have some information recorded differently in the New York City database, and these differences cause charter schools’ numbers of special education and English language learner students to be understated. Nevertheless, the data that we have suggest that, at the time they applied, 11.1 percent of charter school applicants were participating in special education. This is about the same percentage as in the New York City schools overall (12.5 percent). The data we have also suggest that, at the time they applied, 4.2 percent of charter school applicants were classified as English language learners, while 13.6 percent of New York City’s students were classified as such. Because of our concerns about the differences in the recording of English proficiency status, we cannot draw the conclusion that charter schools appeal disproportionately to students who are proficient in English. But the fact that charter schools appeal disproportionately to black students is probably reflected in applicants being more likely to be English speakers.

We do not have good data that would help answer the question of whether charter schools disproportionately draw high or low achievers. Because most students enter charter schools before the 3rd grade when state-mandated testing begins, only 36 percent of applicants in our study have prior test scores on record and this group is not representative of all applicants.

Student Achievement

The basic strategy we use to evaluate the effect of charter schools on student achievement is to compare students who are awarded a seat in a charter school through a lottery with students who enter the lottery but are not awarded a seat. About 91 percent of all charter school applicants participated in lotteries. The random assignment to the two separate groups of students who are otherwise similar—in their measured characteristics and the fact that they expressed a desire to attend a charter school—enables us to isolate the impact of attending a charter school.

We first wanted to confirm that the two groups contained similar students. As expected, when we compared students who were awarded a seat in a charter school to those who were not, we found no statistically significant differences on any of the demographic or predetermined program eligibility characteristics we could measure.

We use common statistical procedures to estimate the effect on math and reading test scores of each additional year of actual attendance at a charter school. Our results therefore reflect the performance of students who, if offered a seat in a charter school, choose to enroll—that is, those who comply with the experimental treatment. In some applications, having an estimate of a program effect that is valid only for compliers is problematic, because it would be useful to know what would happen if the program were expanded to other populations. In the case of charter schools, however, an estimate of their effect on students who enroll is exactly what we want, as the basic idea behind charter school reform is that only students who want to should attend them. Our present approach also assumes that each year of charter schooling has the same effect on student achievement. When we investigated whether each year of attendance at a charter school had a different effect, we found no evidence to support the idea of different effects in different years. However, we plan to return to the question in subsequent analyses when we will have more variation in the number of years students attend charter schools.

We use test-score data from the years 2000–01 to 2005–06 from the 36 charter schools that enroll students in grades 3 through 12. However, because the number of students in grades 9 through 12 is too small to produce statistically significant results at this time, our discussion will focus on the results for the 32 schools that enrolled 3rd through 8th graders in the relevant years. For them, the number of test-score observations included in the analysis ranges from almost 7,800 in grade 5 to 3,000 in grade 8.

We first present our results in the way most often used by researchers: standard scores. These scores, which are generated by dividing a scale score by its standard deviation, are helpful because they allow researchers to compare the effects of charter schools to the effects of other interventions, like class-size reductions. Our results indicate that, on average, New York City’s charter schools raise their 3rd through 8th graders’ math achievement by 0.09 of a standard score and reading achievement by 0.04 of a standard score, compared with what would have happened had they remained in traditional public schools (see Figure 3). We find no evidence that the improvement in achievement differs between boys and girls or between blacks and Hispanics.

To put these results in context, consider the Tennessee STAR Experiment, which produced some of the literature’s highest estimated effects for class-size reduction. The Tennessee experiment suggested that a 10 percent reduction in class size in grades K–3 raised students’ standard scores by 0.06. Furthermore, this was a one-time effect: even if students stayed in smaller classes for multiple years, their achievement rose only once, by 0.06. In contrast, the average charter school student improved by 0.09 in math and 0.04 in reading for each year of charter school attendance.

Another way to present the results is in terms of New York State’s performance levels. In 2005–06, depending on the grade, a student’s math scale score had to rise by an average of 32 points to go from the top of the Performance Level 1 range (“failing” or not meeting learning standards) to the bottom of the Performance Level 3 range (“proficient” or meeting learning standards). The equivalent required rise in a student’s reading score was 44 points.

We estimate that, depending on the grade, students’ math scale scores rise by 3.75 to 3.98 points and their reading scale scores rise by 1.53 to 1.61 points for every year they spend in charter schools. Again, these improvements are measured relative to what would have happened to the same students in traditional public schools. Another way to think about these gains is to understand that, for every year they spend in a charter school, students make up 12 percent of the distance from failing to proficient in math. They make up 3.5 percent of the distance from failing to proficient in reading.

There are several possible explanations for the effects of charter schools being larger in math than in reading. The most likely explanation, we believe, is that schools largely control math education, but that both families and schools exert strong influence over reading skills. If, for instance, the families of students who were and were not awarded a seat through a lottery had the same effect on reading and families controlled half the gains in reading, then the difference between the estimated math and reading effects would be fully explained.

Keep in mind, these annual gains are relative to whatever gains the students would have been expected to make in the traditional public schools had they not been awarded a seat through the lottery. Because most of the students in our study have been attending a charter school for between one and three years and no student has attended for more than six years, we are uncomfortable extrapolating our finding beyond four years of enrollment in a charter school.

We also estimated a separate effect on achievement for each of the 32 charter schools with students in grades 3 through 8. The results for about one-third of these schools are very imprecise, usually because they had very few students in test-taking grades during the analysis years. Based on the remaining schools for which we have reasonably precise estimates, however, we found a good deal of variation in achievement effects. About 19 percent of charter school students attend a school that is estimated to have a positive effect on math that is very large: greater than 0.3 of a standard score per year. Another 56 percent attend a school that is estimated to have a positive effect that is large: between 0.1 and 0.3 of a standard score. 18 percent attend a school with a positive but small to moderate effect. Only 6 percent attend a school that is estimated to have a negative effect on math, and these estimated effects are all small. The effects on reading are similarly distributed across a range, with 80 percent being positive and only 8 percent being negative.

School Policies

The variation in achievement effects among charter schools raises the question of whether one can identify specific policies that are associated with charter school success (see sidebar, “New York City Charter Basics”). To provide hints at possible answers, we conducted some preliminary analysis on the question using the math and reading results from the 32 schools that enrolled elementary and middle school students.

We want to be clear that our analysis cannot establish definitively whether the policies of charter schools cause changes in student achievement. We can describe only associations between policies and achievement effects, and the distinction between association and causation is very important in practice in the charter school context. Charter schools may adopt policies for reasons that we do not observe and it may be that it is these unobserved reasons that actually affect achievement. For instance, suppose that charismatic school leaders were a key cause of positive achievement effects, and suppose that charismatic leaders just happened to like long school years. We cannot measure charisma, but we can measure the length of the school year. Therefore, we might find an association between a long school year and positive achievement effects, even if the charisma, and not the long school year, caused higher achievement. A school that lengthened its school year would be disappointed in the results, not realizing that what it had really needed to do was to hire a charismatic leader.

That caution given, there are a few clear and interesting associations to be noted. We find no relationship between how long a charter school has been in operation and student achievement after controlling for school policies. However, if we do not control for school policies and look at the simple correlation between a charter school’s years in operation and student achievement, we find that older schools have more positive achievement effects. The fact that this correlation disappears when we include such policies in our analysis suggests that the reason older schools have more positive achievement effects is that they adopt more effective policies.

A long school year is associated with positive achievement effects, and we estimate that schools with years that are 10 days longer are associated with average student achievement that is 0.2 standard deviations greater. This is a large effect, and a 10-day difference among school calendars is quite common. In fact, 12 days is the standard deviation in the length of the school year among charter schools. We should note, however, that a long school year tends to go part and parcel with several other policies, such as a longer school day and Saturday school, and this should make us cautious about assigning too much importance to a longer school year in and of itself. A more conservative conclusion would be to think of the package of the three policies having a positive association with student achievement.

We also find that class size, optional afterschool programs, and most math and reading curricula seem to have no relationship to student achievement. Everyday Math and Open Court reading curricula did have negative and statistically significant associations with achievement effects. We discourage readers from interpreting these as causal effects, however, since an equally plausible interpretation is that these are curricula that schools adopt when their students are struggling.

New York City Charter Basics New York City has three charter school authorizers. Of the schools covered in this report, the State University of New York authorized 20 , the chancellor of the New York City schools authorized 19, and the New York State Board of Regents authorized 3. Three types of organizations operate charter schools in New York City: nonprofit community-grown organizations (CGOs), nonprofit charter management organizations (CMOs), and for-profit education management organizations (EMOs). CMOs and EMOs are formal organizations that exist to manage charter schools, and they function somewhat like firms that have a strong brand and that establish fairly independent branches or franchises (see “Brand-Name Charters,” features). CMOs and EMOs typically make overarching curricular and policy decisions, conduct back-office activities, and provide something of a career ladder for teachers and administrators within their network of schools. The CMO with the most schools in New York City in 2005–06 was the KIPP Foundation, and the EMO with the most schools was Victory Schools. CGO schools may be founded by a group of parents, a group of teachers, a community organization that provides local social services, one or more philanthropists, or the teachers union. More often than not, the founding group combines people from a few of the groups listed above. Fifty-six percent of the charter school students covered by this report attend 23 schools operated by CGOs; 19 percent attend 12 schools that are affiliated with CMOs; and 25 percent attend 7 schools run by EMOs. As these percentages suggest, the average school operated by an EMO has considerably larger enrollment than the average school operated by a CGO or a CMO. Missions and Policies Every charter school describes itself in a carefully crafted mission statement that sets out its vision, educational philosophy, and focus. Based on these statements, we can categorize the schools roughly into five groups: those that have a child-centered or progressive educational philosophy and typically seek to develop students’ love of learning, respect for others, and creativity (29 percent of students); those with a general or traditional educational mission and a focus on students’ core skills (28 percent of students); those with a rigorous academic emphasis, which have mission statements that focus almost exclusively on academic goals such as excelling in school and going to college (25 percent of students); those that target a particular population of students, such as low-income students, special needs students, likely dropouts, male students, and female students (11 percent of students); and those in which a certain aspect of the curriculum, such as science or the arts, is paramount (7 percent of students). There are a number of reasons to expect that charter schools will choose different policies and practices: They are independent and fairly autonomous. Their operating agencies have a variety of histories and priorities. All are young schools and more likely to experiment with new policies than are established schools. At the same time, there are reasons to think that New York City’s charter schools will share certain policies. They commonly serve disadvantaged students; they are all under pressure to attract parents and to satisfy a small number of authorizers; one school may deliberately imitate another by adopting a policy that seems to be working in the other school; schools may also imitate one another unconsciously (as when teachers who have worked at one school are hired by another and bring their knowledge with them). The common characteristics of charter schools reveal which innovations seem most promising to urban school leaders empowered to set their own policies (see Figure 4). About 64 percent of students attend a charter school with a school year of 190 days or longer, and 20 percent attend a school with a school year of 200 days or longer. By way of comparison, the modal school year in the United States is 180 days or 36 weeks. About 55 percent of students attend a charter school with a day that lasts eight hours or longer, 67 percent attend one with an optional afterschool program, and about 57 percent attend one with Saturday school that is mandatory for all or at least some students (for instance, students who are struggling academically). About 49 percent of students attend a charter school that has a system of bonuses for successful teachers, and 17 percent of students attend a charter school whose teachers are unionized. Most of the students in charter schools whose teachers are unionized attend one of the five charter schools that were formerly traditional public schools but converted to charter status. In addition, about 91 percent of charter school students attend schools that require uniforms, and about 95 percent attend schools that voluntarily administer standardized exams on a regular basis for diagnostic purposes. The advisory system is used by nearly all the charter schools that serve middle or high school grades. In an advisory system, a teacher or pair of teachers is assigned to a group of students for an entire school year. Teachers meet frequently (often daily) with their students and are responsible for tracking their progress and preventing them from “falling through the cracks.” Because students in elementary grades are assigned to one teacher for most of the school day, advisory systems would be duplicative and are therefore not used by elementary schools. About 52 percent of students attend charter schools that ask their parents to sign “contracts.” Because these contracts are not enforceable, it is best to think of them as a method of trying to ensure that parents know about the school’s policies and expectations. Some parents may also feel morally bound to abide by the contract. Just over half the students attend a charter school that reserves one or more seats on its board for parents. About 21 percent attend one with a disciplinary policy that fits the “no broken windows” school of thinking, which holds that encouraging small courtesies and punishing small infractions (usually at the classroom level) are important. This is in contrast to disciplinary strategies that focus more on preventing or punishing large infractions (often at an administrative level above the classroom). The charter schools employ a variety of math and reading curricula, with no curriculum being dominant. The most popular are Saxon Math (41 percent of students) and Core Knowledge (38 percent of students.) Fifty-four percent of students have an extended English or language arts period of 90 minutes or more, and the same percentage have an extended math period. While the Children First initiative in New York City mandates a daily “literacy block” of 90 minutes for elementary school grades, the city requires that traditional public elementary schools have between 60 and 75 minutes of math instruction daily, depending on the grade.

Conclusion

In sum, in the largest lottery-based evaluation of charter schools to date, we find that charter schools in New York City are having positive effects on the academic progress of the students who attend them. These effects are largest in charter schools that have extended the length of the school year, though we cannot establish definitively that this is the reason for their exceptional performance. We also find that the students applying to charter schools in New York City are more likely to be black and eligible for a free or reduced-price lunch program than students in the public schools in the district.

While it is reasonable to extrapolate the findings to other urban students who are similar to New York City applicants, we would argue against these results being applied to students who differ substantially from applicants to the charter schools. In particular, the results should not be applied to students who are substantially more advantaged or to students who would not be interested in applying to the types of charter schools available in New York City, even if they were conveniently located in the students’ area.

That said, our results provide a strong basis for recommending the continued expansion of charter schooling in the Big Apple and in other large cities with similar student populations.

Caroline M. Hoxby is professor of economics at Stanford University and director of the Economics of Education program at the National Bureau of Economic Research. Sonali Murarka is a project manager at the National Bureau of Economic Research. They are, respectively, principal investigator and project manager of the New York City Charter Schools Evaluation Project.

Not surprisingly, school board members agree that one of their most important goals is to help students learn. According to a 2002 national survey, student achievement ranks second only to financial concerns as school board members’ highest priority. We wondered, though, do voters hold school board members accountable for the academic performance of the schools they oversee? Do they support sitting board members when published student test scores rise? Do they vote against members when schools and students struggle under their watch?

Existing accountability policies assume that they do: states shine light on school performance by providing the public with achievement data. Voters and parents are expected to make use of these data in choosing school districts or schools, and to hold administrators and school board members accountable for the schools’ performance at each election. The idea is that voters will replace incumbents with new members when performance is poor and support incumbents over challengers when performance is strong. Indeed, there are very few other ways in which district officials can be held accountable for school performance. Neither the federal No Child Left Behind Act (NCLB) nor the states impose direct sanctions on members of school boards that oversee large numbers of underperforming schools.

Our questions led us to undertake the first large-scale study of how voters and candidates respond to student learning trends in school board elections. We analyzed test-score data and election results from 499 races over three election cycles in South Carolina to study whether voters punish and reward incumbent school board members on the basis of changes in student learning, as measured by standardized tests, in district schools. In addition, we assessed the impact of school performance on incumbents’ decisions to seek reelection and potential challengers’ decisions to join the race.

We found that in the 2000 elections, South Carolina voters did appear to evaluate school board members on the basis of student learning. Yet in the 2002 and 2004 elections, published test scores did not influence incumbents’ electoral fortunes. As we’ll see, the possible reasons our results differed so dramatically from one time period to the next hold important implications for the design of school accountability policies. But let’s first take a closer look at our methods and findings.

South Carolina

Once we set out to study local school board races, we encountered tall hurdles to obtaining election results. Only one state, South Carolina, centrally collects precinct-level election data for school board races. In all other states, obtaining precinct-level election results requires gathering and organizing election returns from hundreds of individual counties and election districts.

So we took a close look at South Carolina. In most respects, South Carolina elections and school boards are similar to those across the rest of the country. All but 4 of the state’s 46 counties hold nonpartisan school board elections. Approximately 80 percent of school board members receive some compensation, either a salary, per diem payments, or reimbursement for their expenses. Over 90 percent of South Carolina’s 85 school boards have between 5 and 9 members, while the largest board has 11. And, as is common practice in other states, nearly 9 out of 10 South Carolina school districts hold board elections during the general election in November.

Perhaps the most important difference between South Carolina and most other states when it comes to local school politics is the role played by the state’s teachers unions, which are among the weakest in the country. In other states strong teachers unions may mobilize high turnout among members, their families, and friends, and punish and reward board members for their treatment of teachers rather than hold them accountable for student test scores. South Carolina school boards are unlikely to be beholden to the unions, which should make the boards more responsive to the broader public.

Roughly half of the state’s 85 districts hold school board elections in any two-year election cycle. We collected precinct-level election returns for all school board races in three election cycles, 2000, 2002, and 2004. We also obtained school-level student achievement data from the South Carolina Department of Education. We began our analysis with 2000 because it was the first cycle of elections after South Carolina started administering the Palmetto Achievement Challenge Test (PACT) to students in grades 3 to 8 in 1999. These tests, based on the South Carolina Curriculum and Standards, are given in both reading and math. We averaged the reading and math percentile scores to produce a composite score for each school. Because we wanted to examine whether voters are more concerned with student performance districtwide or in their local neighborhood, we computed two measures of average school performance to include in our analysis. The first is the average test score for each district. The second is the average test score for the public school that is located closest to an election precinct.

Searching for Accountability

We began our analysis by comparing the vote shares of incumbent school board members who ran and faced an opponent with the test-score performance of the schools and districts they represented. We were careful to separate the effect of school performance from the effects of other factors that could reasonably influence an incumbent school board member’s vote share. For example, we considered whether voters evaluate student outcomes relative to spending by measuring the effect of changes in the district’s property tax rate. We also took into account features of the election, including whether it was held as part of the November general election or on another date, when turnout is likely to be lower. Additionally, we accounted for the partisanship of the electorate, measured by the Democratic candidate’s share of the presidential vote, and demographic characteristics, such as race, age, and gender. We also adjusted for potential differences in how voters from precincts with higher and lower average test scores respond to changes in test scores. For example, voters from precincts with lower test scores might respond more strongly when test scores improve than do voters from precincts with test scores that already were very high.

In 2000, 67 incumbents from 37 school boards ran for reelection in contested races in South Carolina. Of these 67 incumbents, 50 were reelected, and the median vote share for all incumbents in competitive races was 58 percent.

We found that incumbent school board members won a larger share of the total vote in a precinct when test scores in that precinct improved. We estimate that improvement from the 25th to the 75th percentile of test-score change—that is, moving from a loss of 4 percentile points to a gain of 3.8 percentile points between 1999 and 2000—produced on average an increase of 3 percentage points in an incumbent’s vote share. If precinct test scores dropped from the 75th to the 25th percentile of test-score change, the associated 3-percentage-point decrease in an incumbent’s vote share could substantially erode an incumbent’s margin of victory. In districts where percentile scores had increased in the year preceding the election, incumbents won 81 percent of the time in competitive elections; in districts where scores had declined, incumbents won only 69 percent of the time.

Citizens therefore did seem to base their assessment of incumbents on changes in test-score performance during a board member’s tenure, exactly the type of accountability many supporters of NCLB had hoped for.

We were interested to find that the average school test score for the precinct, rather than the district, had a significant effect on an incumbent’s vote share. The significant relationship with precinct test scores and the absence of a relationship with district scores suggests that voters were more concerned with school performance within their immediate neighborhood than across the district.

The Later Elections

With the evidence from 2000 in hand, we were initially surprised that all indications of a relationship between school performance and an incumbent school board member’s vote share vanished after the passage of NCLB in 2002.

We reanalyzed the data in a number of different ways, but were unable to find any indication that voters cast their ballots based on changes in test scores. We included administrative data from teacher, parent, and student ratings of local schools; we considered the potential relationship between vote share and test-score changes over the previous two or three years; we examined the deviation of precinct test scores from district means; we looked at changes in the percentage of students who received failing scores on the PACT; we evaluated the relationship between vote share and the percentage change in the percentile scores rather than the raw percentile point changes; and we turned to alternative measures of student achievement, such as SAT scores, exit exams, and graduation rates. None of these approaches yielded clear evidence of a link between school performance and voter behavior in school board elections.

Even when we estimated the probability that an incumbent won a majority of the votes in each precinct, or accounted for test-score changes and levels as a function of dollars spent on students, or measured the relationship between an incumbent’s vote share in one election and the previous election, the overwhelming weight of the evidence indicated that school board members were not being judged on improvement or weakening in school test scores.

Strategic Politicians

So far, we’ve discussed the experience of incumbents who ran against an opponent. Many incumbents, however, either did not run for reelection or ran unopposed. For example, in 2000, 42 of the 157 sitting board members in 39 school districts who were up for reelection did not run for office. Among the remaining 112 who sought to retain their seats, more than one-third, 45, did not face a challenger. The 67 incumbents who ran opposed in 2000 represented less than half of the sitting board members whose seats were in play that election.

School performance as measured by test scores may have helped determine which candidates sought reelection and which faced a challenger. If board members and potential challengers anticipate that voters will punish incumbents for poor school performance, declining test scores may lead board members to retire rather than endure defeat. A drop in test scores may also encourage opponents to run for office, either because they believe that incumbents are now vulnerable to defeat or because disgruntled citizens feel compelled to run for office when schools perform poorly.

Although exact election filing dates vary by school district, most candidates for seats on South Carolina’s school boards must decide whether to run by mid-September for a November election. PACT scores, however, are typically released to the public in late September or early October. Incumbents and potential challengers may not know the exact size of precinct or district test-score changes, but they could very well have impressions of the direction and rate of student learning trends. School board members and some challengers have observed the schools firsthand and have listened to accounts from principals and teachers. By monitoring the coverage of education issues on local television and in the print media, candidates may also have a sense of the extent to which voters are likely to use student test-score performance to evaluate candidates. And although we do not know this with any certainty, it is possible that school board members have access to test-score results before they are released to the public.

We decided to assess the relationship between test-score trends and incumbents’ decisions to run for reelection, and then to estimate the effect of test-score trends on the probability that an incumbent who runs faces an opponent. Our basic approach in this analysis was to compare the probability of running (or running and facing a challenger) between incumbents who oversaw districts with stronger and weaker year-over-year test scores. Because candidates either run for election in every precinct or do not run at all, we focused only on district test scores. As with our analysis of the relationship between test scores and vote share, we accounted for a number of factors that could reasonably influence a candidate’s decision to run for office. These included the incumbent’s vote share in the previous election, which might serve as a signal of the likelihood of victory to both the incumbent and potential challengers, and whether board members received compensation for their service, under the assumption that paid positions would be more attractive.

Our results indicate that incumbents may bow out in anticipation of being held accountable for poor test-score performance by schools in their district. During the 2000 election, incumbents were less likely to seek reelection when their district’s test scores declined over the preceding school year. If a district experienced a drop from the 75th to the 25th percentile of test-score change, our results lead us to expect that incumbents will be 13 percentage points less likely to run for reelection. In fact, 76 percent of incumbents sought reelection in districts with improving test scores; in districts with falling scores, only 66 percent did. The results did not hold for the later elections. Just as we found no evidence in the 2002 and 2004 elections that a large block of voters held incumbents accountable for poor test scores, we failed to find any indication that incumbents in 2002 and 2004 based their decisions about running for reelection on student learning trends.

When we looked at the behavior of the challengers, we once again saw evidence of their responding to test scores during the 2000 election, but no indication in 2002 or 2004 (see Figure 1). In 2000, a drop in test scores within the district significantly increased the likelihood an incumbent would face a challenger. If a district’s test-score change fell in the 25th rather than the 75th percentile, we estimate that an incumbent experienced an 18-percentage-point increase in the probability of facing a challenger. On the ground, the data show that 74 percent of incumbents who ran for reelection in districts with declining scores faced a challenger; in districts with improving scores, only 49 percent of incumbents faced a challenger.

What Happened in 2000?

Why did voters, incumbents, and potential challengers care about test scores in 2000 but not in 2002, or in 2004? The most likely explanation involves changes in media coverage of education issues. The amount and content of media coverage of student test scores differed substantially between 2000 and the latter two election years.

The 2000 elections were the first to follow the passage of the state’s accountability system. Journalists devoted ample space to issues that either directly or indirectly concerned student learning trends. Charleston’s Post and Courier, the Herald in Rock Hill, Columbia’s The State, and the Associated Press State & Local Wire, which serves numerous other South Carolina papers, regularly carried stories about the state of South Carolina’s schools. Both incumbents and challengers frequently identified student achievement generally, and test scores in particular, as the single most important issue in the 2000 school board election. Newspaper editorials that endorsed candidates in the 2000 election regularly underscored ways in which individual incumbents and challengers did, or said they would, improve student achievement. And 45 percent of the newspaper articles about school board races in the two months prior to the election mentioned student test scores.

In the 2002 and 2004 elections, however, media coverage shifted to other issues, such as the closing of schools, the racial composition of schools and boards, disciplinary problems, and sports programs. In these years, only 30 and 34 percent of articles, respectively, touched on test scores. The decline in media attention leads us to suspect that concerns about student learning trends probably did not stand at the forefront of voters’ or candidates’ thinking in the 2002 and 2004 elections.

The tone of articles about the state’s accountability system also shifted drastically during the 2002 and 2004 election cycles. From 1998 to 2000, most stories adopted a fairly neutral tone, introducing the public to the new accountability system and offering tepid praise and criticism of the testing regimen. After the 2000 election, journalists portrayed considerably more skepticism in their coverage of student achievement trends. Reporters devoted stories to errors in PACT’s scoring, security breaches in school testing, flaws in the science and social studies portions of PACT, district efforts to get ahead by changing their test dates, confusion regarding the comparability of test scores over time, missing PACT scores, and conflicts between school evaluations under the state and national accountability systems.

At the same time that administrative irregularities and mishaps attracted public scrutiny, teachers, district officials, and various other interest groups began to challenge the value of standardized tests more generally. One 3rd-grade teacher was quoted as saying, “These tests cannot and never will truly measure what a child actually knows, how a child sees the world, what a child genuinely understands and grasps, and what kind of life that child lives outside the school walls.” A school district associate superintendent claimed, “The problem with PACT is it doesn’t tell you what your child knows and doesn’t know.” The Palmetto State Teachers Association questioned the value of the state’s testing regimen, noting on its web site, “The current statewide tests do not provide immediate diagnostic information needed to improve student achievement or provide information to help teachers plan to meet the needs of each student. The testing process is time consuming, and spending weeks on high-stake testing is NOT in the best interest of children.” And as Andrew HaLevi, the Charlestown County School District 2000 Teacher of the Year, wrote in a 2001 op-ed for the Post and Courier, “The PACT needs to be seen for what it is: a vehicle for politicians to say that they are tough on education (and educators). This may make for good politics, but it makes for bad educational policy.” Reacting to the rising criticisms directed toward PACT, voters may have grown disenchanted with the state’s accountability system and removed test-score performance from among the criteria on which they evaluated school board candidates.

There are, of course, several other plausible explanations for why South Carolinians voted based on test score performance in 2000 but not in 2002 and 2004. The timing of the public release of the test scores is one. The 2000 scores were released in late October, whereas scores in 2002 and 2004 were released in early October and early September, respectively. In 2000, the release of scores so close to the election date and the media coverage that followed may have primed voters to evaluate candidates on student test scores. In the other two election years, the gap of a month or two between the release of scores and election day may have allowed the issue of test scores to fade from voters’ minds.

Another possibility is a major change in the reporting of test information. NCLB requires schools to notify parents directly about the performance of their schools. In 1999 and 2000, the first two years of PACT testing, scores were reported in their raw form in the materials that parents received. Beginning in 2001, official PACT reports to parents used a simpler rating scale that classified each school into one of five performance categories ranging from unsatisfactory to excellent. Under this scheme, almost every school received a rating of at least average. Indeed, a Department of Education news release in 2002 ran with the headline, “Schools receive higher Absolute ratings on report cards; 80% average or better.” Although the raw scores were contained deeper in the reports, if most schools appeared to be average or better, parents may not have been prompted to hold incumbents accountable for poor school performance. Incumbents and potential challengers may also have become less responsive to scores when the testing regimen began to give nearly every school a passing mark.

Implications for Policy

The evidence from South Carolina shows that voters do at least sometimes evaluate school board members on the basis of student learning trends as measured by average school test scores. Changes in average school test scores from year to year can affect the number of votes incumbents receive, the probabilities that they run for reelection, and the likelihood that they face competition when they do.

But the absence of a relationship between average school test scores and incumbents’ electoral fortunes in the 2002 and 2004 school board elections raises important questions about the assumptions underlying accountability systems. School board elections give the public the leverage to improve their schools. If voters do not cast out incumbents when local school performance is poor, they forfeit that opportunity. As debate continues over components of NCLB, policymakers should consider whether it is realistic to assume voters will in fact use the polls to drive school improvement.

Christopher R. Berry is assistant professor at the Harris School of Public Policy Studies at the University of Chicago, where William G. Howell is associate professor.

Despite the old habits and the old research, however, school districts across the nation have been slowly but steadily bucking convention. Several large systems, including Chicago (beginning in 1996), New York (2004), and Philadelphia (2005), now require students in particular grades to demonstrate a benchmark level of mastery in basic skills on a standardized test before they can be promoted. Florida (2002) and Texas (2002) have taken the lead among states in forbidding social promotions. In 2000, the most recent year for which national enrollment data are available, these five school systems alone enrolled nearly 20 percent of the nation’s 3rd-grade students. (For more on Chicago’s policy, see Alexander Russo, “Retaining Retention,” features, Winter 2005; and Robin Tepper Jacob and Susan Stone, “Teachers and Students Speak,” features, Winter 2005.)

But is this new approach to grade promotion effective? And what about those studies that say retention doesn’t work? Proponents of the new programs believe that schools do students no favor by
promoting them if they don’t have the skills to succeed at a higher level. But because these arguments, however plausible, have little research to support them, we set out to determine if they have scientific merit. Our findings from Florida suggest that the use of standardized testing policies to end social promotion can help low-performing students make modest improvements in reading and substantial improvements in math.

Florida’s Program to End Social Promotion

Over the past several years Florida has attempted substantial reforms of its struggling public school system, the fourth-largest in the country and one that consistently ranks close to the bottom on academic indicators, including high-school graduation rates and scores on the National Assessment of Educational Progress (NAEP). The Sunshine State had instituted school voucher programs, increased the number of charter schools, and devised a sophisticated accountability system that evaluates schools on the basis of their progress as measured by the Florida Comprehensive Assessment Test (FCAT). But in May 2002, the state legislature made one of its boldest moves, revising the School Code, the state’s education law, to require 3rd-grade students to score at the Level-2 benchmark or above on the reading portion of the FCAT in order to be promoted to 4th grade.

The hurdle created for students was not terribly high. The state’s department of education describes a student who scores at Level 2 (of five levels) as having “limited success” against the state standards; only students who score at Level 3 or above are considered to be proficient for the purposes of evaluating schools under No Child Left Behind. Even so, roughly 24 percent of 3rd graders tested in Florida in 2001–02, the year before the retention policy was introduced, performed below Level 2. This number fell slightly, to 22 percent, in the 2002–03 academic year.

Not all these students were retained, however, even after the policy change. The law allowed for exceptions to the retention policy if a student had limited English proficiency or a severe disability, scored above the 51st percentile on the Stanford-9 standardized test, had demonstrated proficiency through a performance portfolio, or had already been held back for two years. Altogether, roughly 40 percent of the 3rd-grade students who scored below the Level-2 threshold in 2002–03 were promoted.

The Problem with Earlier Studies

Traditionally, the retention of a student, uncommon as it was, resulted from an individual teacher’s assessment of the student’s ability to succeed at the next level. But such teacher discretion, while arguably desirable as a matter of policy, is the primary reason earlier studies of social promotion are flawed. We must assume from studying those retention programs, which are still the predominant practice in schools throughout the United States, that students who were held back were fundamentally different from students who were promoted. Because teachers were considering intangible factors, even when race, gender, family income, and academic achievement are the same, there was no way to isolate the effect of being held back, much less to make reasonable conclusions about the effects of retention on a student’s academic achievement or the probability of his dropping out of high school. Are students who were retained less likely to graduate because they were retained? Or were they retained because of characteristics that also predisposed them to drop out? Because the retention policies were subjective, we will simply never know.

There are also reasons to believe that subjective retention policies affect students differently than policies that use promotion criteria like performance on standardized tests. If promotion depends on an individual teacher’s assessment of a child, then that child is not likely to know what he or she must do to avoid being held back. Also, if few students were being held back, then those students might perform worse because they felt excluded and inferior. A policy that holds back thousands of students might dilute this sense of being singled out. Finally, subjective assessments of students are vulnerable to inappropriate influences, including teachers’ prejudices and pressure brought by parents, in ways that objective criteria of performance might inhibit.

Implementing objective standards, even if they were accompanied by subjective exemptions, might significantly change the effects of retention in ways that previous research could not anticipate or measure. For research purposes, objective retention policies also create a useful comparison group of students not subject to retention. In the case of Florida’s program to end social promotion, for example, we can compare students who were subject to the threat of retention with students who would have been had they been born a year later.

What a Difference a Year Makes

To determine the impact of ending social promotion for 3rd graders in Florida, we compared low-scoring 3rd graders in 2002, the first students to be subject to the program, with low-scoring 3rd graders from the previous year. Of the 43,996 3rd graders in 2002 for whom we have valid test scores on both FCAT math and reading assessments, 60 percent were actually retained. By contrast, of the 45,401 3rd graders in 2001 for whom we have valid test scores, only 9 percent were retained. Our analysis assumes that the students from the two school years should be similar in all respects except for the year in which they happened to have been born. We analyzed the test-score improvements made between each student’s first 3rd-grade year and the following year on both the state’s own accountability exam and the Stanford-9, a nationally normed exam administered at the same time as the FCAT but not used for accountability purposes.

We measure FCAT performance using developmental-scale scores, which allow us to compare the test-score gains of all the students in our study, even though they took tests designed for different grade levels. Developmental-scale scores are designed to measure academic proficiency on a single scale for students of any grade and in any year. For example, a 3rd grader with a developmental-scale score of 1,000 and a 4th grader with a developmental-scale score of 1,000 have the same level of academic achievement; if a student gets a developmental-scale score of 1,000 in 2001 and then gets the same score of 1,000 in 2002, this indicates that the student has not made any academic progress in the intervening year. The developmental-scale scores required to reach Level 2 on the FCAT reading test were consistent for each year’s cohort.

We began by measuring the effect on all low-scoring 3rd graders of simply having been subject to the new policy. That is, we did not distinguish in our initial analysis between students who were actually retained and those who received an exemption and were promoted to the next grade. This analysis provides an estimate of the average impact of the policy change on all students in the state performing below the Level-2 benchmark. It also allows for the possibility that exempted students enjoyed spillover benefits from the retention policy, since they were now being instructed in a system in which fewer students in 4th grade were unprepared to do grade-level work.

To identify the policy’s average impact, we compared the gains in developmental-scale scores made by students who first entered 3rd grade in 2002 and scored below the FCAT benchmark with gains made by students who first entered 3rd grade in 2001 and scored below the FCAT benchmark. In making this comparison, we took into account other factors that could affect achievement gains, such as the student’s race, whether the student received a free or reduced-price school lunch, whether the student was deemed Limited English Proficient, and the student’s precise test score during his first 3rd-grade year. With these differences accounted for, the only distinction between the two groups of students was assumed to be that the former group entered the school system a year later and was therefore subject to the new policy in 3rd grade.

As discussed above, however, many low-scoring 3rd graders were granted exemptions and promoted to the 4th grade even under the new policy. We therefore also evaluated the effect of actually being retained, again controlling for race, eligibility for free or reduced-price lunch, English proficiency, and baseline test scores. In conducting this analysis, we also needed to account for the fact that the students who were held back were a select group of students who could differ in important ways from the promoted students. Presumably, teachers and other decisionmakers expected these students, unlike promoted students, to benefit from an additional year as 3rd graders. Fortunately, the fact that simply having entered school a year later increased the probability of retention for all low-scoring students again provides a way around this obvious selection problem. In essence, the statistical method we use compares those retained students that our data suggest would not have been retained the previous year with a comparable group of students who were not retained. Our results therefore indicate the effect of retention on those students who were held back as a result of the new policy.

During this time, Florida was engaged in other education reforms as well: instituting several school-voucher programs, increasing the number of charter schools in the state, and improving the system used to assign grades to schools based on the FCAT. However, it is reasonable to assume that whatever effect these other policies have on our analyses is minor. In order for the existence of another policy to affect our results significantly, we would have to believe that the program substantially improved the education of the 3rd graders in 2002–03 without having a similar effect on the previous year’s cohort. Moreover, while a sudden policy change could conceivably explain the overall improvements between the two cohorts, it is difficult to see how such a change could cause substantially larger gains among those students actually retained.

Retention Works

Our fundamental findings from an analysis of the 3rd-and 4th-grade data for these two years indicate that the performance of students identified for retention, regardless of whether they were retained or exempted and promoted, exceeded the performance of low-performing students from the previous year who were not subject to the retention policy; and students who were actually retained made the larger relative gains.

Students identified for retention by the Florida policy gained 0.06 of a standard deviation in reading on both the FCAT and Stanford-9 over equally low-performing 3rd graders from the previous school year (see Figure 1). In math, students identified for retention surpassed low performers who were not subject to the policy by 0.15 standard deviations (4.8 percentiles) on the FCAT and 0.14 standard deviations (4.4 percentiles) on the Stanford-9.

Students who were actually retained experienced even larger relative improvements (see Figure 2). Retained students performed better than low-scoring students who were promoted by 0.13 standard deviations (4.10 percentiles) on the FCAT and 0.11 standard deviations (3.45 percentiles) on the Stanford-9 in reading. In math retained students improved 0.30 standard deviations (10.0 percentiles) on the FCAT and 0.28 standard deviations (9.3 percentiles) on the Stanford-9 over promoted students.

Some critics of the new retention policies argued that teachers and schools would respond to them by manipulating test scores, either directly by cheating or indirectly by teaching students skills that would help them to improve their test scores but would not provide real academic proficiency. This argument would have merit only if we found strong gains on the high-stakes FCAT and no similar gains on the low-stakes Stanford-9, for which there is no incentive to manipulate scores. But our results are consistent between the FCAT and the Stanford-9, indicating that there have been no serious manipulations of the high-stakes testing system. If teachers are in fact changing their curricula with the intent to “teach to” the FCAT, they are doing so in ways that also contribute to gains on the highly respected Stanford-9. This would indicate that teachers have made changes resulting in real increases in students’ proficiency.

An unexpected benefit of the retention policy is the improvement in math scores. This might seem odd, given that it is the reading portion of the FCAT that students must pass to earn promotion and that the rhetoric supporting Florida’s retention program emphasizes that it will improve student literacy. Of course, the math gains could simply reflect the fact that math skills are learned primarily in schools, while reading is practiced both in and outside of school. For this reason, evaluations of school reforms frequently find stronger effects in math than in reading. Alternatively, it may be that students who were retained specifically because of their poor reading skills are particularly poor in that subject and that this limits their room for improvement.

We also explored the possibility that the objective retention program could have different effects on students of different races. Our results show gains of similar sizes by the three racial groups for which we have an adequate sample size to have reasonable confidence in our findings: white, black, and Hispanic. The exception is for whites’ performance on the FCAT reading test. It is difficult for us to interpret why white students would fail to benefit from the retention policy as measured by the FCAT reading test but would be shown to benefit as measured by the Stanford-9 reading test.

Our results also suggest that low-scoring Florida 3rd graders who were given an exemption and promoted might have bene­fited from another year in the 3rd grade. This does not mean that it would be wise to eliminate all exemptions to the testing requirement. There are certainly students for whom testing is either inappropriate or whose performance on other academic measures could reasonably indicate that they would be better served by moving on to the next grade. However, our findings do indicate that teachers and school systems should be cautious when granting exemptions.

What It Means

At first glance our findings seem inconsistent with evaluations of Chicago’s program ending social promotion, to our knowledge the only similarly designed retention policy to be evaluated using comparable methods. In Chicago, students in the 3rd, 6th, and 8th grades must exceed benchmarks on the Iowa Test of Basic Skills (ITBS), a respected standardized test, in order to be promoted to the next grade. In a study conducted in 2004 by scholars at the Consortium on Chicago School Research, the performance of 3rd- and 6th-grade students who scored just below the benchmark on the ITBS, most of whom were retained because of the mandate, was compared with the performance of students who scored just above the benchmark, most of whom were promoted. The Chicago researchers were able to measure test-score performance for two years after implementation of the program. They found benefits from the program after one year, similar to what we found in Florida, but discovered that those benefits went away after the second year. Third-grade students were not affected, and 6th-grade students were negatively affected by the policy in their performance on the ITBS reading test. The findings on the Chicago retention program emphasize the importance of following the progress of retained students in Florida over time.

Still, the Chicago policy differs from Florida’s in some respects. In 1999 the Chicago policy stopped allowing students to be retained twice, which Florida’s policy does allow. This difference might reduce teachers’ motivation to work with already retained students, whom they now can expect to be promoted the next year regardless of their performance. Other programs with different and more stable retention policies might show different results.

Finally, while our study provides valuable information about the effectiveness of Florida’s policy to end social promotion, it does not offer a full catalog of the policy’s benefits or of its potential costs. It will be some time before we can examine whether retention increased or reduced the probability of dropping out of school later on. Most important, it does not provide any information about the program’s effects on students’ academic progress the first time they were in 3rd grade. The policy’s greatest benefits could result not from retention itself, but rather from increased efforts on the part of teachers and even students to avoid being retained in the first place.

-Jay P. Greene is professor and head of the Department of Education Reform, the University of Arkansas; he is also a senior fellow at the Manhattan Institute. Marcus A. Winters is a doctoral fellow, the University of Arkansas and a senior research associate at the Manhattan Institute.

Students will test the limits of acceptable behavior in myriad ways better known to school teachers than to judges; school officials need a degree of flexible authority to respond to disciplinary challenges; and the law has always considered the relationship between teachers and students special.
— Supreme Court Justice Stephen Breyer

In Morse v. Frederick, a 2007 First Amendment student free speech case, the Supreme Court held that a school official may restrict student speech at a school-supervised event when that speech is viewed as promoting illegal drug use. Filing a separate opinion, Justice Stephen Breyer echoed concerns expressed by his conservative colleagues that school authority was being undermined by legal challenges. Since the 1960s, courts have become increasingly involved in regulating U.S. schooling in general, but especially in the area of school discipline. Justice Breyer noted in his opinion, “Under these circumstances, the more detailed the Court’s supervision becomes, the more likely its law will engender further disputes among teachers and students.”

School discipline is a critical area for research, as student interaction with school institutional authority is one of the primary mechanisms whereby young people come into contact with and internalize societal norms, values, and rules. It is thus significant that the number of cases reaching state and federal appellate courts has surged back up to levels attained during the early 1970s when civil rights cases had a central place on the national political agenda (see Figure 1). Our research indicates that both educators and students understand the former’s authority to be more limited and the latter’s rights more expansive than has actually been established by case law.

School Discipline in Court
Until the late 1960s, parents and students rarely challenged the disciplinary actions of school authorities, viewing common schools as providing instruction, instilling virtue, and fostering the ideals of our nation. Then, as conceptions of youth rights began to shift, and as institutions that provided support for the expansion of these rights emerged, students and parents, with the support of public-interest lawyers, began to question and challenge school disciplinary practices in court.

Table 1 summarizes key school-related rulings from the Supreme Court over the last 40 years. From 1969 to 1975, amid increasing legal challenges to the regulation of student expression in school, the Court’s rulings largely confirmed students’ rights to various free expression and due process protections. The most important decision affecting how schools approach student discipline was Goss v. Lopez, decided by the Supreme Court in 1975. During a patriotic assembly at Central High School in Columbus, Ohio, in 1971, expressions of student unrest over the lack of African American curricula turned into a week of demonstrations and disturbances. Dozens of students were suspended for up to 10 days without formal hearings or notification of the specific charges against them. The Supreme Court case hinged on whether the disciplinary actions improperly denied students their rights to a public education. In ruling for the students, the Court granted “rudimentary” due process rights to those suspended from school for fewer than 10 days, as well as “more formal protections” for students facing longer exclusions.

In recent years, courts at all levels have dealt with cases challenging the enforcement of “zero-tolerance” policies that establish severe and nondiscretionary punishments for violations involving weapons, violence, drugs, or alcohol. At the same time, an increasing number of cases have appeared in lower courts that involve students and families suing schools for failing to provide adequate discipline within school facilities. These cases have alleged climates that permit bullying, sexual harassment, or other forms of school violence (including school shootings). Thus, in recent years, schools have been sued for both disciplining students and not disciplining them.

Since 1975, the Supreme Court has generally been less favorable toward students than it was during the early years of the civil rights movement.. This shift in orientation occurred for diverse reasons, including growing public concern about the level of violence and disorder in public schools, the changed political climate following the end of the Vietnam era, and a pattern of increasingly conservative judicial appointments during the Nixon, Reagan, and Bush administrations. The Supreme Court’s 2007 decision in Morse v. Frederick continued the post-1975 pattern of sympathy with schools that are facing challenges to their disciplinary authority, but did not, as some of the media coverage implied, alter the general contours of student rights as previously established. Its June 2009 decision in Safford United School District v. Redding, in which eight justices agreed that a near strip-search of an 8th-grade girl suspected of concealing prescription-strength ibuprofen was unconstitutional, at first glance appears to be an exception—a sign that the courts will continue to watch over the shoulders of school officials to ensure that reasonableness and proportionality prevail. Yet a majority on the court ruled that the administrators who conducted the search could not be held personally liable because of the uncertainty of the law in this area.

Appellate Case Patterns
While Supreme Court decisions are important because every school in the nation must adhere in principle to its rulings, these few landmark cases do not encompass the universe of legal challenges regarding school discipline and related policies. To discern the larger contours of the legal climate facing schools, we analyzed all appellate-level federal and state court cases in which school efforts to discipline and control students have been challenged. As a whole, decisions in these cases are often complex and contradictory in providing practical guidance to schools regarding specific disciplinary matters. We included cases involving the use of state agents (such as the police) acting on behalf of school authorities to deal with students in the vicinity of school grounds. We excluded instances of conflicts between schools and teachers (such as teacher dismissal cases) and between schools and nonstudent outsiders (such as drug- and weapon-free-zone cases that did not involve students), as well as student rights cases focused exclusively on free speech issues (that is, those not combined with the school’s use of suspension, expulsion, corporal punishment, and transfer). We also excluded cases in which students allege that school authorities have breached their duty to maintain safety in the school and to protect students from harm.

Of course, we did not include the vast majority of litigation, which was either settled before hearing or never reached state and federal appellate courts. Still, our methods provide a way to gauge the general character and broad trends in legal challenges that contemporary educators face. Appellate-level court cases define case law, generate media coverage, influence public perceptions, and can be tracked over time as an empirical indicator of the broad parameters of court climate toward school discipline. We found that not only has the frequency of legal challenges greatly varied over time, but the content and direction of outcomes has shifted as well.
The newfound willingness to challenge school authority became evident in the surge of litigation during the late 1960s. In part because of increased institutional support from public-interest legal advocacy groups and the legal services program of the Office of Economic Opportunity, from 1968 to 1975 an average of 39.1 public school K—12 cases per year reached the appellate level. After important legal precedents were set and institutional support waned, the average number of cases declined but then took a sharp upturn from 1993 on, with a peak of 76 cases in 2000 and a total of 65 in 2007. We present here the overall number of cases rather than a relative measure accounting for public school enrollment, given that media coverage and individual understandings reflect the former indicator. Nevertheless, a measure of state and federal court cases calculated per enrolled student would demonstrate similar upward trends, more than doubling from the years 1976—1992 to the 2003—2007 period.

The substance of the cases brought before the courts has also varied over time, with protest and free expression cases decreasing markedly through 1992 (see Figure 2a). Recently, courts have witnessed a reemergence of these issues. Cases involving alcohol and drugs rose during the intermediate time periods that coincided with national attention to the “War on Drugs” and then diminished. Those involving weapons and violence have increased to nearly 40 percent of all K—12 public school discipline cases since 1993. In addition, school discipline court cases increasingly have involved student disability. From 2003 to 2007, 18 percent of cases included discussion of student disability status. Since the 1970s, legal entitlements and protections have grown for students classified as disabled because of learning, physical, or behavioral handicaps (including psychological disorders that are associated with the manifestation of student misbehavior). Special education students thus gained additional protections related to school discipline, particularly in cases in which infractions could be attributed to the individual’s disability.

Over time, we found that courts in general have become less favorable to student claims across these areas of litigation (see Figure 2b). However, since the number of court challenges has increased in recent decades, the likelihood of a school facing a legal environment in which a student has recently been successful in a court challenge over school discipline has not significantly diminished.

Socioeconomic Disparities
Many of the early school discipline cases were brought to ensure that the rights of less-advantaged students were protected. New evidence suggests, however, that litigation is increasingly used strategically and instrumentally by families from relatively privileged origins to promote the interests of their children. Research (by Irenee Beattie, Josipa Roksa, and Richard Arum) that examined appellate court cases from 2000 to 2002 found that, on average, those cases emerged from secondary schools with 29 percent nonwhite students compared to 37 percent nonwhite students in the national population of secondary schools (the latter weighted for enrollment size to be comparable to the court case data); appellate cases also emanated from schools with more educational resources per student (student/teacher ratios of 16.3 compared to 17.5 nationally).

National surveys of teachers and administrators reveal a similar middle-class bias in legal challenges. A reanalysis of a Harris survey of teachers and administrators conducted by Melissa Velez and Richard Arum for Common Good in 2003 examined the proportion of public school educators (a combined sample of teachers and administrators) who reported that either they or someone they knew personally had been sued by a student or parent. Educators in suburban schools with less than 70 percent nonwhite students had a 47 percent probability of having experienced contact with an adversarial legal challenge compared to a 40 percent chance for educators in all other schools. Although much of the development of student rights originally emerged from concern about nonwhite students in urban areas, educators in those settings had only a 41 percent probability of contact with a legal challenge.

In collaboration with colleagues working on the School Rights Project (Lauren Edelman, Calvin Morrill, and Karolyn Tyson), we conducted a national telephone survey of 600 high school teachers and administrators and site-based surveys of 5,490 students and 368 educators on perceptions and experiences of the law in schools. In our site-based work, which included in-depth interviews and ethnographic fieldwork, we examined 24 high schools with varying legal environments situated across three states (New York, North Carolina, and California), stratified by school type (traditional public, charter, and Catholic) as well as by student socioeconomic composition. We found that 15 percent of public school teachers and 55 percent of public school administrators have been threatened with a legal suit over school-related matters. For administrators with more than 15 years of experience in the position, the figure rose to 73 percent. Administrators’ actual experience with being sued for school-related matters occurs at a lower rate (14 percent), but is still the source of considerable professional anxiety, given that these cases—following Wood v. Strickland (1975)—include vulnerability to personal liability claims. We again found that legal challenges are concentrated in schools with more-privileged students. When we looked solely at administrators working in urban public schools with more than half of students eligible for free lunch, we found—albeit with a sample of only 16 cases—not a single report of administrators being sued for a school-related matter.

That legal mobilization is dependent on economic resources needed to pursue such challenges is in general not surprising. We documented evidence of this association, however, to illustrate that regardless of the institutional and political origins of student rights, today legal mobilization in schools largely reflects patterns of socioeconomic inequalities. In the School Rights Project, we found that white students were nearly twice as likely as nonwhite students to report having pursued a formal legal remedy for a perceived rights violation.

Legal Understandings and School Practices
Legal mobilization is a relatively rare occurrence, a small tip of a much larger legal-dispute pyramid. School discipline today is profoundly shaped by legal understandings that are only partially and indirectly related to formal regulation and case law. We highlight here the extent to which both students and educators have developed an expansive definition of legal rights of students, the relationship between this sense of legal entitlement and school disciplinary practices, and perceptions of the fairness and legitimacy of various school disciplinary practices.

The institutionalization of student due process protections goes well beyond appellate case law, having been enshrined in extensive state statutes and administrative regulations. The accompanying sidebar (page 65) provides a sense of the extent to which law has come to permeate school practices by highlighting codified disciplinary procedures in New York City. While discipline policies vary across schools, districts, and states—and as the nation’s largest school district the New York City public schools are likely more bureaucratized and formalized in matters of school discipline than smaller districts—the scale, scope, and level of complexity of the legal regulations affecting day-to-day school practices appear quite formidable.

Generally speaking, educators and students have developed a set of legal understandings that assumes a broad and expansive definition of student legal entitlements. Following the Goss decision, students have been granted rudimentary due process protections when facing minor discipline and more formal due process protections when facing more serious forms of discipline (such as long-term expulsion or suspension). The Goss decision delineated procedural safeguards, stating that “the student be given oral or written notice of the charges against him and, if he denies them, an explanation of the evidence the authorities have and an opportunity to present his side of the story.” More formal due process protections may include the right of students to “summon the accuser, permit cross-examination, and allow the student to present his own witnesses. In more difficult cases, he (the disciplinarian) may permit counsel.”

We are interested in individuals’ perceptions of such protections, since students’ and educators’ beliefs about rights likely have real consequences for school authority and disciplinary procedures. In the School Rights Project, we specifically asked students and teachers which due process protections were required when students faced various disciplinary sanctions. We found that while expectations of formal due process protections were broadly diffused for students when facing major disciplinary actions, many of them had also come to expect these legal entitlements when facing minor day-to-day discipline. For example, 62 percent of public school students in our sample believed that, if faced with long-term suspension or expulsion, they were legally entitled to at least one of the following: a formal disciplinary hearing, opportunity to be represented by legal counsel, opportunity to confront and cross-examine witnesses bringing the charges, or opportunity to call witnesses to provide alternative versions of the incident. Approximately one-third of students also believed that they were legally entitled to some form of formal due process protection when they had their grades lowered for disciplinary reasons (33 percent), were suspended from extracurricular activities (36 percent), or faced in-school suspension (35 percent).

We found that students’ sense of legal entitlement was expansive, and that teacher and administrator expectations of required student due process protections were even more so. For example, when asked about lowering student grades for disciplinary reasons, approximately half of public school teachers and administrators responded that this action was prohibited; among the educators who did think such disciplinary actions were permissible, 32 percent reported that students subject to such disciplinary sanctions were entitled to formal due process protections.

In the School Rights Project, we found that increased perceptions of student legal entitlements correlate with decreased reports of the fairness of school discipline. This conclusion mirrors James Coleman’s finding that Catholic school students in the 1980s were significantly more likely to perceive school discipline to be fair than public school students, who possessed far greater formal legal protections. Educators and students have developed a generalized sense of legal entitlements, while school practices have, in many settings, become increasingly authoritarian, with student misbehavior often subject to criminalization and formal legal sanction. These internal contradictions enhance students’ sense of the unfairness of school discipline. Longitudinal research has demonstrated that students who perceive school discipline as unfair are more likely to disobey teachers, disrupt classroom instruction, and in general fail to develop behaviors conducive to educational success and related positive outcomes.

Also, in recent decades schools have moved away from disciplinary practices that rely on the judgment, discretion, and action of professional educators and have turned instead to reliance on school security guards, uniformed police, technical surveillance, security apparatus, and zero-tolerance policies. The latter techniques are ill suited to the pedagogical task of enhancing the moral authority of educators to support the socialization of youth, that is, the internalization of norms, values, and rules.

Conclusion
Citizens, legislators, judges, and policymakers have begun to recognize and question legal interventions in situations involving school discipline and authority. We add to this discussion our findings that the legal understandings underlying school discipline policies depart in significant ways from the case law on which they are assumed to be based, according expansive rights and protections to students, even as the courts have tended to side with school authorities. We also document that although public-interest lawyers were initially motivated to expand student legal rights as part of a larger strategy to reduce social inequality, legal challenges to school disciplinary actions are disproportionately the province of white and higher-income students and their families.

The expansion of student legal entitlements has been accompanied by the increasing formalization and institutionalization of school discipline. As educators’ discretionary authority over school discipline has been challenged and undermined, counterproductive authoritarian measures such as zero-tolerance policies have been implemented in its place. But to be educationally effective, school discipline requires that educators have moral authority and students perceive their actions as legitimate and fair. Ironically, the expansion of student legal rights, rather than enhancing youth outcomes, has increased the extent to which schools have relied on authoritarian measures, decreased the moral authority of educators, and diminished the capacity of schools to socialize young people effectively.

As various social and political actors consider legal regulatory reforms, it is important to recognize that the expansion of students’ legal entitlements has also increased the potential for student dissent in U.S. schools, whether of a political, religious, or ideological character. At the same time, individual students and families with sufficient resources are able to contest what they perceive as unfair disciplinary sanctions or rights violations. These gains have come at a pedagogical and societal cost, as the resolution of school disciplinary matters has increasingly moved—as Justice Breyer feared—from the schoolhouse to the courthouse.

Richard Arum is professor of sociology and education at New York University, where Doreet Preiss is a research fellow and doctoral candidate. This essay is adapted from “Still Judging School Discipline,” in Joshua M. Dunn and Martin R. West, eds., From Schoolhouse to Courthouse: The Judiciary’s Role in American Education (Brookings, 2009).

Understanding whether troubled children in fact generate spillover effects in school is important for two reasons. First, the existence of substantial spillovers caused by family problems such as domestic violence would provide an additional compelling reason for policymakers to find ways to help troubled families. Second, because many education policies change the composition of school and classroom peer groups, it is important to understand how such changes may affect student achievement. For example, a common concern regarding the ongoing push to “mainstream” emotionally disturbed students in regular classroom settings is that doing so may undermine the performance of other students. Similarly, the tracking of students into classrooms based on ability or academic performance may group disadvantaged children with the most disruptive students. The validity of these concerns hinges on whether and how classroom exposure to troubled peers affects student achievement and behavior.

Credibly measuring negative spillovers caused by troubled children has been difficult. Most data sets do not allow researchers to identify troubled children. Even when such students are identified in the data, it is difficult to determine if a disruptive child causes his classmates to misbehave or if his classmates cause him to be disruptive, what scholars of peer effects call the “reflection problem.” In addition, troubled children are likely to attend the same schools as other disadvantaged children. One must rule out the possibility that the disruptive student and his classmates misbehave due to some common unobserved factor.

We overcome these problems in this study by utilizing a unique data set in which information on students’ academic achievement and behavior is linked to domestic violence cases filed by their parents. This data set allows us to identify troubled children more precisely than we could by using conventional demographic measures. Moreover, we can identify children who are troubled for specific family reasons and not because of their peer group. This allows us to measure peer effects free from the reflection problem, providing a rare opportunity to test the notion that even one “bad apple” impedes the learning of all other students.

Our results confirm, first, that children from troubled families, as measured by family domestic violence, perform considerably worse on standardized reading and mathematics tests and are much more likely to commit disciplinary infractions and be suspended than other students. We find also that an increase in the number of children from troubled families reduces peer student math and reading test scores and increases peer disciplinary infractions and suspensions. The effects on academic achievement are greatest for students from higher income families, while the effects on behavior are more pronounced on students who are less well-off. The results of our analysis provide evidence that, in many cases, a single disruptive student can indeed influence the academic progress made by an entire classroom of students.

Data

In our study, we use a confidential student-level data set provided by the school board of Alachua County in Florida. This data set consists of observations of students in the 3 rd through 5th grades from 22 public elementary schools for the academic years 1995–96 through 2002–03. The Alachua County school district is large relative to school districts nationwide, with roughly 30,000 students; in the 1999–00 school year, it was the 192nd largest among the nearly 15,000 districts nationwide. The student population in our sample is approximately 55 percent white, 38 percent black, 3.5 percent Hispanic, 2.5 percent Asian, and 1 percent mixed race. Fifty-three percent of students were eligible for the federal free or reduced-price lunch program.

The test-score data consist of reading and mathematics scores from the Iowa Test of Basic Skills and the Stanford 9, both nationally normed exams. Reported scores indicate the percentile ranking on the national test relative to all test-takers nationwide. Because the reading and math results are so similar, we use a composite score calculated by taking the average of the math and reading scores. The average student in our data scored at the 53rd percentile, or just above the national norm.

Yearly disciplinary records, which include incident type and date, are available for every student in our sample. Incidents are reported in the system if they are serious enough to require intervention by the principal or another administrator. We focus on three behavioral outcomes from these records: the probability the student was involved in a disciplinary incident, the total number of disciplinary incidents per student, and the probability the student was suspended. In a typical year, 18 percent of the students in our data set were involved in a disciplinary incident, the average student was involved in 0.56 incidents, and 9 percent of students were suspended.

We gathered domestic violence data from public records information at the Alachua County courthouse, which included the date filed and the names and addresses of individuals involved in domestic violence cases filed in civil court in Alachua County between January 1, 1993, and March 12, 2003.Cases are initiated when one family member (typically the mother) petitions the court for a temporary injunction for protection against another member of the family (most often the father or boyfriend). Students were linked to cases in which the petitioner’s first and last name and the first three digits of the residential address matched the parent name and student’s residential address in the annual school record. In that way, we were able to identify the set of students who could be matched to a domestic violence case from1993 to 2003. In total, 4.6 percent of the children in our data set were linked to a domestic violence case filed by a parent, split equally between boys and girls. Sixty-one percent of these children were black, while 85 percent were eligible for subsidized school lunches.

Students linked to a domestic violence case performed at lower levels academically and were more likely to have been involved in a disciplinary incident than other students in the district. Boys exposed to domestic violence, for example, performed at the 37th percentile academically, as compared with the 52nd percentile for boys who were not exposed. Forty-three percent of boys exposed to domestic violence were involved in a disciplinary incident, as compared with 25 percent of boys who were not exposed. Girls exposed to domestic violence performed at the 41st percentile academically and 19 percent of them were involved in a disciplinary incident, as compared with the 55th percentile and 11 percent for girls who were not exposed to domestic violence (see Figure 1).

Measuring Peer Effects

Our main analysis examines the impact of troubled children on their peers. We assume there is no feedback loop in which a student’s peers cause the domestic violence in the household. This assumption appears reasonable, as none of the most likely determinants of domestic violence can plausibly be caused by an elementary school child or her peers.

To overcome the bias that results from self-selection into peer groups, our main analysis compares cohorts of students in the same grade at the same school in different years. For example, we compare the 3rd graders in a given school this year with the 3rd graders in the same school last year to see whether the cohort with more students exposed to domestic violence had higher or lower student achievement. Restricting the comparisons to students attending the same school ensures that any effects we observe reflect the impact of troubled students and not the fact that schools with more such students differ in unobserved ways from other schools. We measure peer domestic violence at the cohort level (that is, across all students in a grade at a school) as opposed to the classroom level due to the possible sorting of students into classrooms according to their achievement and behavior. We also adjust for differences among students in a large set of individual characteristics—most importantly whether particular students had been directly exposed to domestic violence—but also race, gender, subsidized lunch status, and median zip code income.

Results

Our results indicate that troubled students have a statistically significant negative effect on their peers’ reading and math test scores. Adding one troubled student to a classroom of 20 students results in a decrease in student reading and math test scores of more than two-thirds of a percentile point (2 to 3 percent of a standard deviation). The addition of a troubled peer also significantly increases misbehavior of other students in the classroom, in effect causing them to commit 0.09 more infractions than they otherwise would, a 16 percent increase. These are effects that could accumulate over time if the same students are repeatedly exposed to troubled peers.

These average effects also mask a few interesting differences across student groups. We find that troubled peers have a large and statistically significant negative effect on higher income children’s math and reading achievement, but only a small and statistically insignificant effect on the achievement of low-income children. However, we find the opposite pattern for disciplinary outcomes. The presence of troubled peers significantly increases the misbehavior of low-income children, but does not increase the disciplinary problems of higher-income children (see Figure 2).

Results of examining the differential effects of peers from troubled families by race and gender show relatively large negative and statistically significant test-score effects on white boys and statistically insignificant effects on black boys, black girls, and white girls. Adding one troubled peer to a classroom of 20 students reduces white boys’ reading and math scores by 1.6 percentile points and black boys’ reading and math scores by 0.9 percentile points (the effects on girls are negligible). Troubled peers increase disciplinary problems for all subgroups except for white girls. The effects are largest for black girls. One troubled peer added to a classroom of 20 students increases the probability that a black girl commits a disciplinary infraction by 2.2 percentage points (an increase of 10 percent over what would otherwise be the case).

Finally, we examined whether troubled boys affect their peers differently than do troubled girls. Across all outcome variables, both academic and behavioral, the negative peer effects appear to be driven primarily by the troubled boys, and these effects are largest on other boys in the classroom. The results indicate that adding one troubled boy to a classroom of 20 students decreases boys’ test scores by nearly 2 percentile points (7 percent of a standard deviation) and increases the probability that a boy will commit a disciplinary infraction by 4.4 percentile points (17 percent). Apparently, troubled boys generate the strongest adverse peer effects, and other boys are most sensitive to their influence.

Testing Key Assumptions

Of critical importance to our method is the assumption that students are not systematically placed into or pulled out of a particular grade cohort within a school depending on the domestic violence status of the student or his peers. For example, if parents who really value education were more likely to pull their children out of a cohort with a particularly high proportion of peers from troubled families, such nonrandom selection would cause us to erroneously attribute lower performance to the presence of the troubled peers.

We performed several additional analyses to probe the robustness of our results to this critical assumption. As a first test for nonrandom selection of students into or out of particular schools and cohorts of students, we examined whether peer family violence appears to have an effect on cohort size or student characteristics such as race, gender, and income. In the absence of nonrandom selection, we expect to find no correlation between these characteristics and the peer family violence variables. This is indeed what we find.

Next, we noted that some parents may be more likely than others to put their children in private schools or move to a different school zone because of a particularly bad cohort, but that parents may be less likely to pull one child out of the school due to a particularly bad cohort when that child has a sibling in the same school. When we calculated peer effects only on children with siblings in the school, the results were essentially the same as those for the full sample.

One might also be concerned that some families are, for some reason, unable to remove their children from cohorts with a large number of troubled peers. To check this potential cause of nonrandom selection, we calculated results based only on comparing students to their siblings. We found that the sibling in the cohort with more children from troubled families has lower test scores and more disciplinary problems. These within-family results are roughly two-thirds the size of the estimates for the full sample, but the differences between the two sets of results are not statistically significant.

For a final check, we added controls for a full set of cohort-level variables, including race, gender, participation in the federal subsidized lunch program, and median zip code income. These variables control for any potential changes in cohort characteristics not captured by our full set of individual controls in the main analysis. In addition, this allows us to examine whether the presence of children exposed to domestic violence is merely a proxy for other peer characteristics, such as family income. The results indicate that the negative peer effects are not likely driven by observable factors, such as family income, that are correlated with domestic violence.

Collectively, these tests provide strong evidence that our findings are not the result of families changing schools in response to the number of children from troubled families in their child’s grade at an assigned school.

Discussion

In addition to knowing how children from troubled homes affect their peers through interaction with their cohort at school, one may also wish to know the precise way in which the troubled families cause the peer effects. This is a particularly challenging task given that researchers have consistently found, as we have, that domestic violence is correlated with other negative family characteristics, such as poverty, unemployment, low levels of education, and substance abuse. While we cannot conclusively attribute the effects found to the causal effect of domestic violence per se, we can exploit the timing of the domestic violence filings to provide suggestive evidence of whether the negative spillovers are due to domestic violence or some other factor correlated with it.

Specifically, we examine whether the negative spillovers associated with children from troubled families are smaller after the parent files the case than before the case is filed. Survey research shows that on average, violence had occurred in the family for more than four years prior to the reporting of the incident. However, 87 percent of the respondents indicated that the reporting of the incident “helped stop physical abuse.” Consequently, if domestic violence itself is causing the negative spillovers on the child’s classmates, then we would expect the spillovers to be smaller when the parent of the peer had already filed for the injunction against domestic violence.

To investigate whether exposure to domestic violence is the potential mechanism through which the spillovers occur, we constructed two peer domestic violence variables: reported and as yet unreported violence. By definition, reported domestic violence means that the petition for the injunction was filed before the student test was taken and unreported domestic violence signals that the filing occurred after the test date.

We find substantially larger effects for the proportion of peers with unreported domestic violence (that is, those whose parents had not yet filed for the injunction) than for those with past domestic violence. For example, the test-score effects for troubled boy peers on boys are statistically insignificant for reported violence, while they are large and highly significant for unreported violence. The larger peer effects for unreported domestic violence suggest that the violence in the home may itself be playing a role in driving the effects. However, we remain cautious with this interpretation, as we have no direct information regarding the details of the family environments for students in our sample.

Conclusion

Our findings have important implications for both education and social policy. First, they provide strong evidence of the validity of the “bad apple” peer effects model, which hypothesizes that a single disruptive student can negatively affect the outcomes for all other students in the classroom. Second, our results suggest that policies that change a child’s exposure to classmates from troubled families will have important consequences for his educational outcomes. Finally, our results provide a more complete accounting of the social cost of family conflict. Any policies or interventions that help improve the family environment of the most troubled students may have larger benefits than previously anticipated.

Scott Carrell is assistant professor of economics at the University of California–Davis. Mark Hoekstra is assistant professor of economics at the University of Pittsburgh.

The federal law No Child Left Behind (NCLB) requires states to “restructure” any school that fails for six years running to make Adequate Yearly Progress (AYP) toward full proficiency on the part of all students by the year 2014. The law provides a number of restructuring options, including turning over the school’s management to a private for-profit or nonprofit entity. Only a few school districts nationwide have sought help from either type of organization in the management of low-performing schools. But in 2002 the School District of Philadelphia, at the request of the state of Pennsylvania, asked entities of both types to participate in a substantial restructuring of many of its lowest performing schools. The restructuring initiative was directed by the Philadelphia School Reform Commission (SRC), which contracted with for-profit organizations to manage 30 elementary and middle schools and with nonprofit organizations to manage 16 schools.

The policy intervention in Philadelphia raises questions of general interest: Do students at schools assigned to for-profit or nonprofit managers learn more than would be expected had those schools remained under school district management? Is for-profit management more or less effective at raising achievement than nonprofit management? Told most simply, the Philadelphia story provides a threefold answer to these questions: 1) for-profits outperform district-managed schools in math but not in reading; 2) nonprofits probably fall short of district schools in both reading and math instruction; and 3) for profits outperform nonprofits in both subjects. However, the answers require both explication and qualification.

The Theoretical Debate

The distinction between for-profit and nonprofit management has been a topic of continuing discussion in the scholarly literature on school reform. Nobel Prize–winning economist Milton Friedman theorized that for-profit firms are more effective because they have clear economic incentives to lift student performance. The firm can build its reputation (and in the long run generate a profit) only if it becomes known for running effective schools. Others have suggested, however, that for-profit firms are likely to cut costs and thereby shortchange students in order to benefit the firm’s owners and shareholders. The debate over nonprofit organizations takes a different form. Some have argued that nonprofit managers are likely to be effective because they have close ties to the community in which they are embedded and can enlist the energies of committed entrepreneurs, who devote all available resources to enhance student performance. But others caution that nonprofit managers may not have the experience, resources, or economic incentives necessary for building quality educational institutions.

The Intervention

Only after an intense political struggle did the Philadelphia school district ask for-profit and nonprofit managers to assume responsibility for a number of its schools. In 2001 Pennsylvania governor Tom Ridge, a Republican and school voucher supporter, indicated he would not support any increase in funding for the Philadelphia school district until an independent entity had assessed its financial practices and educational effectiveness. Philadelphia’s mayor at the time, Democrat John Street, knowing the district was facing a $215 million deficit, agreed to the study, and the state department of education asked Edison Schools to carry it out. Edison, a for-profit firm that manages charters and other schools under contracts with school districts, reported that the Philadelphia school district had spent $10 billion over a decade without being held accountable for the results. Governor Ridge refused to distribute any more state aid beyond current levels unless the school district agreed to a new partnership with the state.

When the local press reported that Edison was expected to assume management of many of the city’s schools, the local teachers union mobilized in opposition, and groups of parent and student activists held rallies throughout the city. As the turmoil was reaching its climax, Governor Ridge resigned from office to become the nation’s secretary of homeland security, and Pennsylvania’s lieutenant governor, Mark Schweiker, became governor. Upon assuming office, Schweiker said he would not serve beyond the current term, ending in 2002, a decision that weakened his leverage vis-à-vis the Philadelphia school district.

The stage was set for a compromise that would save face for all the parties involved. It was agreed that the school board should be replaced by a School Reform Commission, three members of which would be appointed by the governor and two by the mayor. The SRC decided that only a limited number of the lowest-performing schools in the district would be turned over to private management. Edison Schools was not to be the only private provider. Instead, seven entities—three for-profit and four nonprofit—were chosen. The SRC explained its decision by saying that multiple providers would yield information on the kind of management that was most effective. The SRC asked Edison Schools to manage 20 of the schools; another 5 each would become the responsibility of two other for-profit companies, Victory Schools and Chancellor Beacon Academies. Sixteen of the low-performing schools would be managed by nonprofit entities—the University of Pennsylvania (3 schools), Temple University (5 schools), Foundations (5 schools), and Universal (3 schools).The SRC also appointed a reform-minded superintendent, the energetic and outspoken Paul Vallas, who had instituted a series of reforms in Chicago at the behest of Mayor Richard Daley.

The for-profit firms were more experienced in running schools but they had fewer local political connections than did the nonprofit entities, as none had operated programs within the city itself. Though Edison was held in high regard by the Republican state secretary of education, it faced strong opposition within Philadelphia, especially from the local teachers union. Edison Schools could claim considerable experience at running schools, however, as it was the manager of 100 district and charter schools nationwide. Victory Schools, a company that offered single-sex education within classrooms, was the manager of schools in New York state and Baltimore, Maryland. To strengthen its local connections, Victory hired a former district employee to head up its Philadelphia effort. Chancellor Beacon operated some 80 private and charter schools, but it had not previously managed schools under contract with a school district. Shortly after the intervention began, Superintendent Vallas canceled the district’s contract with Chancellor Beacon, and its five schools were either brought back under district control or assigned to other providers.

By contrast, the nonprofit entities were—and have remained—politically well-connected institutions. The University of Pennsylvania is a Philadelphia icon, a highly prestigious Ivy League institution with a history dating back to Benjamin Franklin. Temple University’s status is less exalted, but it is nonetheless an established Philadelphia institution of higher learning. Foundations was created by one of the school district’s former associate superintendents and is staffed by many former district employees. Foundations also had close ties to a politically influential state representative active in community development programs. Universal, a community development corporation founded by Kenny Gamble, an immensely successful writer of soul music, has strong ties to Islamic leaders within Philadelphia’s black community.

Temple University and the University of Pennsylvania drew on the resources of their schools of education. Rather than taking on a general reorganization of the schools, they focused mainly on providing professional development to teachers, formal and informal assessment feedback to teachers, and within-classroom coaching services to students. Foundations operated afterschool programs and favored a teaching approach that relied on computer-based learning in which students progress at their own pace. Universal was known for boosting economic development and providing social services, but it had only limited experience managing schools.

The school district restricted the prerogatives of both for-profit and nonprofit managers in a number of ways. Management had to operate within the framework of the district’s collective bargaining agreements with its union employees, and teachers were allowed to transfer to other schools within the district if they wished. Many teachers chose to transfer, as could be expected given the fact that the schools in which they had been teaching were considered most in need of new management. The district also retained control over many aspects of school management, including the school calendar and the code of conduct for teachers and students.

All of that happened in 2002. Then over the following six years, several leadership changes occurred. Sandra Dungee Glenn was appointed by Democratic governor Edward Rendell (who had succeeded Schweiker) as the SRC’s new chair. Glenn was a former community organizer and active in Democratic politics. Superintendent Paul Vallas left for New Orleans, where he took on the daunting task of rebuilding the city’s post-Katrina school system. And Arlene Ackerman, who had previously served as superintendent in San Francisco and Washington, D.C., was hired by the SRC as the new superintendent.

Under its new leadership, the SRC removed five schools from the management of the for-profit firms (four from Edison, one from Victory) and one school from the management of a nonprofit entity (Temple). Acting upon the superintendent’s recommendation, the SRC decided that the six schools should come under direct district control once again because they had not made Adequate Yearly Progress (AYP), as required by No Child Left Behind. “It’s been six years—it is time to sort it out,” said SRC chair Glenn. Ackerman indicated that many more schools could come back under district control once a full-scale review had been undertaken.

Factors other than educational considerations may have influenced the selection of schools the district brought back under its control. As noted above, the nonprofit schools were well connected politically, while the for-profit firms were not. But it is also possible that the nonprofits were the more effective educational institutions. To see whether that was the case, we use a rigorous research design to estimate the impact of for-profit and nonprofit management in Philadelphia.

The Data

The Philadelphia school district supplied the information on which we base our analysis. Test-score, demographic, and school enrollment information on Philadelphia students in grades 2 to 8 from2001 through 2008 are available for each student. The test-score data come from three different tests. The Pennsylvania System of School Assessment (PSSA) is the test currently used for holding schools accountable for improved student learning in Philadelphia. But when the private management intervention began in the fall of 2002, that system of measuring school performance was still a work in progress. However, two other tests were given to some Philadelphia students between 2001 and 2006: the Stanford 9 and the TerraNova, both of which are nationally normed. In order to place the information from these tests on a common scale, we followed the standard practice of standardizing all scores by test, grade, and year to have a mean of zero and standard deviation of one.

We next classified schools as under for-profit management, under nonprofit management, or under regular district management. The average combined reading and math test scores one year prior to the management change at schools assigned to for-profit and nonprofit entities were 0.39 and 0.13 standard deviations below the Philadelphia average, respectively, while the pre-intervention scores of the full set of 142 regular public schools were 0.19 standard deviations above the district average. Because of that disparity, we limited the schools included in the comparison group to the lower half of all regular district schools. Those 71 schools had prior test scores that were 0.15 standard deviations below the district average, a level of performance much closer to those at the schools placed under new management. Restricting the comparison group in this way allows us to make a cleaner, if not an exact, comparison while maintaining a sufficient number of schools to be able to detect sizable management impacts at conventional levels of statistical significance.

Method

The Philadelphia intervention does not provide the opportunity for a random assignment study of the impact of for-profit and nonprofit management. Schools assigned to intervention status were not chosen randomly, but selected on the grounds that they were in greatest need of intervention. We therefore employed a “difference-in-differences” analysis to estimate the impact of attending a for-profit or nonprofit privately managed school (relative to attending that school had it remained under district management). The treatment groups consist of schools managed by each type of private provider, and the comparison group includes the regular public schools with test scores below the median for all regular district schools, as discussed above. To identify the effect of treatment, we calculate the difference between average annual test-score gains made by students at treated and comparison schools before and after the intervention began. So, for example, if test-score gains at the schools treated by for-profit management were 20 percent of a standard deviation higher than gains before treatment, while comparably measured gains at the comparison schools were only 15 percent of a standard deviation higher, the estimated effect of for-profit management would be the difference between them, or 5 percent of a standard deviation.

Our use of annual gain scores provides an estimate of treatment effects based on the extent to which students at each school do better or worse than would be expected, given their initial test scores. We also include student fixed effects, which account for changes in the composition of the schools’ student populations over time that cannot be explained by the limited set of student characteristics for which information was available in the district’s database. Finally, we control for the demographic characteristics of students’ peers and whether students have recently moved from one school to another.

The inclusion of student fixed effects means that students are compared only to themselves over time when estimating the effect of each kind of management. Estimates that include student fixed effects require at least three years of scores, which provide two changes in scores from one period to another, typically called gain scores (even though in some cases they are losses, not gains). In our analysis, there are 68,677 students for whom the data allow us to compute at least two gain scores in math (and a similar number for the reading estimation). For another 46,875 students one gain score is available. With one gain score, it is possible to use models that control for observable background characteristics, but those models cannot adjust for unobserved student characteristics (such as parental education or student commitment). We nonetheless use such models to estimate informally the impacts on all students for whom at least one gain score can be calculated. Those informal estimates require assumptions that are quite restrictive, however, so the best estimates available to us are only for the approximately 68,000 students that could be included in the model that employed a fixed-effects analysis.

Despite the tens of thousands of student observations, those in schools under private management are clustered within only 30 schools managed by for-profit organizations and 16 managed by nonprofit ones. As a result, the annual impacts of the intervention must be as much as 20 to 30 percent of a standard deviation in order to be detected at conventional levels of statistical significance. That substantial an annual impact is seldom detected for large-scale structural interventions in education. For that reason, we also discuss large impacts that fall short of statistical significance when the pattern of results is consistent over the six year time period.

Results

The impact of nonprofit management appears to have been negative. At schools under nonprofit management, students learned, on average for the six years, 21 percent of a standard deviation less in math each year than they would have had their school remained under district management. In terms of years of schooling, the negative impacts on math performance were, on average, approximately half a year’s worth of learning each year, a large effect (see Figure 1). However, the negative impact was statistically significant in only the first year after the intervention began. In reading, the average adverse impact of nonprofit management was roughly 10 percent of a standard deviation annually, about 32 percent of a year’s worth of learning. The adverse effect on reading performance was statistically significant in only the first year after the intervention began.

As mentioned earlier, impacts may have been somewhat different for the students with fewer test scores, who could not be included in our preferred model. Adjusting for that possibility yields an adverse impact of nonprofit management of 18 percent of a standard deviation on math performance and 14 percent on reading scores.

The effect of for-profit management was generally positive, although only the math impacts are statistically significant. At schools under for-profit management, students learned on average 25 percent of a standard deviation more in math each year of the six years of the intervention than they would have had the school been under district management. The estimated impact each year was roughly 60 percent of a year’s worth of learning, a large, statistically significant impact. Our adjustment using results from an alternative model that includes the larger number of students yields a positive annual impact of for-profit management on math performance of 12 percent of a standard deviation, 29 percent of a year’s worth of learning.

The estimated average annual impact on reading performance of for-profit management relative to district management is a positive 10 percent of a standard deviation, approximately 36 percent of a year’s worth of reading. However, that impact is not statistically significant. The adjusted impact was just 4 percent of a standard deviation, about 14 percent of a year’s worth of learning.

The differential impact of for-profit and nonprofit management is especially sizable. Using the estimates given above, students in schools under for-profit management gained between 70 percent and greater than a full year’s worth of learning in math more each year than they would have had the schools been under nonprofit management. All of these differences are statistically significant. In reading, students learned approximately two-thirds of a year more in a for-profit school than they would have had the school been under nonprofit management. All but one of the differences are statistically significant.

Taking Back Five Schools

Our analysis provides compelling evidence that schools do much better under for-profit than under nonprofit management. Year after year, students learned substantially more in reading and math if they attended a school under for-profit rather than one under nonprofit management. Yet in 2008 the district reassumed control of only one school under nonprofit management while not renewing the contracts for five schools under for-profit management. To ascertain whether that decision had a strong educational basis in the district’s own test-score database, we used the same methodology to estimate the impact on student learning of the five schools for which the for-profit management contract had been terminated.

The results are mixed but provide little support for the district’s decisions. The reading performance of students at the five schools was, on average, 18 percent of a standard deviation below what could have been expected had the schools been under district management, a difference that is statistically significant in three of the six years. Also, nonprofit schools whose contracts were not revoked had an impact on reading performance that was 10 percent of a standard deviation more positive than that of the for-profit schools whose contracts were terminated. However, that difference is not statistically significant in any year.

Math performance of students at the for-profit schools was 35 percent of a standard deviation higher than would have been the case had the schools been under district management. Also, it was 56 percent of a standard deviation higher than would have been the case had the schools been under nonprofit management. Those large differences are statistically significant in most years.

Our results indicate that nonprofits outperformed the five for-profits in reading in five out of six years, although the difference was only 3 percent of a standard deviation in 2008, and in no year were the differences statistically significant. In math, the five for-profits had strongly positive impacts in all years, while the nonprofits had decidedly negative ones, leading to very large, statistically significant differences between the two groups of schools in all years. The large differences in math clearly offset the statistically insignificant differences in reading.

If the Philadelphia school district cares only about reading results, and places no weight on math results, our data could be used to support the policy choice that was made, provided no attention is paid to the statistical insignificance of the reading finding. But if the two subjects are given equal weight in evaluating a school, our results provide no support for the decisions made by the school district with respect to renewing for-profit and nonprofit management contracts.

Discussion

Care should be taken before generalizing from the Philadelphia experience concerning the relative advantage of for-profit and nonprofit management. It is possible that for-profit entities have a greater vested interest in enhancing student achievement, because only in that way are they likely to survive over the longer run. But other factors in Philadelphia could easily account for the same result. The two main for-profit providers had much more experience with school management than did any of the nonprofit organizations. The nonprofits seem to have been selected more for their strong political ties than for any history of effectiveness at delivering educational services. Others have reported that newly formed charter schools under both for-profit and nonprofit management appear to become more effective as they gain in experience. That could easily account for the pattern of results reported here. Still, it is disconcerting to discover that impacts of non-profits compared unfavorably with those of the for-profits six years after the intervention began, presumably a long enough period for a new school manager to learn from experience.

Paul E. Peterson, director of the Harvard Program on Education Policy and Governance (PEPG), is editor-in-chief of Education Next.
Matthew M. Chingos is a PEPG research fellow.

Scholars have attempted to discern the effects of competition between the public and private sectors within the United States and in other countries, but no study, to our knowledge, has attempted to measure systematically the causal impact of competition by looking at variation across countries. Until now, research has been stymied by the fact that any simplistic statistical correlations between the extent of competition and student achievement that might be found are suspect. Countries where more people choose to invest in private schools may have other attributes, such as more income or a greater commitment to education, that lead to higher levels of achievement. If this is the case, any positive correlation between private schooling and student achievement could reflect a country’s income or educational commitment rather than any beneficial effects of competition. Or it may be the case that low-quality public schools increase the demand for private schooling. If so, then it could appear that competition lowered the quality of public schooling when in fact the causal connection was in the opposite direction.

In this study, we solve this conundrum by taking advantage of the historical fact that the amount of competition in education today varies from one country to another for reasons that have little to do with contemporary school quality, or national income, or commitments to education. The extent of private schooling stems in large part from the Catholic Church’s decision in the 19th century to build an alternative system of education wherever they were unable to control the state-run system.

Nineteenth-century Catholic doctrine strongly opposed Catholic attendance at state-run schools that were not controlled by the Church. In the United States, for example, Catholics perceived government-operated “common schools” to be Protestant-dominated institutions that were only ostensibly nonsectarian. Local parishes responded by establishing separate schools in which children received Catholic-infused instruction. The United States was not the only country where this happened. Catholic school systems developed in many other countries, but their size depended on the percentage of Catholics living in that country during this critical period (see sidebar). (In countries where Catholicism was the state religion, there was no perceived need for private schools, however.) As a result, even today the size of the private education sector—and thus the amount of competition between public and private schools—is related to the size of the Catholic population in 1900.

To connect the historical past to competition’s effect on achievement today requires two analytic steps. We first estimate the statistical relationship between the size of the Catholic population in 1900 and the extent of private schooling today in order to capture only that share of the private sector’s size that can be attributed to 19th-century Catholic policies—policies we assume to be otherwise unrelated to contemporary student achievement. Having estimated this relationship between Catholicity in the past and competition in the present, we then use that estimate to isolate the causal effect of private school competition on the achievement of individual students across 29 countries.

Our results confirm that countries with larger shares of Catholics but without an official Catholic state religion in 1900 have significantly larger shares of privately operated schools in 2003. More important, private school competition attributable to past Catholic policies generates higher student achievement in mathematics, reading, and science today. We also show that competition between the public and private sector positively affects the achievement of students attending public schools. Spending on education is also reduced, suggesting that school systems are more productive if they are more competitive.

PISA 2003

For the information on contemporary student achievement we rely on the well-regarded data sets compiled by the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) in 2003. Working closely with official government agencies, PISA gathered information on the mathematical, scientific, and reading literacy of nationally representative student populations in all 30 OECD countries. The term “literacy” signifies that the PISA measured not only the students’ knowledge of math, reading, and science, but also their ability to use that knowledge to meet real-life challenges. In 2003, PISA made a special effort to measure math literacy, allocating 70 percent of testing time to questions in this subject. PISA assessed the achievement of 15-year-old students in each country, regardless of the grade they attended. This means that, in most participating countries, PISA tested students nearing the end of compulsory schooling.

For purposes of this analysis, we constructed a data set that contained pupil-level test scores for about 220,000 students. We also were able to obtain from PISA student reports of their background characteristics and administrator reports on the characteristics of each student’s school, including such things as school resources and whether the school was public or private. All that information was available from 29 of the 30 OECD countries. (France had to be dropped from the analysis because it did not supply any information on the characteristics of its participating schools.)

We defined a school as private if the principal reported that it was managed directly or indirectly by a nongovernment organization (e.g., a church, trade union, business, or other private entity). A public school was defined as one being managed directly or indirectly by a public education authority, government agency, or governing board appointed by government officials or elected by public franchise. We used these definitions to calculate the share of private schools in a country. Throughout our study, this figure serves as our measure of the extent of contemporary private school competition in each country.

The size of the private sector so defined ranges widely across countries. In the Netherlands, more than three-quarters of 15-year-old students attend privately operated schools. Private school shares in Belgium, Ireland, and Korea are also well above one-half. By contrast, the share of students attending privately operated schools in Greece, Iceland, Italy , New Zealand, Norway, Poland, Sweden, and Turkey is below 5 percent. Just over 6 percent of the American 15-yearolds sampled by PISA attended private schools, a figure that corresponds closely to official estimates of private enrollment at the secondary level from the U.S. Department of Education (see Figure 1).

ESTIMATING COMPETITIVE EFFECTS

Recall that our analysis involves two steps. First, we estimate the amount of contemporary private school competition across our 29 countries that can be accounted for by the share of each country’s population that was Catholic in 1900. Where Catholicism was the official state religion, we assign a value of zero for this variable (even though the size of the Catholic population was quite large). That decision is not as odd as it sounds, as we are interested in Catholicism only insofar as it was a factor contributing to the creation of a private sector, something that clearly was not the case in those countries where Catholicism was the state religion and Catholics had no reason to object to the education provided in state-run schools.

The second step uses the connection between past Catholicism and the contemporary size of the private sector to estimate the impact of competition on student achievement. Specifically, we measure the relationship between Catholic-induced private school competition in a country and the PISA test scores of individual students in math, reading, and science.

In taking this approach, we assume that the density of Catholics in 1900 is not directly related to student achievement today, independent of effects that may occur via school competition. While this assumption cannot be proven, there are good reasons to believe it is well founded. Protestant Christians have historically placed a greater emphasis than have Catholics on the value of education, because Protestants thought individual

Bible reading helped one along the road to salvation. Catholics placed greater emphasis on remaining connected to the traditions and practices of the Church. Interestingly enough, in those 22 majority-Christian countries for which data on literacy in 1900 are available, one finds a strong negative association between Catholic population shares and literacy rates. This strong negative correlation exists even after accounting for the lower gross domestic product (GDP) per capita, which is associated with lower literacy rates, in countries with larger Catholic population shares. So to the extent that we find any beneficial effect of Catholic-induced private school competition, its size is probably depressed by cultural values related to Catholicism. In other words, our approach is more likely to yield underestimates than overestimates of competitive effects.

Of course, the historical prevalence of Catholicism could also have had other consequences, apart from a greater reliance on private schooling, that indirectly affect student achievement. For example, the share of Catholics in a country could have an effect on current GDP per capita or education spending per student. We therefore account for the effect of both of these factors in all of our analyses.

In estimating the effect of private school competition on student achievement, we also adjust for the effects of a host of other factors that can affect individual student performance. In addition to the country-level factors of per capita GDP and education spending per student, we include in our analysis information on the presence or absence of external exit exams (which research suggests are associated with higher achievement) and information on whether the country had a Communist government in 1970 (which may have affected both the size of the private sector and achievement). Student and family background characteristics used in the analysis include a student’s gender, immigration status, exposure to early childhood education, the number of books in the home, and parental occupation and work status. Finally, we account for school resources such as class size, availability of materials, teacher certification and preparation, and amount of time for instruction.

PRIVATE SCHOOL COMPETITION AND STUDENT ACHIEVEMENT

The first step of our analysis confirmed the existence of a statistically strong relationship between the extent of private school competition in 2003 and a country’s Catholic population in 1900, much as the historical record would suggest. A 10-point increase in the percentage of Catholics in 1900 is associated with a 4.7-percentage-point increase in the share of students enrolled in privately operated schools in 2003 (see Figure 2). These results support our basic reasoning that as long as Catholics could not be sure that the emerging public school systems of the 19th century would provide education in line with their church’s demands, they tended to resist state schooling and establish their own private schools alongside the state sector. The consequences of historical differences in denominational shares across countries persist to this very day.

The results from the second step of our analysis are equally striking. Let us begin with the results related to student achievement in mathematics, the subject most extensively assessed in PISA 2003. A 10-percentagepoint increase in the share of national student enrollment in private schools attributable to a historically larger share of Catholics induces an improvement in PISA math scores of 9.1 percent of a standard deviation (see Figure 3). As a benchmark for interpreting the magnitude of this effect, note that the difference in average mathematics test scores between the two grades with the largest share of 15-year-olds (9th grade and 10th grade) in the PISA study was 22.1 percent of a standard deviation. This “grade-level equivalent” provides a rough idea of how much a typical student learns during one school year. By this metric, our estimate of the effect of a 10-percentagepoint increase in private school enrollment is equivalent to 41 percent of a year’s worth of learning in high school.

Because we are able to draw on evidence from a relatively small sample of only 29 countries, the statistical precision of our estimate is not very high. That is, we can say with 95 percent confidence that the effect of a 10-percentage-point increase in the private school share is between 3.9 and 14.2 percent of a standard deviation in test scores. Still, this means we have a very high degree of confidence that the real effect is larger than zero. The bottom line is that students in countries whose larger shares of Catholic population in 1900 induced them to have larger shares of privately operated schools today performed significantly better on the PISA 2003 math test.

As an additional step to address any lingering concerns about Catholicism’s direct influence on student achievement, we conducted both stages of our analysis again, this time accounting for the relationship between contemporary differences in the share of Catholic adherents in a country and student achievement. We found that historical Catholic shares continue to be a strong predictor of the extent of private school competition in a country. In addition, the estimated effect of Catholic-induced private school shares on student achievement increases relative to our first version of the analysis, which did not account for contemporary Catholic adherence. There is now a 12.2 percent of a standard deviation increase in test scores for each 10-percentage-point increase in the private school share in a country. This larger estimate suggests that the true effect may be closer to the upper bound of the interval we identified above.

Why the stronger relationship between private school competition and student achievement? We reason that this may be because, in the latter approach, the Catholic-induced school share was reflecting the slightly negative direct effect of contemporary Catholic adherence on student achievement, a relationship that reveals itself in this version of the analysis, although the estimated effect is just shy of statistical significance. Considered together, these results increase our confidence that we are describing a real, causal relationship between private competition and student performance, rather than effects of cultural differences related to religious adherence.

The estimated effects of the private school share on student achievement are somewhat smaller in science and reading than in math, but they remain substantial, positive, and statistically significant (see Figure 2). A change in the historical Catholic population share that produces a 10-percentage-point increase in the extent of contemporary private school competition generates an improvement of about 5.5 percent of a standard deviation in both science and reading—or more than one-fifth of a grade-level equivalent in these subjects.

To gain additional insights, we also re-ran both stages of our analysis while accounting for the average share of funding that private schools receive from the government. The inclusion of this variable hardly affects our results, suggesting that our findings reflect competitive effects stemming from the private operation of schools and not from differences in funding policies.

EFFECT ON PUBLIC SCHOOL STUDENTS

The previous portions of our study investigated the impact of private competition on student achievement in the educational system as a whole. But what about the effect of private school competition on public schools? To answer this question, we removed all students attending a privately operated school from the sample in each country and analyzed only the academic achievement of students in the public sector.

These results are somewhat more difficult to interpret than our findings above, as they combine the effects from competition with the consequences of student sorting. In other words, some of what we find may be due to high-ability students (and their parents) being more likely to choose private schools, leaving the weaker students in the public sector.

Nonetheless, the results suggest that public school students profit nearly as much from increased private school competition as do a nation’s students as a whole. While our estimates of the effects are somewhat smaller than the estimates for students in both the private and public sectors, the results are not statistically distinguishable. It therefore appears that much of the increased performance of education systems with higher levels of private school competition accrues to students who attend public schools.

EDUCATION SPENDING

The analysis so far has been limited to educational outcomes, estimating the effect of private school competition on students’ achievement. In doing so, we have controlled for possible effects of differences in educational inputs such as class sizes, availability of materials, and aggregate expenditure per student in the country. We wondered, though, whether private school competition also affects the input side of the educational process, specifically educational spending per student.

We again used a two-stage process, with the first stage using historical Catholicism to predict the Catholic-induced share of current private school competition in each country. Then, in a second stage, we measured the relationship across countries between the Catholic-induced share of competition and the cumulative educational expenditure per student up to age 15—a measure that includes both public and private spending. We continued to account for a range of country- and student-level characteristics when making these comparisons, but we now excluded measures of school resources that are likely to be affected by spending levels.

Our results show that private school competition, in addition to raising student achievement, substantially reduced the average spending level of the educational system. Changes in historical shares of Catholics in the population that are associated with a 10-percentage-point increase in the private school share today lead to a $3,209 reduction in cumulative spending per student, or 5.6 percent of the average OECD spending level of $56,947 (see Figure 3).

CONCLUSION

Our findings from an international study of 29 countries speak quite clearly. Competition from private schools improves student achievement, and appears to do so for public school as well as private school students. And it produces these benefits while decreasing the total resources devoted to education, as measured by cumulative educational spending per pupil. Under competitive pressures from private schools, the productivity of the school system measured as the ratio between output and input increases by even more than is suggested by looking at educational outcomes alone. Ironically, although Catholics historically placed less emphasis on education than did adherents of many other religions, their resistance to state-run schooling in many countries helped create institutional configurations that continue to spur student achievement.

Martin R. West is assistant professor of education at Brown University and an executive editor of Education Next.
Ludger Woessmann is professor of economics at the University of Munich and heads the Department of Human Capital and Innovation of the Ifo Institute for Economic Research.

The explanation for this odd fact: since 1981 Chile has had a more comprehensive school choice system than any other country in the world, as well as a system of publicly available information on student test performance. Scholars have thus seen Chile as a place to test theories of school choice. Do students with vouchers learn more in private schools or in those run by municipalities? What is the impact of a voucher system on equality of educational opportunity? The answers to these and related questions have been just about as varied as the number of scholars who have inquired into the matter. On balance, the bulk of the research shows a small educational advantage for students who attend privately operated voucher schools rather than municipal ones. But hardly any study looks at differences among the voucher schools, and none has examined differences between private schools in networks and those that operate on a stand-alone basis. Yet interest in school networks has escalated since many operators of charter schools in the United States have begun to expand their operations beyond a single school. Some have argued that this is the ideal way for connecting school choice to school improvement. If effective schools can expand either by setting up or by “franchising” other schools, school quality can gradually improve. But others say that the formation of networks of schools will lead to a standardization that will undermine the vitality of individual school communities.

Chile is an ideal place for exploring these questions. In 2002, only 53 percent of students were still being educated in municipally run schools, which nonetheless received a good deal of their funding from the vouchers paid for by the national government. Another 9 percent of students attended fee-based private schools that were independently operated and received no government assistance whatsoever. For the most part, these were schools with well-established reputations that served the country’s upper class. The remaining students attended what might be called voucher schools, because the schools, while private, had been since 1981 heavily dependent on the subsidy that the schools received from the national government for each student they enrolled. This sector is the fastest growing segment of the Chilean educational system.

Like American charter schools (see “Brand-Name Charters,” features), Chile’s privately run voucher schools may be part of a larger organization or school network, or operate on their own. Most schools are of the stand-alone or “mom-and-pop” variety: 25 percent of all students in Chile attend such schools. But another 13 percent of students attend schools that are part of a network of two or more schools.

The schools, inside and outside of networks, vary from one another in many ways. Some are operated by teachers who once worked in municipal schools. Others are run by business entrepreneurs. Fifty-nine percent of the network schools are run by nonprofit entities, either religious or secular. For-profit organizations operate the remainder. Some schools as well as some networks are religious. Most networks, and especially those in rural areas, consist of just two or three schools. Only about 20 percent of primary (K–8) private voucher school students attend schools that belong to networks that have more than three schools (see sidebar).

Proponents of school networks say that the networks facilitate the flow of information (such as research on best practices) among schools and provide political benefits, credibility, and legitimacy in the eyes of the community. They argue that larger schooling operations have more access to private investments and loans to expand than smaller operations do. Supporting this view, research on public charter schools in the United States indicates that well-established charter school networks can build credibility for fund-raising more easily than stand-alone charter schools can. The underlying hypothesis is that, all else bring equal, the more a schooling organization facilitates transactions between members of a school’s community, the better the school’s performance. The research conducted so far shows that stand-alone charter and brand-name schools, like their district counterparts, vary widely in quality.

Critics of school networks fear unintended negative consequences. They argue that large centralized operations create hard-to-manage bureaucracies and make it difficult to maintain order and create a sense of community among students, parents, teachers, and administrators. Opponents also claim that large schooling operations grant too much authority to administrators and other professionals far removed from the classroom. Some critics are concerned that consolidation encourages standardization. For instance, they maintain that school networks must establish a brand to be successful, which necessitates relatively uniform operations and services from site to site. They argue that this branded approach to education stifles innovation.

Very little factual information is available to sort out the credibility of these claims and counterclaims. It is thus of interest to examine the Chilean experience, where both network and stand-alone voucher-subsidized schools have been operating for several decades. Information on more than one-quarter million students who were 4th graders in 2002 allows us to compare Spanish language and mathematics achievement in network and stand-alone voucher-subsidized schools. Our findings suggest that network schools in Chile are more effective than stand-alone schools, and that larger networks tend to outperform smaller networks. While we cannot be certain whether the higher performance of network schools is because good schools were the ones to expand or whether networking, by itself, had a positive impact, our results nonetheless add considerably to the sparse information currently available on a question of substantial policy interest.

School Reform in Chile

During the 1980s, the Chilean government decentralized the administration of schools, transferring responsibility for public school management from the Ministry of Education to municipalities (recognized neighborhoods in Chile around which municipal services are organized). The government also changed the education financing scheme. Municipalities began to receive funding from the central government according to the number of students who chose to attend the municipal schools. Any enrollment loss had a direct effect on their education budgets. Equally important, privately run schools that had not charged tuition began receiving the same per-student voucher as the public schools. Tuition-charging (elite) private schools mostly continued to operate without public funding.

Despite the parity in funding, significant differences remained between municipally run schools and privately run voucher schools. First, starting in 1994 municipal elementary schools were not allowed to charge parents fees, while all privately run voucher schools could. Second, municipal schools were required by law to accept all who applied. Private voucher schools, in contrast, were allowed to consider results from admissions tests and parent interviews when making admission decisions. Third, municipal schools had to comply with labor laws that made it virtually impossible to fire a low-performing teacher. Privately run voucher schools had greater freedom to terminate employment. In addition, municipal school teachers received salary increases incrementally, based on years of experience. There were no rules with regard to incremental salary increases in the private school sector.

The policies sparked a movement of students from municipal to previously existing private schools as well as the establishment of new institutions. In 1981, 15 percent of the nearly 2.9 million Chilean K–12 students had been attending private schools that received some public subsidy, and another 7 percent attended elite, unsubsidized private schools. By 1990, 34 percent of students attended privately run voucher schools; by 2002, enrollment in such schools reached 38 percent of the roughly 3.4 million in total enrollment (see Figure 2). The trend continued after 2002, the year in which the data for this study were collected. By 2005, 43 percent of students were enrolled in privately run voucher schools. As indicated above, about one-third of the voucher schools belong to networks, while the remaining two-thirds operate independently.

Beginning in 2003, after our data were collected, the Chilean government sought to alter several features of the system, although not all of the changes have been fully implemented: Rather than providing vouchers at a flat rate, voucher amounts are to be tied to family income. Private voucher schools can no longer select students in primary school; for secondary school admission, they can administer tests, but they cannot conduct parent interviews. In addition, Congress recently passed legislation that will hold schools accountable for student achievement and improvement over time.

The political debate continues. In 2006, widespread student protests of inequalities in the education system prompted debate over whether entrepreneurs should be able to own and run private voucher schools for profit. Proposed legislation, which initially prohibited for-profit education organizations, now would require that such entities make available to the public information on their profitability as well as their use of voucher funds.

Data and Methodology

Our study is based on student-level data from Chile’s national standardized test, Sistema de Medición de la Calidad de la Educación (Educational Quality Measurement System—SIMCE), which assesses students in grades 4, 8, and 10 in language, mathematics, history and geography, and natural sciences. In 2002, SIMCE evaluated 274,863 4th graders. Complementing student test scores are parent and teacher questionnaires, which include socioeconomic and environmental information on the students, their families, their peers, and their schools.

Because we lacked complete data for some schools, our study includes 252,202 students. Fifty-eight percent of the students were attending municipal schools, 24 percent were attending stand-alone schools, and 18 percent were attending network schools. In addition to separating out municipal and stand-alone private schools, our analysis subdivides the network schools into five groups—those that are in networks of two, three, four, five, and more than five schools—for a total of seven categories. (We have excluded information from the elite, independent schools that receive no government subsidy.) We compare the test scores of students in each of the seven categories, taking into account differences in the students’ socioeconomic characteristics, including parent schooling, self-reported household income, the number of non-school books in the home, and the quality of the peer groups (calculated by averaging family background and home resources for all students in the classroom). We also included some school-level control variables—whether or not the school was located in a rural area, the total number of students per school, and the average monthly tuition a school charges.

If these variables fully account for differences in student and peer demographics across the various categories of schools, then this strategy will provide unbiased evidence on the relative effectiveness of municipal, stand-alone, and network schools. We cannot account for other factors that could be significant. For example, the average student attending a privately run voucher school, whether network or stand-alone, may have parents who place a higher value on education than those of the average student attending a municipal school. Because we do not have a measure of parent commitment to education, we may confuse the effect of having a committed parent with that of attending a private school. Similarly, the “brand name” value attached to network schools may enable them to select more-qualified students, on average, than their independent counterparts.

As a result, simple comparisons of student outcomes in municipal, stand-alone, and network schools might give misleading estimates of the impact of schools on student achievement, even after adjusting for the measured characteristics of the students who attend each type of school. In order to correct for this selection bias, we restrict our analysis to differences across students in the type of school they attend that result from the types of schools available to them. More specifically, we assume that an individual’s probability of choosing a given school type is affected by the school density (that is, the number of schools per square kilometer) of each type in her municipality. All else being equal, students are more likely to choose schooling alternatives that are more densely concentrated in their municipalities. The crucial assumption made by our method is that school choice is influenced by local school supply, but school densities at the community level do not directly influence student achievement.

Though every precaution has been taken to make the comparison exact, it is still possible that our results overestimate the benefits of privately subsidized schools over municipal ones. For example, it is possible that voucher schools are to be found in greater density in higher income areas, where more parents are willing to pay additional fees for their children to attend higher-quality schools. To the extent that is happening, our results could be biased toward finding greater voucher benefits than is actually the case. For that reason, our comparisons between private and municipal schools should be interpreted cautiously. However, that potential source of bias is unlikely to affect comparisons between stand-alone private schools and network ones, the main focus of this analysis.

Results

We report our results in terms of standard deviations of student test scores. The difference in performance between American 4th and 8th graders on the National Assessment of Educational Progress is about one full standard deviation, suggesting that students improve by one quarter of a standard deviation each year. Although comparable measures of the rate of student learning are not available for Chile, researchers studying the Chilean school system typically consider a difference in student achievement of 10 percent of one standard deviation to be a small to moderate effect. Without accounting for any differences in students’ socioeconomic status, the Spanish language and mathematics test scores of students who attend network schools are considerably higher than the scores of those attending stand-alone schools. After controlling for student and peer attributes and for selection bias, we still find a substantial positive and statistically significant effect of attending a network school on student achievement. Students at network schools score 19 percent and 25 percent of a standard deviation higher than students at stand-alone schools in Spanish language and math, respectively. We also find that students at municipal schools do significantly worse than students at stand-alone schools on achievement tests (19 percent and 16 percent of a standard deviation in Spanish language and math, respectively), although, as discussed above, we are less confident in these results because of the difficulties of accounting for the selection of students into and by private schools.

Although these results provide some evidence of the effectiveness of school networks, a more precise analysis is needed to understand the optimal size of a network. We examined whether larger networks are more effective than smaller ones and found that, both with and without correcting for student and peer socioeconomic characteristics and selection bias, students at schools that are part of networks of three or more schools consistently outperform students at schools in networks of only two schools.

Figure 3 shows the results from our estimations. Students in schools in larger networks generally learned more than students in stand-alone schools. The results for Spanish language achievement show students in schools in networks with three schools learn 24 percent of a standard deviation more, those in networks of five schools learned 50 percent of a standard deviation more, and those in networks of more than five schools learned 23 percent of a standard deviation more. The effects on mathematics achievement are similar. Students who attend schools in networks with three schools learn 37 percent of a standard deviation more than students in stand-alone schools. The percentages for those in networks of five and more than five schools are 36 and 34, respectively.

Prior research in Chile and in the United States has demonstrated that, all else being equal, Catholic schools outperform public schools and other private schools. Since some of the network schools were affiliated with Catholic churches, that fact could be the explanation for the apparent positive benefits that come from networking. To determine whether that was the case, we checked whether the school owners were Catholic. Only 13 percent of the students attended such schools, however. And after adjusting for Catholic affiliation, the differences between network and stand-alone schools remained large and significant. In other words, the superior performance of network schools is not driven by the number of them that are Catholic.

Policy Implications

This paper compares the academic achievement of 4th graders in municipal schools, stand-alone schools, and network schools. Controlling for individual and peer characteristics as well as selection bias, we find that students in network schools outperformed those in stand-alone schools in both Spanish language and math. The stand-alone schools outperformed municipal schools but not by as large a margin. It also is of interest that students generally performed better in networks of large size. Most clearly, those in networks that contained three or more schools generally outperformed those in networks with only two schools.

Possible explanations for the positive school network effect include the substantial benefits of scale for employing education professionals and administrators, the bulk purchases of supplies and equipment, and the costs of implementing innovations in the curriculum. School networks may also benefit from greater access to credit and private investment than that extended to small individual schools in Chile. In addition, it may be that operating within a larger communal organization reduces agency problems; encourages interactions between parents, teachers, administrators, and students; and influences the development of professional school communities.

Of course, it is also possible that good schools are invited to join networks, while weaker schools are left on their own. In a competitive schooling environment, low-quality schools may be unable to attract students and additional resources needed to expand operations.

The results of this paper add evidence to the debate in the United States over the desirability of creating networks of charter and voucher schools. The findings provide some ground for optimism about the effects of networking on student achievement. Policies that provide incentives for schools to establish a network or to be managed by an organization that runs a network of schools may have the potential to increase educational outcomes.

Gregory Elacqua is professor at the Universidad Diego Portales in Santiago, Chile, and former policy advisor to the Minister of Education of Chile. Dante Contreras is senior researcher at the United Nations Development Program (UNDP) Chile and associate professor, Universidad de Chile. Felipe Salazar is researcher at the Universidad Diego Portales.

According to the latest international math and science assessment conducted by the Organisation of Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) (see Figure 1), the United States remains a long distance from that target. Rather than worrying about the consequences, some have begun to question what all the fuss was about. Education researcher Gerald Bracey, for example, has argued that no one has “provided any data on the relationship between the economy’s health and the performance of schools. Our long economic boom suggests there isn’t one—or that our schools are better than the critics claim.”

Truth be told, the Bracey critique is not entirely misplaced. Most commentators rely more on the commonsense understanding that countries must have good schools to succeed economically rather than presenting conclusive empirical evidence that connects what students learn in school to what subsequently happens in a nation’s economy. Even economists, the people who think the most systematically about the way in which “human capital” affects a nation’s economic future, have skirted the heart of the question by looking only at “school attainment,” namely the average number of years students remain in school.

Using average years of schooling as an indicator of a country’s human capital has at least two major drawbacks. First and foremost, the approach assumes that students in diverse school systems around the world receive the same educational benefits from a year of schooling. A year of schooling in Papua New Guinea and a year of schooling in Japan are treated as equally productive. Second, this measure does not account for learning that takes place outside the classroom—within families, among peers, or via the Internet, for example.

A more direct measure of a country’s human capital is the performance of students on tests in math and science, something that might be called the average level of “cognitive skills” among those entering a country’s work force. At one time, internationally comparable information on student performance was not available for a sufficient number of countries over a long enough period of time to allow for systematic study, which is why economists relied upon the less informative measures of school attainment. Now that test-score data for many countries over an extended period of time are readily available, it is possible to supplement measures of educational attainment with these more direct measures of cognitive skills.

In a series of studies conducted over several years, the four of us have explored the role of both school attainment and cognitive skills in economic growth. Beginning in the mid-1960s, international agencies started conducting tests of students’ performance in mathematics and science at various grade levels. We used performance on 12 of these standardized tests as rough measures of the average level of cognitive skill in a given country. With this information, we could assess how human capital relates to differences in economic growth for 50 countries from 1960 to 2000, more countries over a longer period of time than any previous study. We were also able to pay close attention to institutional factors that influence economic growth, such as openness of the economy and protection of property rights.

What we discovered gives credence to the concerns expressed in A Nation at Risk. The level of cognitive skills of a nation’s students has a large effect on its subsequent economic growth rate. Increasing the average number of years of schooling attained by the labor force boosts the economy only when increased levels of school attainment also boost cognitive skills. In other words, it is not enough simply to spend more time in school; something has to be learned there.

We also discovered that the size of the impact of cognitive skills depends on whether a nation’s economy is open to outside trade and other external influences. The more open the economy, the more important it is that a country’s students are acquiring high levels of cognitive skills. As the world becomes increasingly interdependent or “flat,” to use New York Times columnist Thomas Friedman’s familiar terminology, enhancing human capital will become increasingly critical. As the world continues to change, the United States can ill afford to rest easily on its past accomplishments.

Measuring Cognitive Skills

Reaching these conclusions required a multistep analysis. The first step was to use the 12 PISA and other international math and science assessments, dating back to 1964, to construct an index of cognitive skill levels for a large sample of countries at various points in time. Because the number of countries participating in the 12 test administrations changed from one administration to the next, and because testing agencies have made no attempt to link their results to one another, we needed to develop comparable scores for each test. This required a norm against which each test could be calibrated. Fortunately, we could construct that norm by using information from tests in the United States, the country that has had the earliest, most sophisticated, and most comprehensive system of testing. The United States has participated in all of the international tests since 1964, and it has also maintained a separate longitudinal testing system of its own, the National Assessment of Educational Progress (NAEP). With that information in hand, it was possible to calibrate scores on each of the separate international tests to one another via the connection of those tests to the NAEP. To obtain further precision, we used the variation in scores across a subset of the more-advanced developed countries to obtain an estimate of the spread in scores across countries. By following these two steps, we were able to aggregate all available scores for each country into measures of average cognitive skill levels for each country.

The 50 countries for which we were able to develop a comparable measure of cognitive skill levels include the 30 democracies that have market economies and have been accepted as members of the OECD, most of which are at a relatively high level of economic development. The other 20 countries are at lower levels of economic development. In Figure 2, you can identify top performers like Finland and Japan, average performers such as the United States and Germany, and low performers that include Albania, the Philippines, and South Africa.

Impact on Economic Growth

We wanted to use this new information to compare the economic benefits of higher levels of just school attainment with the benefits of higher levels of cognitive skills. We therefore took measures of average educational attainment and average cognitive skill levels for as many countries as possible and examined their relationship to the average annual growth rate in the country’s gross domestic product (GDP) per capita from 1960 through 2000.

First, we looked just at the impact of average school attainment on the economic growth rate. (An adjustment was made for the initial level of GDP because it is “easier” to grow if you are starting out at a lower level; that is, it is easier to copy more productive technologies than to initiate progress on your own.) When we performed this analysis, we found, as other economists before us, that when the average number of years of schooling in a country was higher, the economy grew at a higher annual rate over subsequent decades. Specifically, we found that, across the 50 countries, each additional year of average schooling in a country increased the average 40-year growth rate in GDP by about 0.37 percentage points.

That may not seem like much, but consider the fact that since World War II, the world economic growth rate has been around 2 to 3 percent of GDP annually. Lifting it by 0.37 percentage points is a boost to annual growth rates of more than 10 percent of what would otherwise have occurred, a significant amount.

But the impact of improved cognitive skills, as measured by the performance of students on math and science tests, is considerably larger. When we performed the analysis again, this time also including the average test-score performance of a country in our model, we found that countries with higher test scores experienced far higher growth rates. If one country’s test-score performance was 0.5 standard deviations higher than another country during the 1960s—a little less than the current difference in the scores between such top-performing countries as Finland and Hong Kong and the United States—the first country’s growth rate was, on average, one full percentage point higher annually over the following 40-year period than the second country’s growth rate. Further, once the impact of higher levels of cognitive skills are taken into account, the significance for economic growth of school attainment, i.e., additional years of schooling, dwindles to nothing (see Figure 3). A country benefits from asking its students to remain in school for a longer period of time only if the students are learning something as a consequence.

Another indication of the importance of education quality to economic growth lies in our ability to explain global variation in GDP growth. When we tried to account for economic growth with information only about school attainment levels and the level of a country’s GDP in 1960, we were able to explain only one-quarter of the differences we saw among countries. But when we also included cognitive skills in our statistical models of economic growth, we were able to attribute nearly three-quarters of the differences among countries to these three factors. In other words, higher levels of cognitive skill appear to play a major role in explaining international differences in economic growth.

Of course, the initial level of economic development, schooling attainment, and cognitive skills are not the only factors that affect economic growth. Could it be that some other factor we have overlooked is responsible for the close connection between test scores and economic growth?

Other economic research has identified two additional factors that affect a country’s economic growth rate: the security of its property rights and its openness to international trade. When those two factors are taken into account, the positive effect of cognitive skills on annual economic growth becomes somewhat smaller, but is still 0.63 percentage points per half of a standard deviation of test scores. This is the best available estimate of the size of the impact of cognitive skills on economic growth.

Other commonly discussed determinants of economic growth are fertility and geography. However, when we took into account the total fertility rate and common geographical proxies, such as latitude or the fraction of the land area of a country that is located in the tropics, neither of these additional variables was significantly associated with economic growth. Once again, the strong effect of cognitive skills remained clear.

We performed a variety of additional tests to assess the validity of these basic results. For example, we estimated the relationships over shorter periods of time, used different subsets of international tests, and compared smaller groups of the 50 countries.

One of our tests was particularly interesting. We thought it possible that the effect of cognitive skills could be the result of the presence in our sample of East Asian countries, most of which have both high levels of cognitive skill and rapidly growing economies. To see whether the inclusion of those countries in our study influenced our results, we excluded them from one of our models. The impact of cognitive skill remained very powerful, albeit diminished.

We also looked at cognitive skills as measured in the 1960s through the mid-1980s to see what their impact was on growth between 1980 and 2000, ensuring that the cognitive skills themselves were not caused by the economic growth. Again, our basic findings remained intact. Finally, we looked at whether a country’s estimated cognitive skills affected the earnings of immigrants working in the United States. Higher home country cognitive skills translated into higher earnings if the immigrants were educated in their homeland but not if educated in the United States.

Our commonsense understanding of the importance of good schools can thus be documented quite precisely. A highly skilled work force can raise economic growth by about two-thirds of a percentage point every year.

More Rocket Scientists or Basic Skills for All?

To gain additional insight into the relationship between cognitive skills and economic growth, we examined the separate impact of improvements at different levels of a nation’s distribution of skills. Loosely speaking, is it a few “rocket scientists” at the very top of the distribution who spur economic growth, or is it “education for all” that is needed?

To address this question, we measured the share of students in each country who reach a threshold of basic competency in mathematics and science, as well as the share of students who perform at very high levels. To estimate the importance of basic competency, we identified the share of students performing at least at a very basic level, that is, no more than one standard deviation below the international average of all OECD countries. In the average OECD country in our study, 89 percent of the students achieved at least at this very basic level. The share of students with at least basic skills ranged widely among countries, from as low as 18 percent in Peru to 97 percent in the Netherlands and Japan. To show a country’s ability to develop a large cadre of high-performing students, we identified the share of students performing at very high levels—at or above one standard deviation over the OECD average. On average across all countries, 6 percent of students performed at that high level. Once again, countries varied enormously in this respect, the percentage ranging from as low as 0.1 percent in Colombia and Morocco to 18 percent in Singapore and Korea and 22 percent in Taiwan.

Which is more important for growth—having a substantial cadre of high performers or bringing everyone up to a basic level of performance? The answer, it seems, is not one or the other but both! When we estimated the importance of each within the same model, we found each of them to be separately important to economic growth. That is, both the performance of countries in ensuring that almost all students achieve at basic levels and their performance in producing high-achieving students seem to matter.

The reasons that a substantial cadre of highly skilled citizens and near-universal basic skills matter are not difficult to imagine. Even if a country is simply making use of new technologies developed elsewhere, as is often the case in developing parts of the world, the more workers that have at least basic skills, the easier it will be for them to make use of those new technologies. Some workers need a high level of skill so they can help adapt the new technologies to their countries’ particular situation. In countries on the technological frontier, substantial numbers of scientists, engineers, and other innovators are obviously needed. But so is a labor force that has the basic skills needed to survive in a technologically driven economy.

But even if the results seem intuitively correct, they should be taken as suggestive rather than definitive, because the two measures of cognitive skills are closely related to one another and our models have difficulty in separating out the precise impact of each individually. Most countries that have a high percentage of students with very high cognitive skills also are ones in which basic skills are near universal. Conversely, countries with a substantial percentage of students lacking even basic skills tend to be those that have only a small percentage of highly skilled students. Still, that pattern is not a perfect one, so we are able, at least tentatively, to identify the impact of each type of human capital, and we are quite confident that we can recommend that countries both concentrate resources on their “best and brightest” and make sure that “no child is left behind.”

The Impact of Becoming a World Leader

What would it mean for economic growth, then, if a country like the United States, currently performing somewhat below the average of OECD countries, managed to increase its performance by 50 points (or 0.5 standard deviations) so that it would score alongside the world leaders? (On average on the PISA 2006 math and science exams, countries such as Canada and Korea scored about 50 points higher than the U.S., Hong Kong and Taiwan about 60 points higher, and Finland as many as 74 points higher.) That increase of 50 test points is exactly what George H. W. Bush and the nation’s governors together promised in 1989 the United States would achieve by the year 2000.

Unfortunately, no such gains were realized. But had the promise been fulfilled by 2000, our results suggest that GDP would by 2015 be 4.5 percent greater than in the absence of any such gains (see Figure 4). That 4.5 percent increment in GDP is equal to the total the U.S. currently spends on K–12 education. In other words, had that money effectively raised cognitive skills by the 50 test points that would have brought the United States close to world leadership, the economic returns to the country would probably have been enough to cover the entire cost of education in 2015 and after.

Figure 4 shows that the benefits of successful reform grow even more vivid when we look farther out. Over 75 years, even a reform that takes effect in 20 years (instead of the governors’ 10 years) yields a real GDP that is 36 percent higher than it would be if there was no change in the level of cognitive skills.

None of this is meant to suggest that schooling is the only factor contributing to a society’s cognitive skill development. Family, individual ability, and health combine with school quality to determine a student’s level of achievement. Yet there is every reason to believe that the single best route to higher levels of cognitive skill is strengthening a country’s education system. After all, most people think that is the system’s primary purpose.

An American Exception?

The United States has never done well on international assessments of student achievement. Instead, its level of cognitive skills is only about average among the developed countries. Yet the country’s GDP growth rate has been higher than average over the past century. If cognitive skills are so important to economic growth, how can we explain the puzzling case of the U.S.?

Part of the answer is that the United States may be resting on its historic record as the world’s leader in educational attainment. In addition, the United States has other advantages, some of which are entirely separate and apart from the quality of its schooling. The U.S. maintains generally freer labor and product markets than most countries in the world. There is less government regulation of firms, and trade unions are less powerful than in many other countries. Put more broadly, the U.S. has generally less intrusion of government in the operation of the economy, including lower tax rates and minimal government production through nationalized industries. Taken together, these characteristics of the U.S. economy encourage investment, permit the rapid development of new products and activities by firms, and allow U.S. workers to adjust to new opportunities.

Those economic institutions seem to matter on their own and in conjunction with cognitive skills. Our analyses suggest that the value of a high-quality education system is substantially diminished in closed economies. We estimate that the effect of a one-standard-deviation improvement in cognitive skills on annual economic growth is 0.9 percentage points per year in closed economies, identified by heavy restrictions on international trade, but 2.5 percentage points in open economies. It may be that rich human capital combines with a laissez-faire economy to foster robust economic growth.

It is also the case that, over the 20th century, the expansion of the U.S. education system outpaced the rest of the world. The U.S. pushed to open secondary schools to all citizens. Higher education expanded with the development of land grant universities, the GI Bill, and direct grants and loans to students. The extraordinary U.S. higher-education system is a powerful engine of technological progress and economic growth in the U.S. not accounted for in our analysis. By most evaluations, U.S. colleges and universities rank at the very top in the world.

Although the strengths of the U.S. economy and its higher-education system offer some hope for the future, the situation at the K–12 level should spark concerns about the long-term outlook for the U.S. economy, which could eventually have an impact on the higher-education system as well. The U.S. higher-education system may also be challenged by improvements in higher education across the world. Other countries are doing more to secure property rights and open their economies, which will enable them to make better use of their human capital. Most obviously, the historic advantage of the U.S. in school attainment has come to an end, as half of the OECD countries now exceed the U.S. in the average number of years of education their citizens receive. Those trends could easily accelerate in the coming decades.

Not Just a Matter of Money

Our evidence of a clear, strong relationship between cognitive skills and economic growth should encourage continued reform efforts. Improvements in mathematics performance called for by No Child Left Behind would matter, contrary to what critics sometimes suggest. Yet reformers should bear in mind that money alone will not yield the necessary improvements. Many expensive attempts around the world to improve schooling have failed to yield actual improvements in student achievement.

Economic growth flows only from reforms that bring actual improvements in cognitive skills. Identifying what works and how to implement it on a society-wide scale remains a challenge, not only for the U.S. but also for many nations across the globe. But, if we are to remain economically competitive, we need to solve the puzzle of our schools and meet the governors’ challenge. We should not, simply because we have failed to meet them in the past, decide that the goals were not legitimate or important.

Eric A. Hanushek is a senior fellow at the Hoover Institution of Stanford University. Dean T. Jamison is professor of health economics in the School of Medicine at the University of California, San Francisco. Eliot A. Jamison is an investment professional at Babcock & Brown. Ludger Woessmann is professor of economics at the University of Munich and heads the Department of Human Capital and Innovation of the Ifo Institute for Economic Research. The opinions expressed in this article are those of the authors and do not necessarily reflect those of their employers.

But the success of the charter school movement has been as uneven as it has been widespread (see Figure 1). There are remarkable differences in the number of charter schools and enrollment between states, and even between school districts within the same state. Take Arizona and Minnesota. The two states were early leaders in the charter school movement, both passing legislation highly favorable to the establishment and support of charter schools. Yet, in the 2005–06 school year, more than 10 percent of Arizona’s enrollment was in charter schools, while only 3 percent of Minnesota students attended a charter school.

The patchwork pattern of success for the charter school movement in the United States raised two big questions in our minds. What factors led some states to grant charter schools a great deal of latitude and provide solid financial support, while others adopted less permissive legislation? And, why, even among states with similar enabling legislation, do charter schools flourish in some places but not in others?

Several well-regarded researchers have tried to explain the differences in charter school legislation. We decided to build on their work, in an effort to produce a more complete account of the politics of the charter school movement. Like those conducting the previous studies, we considered the role of state demographics and party politics. We also used new data to see whether the academic performance of students in traditional public schools and the influence of teachers unions affect the strength of charter school legislation in a state. But this was only one part of our larger project. Marshalling demographic, financial, political, and school performance data from 1990 to 2004, we took the novel step of assessing patterns in the presence of charter schools and in their enrollments at both the state and local levels.

Our Approach

The first thing we needed to do was identify U.S. charter schools and their locations and determine their enrollments. The most recent comprehensive catalog of charter schools and school enrollments is the National Center for Education Statistics (NCES) Common Core of Data. We combined the NCES file with the Center for Education Reform (CER) directory of charter schools to create a master database of 3,066 schools for the 2003–04 school year. Next, we calculated the total number of charter schools and the total enrollment in charters and traditional public schools in each school district. In all, at least 1,000 of the 14,000 districts in the U.S. contained at least one charter school.

With this information in hand, we set out to answer three questions about the charter school movement at the state level. One, why did some states pass charter laws earlier than others? To answer this question, we first studied how D.C. and the 37 states that passed charter laws before 1999 differed from the remaining states that had not adopted charter laws by 1999. For the 40 states that passed a charter law by the 2003–04 school year, we also investigated how earlier and later adopters, grouped by year of the law’s enactment, differ from one another. We wanted to know, for instance, how Minnesota, the state that passed the nation’s first charter school law in 1991, is different from Maryland, which passed the most recent enabling legislation in 2003.

Two, we wondered why some states enacted laws highly favorable to charter schools while others passed more-restrictive statutes. In this part of our study, we compared states based on the rating of their laws by CER, which is an advocacy organization for charter schools. CER rates the “strength,” or permissiveness, of the laws’ provisions. CER judges each law against 10 criteria, each scored on a 1–5 scale, with a total possible score of 50 for laws most favorable to charter schools. These criteria include whether the state grants charter schools an exemption from collective bargaining, the number of chartering authorities beyond local school boards, the number of new charter schools permitted, and whether charters are granted waivers from certain state and local laws. Arizona, Michigan, and Minnesota have enacted relatively “strong” legislation, that is, legislation that provides considerable latitude to charter schools. Other states, such as Kansas, Tennessee, and Virginia, have adopted charter legislation with much more restrictive provisions. In our study, we assigned states without charter school laws a CER law strength score of 0. Although an advocacy organization may have an incentive to understate the strength of these laws, it is unlikely that any overall downward bias would create problems for our assessment of states relative to one another (and, in fact, these scores have been used in a numb