Checked by Eric A. Hanushek

The No Child Left Behind Act of 2001 (NCLB) was scheduled for reauthorization in 2007, and its future has in recent months garnered renewed attention. Yet so far, Congress has found it impossible to reach sufficient consensus to update the legislation, as competing groups want to a) keep all the essential features of the current law as a way of maintaining the pressure on schools to teach all students, b) modify the federal law by moving to a value-added or some alternative testing and accountability system, or c) eliminate federal testing and accountability requirements altogether, reverting to the days when the compensatory education law was simply a framework for distributing federal funds to school districts. Critics of NCLB’s testing and accountability requirements have a litany of complaints: The tests are inaccurate, schools and teachers should not be responsible for the test performance of unprepared or unmotivated students, the measure of school inadequacy used under NCLB is misleading, the tests narrow the curriculum to what is being tested, and burdens imposed upon teachers and administrators are excessively onerous.

But in all the acrimonious discussion surrounding NCLB, surprisingly little attention has been given to the actual impact of that legislation and other accountability systems on student performance. Now a reputable body, a committee set up by the National Research Council (NRC), the research arm of the National Academy of Sciences, has reached a conclusion on this matter. In its report, Incentives and Test-Based Accountability in Education, the committee says that NCLB and state accountability systems have been so ineffective at lifting student achievement that accountability as we know it should probably be dropped by federal and state governments alike. Further, the committee objects to state laws that require students to pass an examination for a high school diploma. There is no evidence that such tests boost student achievement, the committee says, and some students, about 2 percent, are not getting their diplomas because they can’t—or think they can’t—pass the test. The headline of the May 2011 NRC press release is frank and bold in the way committee reports seldom are: “Current test-based incentive programs have not consistently raised student achievement in U.S.; Improved approaches should be developed and evaluated.”

Needless to say, the report can be expected to play an important role in the continuing debate over NCLB. Upon its initial release, the report captured top billing, appearing on Education Week’s front page. Certainly, the NRC intends for the report to influence the NCLB conversation, rushing a draft version to the media five months before the completed report was available to the public.

Unfortunately, the NRC’s strongly worded conclusions are only weakly supported by scientific evidence, despite the fact that NRC’s stated mission is “to improve government decision making and public policy, increase public understanding, and promote the acquisition and dissemination of knowledge.”

The Report

Reports from the NRC are generally treated as highly credible. The NRC convenes panels of outside experts who volunteer their time to provide consensus opinions on issues of policy significance. And this particular panel includes a number of especially qualified researchers (see sidebar). The committee chair, Michael Hout, is a member of the National Academy of Sciences; 7 of the 17 panel members have named professorships; 2 are deans (of law and education schools); and a majority have published articles about testing, accountability, or incentives.

When it comes to gathering together the general literature, both theoretical and empirical, on the use of incentives in various contexts, the committee’s work is solidly constructed. But this strong scientific discussion of theory and empirical analysis of incentives and accountability breaks down when it comes to the committee’s core purpose: evaluating accountability regimes in education that employ incentives and tests.

The report comes to two policy conclusions: NCLB and state accountability systems have proven ineffective and state-required high-school exams are counterproductive. The unequivocal presentation of the conclusions is clearly designed to leave little doubt in the minds of policymakers. When the underlying evidence is examined, however, it becomes apparent that neither conclusion is warranted. Instead of weighing the full evidence before it in the neutral manner expected of an NRC committee, the panel selectively uses available evidence and then twists it into bizarre, one might say biased, conclusions.

Selecting Evidence

To get a grasp of the bias that motivated the report’s authors, consider how its first conclusion is phrased:

Test-based incentive programs, as designed and implemented in the programs that have been carefully studied, have not increased student achievement enough to bring the United States close to the levels of the highest achieving countries.

Note especially that the conclusion does not say that there is no evidence that testing and accountability work. It says that testing and accountability, by themselves, cannot lift the United States to the level of accomplishment reached by the world’s highest-achieving countries, an extraordinary standard for evaluating a policy innovation. To catch up to the leading countries would require gains of at least half of a standard deviation, or roughly two years of learning (see “Are U.S. Students Ready to Compete?” features, Fall 2011). No individual reform on the public agenda—neither merit pay, class size reduction, salary jumps for teachers, nor Race to the Top—can claim or even hope for anything close to that level of impact. The appropriate question is not whether testing and accountability is a panacea, but whether it has proven worthwhile.

By that more appropriate standard of judgment, the committee’s own data indicate that testing and accountability have proven effective, if not quite the spectacular success promised by those who enacted NCLB into law. The committee report tells us that the average estimated impact of these interventions is 0.08 standard deviations of student achievement. In other words, the average student in a state without accountability would have performed at the 53rd percentile of achievement had that student been in a state with an accountability system, all other things being equal.

That estimate may well be too low. The report states that “our literature review is limited to studies that allow us to draw causal conclusions about the overall effects of incentive policies and programs,” and then it goes on to describe several types of studies that would be excluded by this criterion. Where does the 0.08 come from? The committee considers a review from 2008 of 14 studies, and 4 studies conducted after that review. The review presents an average impact of 0.08. The NRC committee apparently felt no need to look any further and ignored the fact that a majority of the 14 studies would not come close to meeting its standard of enabling a “causal conclusion.” The committee determines that one of the more recent studies also supports an estimate of 0.08, although that study’s authors prefer estimates that are much higher. The 14 earlier studies and the 4 later ones produce a wide distribution of estimated impacts, but the committee makes no attempt to investigate whether the unusual estimates suggest circumstances under which accountability seems particularly effective (or ineffective). The committee chooses to emphasize the studies with negative findings (10 percent) while downplaying a number of those that have positive findings (90 percent). Thus the NRC mantra, repeated with slightly different wording throughout the report: “Despite using them for several decades, policymakers and educators do not yet know how to use test-based incentives to consistently generate positive effects on achievement and to improve education.” Apparently, the inconsistent results heralded in the press release reflect the 10 percent of studies that differed from the overwhelming majority.

Small Gains Add Up

Let us put this concern aside and consider the increment in student performance of 0.08 standard deviations of individual achievement that the committee presents as its best estimate. Is that so small an effect that it cannot justify continuation of testing and accountability? Consider that this is the average effect of a program that has been implemented on a national scale, affecting students across the country. We are hard pressed to come up with any other education program working at scale that has produced such results. Moreover, these average gains are the result of accountability systems that many people believe have important flaws. Even larger gains might be expected if those flaws could be corrected, as many experts, though not the NRC panel, have suggested.

The estimated benefits from a 0.08 standard deviation gain in student performance vastly outweigh its estimated costs. The cost of designing, administering, grading, and reporting the results from statewide examinations have been estimated at between $20 and $50 per pupil, a trivial sum considering that per-pupil education expenditures in the United States run above $12,000 annually. Most reforms—including class size reduction, merit pay, across-the-board raises for teachers, in-service training programs, or the scaling up of charter schools—would cost many, many times as much. For these innovations to have the same kick for every dollar invested, results would have to be improbably large.

The NRC, instead of considering these actual costs, suggests that implicit costs in the form of narrowed curricula are the most important, but it provides no evidence for its view.

What might the economic impact of a 0.08 standard deviation improvement in average achievement nationwide be? Along with University of Munich professor Ludger Woessmann, I have estimated the impact on U.S. Gross Domestic Product (GDP) of higher levels of student achievement. These estimates project the historical pattern of growth to determine the result of gains in student achievement, calculate the additions to GDP over the next 80 years, and discount them back to today so that they are comparable to other current investments. A 0.08 improvement has a present value of some $14 trillion, very close to the current $15 trillion level of our entire GDP, and equivalent to $45,000 for every man, woman, and child in the U.S. today. In other words, an inexpensive program that affects every student nationwide can, over the long run, have a very large impact, even if its average effect seems at first glance to be quite small. Indeed, if we figured testing cost $100 per student each year for the next 80 years and we tested all students rather than the limited grades tested now, the rate of return on the investment would be 9,189 percent. Google investors would be envious.

Several omissions from the report are also noteworthy. The report gives only passing attention to the positive impact of NCLB on the education of the most disadvantaged students, a consequence of the requirement to report performance by specific subgroups (e.g., racial and ethnic groups and the economically disadvantaged). The NRC report’s main reference to this feature of current accountability systems is that consideration of subgroup performance has added analytical difficulties because of the smaller samples.

Perhaps more telling, this panel of experts on testing and incentives makes absolutely no effort to describe how accountability programs could be improved. Being good researchers themselves, they do favor continued research on testing, however, and provide recommendations on what research should be done, which not surprisingly matches their own interests and expertise.

Lower the Bar?

The report also addresses a second, widely used accountability policy: high-school exit exams that hold students responsible for meeting a set of content standards. The report’s second conclusion reads,

The evidence we have reviewed suggests that high school exit exam programs, as currently implemented in the United States, decrease the rate of high school graduation without increasing achievement.

The panel strongly suggests that states that impose an exit exam should repeal this requirement. To understand this conclusion, it is necessary to understand the exams themselves and to evaluate the evidence behind the committee’s conclusion.

Currently, more than half of the states require that students pass a test of some sort to obtain a normal diploma (see Figure 1), and virtually all of these current requirements have been put in place since 2000. The tests almost always cover English and math, but many states add science and history. Test difficulty varies by state, but the modal level is grade 10. Although that standard may seem low, it is considerably more stringent than the standards that existed prior to 1990, when no state had a test reaching even the 9th-grade level. The current tests are not as high a barrier to high school graduation as they are often alleged to be, as a student may generally take the exam multiple times in order to achieve a passing score. And in all but three states (South Carolina, Tennessee, and Texas), students can either appeal the test result, if they feel the score misrepresents their accomplishments, or obtain a diploma by some alternative path.

The motivations for administering exit exams are to create incentives for students to apply themselves to the task of learning and to set uniform (minimum) quality standards for the state’s schools. Such content standards provide guidelines to schools about what to teach. They also indicate to colleges and universities what knowledge and abilities a graduate can be expected to possess. And they give similar information to prospective employers.

According to the best available evidence (discussed below), perhaps 2 percent of students are induced to drop out of school either because of failure to pass the exam or because of fear of not being able to pass the exam. Implicitly, the committee assumes this consequence does considerable harm to the affected students, given the substantial economic rewards that accrue, on average, from receiving a high school diploma. But average effects do not necessarily apply to the 2 percent on the border line between graduating and failing to graduate from high school. The impact for this particular group of students is likely to be much less, unless you make the bizarre assumption that it is only the diploma—not what the student learns—that affects job prospects and future income. The people who are induced to drop out because they cannot pass a 10th-grade exam would most likely be near the bottom of the earnings distribution of graduates were they to be handed a diploma. The economic impact on these students will be much lower than the average difference between graduate and dropout.

Perhaps the best argument against exit exams is simple: If a student shows up for school for 12-plus years and cannot pass a 10th-grade exam, it must be the school’s fault, and it would be unfair to hold the student responsible. This argument, interestingly enough, is the precise opposite of one of the primary arguments against the testing and accountability provisions of NCLB: We should not hold schools responsible for low achievement, because achievement is affected by student motivation and family background characteristics beyond the school’s control. Taken together, the arguments embedded in the committee’s two conclusions imply that nobody—not schools, not teachers, not even students themselves—bears responsibility for low student achievement.

Interestingly, the committee’s conclusion with respect to exit exams does not pick up on the full report’s emphasis on the importance of the design features of incentive systems, which include warnings that tests aimed at ensuring minimum competency may lower expectations, and concerns about both the potential narrowing of the curriculum and the tendency for score inflation on a known test. Instead, the presumed problem is the inherent unfairness of denying a diploma to a student who has met the attendance and course distribution requirements for a diploma.

If the main objective is to maximize high school graduation, there are many ways to do that. We could eliminate all exams, even those administered by teachers. We could loosen up course requirements. We could offer the diploma after 10 or 11 years of schooling, instead of 12. Of course, nobody is willing to take such steps, even though class exams, course requirements, and the inclusion of the 12th grade of schooling all have negative impacts on graduation rates. So why then does the NRC promote the idea of eliminating a 10th-grade-level examination as a requirement for high school graduation on the narrow basis that a few students will, as a result, not earn the degree? Is the NRC also against the movement of many states toward increasing the required amount of math or moving to college and career-ready standards?

The Data Shuffle

Let’s examine the evidence the committee supplies for its exit exam conclusion. The report marshals three studies that explore the issue: two on dropouts and one on achievement. Evaluating the impact of exit exams on achievement is inherently difficult. Because the exams apply to everybody in a state at the same time, it is not possible to compare students of the same age within the same state to find out the impact of exams. It is possible, however, to look at different cohorts of students, for example, those who attended school before the exam was in place and those who attended after, and to compare these to similar cohorts in other states where no such change in policy took place. In conducting this type of study, one must rule out other differences, such as those in family background or those in state education policies that might also affect student performance over time. Even when these challenges are met, one cannot be entirely sure of the results, as exit exams may influence student and school performance even before they come into effect, if teachers and students know that they will soon be introduced, which is usually the case.

The committee tosses out every exit-exam study (save three) that has ever been conducted on the grounds that it is not possible “to draw causal conclusions about the overall effects of test-based incentives” (that is, the very same criteria the committee ignored in considering school-level accountability). Some of the excluded studies use the well-regarded quasi-experimental technique known as regression discontinuity analysis. In the committee’s view, “Such regression discontinuity studies provide interesting causal information about the effect of being above or below the threshold, but they do not provide information about the overall effect of implementing an incentives program.” That criticism is odd, since the impact of an exit examination is of special interest for exactly those students on the cusp of adequate levels of achievement. While these excluded studies are not really appropriate for studying achievement, they tend to show little impact of exit exams on dropout behavior or graduation outcomes.

The committee relies for its conclusion regarding exit examinations exclusively on a 2009 study by Eric Grodsky, John Robert Warren, and Demetra Kalogrides. Because of the significance of this piece of research for the committee project as a whole, it is worth considering in some depth. The Grodsky team identified trends in student achievement in each state that administers an exit examination by drawing on data provided by the long-term trend assessments of the National Assessment of Educational Progress (NAEP). The long-term NAEP, begun in the late-1960s and continued with testing every few years, was designed to provide consistent score information to judge achievement of the nation as a whole. It was not designed to be used to evaluate the schools of any particular state or district. As a result, NAEP never collected in its long-term trend assessment a representative sample of students for any specific state, and the median number of tested students in each state was very small.

Grodsky et al. pretend that the NAEP provides them with just that: a representative sample of students for each state. They assume that the average performance of students in each state on the long-term NAEP provides an accurate measure of the average performance of students in that state, thereby violating the first principle of statistical sampling.

They then merge the information with information on the timing of the adoption of an exit exam by a state between 1971 and 2004. The study includes observations of math and reading achievement at 9 and 10 different points in time, respectively. The researchers report results for achievement of 13-year-olds and 17-year-olds separately, acknowledging that there are limitations to using either cohort. Thirteen-year-olds may be too young to detect the impact of exit exams, while the sample of 17-year-olds suffers from the noninclusion of school dropouts.

The Grodsky analysis encounters a further difficulty. For the most part, the researchers consider only the very early years, when exit exams were first introduced, a time when the exams were set at a very low level of difficulty, below that of a 9th-grade student. Only 1 percent of the observations included in their analysis are for states that had an exit exam rated at the 9th-grade level or higher, as most current examinations are.

Not only does the Grodsky team rely on inadequate data, but the analysis itself is flawed. Any attempt to see the effects of state tests should compare the changes that occur in the states that introduce them with changes in the states that do not. But the Grodsky study effectively tosses out all the information available for the 27 states that do not have an exit examination before 2004. As important, the analysis does not consider any measures of state policies except for exit exams, implying that any other policy changes for the three decades between 1971 and 2004 are either irrelevant for student performance or are not correlated with the introduction and use of exit exams.

The central finding is that exit exams do not have a statistically significant effect on test scores. But this insignificance could arise because of any or all of the above-mentioned problems rather than the absence of an effect of exit exams, as the NRC committee wants us to presume.

The committee’s estimate of the effects of exit exams on school dropout rates is less controversial. It relies on two quite reliable studies, although they are not without limitations: they study the effects of specific exit exams, which may not generalize to other arrangements. The studies indicate that perhaps 2 percent of potential high-school graduates would have received the diploma had it not been for the exit exams.

The committee touts the possibility of alternative incentives to exit exams: “Several experiments with providing incentives for graduation in the form of rewards, while keeping graduation standards constant, suggest that such incentives might be used to increase high school completion.” The key of course is just what the phrase “while keeping graduation standards constant” means. The idea behind exit exams is to ensure a minimum level of quality, as distinct from meeting the course completion requirements. Moreover, the report never makes the case that exit exams and other potential incentive programs are mutually exclusive. In principle, nobody would argue against employing other incentive programs as long as they were worth the expense and, as the committee says elsewhere, do not introduce perverse incentives of one kind or another.

The Takeaway

The NRC clearly wants to enter into the current debate about the reauthorization of NCLB. And the NRC has an unmistakable opinion: its report concludes that current test-based incentive programs that hold schools and students accountable should be abandoned. The report committee then offers three recommendations: more research, more research, and more research. But if one looks at the evidence and science behind the NRC conclusions, it becomes clear that the nation would be ill advised to give credence to the implications for either NCLB or high-school exit exams that are highlighted in the press release issued along with this report.

The framing of policy in the NRC report is simple: “The small or nonexistent benefits that have been demonstrated to date suggest that incentives need to be carefully designed and combined with other elements of the educational system to be effective.” Nobody would oppose careful design of incentives. Nobody would oppose evaluating the intended and unintended outcomes of incentives. And nobody would oppose combining carefully designed incentives with “other elements of the educational system to [make them] effective.”

The NRC is careful to offer no guidance on how NCLB or state exit exams might be modified to make them more effective. And the NRC is very careful not to offer any guidance on “other elements of the educational system.” The message that comes through is clear: keep working on test development, but never use tests for any incentive or policy purposes.

A better takeaway message might be, “Never rely on the conclusions of this NRC report for any policy purpose.”

Eric Hanushek is senior fellow at the Hoover Institution of Stanford University and member of the Koret Task Force on K–12 Education.

A  more complete integration of testing, accountability, and teaching would be superior to dealing with the integrity of testing in isolation.   Let’s put the tests out in the sun instead of trying to lock them up in more and more secure rooms.

The use of student outcome measures for accountability is now firmly entrenched and is not about to go away.  But a variety of complaints about the current testing system exist.   First, the tests tend to narrow the teaching to just what is expected on the tests – with excessive teaching to the test and drilling on  practice tests.  Second, the tests are too easy for students in some schools and too hard for others, either wasting the time of some or frustrating others.  Third, using tests for accountability purposes encourages cheating and other ways to evade scrutiny.

To address these different issues, we need to think differently about aims and means.  Here is a brief proposal to deal with all of the problems.  It starts with developing a large item bank of test questions of varying difficulty.  Imagine 1,500 questions for fourth grade math that cover the entire scope of appropriate material from basic to advanced topics.  Next, make all of the test items – not just sample items – publicly available and encourage teachers to teach to the test, because the items cover the full range of the desired curriculum.  Making the items public will also ensure the quality of the test items.  One could invite feedback ratings or open sourcing to provide a path to improving the questions over time.  Then, move to computerized adaptive testing, where answers to an initial set of questions move the student to easier or more difficult items based on responses.  This testing permits accurate assessments at varying levels while lessening test burden from excessive questions that provide little information on individual student performance.  Such assessments would not be limited to minimally proficient levels that are the focus of today’s tests, and thus they could provide useful information to districts that find current testing too easy.  Students would be given a random selection of questions, and the answers would go directly into the computer – bypassing the erasure checks, the comparison of responses with other students, and the like.

This proposal actually follows the current testing by the FAA of knowledge needed to obtain a private pilot license.  While there are commercial books on these tests, replete with questions and answers, the efficient way to prepare for the tests is simply to learn the underlying concepts.  It is not to attempt to memorize the answers, because it is easy to confuse such an attempt.

What are the potential problems?  Some say developing test items would make this too costly, but remember that it is only necessary to have one item bank, not the continually changed banks of today.  Some worry about ensuring that sufficient computers are available in all schools, but with all of the digital devices currently in use, surely there are a range of possibilities to deliver the tests effectively and efficiently.  There is the problem that the testing companies would not particularly like the proposal.   They find they are happy with mindlessly developing slightly different variants of existing tests for different states, years, and administrations.  But, maybe there are more productive ways for them to enter into the process.

The proposed system would yield quick and reliable feedback on student achievement, would deal with the various cheating and gaming issues, and would more effectively define what students should know than the currently available standards.

-Eric A. Hanushek

Eric Hanushek is a senior fellow at the Hoover Institution of Stanford University and member of the Koret Task Force on K-12 Education.

On August 17, 2011 Paul Peterson discussed the findings of this study in a free online webinar. An archived recording of this webinar can be found here.

At a time of persistent unemployment, especially among the less skilled, many wonder whether our schools are adequately preparing students for the 21st-century global economy. Despite high unemployment rates, firms are experiencing shortages of educated workers, outsourcing professional-level work to workers abroad, and competing for the limited number of employment visas set aside for highly skilled immigrants. As President Barack Obama said in his 2011 State of the Union address, “We know what it takes to compete for the jobs and industries of our time. We need to out-innovate, out-educate, and out-build the rest of the world.”

The challenge is particularly great in math, science, and engineering. According to Internet entrepreneur Vinton Cerf, “America simply is not producing enough of our own innovators, and the cause is twofold—a deteriorating K–12 education system and a national culture that does not emphasize the importance of education and the value of engineering and science.” To address the issue, the Science, Technology, Engineering, and Math (STEM) Education Coalition was formed in 2006 to “raise awareness in Congress, the Administration, and other organizations about the critical role that STEM education plays in enabling the U.S. to remain the economic and technological leader of the global marketplace.” Tales of shortages of educated talent appear regularly in the media. According to a CBS News report, 22 percent of American businesses say they are ready to hire if they can find people with the right skills. As one factory owner put it, “It’s hard to fill these jobs because they require people who are good at math, good with their hands, and willing to work on a factory floor.” According to a Bureau of Labor Statistics report, of the 30 occupations projected to grow the most rapidly over the next decade, nearly half are professional jobs that require at least a college degree. On the basis of these projections, McKinsey’s Global Institute estimates that over the next few years there will be a gap of nearly 2 million workers with the necessary analytical and technical skills.

In this paper we view the proficiency of U.S. students from a global perspective. Although we provide information on performances in both reading and mathematics, our emphasis is on student proficiency in mathematics, the subject many feel to be of greatest concern.

Student Proficiency on NAEP

At one time it was left to teachers and administrators to decide exactly what level of math proficiency should be expected of students. But, increasingly, states, and the federal government itself, have established proficiency levels that students are asked to reach. A national proficiency standard was set by the board that governs the National Assessment of Educational Progress (NAEP), which is administered by the U.S. Department of Education and generally known as the nation’s report card.

In 2007, just 32 percent of 8th graders in public and private schools in the United States performed at or above the NAEP proficiency standard in mathematics, and 31 percent performed at or above that level in reading. When more than two-thirds of students fail to reach a proficiency bar, it raises serious questions. Are U.S. schools failing to teach their students adequately? Or has NAEP set its proficiency bar at a level beyond the normal reach of a student in 8th grade?

One way of tackling such questions is to take an international perspective. Are other countries able to lift a higher percentage—or even a majority—of their students to or above the NAEP proficiency bar? Another approach is to look at differences among states. What percentage of students in each state is performing at a proficient level? How does each state compare to students in other countries?

In this article, we report results from our second study of student achievement in global perspective conducted for Harvard’s Program on Education Policy and Governance (PEPG). In our 2010 PEPG report, we compared the percentage of U.S. public and private school students in the high-school graduating Class of 2009 who were performing at the advanced level in mathematics with rates of similar performance among their peers around the world (see “Teaching Math to the Talented,” features, Winter 2011). The current study continues this work by reporting proficiency rates in both mathematics and reading for the most recent cohort for which data are available, the high-school graduating Class of 2011.

Comparing U.S. Students with Peers in Other Countries

If the NAEP exams are the nation’s report card, the world’s report card is assembled by the Organization for Economic Co-operation and Development (OECD), which administers the Program for International Student Assessment (PISA) to representative samples of 15-year-old students in 65 of the world’s school systems (which, to simplify the presentation, we shall refer to as countries; Hong Kong, Macao, and Shanghai are not independent nations but are nonetheless included in PISA reports). Since its launch in 2000, the PISA test has emerged as the yardstick by which countries measure changes in their performance over time and the level of their performance relative to that of other countries.

Since the United States participates in the PISA examinations, it is possible to make direct comparisons between the average performance of U.S. students and that of their peers elsewhere. But to compare the percentages of students deemed proficient in math or reading, one must ascertain the PISA equivalent of the NAEP standard of proficiency. To obtain that information, we perform a crosswalk between NAEP and PISA. The crosswalk is made possible by the fact that representative (but separate) samples of the high-school graduating Class of 2011 took the NAEP and PISA math and reading examinations. NAEP tests were taken in 2007 when the Class of 2011 was in 8th grade and PISA tested 15-year-olds in 2009, most of whom are members of the Class of 2011. Given that NAEP identified 32 percent of U.S. 8th-grade students as proficient in math, the PISA equivalent is estimated by calculating the minimum score reached by the top-performing 32 percent of U.S. students participating in the 2009 PISA test. (See methodological sidebar for further details.)

What It Means to Be Proficient

According to the National Center for Education Statistics (NCES), which administers NAEP, the determination of proficiency in any given subject at a particular grade level “was the result of a comprehensive national process [which took into account]…what hundreds of educators, curriculum experts, policymakers, and members of the general public thought the assessment should test. After the completion of the framework, the NAEP [subject] Committee worked with measurement specialists to create the assessment questions and scoring criteria.” In other words, NAEP’s concept of proficiency is not based on any objective criterion, but reflects a consensus on what should be known by students who have reached a certain educational stage. NAEP says that 8th graders, if proficient, “understand the connections between fractions, percents, decimals, and other mathematical topics such as algebra and functions.”

PISA does not set a proficiency standard. Instead, it sets different levels of performance, ranging from one (the lowest) to six (the highest). A student who is at the proficiency level in math set by NAEP performs moderately above proficiency  level three on the PISA. (See sidebar for a statement of the 8th-grade proficiency standard and sample questions from PISA and NAEP that proficient students are expected to pass.)

Crossing the Proficiency Bar

Given that definition of math proficiency, U.S. students in the Class of 2011, with a 32 percent proficiency rate, came in 32nd among the nations that participated in PISA. Performance levels among the countries ranked 23rd to 31st are not significantly different from that of the U.S. in a statistical sense, yet 22 countries do significantly outperform the United States in the share of students reaching the proficiency level in math. Six countries plus Shanghai and Hong Kong had majorities of students performing at least at the proficiency level, while the United States had less than one-third. For example, 58 percent of Korean students and 56 percent of Finnish students performed at or above a proficient level. Other countries in which a majority—or near majority—of students performed at or above the proficiency level included Switzerland, Japan, Canada, and the Netherlands. Many other nations also had math proficiency rates well above that of the United States, including Germany (45 percent), Australia (44 percent), and France (39 percent). Figure 1 presents a detailed listing of the scores of all participating countries as well as the performance of the individual states within the United States.

Shanghai topped the list with a 75 percent math proficiency rate, well over twice the 32 percent rate of the United States. However, Shanghai students are from a prosperous metropolitan area within China, so their performance is more appropriately compared to Massachusetts and Minnesota, which are similarly favored and are the top performers among the U.S. states. When this comparison is made, Shanghai still performs at a distinctly higher level. Only a little more than half (51 percent) of Massachusetts students are proficient in math, while Minnesota, the runner-up state, has a math proficiency rate of just 43 percent.

Only four additional states—Vermont, North Dakota, New Jersey, and Kansas—have a math proficiency rate above 40 percent. Some of the country’s largest and richest states score below the average for the United States as a whole, including New York (30 percent), Missouri (30 percent), Michigan (29 percent), Florida (27 percent), and California (24 percent).

Click image to enlarge

Proficiency in Reading

According to NAEP, students proficient in reading “should be able to make and support inferences about a text, connect parts of a text, and analyze text features.” According to PISA, students at level four, a level of performance set very close to the NAEP proficiency level, should be “capable of difficult reading tasks, such as locating embedded information, construing meaning from nuances of languages critically evaluating a text.” (See sidebar for more specific definitions and sample questions.)

Click image to enlarge

The U.S. proficiency rate in reading, at 31 percent, compares reasonably well to those of most European countries other than Finland. It takes 17th place among the nations of the world, and only the top 10 countries on PISA outperform the United States by a statistically significant amount. In Korea, 47 percent of the students are proficient in reading. Other countries that outrank the United States include Finland (46 percent), Singapore,  New Zealand, and Japan (42 percent), Canada (41 percent), Australia (38 percent), and Belgium (37 percent).

Within the United States, Massachusetts is again the leader, with 43 percent of 8th-grade students performing at the NAEP proficiency level in reading. Shanghai students perform at a higher level, however, with 56 percent of its young people proficient in reading. Within the United States, Vermont is a close second to its neighbor to the south, with 42 percent proficiency. New Jersey and Montana come next, both with 39 percent of the students identified as proficient in reading. The District of Columbia, the nation’s worst, are at the level achieved in Turkey and Bulgaria, while the one-eighth of our students living in California are similar to those in Slovakia and Spain. (See Figure 2 for the international ranking of all states.)

Click image to enlarge

Ethnic Groups

The percentage proficient in the United States varies considerably among students from different racial and ethnic backgrounds. While 42 percent of white students were identified as proficient in math, only 11 percent of African American students, 15 percent of Hispanic students, and 16 percent of Native Americans were so identified. Fifty percent of students with an ethnic background from Asia and the Pacific Islands, however, were proficient in math, placing them at a level comparable to students in Belgium, Canada, and Japan.

In reading, 40 percent of white students and 41 percent of those from Asia and the Pacific Islands were identified as proficient. Only 13 percent of African American students, 5 percent of Hispanic students, and 18 percent of Native American students were so identified.

Given the disparate performances among students from various cultural backgrounds, it may be worth inquiring as to whether differences between the United States and other countries are due to the presence of a substantial minority population within the United States. To examine that question, we compare U.S. white students to all students in other countries. We do this not because we think this is the right comparison, but simply to consider the oft-expressed claim that education problems in the United States are confined to certain segments within the minority community.

While the 42 percent math efficiency rate for U.S. white students is considerably higher than that of African American and Hispanic students, they are still surpassed by all students in 16 other countries. White students in the United States trail well behind all students in Korea, Japan, Finland, Germany, Belgium, and Canada.

White students in Massachusetts outperform their peers in other states; 58 percent are at or above the math proficiency level. Maryland, New Jersey, and Texas are the other states in which a majority of white students is proficient in math. Given recent school-related political conflicts in Wisconsin, it is of interest that only 42 percent of that state’s white students are proficient in math, a rate no better than the nation as a whole.  (Results for all states are presented in the unabridged version of the paper.)

In reading, the picture looks better. As we mentioned above, only 40 percent of white students are proficient, but that proficiency rate would place the United States at 9th in the world. Its proficiency rate does not differ significantly (in a statistical sense) from that for all students in Canada, Japan, and New Zealand, but white students trail in reading by a significant margin all students in Shanghai, Korea, Finland, Hong Kong, and Singapore. In no state is a majority of white students proficient, although Massachusetts comes close with a 49 percent rate. The four states with the next highest levels of reading proficiency among white students are New Jersey, Connecticut, Maryland, and Colorado.

Are the Proficiency Standards the Same for Math as for Reading?

Has NAEP set a lower proficiency standard in math than in reading? If so, is the math standard too low or the reading bar too high?

At first glance it would seem that the standard is set at pretty much the same level. After all, 32 percent of U.S. students are deemed proficient in math and 31 percent are deemed proficient in reading.

But that coincidence is quite misleading. When compared to peers abroad, the U.S. Class of 2011 performed respectably in reading, trailing only 10 other nations by a statistically significant amount. Admittedly, the U.S. trails Korea by 16 percentage points, but it’s only 10 percentage points behind Canada. Meanwhile, U.S. performance in math significantly trails that of 22 countries. Korean performance is 26 percentage points higher than that of the United States, while Canadian performance is 18 percentage points higher. Judged by international standards, U.S. 8th graders are clearly doing worse in math than in reading, despite the fact that NAEP reports similar percentages proficient in the two subjects.

A direct comparison of NAEP’s proficiency standard with PISA’s proficiency levels three and four also indicates that a lower NAEP bar has been set in math than in reading. To meet NAEP’s standards currently, one needs to perform near the fourth level on PISA’s reading exam, but only modestly above the third level on its math exam.

Clearly, the experts set an 8th-grade math proficiency standard at a level lower than the one set in reading. Perhaps this is an indication that American society as a whole, including the experts who design NAEP standards, set lower expectations for students in math than in reading. If so, it is a sign that low performance in mathematics within the United States may be deeply rooted in the nation’s culture. Those who are setting the common core standards under discussion might well take note of this.

Of course, it could be argued that the math proficiency standard is correct but the reading standard has been set too high. In no country in the world does a majority of the students reach the NAEP proficiency bar set in 8th-grade reading.

What Does It Mean?

Many have concluded that the productivity of the U.S. economy could be greatly enhanced if a higher percentage of U.S. students were proficient in mathematics. As Michael Brown, Nobel Prize winner in medicine, has declared, “If America is to maintain our high standard of living, we must continue to innovate…. Math and science are the engines of innovation. With these engines we can lead the world.”

But others have argued that the overall past success of the U.S. economy suggests that high-school math performance is not that critical for sustained growth in economic productivity. After all, U.S. students trailed their peers in the very first international survey undertaken nearly 50 years ago. That is the wrong message to take away however. Other factors contributed to the relatively high rate of growth in economic productivity during the last half of the 20th century, including the openness of the country’s markets, respect for property rights, low levels of political corruption, and limited intrusion of government into the operations of the marketplace. The United States, moreover, has always benefited from the in-migration of talent from abroad.

Furthermore, the United States has historically had far higher levels of educational attainment than other countries, with many more students graduating from high school, continuing on to college, and earning an advanced degree. It appears that in the past the country made up for low quality in elementary and high school by educating students for longer periods of time.

As we proceed into the 21st century, none of these factors remains as favorable to the United States. While other countries are lifting restrictions on market operations, the opposite has been occurring within the United States. The U.S. has also placed sharp limits on the numbers of talented workers that can be legally admitted into the country. Our higher education system, though still perceived to be the best in the world, is recruiting an ever-increasing proportion of its faculty and students from outside the country. Meanwhile, educational attainment rates among U.S. citizens now trail the industrial-world average.

Even if some of these trends can be reversed, that hardly gainsays the desirability of enhancing the mathematical skills of the U.S. student population, especially at a time when the nation’s growth in productivity is badly trailing growth rates in China, India, Brazil, and many smaller Asian countries. Eric Hanushek and Ludger Woessmann have shown elsewhere that student performance on international tests such as those we consider here is closely related to long-term economic growth (see “Education and Economic Growth,” research, Spring 2008). Assuming past economic patterns continue, the country could enjoy a remarkable increment in its annual GDP growth per capita by enhancing the math proficiency of U.S. students. Increasing the percentage of proficient students to the levels attained in Canada and Korea would increase the annual U.S. growth rate by 0.9 percentage points and 1.3 percentage points, respectively. Since current average annual growth rates hover between 2 and 3 percentage points, that increment would lift growth rates by between 30 and 50 percent.

When translated into dollar terms, these magnitudes become staggering. If one calculates these percentage increases as national income projections over an 80-year period (providing for a 20-year delay before any school reform is completed and the newly proficient students begin their working careers), a back-of-the-envelope calculation suggests gains of nothing less than $75 trillion over the period. That averages out to around a trillion dollars a year. Even if you tweak these numbers a bit in one direction or another to account for various uncertainties, you reach the same bottom line: Those who say that student math performance does not matter are clearly wrong.

Given the integration of the world economy, a global perspective is needed for assessing the performance of U.S. schools, districts, and states. High-school graduates in each and every state compete for jobs with graduates from all over the world. Charles Vest, president of the National Academy of Engineering and president emeritus at Massachusetts Institute of Technology, has warned, “America faces many challenges…but the enemy I fear most is complacency. We are about to be hit by the full force of global competition. If we continue to ignore the obvious task at hand while others beat us at our own game, our children and grandchildren will pay the price. We must now establish a sense of urgency.”

Paul E. Peterson is the director of Harvard’s Program on Education Policy and Governance and senior fellow at the Hoover Institution. Ludger Woessmann is professor of economics at the University of Munich. Eric A. Hanushek is senior fellow at the Hoover Institution of Stanford University. Carlos X. Lastra-Anadón is a research fellow at the Program on Education Policy and Governance at Harvard University. An unabridged version of this paper is available here.

Let us look at what is happening, since this might be a direction in which other states consider moving.  As most people are aware, the budget problems in California were clear even before the recession put added pressure on the state.  The traditional California way to deal with these budget problems has been to shroud them in smoke and mirrors – doing budgetary manipulations that make it look like the budget is balanced but that only move the problem into the future.

This year, the new Governor, Jerry Brown, set out to deal with the budget honestly, including putting in place contingency spending reductions if revenues did not come in at the level anticipated in the budget.  This action leads to uncertainty in school district budgets, because there is a reasonable chance that the state may reduce funding midyear.

And here is where the California legislature showed the kind of leadership that makes a mockery of the idea that school policy is about the kids.  At the behest of the California Teachers Association, the legislature declared that to deal with the fiscal situation, none of the thousand school districts in California is permitted to lay off any teachers.  What can it do?  By this legislation, it can eliminate up to seven days from the school year (as long as the local union agrees to that action).

California is the 47^th ranked state in terms of achievement, and it educates one-eighth of the nation’s students.  Perhaps the ranking is more obvious to everybody after we see the kind of policies that are being made.

Nationally, the fiscal problems of schools have left many districts scrambling to figure out how to deal with budget shortfalls.  Hopefully no other state will follow this precedent.

-Eric A. Hanushek

The fundamental problem with the report (Incentives and Test-based Accountability in Education) is dramatically evidenced in the first sentence of the first conclusion:

Test-based incentive programs, as designed and implemented in the programs that have been carefully studied, have not increased student achievement enough to bring the United States close to the levels of the highest achieving countries.

This statement invites a sound bite such as ‘we should move away from accountability, because it has not done much.’  Indeed, the first sentence of the National Academies own press release says “Despite being used for several decades, test-based incentives have not consistently generated positive effects on student achievement.”

When you parse the conclusion, you see that it refers to programs: (1) as designed; (2) as implemented; (3) that have been carefully studied; and – the topper – (4) that have not by themselves closed the achievement gaps with the highest performing countries.   Digging into the report, one finds a nice research review suggesting that conditions (1)-(3) are based on a fairly thin evidentiary base but one that generally suggests positive impacts of accountability.

The biggest problem with the conclusion (and the report) lies in the casual “compared to what” measure that is adopted.   Why would we discard an effective program just because it falls short of our hopes of producing the world’s best education?  It is generally inappropriate to use words like “silly” in academic discussions, but . . .

Nowhere does the report indicate an alternative educational program that leads to as large an improvement in overall U.S. achievement as accountability.  Nowhere does the report suggest any single program or package of reforms that would close the achievement gap with the highest performing countries.  Nowhere does the report really make the case that alternative reform packages should not include an accountability component.

Let’s quickly put the report’s overall judgment into perspective.  The report speaks quite dismissively of the estimates of achievement gains of 0.08 standard deviations.   Ludger Woessmann and I have analyzed how achievement relates to national economic growth.  If the future follows the patterns we have seen historically, the present value of achievement gains of this magnitude would be over $13 trillion.  Given that our GDP is currently $15 trillion, I personally think such gains are worth considering.

The second major shortcoming of the report follows from their nice review of incentive and testing issues.  Most scholars would agree that it is insufficient simply to trot out the well-known list of potential problems with accountability and incentive schemes.  Many have done this before.  Moreover, others have taken the list and the available evidence as the basis for framing how the existing but imperfect accountability schemes could be modified in order to improve on the first generation of plans.  Unfortunately, that was not the focus or interest of this panel.  They make virtually no effort to provide their professional opinions about how the incentives of accountability systems could be strengthened or how the testing of achievement could be improved.  It seems like Shakespeare all over:  “I come to bury Caesar, not to praise him.”

The remarkable conclusion to be drawn from the evidence presented in the report is how much can be gained from a flawed accountability system – again, think trillions of dollars.  Imagine what might be possible if we improved the system along the lines that many others have described (and that can be inferred from the analysis buried within this report).

-Eric A. Hanushek

The rhetoric of class size policy has been virtually constant for the decade-and-a-half before this year.  If one carefully culls the research literature, it is possible to find a set of studies that conclude that achievement will improve with smaller classes.  It did not take much of a sales pitch to convince parents, school officials, and legislators that everything should be done to bring class sizes down further, resulting in a steady decrease in class size.  And with the help of federal stimulus funds, most districts managed to keep prior reductions, even as state fiscal conditions deteriorated.  Handing out pink slips to teachers in the spring (and rescinding them later) was the perennial political maneuver to ensure that education takes small if any funding cuts.

Until now.  Without further federal stimulus, and without recovery from the recession, schools have begun to feel the budget pressure for the very first time, and the obvious way to deal with any budget slowdown (or actual reduction) is to let class sizes drift up a little.  But this has reinvigorated the political efforts to hold education harmless from any fiscal exigencies.  This situation has led to repeated news media coverage of classrooms with students sitting in the hallways, of testimonials about how it has become impossible to teach fractions with so many students, of how ten years ago they could grade papers but no longer, of . . . .   It has also led to the class size reduction lobbyists quoting back their evidence with the twist of how this is the worst thing that could happen to schools.

Why is an increase different than a reduction?  When reducing class size, one must hire more teachers, which means that the school system will essentially get a random draw that is expected to yield an average teacher.  But increasing class size means that some current teachers must be laid off, and here the schools have a tremendous advantage.  They know how effective their teachers are, so they are not forced to lay off an average teacher.  They can, in fact, lay off below average teachers.

Laying off the worst teachers would lead to dramatic improvements in student achievement.  As I have described elsewhere, replacing the worst 5-8 percent of our teachers with average teachers would be expected to move student outcomes near to–if not at–the top of the international league tables for math and science performance.  And this would have enormous benefits for the U.S. economy and for the students who now have greater skills when they enter the labor force.

But wouldn’t the increased class sizes offset any gains?  In simplest terms, no.  The evidence has been rehashed many times.  The latest Brookings study, for example, concludes once again that the small class size increases from the current fiscal pressures would be virtually undetectable.

Part of the confusion and dissonance over the outcomes arises from the unwillingness (or inability) of schools to make decisions based on the effectiveness of teachers.  By applying LIFO rules (last in, first out) to any dismissals, schools almost completely eliminate the chance to improve the learning of our children.  Specifically, they insure that the largest number of teachers is laid off, while not affecting the average quality of the teaching force.

Moreover, the difficulties are reinforced by news media stories (which appear to be getting data from each other’s stories) that breathlessly cite classes of 45, 50, and even 60.  To the extent that these reports are accurate, we might even applaud the decisions.  One of the biggest problems of the class size reduction movement was that it called for laws and regulations that insisted on uniform reductions without regard to the particular classes, students, and teachers and without regard to where large classes might be appropriate and where small classes might be appropriate.  With increases, school decision makers can at least avoid these damaging rules and can make the changes where they have the least impact on students.

Any such large classes are truly decisions that schools are making.  To obtain a five percent savings in budget, schools must typically let average student-teacher ratios drift up by less than one student per teacher.  This would put student-teacher ratios back roughly to where they were five or six years ago – larger yes, but hardly the dark ages.  It certainly does not require a doubling of class sizes, as some of the media accounts might suggest.  The real data show that student-teacher ratios and class sizes have been falling throughout the past decade – and the recent changes are not in any way simply a continuation of a long slide toward larger classes.

Doing the right thing does require active decision making by schools and policy makers.  Some of this may become easier as legislatures in Wisconsin, Illinois, Ohio, and more revisit the rules on hiring, retention, and school decision making.  But it is not automatic.

- Eric A. Hanushek

The headline story has been fiscal issues – salaries, retirement and health benefits, and the bargains agreed to by legislatures past.  But these issues have morphed into issues more fundamentally threatening to the unions – the right to strike, the ability to bargain about nonsalary issues, and the like.  In response, the teachers unions have mounted a concerted counter-attack aimed at restoring their prior position.

The fiscal issues are important, but I do not think they are the most important ones.  In a recent article in Education Next, “Valuing Teachers,” I presented evidence about the huge economic impacts of highly effective teachers.  A parallel calculation also reveals the huge costs to highly ineffective teachers.  To me, this is what we should be talking about.  The quality of our teaching force determines the level of student achievement, and student achievement directly determines how our economy will develop in the long run.

I argue elsewhere that the teacher unions would be better off getting in front of the teacher quality issue.  The low public regard for teacher unions is, I would argue, a result of public perceptions that concern for student outcomes ranks very low relative to the income, convenience, and preferences of the teachers themselves.  The public – generally very supportive of teachers – does not understand union positions that over-protect the small number of teachers who are harming kids.  The unions can try to rebuild their image (while doing good for America) by actively participating in efforts to figure out how to evaluate teachers and how schools can make personnel decisions based on those evaluations.

But, it should also be recognized that others in the schools are not innocent.  First, the current fiscal problems of school systems, with excessive retirement and health packages, were the result of prior agreements by legislatures, administrators, and school boards.  They were not unilaterally imposed by the unions.

Second, even in states without collective bargaining, there are precious few decisions made on the basis of teacher effectiveness.    There is scant evidence that performance in states without collective bargaining is better than in states with strong collective bargaining.

Returning to the opening question:  what should the current discussions be about?  They should, in my mind, focus on how the incentives, rules, and actions can be arranged to ensure that there is indeed an effective teacher in every classroom.  This in turn really means focusing on student learning.

The unions have to quit defending the worst of the worst.  The majority of very good teachers need to quit tolerating the few bad teachers in their midst. The administrators have to quit hiding behind the “it’s all the unions’ fault” slogan and figure out how to evaluate teachers and to use that information in pay and retention decisions.  The districts must hold administrators responsible for their decisions and set incentives for them that parallel those for teachers.  The legislatures must reward districts for getting it right, not for getting it wrong.

The switch to a focus on student outcomes would be a dramatic change for all parties.  And, returning to my underlying motivation, whether or not we can do this will have a lot to say about the future economic well-being of America.  The contrasting futures of America with and without improvement of our schools are dramatically different.

-Eric Hanushek

Opinion: In an Ed Week commentary, Eric Hanushek discusses some policy implications of his findings about the impact of good and bad teachers.

According to the president in this year’s State of the Union address, this is “our generation’s Sputnik moment,” the time when we realize the urgent need to step up the performance of our education system. Only today, unlike in the 1950s, we have a clear idea of what it takes to improve achievement. The quality of the teachers in our schools is paramount: no other measured aspect of schools is nearly as important in determining student achievement. The initiatives we have emphasized in policy discussions—class-size reduction, curriculum revamping, reorganization of school schedule, investment in technology—all fall far short of the impact that good teachers can have in the classroom. Moreover, many of these interventions can be very costly.

Indeed, the magnitude of variation in the quality of teachers, even within each school, is startling. Teachers who work in a given school, and therefore teach students with similar demographic characteristics, can be responsible for increases in math and reading levels that range from a low of one-half year to a high of one and a half years of learning each academic year.

But while most parents are able to distinguish a good teacher from a bad one, few have any idea what difference it makes in the lives of their children. And researchers do not help, tending to talk in terms of standard deviations of achievement and effect sizes, phrases that simply have no meaning outside of the rarefied world of research. Here, I translate the researchers’ shorthand into concepts that might be more readily understood: the impact of teachers on the earnings of individuals and on the future of the economy as a whole.

Measuring Teachers’ Impact

Many of us have had at some point in our lives a wonderful teacher, one whose value, in retrospect, seems inestimable. We do not pretend here to know how to calculate the life-transforming effects that such teachers can have with particular students. But we can calculate more prosaic economic values related to effective teaching, by drawing on a research literature that provides surprisingly precise estimates of the impact of student achievement levels on their lifetime earnings and by combining this with estimated impacts of more-effective teachers on student achievement.

Let’s start with the researcher’s point of view. With a normal distribution of performance (the classic bell curve), a standard deviation is simply a more precise measure of how spread out the distribution is. Somebody who is one standard deviation above average would be at the 84th percentile of the distribution. If we then turn to the labor market, a student with achievement (as measured by test performance in high school) that is one standard deviation above average can later in life expect to take in 10 to 15 percent higher earnings per year.

That estimate may be deemed conservative for two reasons. First, it does not account for increases in years of education that may result from having a higher level of performance early on. Also, the estimate is based on information from people’s wages and salaries early in their careers, before they have reached their full earnings potential. Other calculations that take into account earnings throughout entire careers estimate 20 percent increases over the course of a lifetime.

Does 10 to 15 percent amount to much? For the average American entering the labor force, the value of lifetime earnings for full-time work is currently $1.16 million. Thus, an increase in the level of achievement in high school of a standard deviation yields an average increase of between $110,000 and $230,000 in lifetime earnings.

How do increases in teacher effectiveness relate to this? Obviously, teacher quality is not the only factor that affects student achievement. The student’s own motivations and support from family and peers play crucial roles as well. But researchers have worked hard to isolate the impact of teachers from these other influences. Rigorous studies consistently show that the impact of a more-effective teacher is substantial A high-performing teacher, one at the 84th percentile of all teachers, when compared with just an average teacher, produces students whose level of achievement is at least 0.2 standard deviations higher by the end of the school year. In fact, the impact of having such a teacher could plausibly be as large as 0.3 standard deviations.

Those impacts attenuate somewhat over time, however. The literature, though less than definitive, suggests that perhaps 70 percent of the gains achieved that year are retained in the long run by the student. The persistence of achievement gains is important, because the more sustained that these increases are, the greater the positive impact teachers will have on the lifetime skills and therefore the earnings of students. Put together, this evidence suggests that a teacher in the top 16 percent of effectiveness will have a positive impact (as compared to an average teacher) on longer-term student achievement that is 70 percent of the immediate gain, which as noted is at least 0.2 standard deviations.  That lower bound of the estimated effect is what we will use as we calculate the economic worth of a teacher by combining a teacher’s impact on achievement with the associated labor market returns.

Let’s start with some conservative estimates of the impact on an individual student. Take a good but not great teacher, one at the 69th percentile of all teachers rather than at the 50th percentile (that is, a teacher who is half a standard deviation above the average). She produces an increase of $10,600 on each student’s lifetime earnings. Even a modestly better than average teacher (60th percentile) raises individual earnings by $5,300, compared to what would otherwise be expected.

While those numbers are not trivial, they burgeon dramatically once we recognize that every student in the class can expect such increases in earnings. Consider, for example, a teacher with a class of 20 students. Under such circumstances, the teacher at the 60th percentile will—each year—raise students’ aggregate earnings by a total of $106,000. The impact of one at the 69th percentile (as compared to the average) is $212,000, and one at the 84th percentile will shift earnings up by more than $400,000.

But there is also symmetry to these calculations. A very low performing teacher (at the 16th percentile of effectiveness) will have a negative impact of $400,000 compared to an average teacher.

Moreover, the economic value of an effective teacher grows with larger classes, as do the economic losses of an ineffective teacher. Figure 1 illustrates the aggregate impact on students’ lifetime earnings for higher- and lower-performing teachers. As we will discuss below, these results are all very large compared with, for instance, the $52,000 annual salary U.S. teachers were paid on average in 2008.

An Alternate Thought Experiment

We can also approach this valuation calculation from the perspective of the impact of teacher effectiveness on the U.S. economy as a whole, rather than just on the future earnings of students. As noted above, student achievement, which provides a direct measure of later quality of the labor force, is strongly related to economic growth. Improving achievement leads to a better prepared workforce and to greater growth, and this growth translates into higher levels of national income.

Starting again with the estimates of the difference in effectiveness of teachers, it is possible to calculate the long-term economic impact of policies that would focus attention on the lowest-quality teachers from U.S. classrooms. Let us propose the following thought experiment: What would happen if the very lowest performing teachers could be replaced by just average teachers? Based on the estimates of variation in teacher quality identified above, Figure 2 shows the overall achievement impact through a cycle of K–12 instruction. Assuming the upper-bound estimate of teachers’ impact, U.S achievement could reach that in Canada and Finland if we replaced with average teachers the least effective 5 to 7 percent of teachers, respectively. Assuming the lower-bound estimate of teachers’ impact, U.S achievement could reach that in Canada and Finland if we replaced with average teachers the least effective 8 to 12 percent of teachers, respectively.

Here the estimated value almost loses any meaning. Closing the achievement gap with Finland would, according to historical experience, have astounding benefits, increasing the annual growth rate of the United States by 1 percent of GDP. Accumulated over the lifetime of somebody born today, this improvement in achievement would amount to nothing less than an increase in total U.S. economic output of $112 trillion in present value. (That was not a typo—$112 trillion, not billion.)

Admittedly, these estimates are subject to some uncertainty. So if you think those that are given here are too high, even though they are based on the best of contemporary research, then just cut them in half. You will still have effects on growth of one-half of 1 percent per year, which produces impacts of $56 trillion over the lifetime of today’s child. In other words, to make the very large effects disappear, you have to make either the very strong assumption that student learning has little effect on the U.S. economy or the equally strong assumption that teachers have little impact on students.

What Would It Take?

The majority of our teachers are hardworking and effective. The previous estimates point clearly to the key imperative of eliminating the drag of the bottom teachers. Here we can offer several alternatives.

One approach might be better recruitment so that ineffective or poor teachers do not make it into our schools. Or, relatedly, we could improve the training in schools of education so that the average teaching recruit is better than the typical recruit of today. Unfortunately, we have relatively few successful experiences with either approach as compared to considerable wishful thinking, particularly among school personnel.

An alternative might be to change a poor teacher into an average teacher. This approach is in fact today’s dominant strategy. Schools hope that through mentoring of incoming teachers, professional development, or completion of further graduate schooling, ineffective teachers can be transformed into acceptable (average) teachers. Again, however, the existing evidence is not very reassuring. While such efforts undoubtedly help some teachers, there is no substantial evidence that certification, in-service training, master’s degrees, or mentoring programs systematically make a difference in whether teachers are in fact effective at driving student achievement.

The final option is a clearer evaluation and retention strategy for teachers. Today, obtaining an entry job into teaching is virtually tantamount to an indefinite contract that stays in force regardless of actual effectiveness in the classroom. Yet the calculations above show the enormous value to individuals and society of “deselecting” the least effective teachers.

Is such a policy change feasible? If we contemplate asking 5 to 10 percent of teachers to find a job at which they are more effective so they can be replaced by teachers of average productivity, states and school districts would have to change their employment practices. They would need recruitment, pay, and retention policies that allow for the identification and compensation of teachers on the basis of their effectiveness with students. At a minimum, the current dysfunctional teacher-evaluation systems would need to be overhauled so that effectiveness in the classroom is clearly identified. This is not an impossible task. The teachers who are excellent would have to be paid much more, both to compensate for the new riskiness of the profession and to increase the chances of retaining these individuals in teaching. Those who are ineffective would have to be identified and replaced. Both steps would be politically challenging in a heavily unionized environment such as the one in place today.

Salary Politics

The above discussion also highlights the difficulties in recruiting high-quality teachers, due in part to the difficulties of paying them well. Collective bargaining mechanisms do not provide incentives for the best people to enter or remain in the profession and likely hold the average pay down: given the uniform salary structure, increases in salary are bound to be unrelated to increases in effectiveness, making large pay raises raises politically problematic. This is likely one of the main reasons that teacher salaries now lag those in other professions. In the 1940s, the salaries of male teachers were slightly above the average pay for all male college graduates, and female teachers had higher salaries than 70 percent of other female college graduates. Today, despite the collective bargaining process, the salaries of male teachers are at the 30th percentile of the distribution of all college graduates, and women who teach are at the 40th percentile of their college-educated peers.

Teachers’ salaries today are based on credentials and years of experience, factors that are at best weakly related to productivity. In a competitive marketplace, a firm must compensate employees according to their productivity or risk bankruptcy. Yet no school district goes out of business if it retains ineffective teachers and pays them as much as effective ones. Salaries become political footballs, and it is often awkward for politicians to explain why a large pay increase goes equally to ineffective and effective teachers.

The challenge of implementing reform of the teaching profession remains considerable. Most of the benefits of implementing the “thought experiment” explored here would be fully realized only many decades later, while the costs of economic, and especially political, reform must be paid at the beginning. These costs would be steep, as they would likely negatively affect some of the most vocal constituents in education policy: current teachers.

The magnitude of the above valuations of teacher effectiveness, however, suggest that we should be willing to consider more radical reforms than have been commonplace in recent decades. Salaries several times higher than those paid teachers today would be economically justified if teachers were compensated according to their effectiveness. But unless we can replace the current system with one that better links teacher recruitment, compensation, and retention to effectiveness, we should expect both our schools and our economy to underperform relative to their potential. The cost to the nation at a time of intensifying international competition is high indeed.

Eric A. Hanushek is a senior fellow at the Hoover Institution, Stanford University.

Even as parents and policy makers repeat the line that good teachers are very important, they seldom understand just how important.  Discussions of standard deviations of test scores simply have little meaning to most people (including assessment people themselves).

I have tried to describe the value of teachers in a narrow but important and understandable way.  In an article published today in Education Next, I calculate the value of a good (and a bad) teacher by tracing the economic ramifications of differences in student achievement.  It turns out that this involves a fairly straightforward set of calculations.  A good teacher gets above average achievement out of her students.  We have clear and consistent measures of the achievement impact of a good teacher.  We also have very consistent measures of the impact on an individual’s future earnings of having higher achievement.  Put these two things together and one gets a direct estimate of how much the average student can expect to gain from having a top teacher.

The numbers are astounding.  A teacher at the 85^th percentile can, in comparison to an average teacher, raise the present value of each student’s lifetime earnings by over $20,000–implying that such a teacher with a class of 20 students generates over $400,000 in economic benefits, compared to an average teacher, for each year that she gets such achievement gains.

Gains go up and down with how good the teacher is and with how many students she has.  And the gains are symmetrical in comparison to the average teacher – a teacher at the 15^th percentile subtracts $400,000 in value from her class of 20 students.

An alternative way to value of teachers simply focuses on the aggregate costs to the U.S. economy of tolerating poor teachers in the classroom.  Again, by using information about quality differences among teachers, if we could replace the bottom 5-8 percent of our teachers with average teachers, we could move our students’  achievement up to that of Canada.  Because the quality of the labor force is directly related to what students know (as measured by math and science tests), the economy would by historical patterns tend to growth more rapidly with improvements in achievement (see the discussion in “Education and Economic Growth” ).   We can then use historical patterns of growth to estimate the aggregate economic value of greater achievement.  Moving to the level of Canada has a present value of over $75 trillion.

These calculations are described in detail in the Education Next article, “Valuing Teachers.”

The simple argument is that these gains – to individuals and to society – seem large enough that we might consider more fundamental changes to the way we run our schools.  We need to pay much more attention to ensuring that there is an effective teacher in every classroom.

-Eric Hanushek

Hanushek finds that the top 25 percent of teachers (teachers in the top quarter of the effectiveness distribution) contribute $16,000 per year in income to the average student (compared to what those students would earn if they had an average teacher). If a class has 25 students, that means that $400,000 in added income will accrue to students because they had a good teacher for one year.

If we could replace the bottom 5-8 percent of teachers with average teachers, we could perform as a nation near the top of the international math and science tests. In dollar terms, this would mean adding over $50 trillion to our nation’s gross domestic product.

For more, please see “Valuing Teachers,” by Eric Hanushek, which appears in the Summer 2011 issue of Education Next.

Click here for a free subscription to the Education Next podcasts on iTunes.

The argument in the courts — playing out now in New Jersey and likely soon in New York — is simple: The state Constitution protects us from taking any share of the pain of the fiscal calamity. This kind of logic may indeed spread to other states.

The common public line is that, because of budget pressures, class sizes will rise to the extent that learning is virtually impossible. In reality, until this year, class sizes across America have fallen for the last 15 years, to new lows. This ploy is simply part of the political bargaining that is designed to separate schools from any budget problem.

Indeed, school spending in America has risen continuously for over a century, with the one exception of a slight fall during the Great Depression of the 1930s.

But recent fiscal pressures have started to take their toll. In New Jersey, Gov. Chris Christie had the audacity to cut state funding for schools, reductions that amounted to some 5 percent of total school spending, with the most disadvantaged districts being shielded from the brunt of the cuts. In New York, Gov. Cuomo and the Legislature just enacted a budget that also has modest year-over-year reductions in state aid to schools — and that also reneges on school-spending increases promised in years past.

Having lost in the media and the political bargaining, New Jersey schools turned to the courts. Garden State judges have ruled over school finance for 40 years, and the schools — especially the highest-poverty schools — have had a friend in court, being allowed to spend virtually whatever they want.

In 2008, the latest year for which data are available, New Jersey was the highest-spending state in the nation, with per-pupil expenditures 70 percent above the national average. The poor districts (known as “Abbott districts” after a long-running court case) spend some 2½ times the national average.

Faced with a 5 percent cut, the schools went back to the courts to describe the hardships that would result, claiming the state Constitution’s educational guarantees would be violated by the lower funding.

Accompanying these claims have been news stories suggesting that class sizes might reach 40 or even, in some places, 60 students. Perspective is needed: The 5 percent cut could be accomplished by allowing pupil-teacher ratios to rise by just one student per teacher — a change that’s virtually unnoticeable by most teachers.

Such cuts would only move schools back to where they were before the recession in terms of class sizes and pupil-teacher ratios. It certainly wouldn’t move them to the level of early last century or of a developing African country.

The exact impact, of course, depends on how the districts adjust. To save money, some teachers will be let go, but which teachers is important. In the ludicrous extreme, a district could lay off its best teachers, leading to the maximum possible harm to students.  But no district is going to do that.

The standard rule, put into state law and into labor contracts, is LIFO: “Last in, first out” — dismiss the newest hires. Note, first, that this policy will lead to the maximum number of dismissals, since the youngest teachers also have the lowest salaries. Second, in most cases, it will have little impact on average teacher quality, and thus on student outcomes, because the youngest teachers average essentially as good as the oldest teachers.

One possible exception: If a school system — like New York City’s — has recently made a major push to improve teacher quality, perhaps drastically increasing pay to attract the best possible new hires, then LIFO layoffs could remove many of the best teachers (even as the earlier-hired ones still enjoy the higher pay that helped attract the now-departed educators).

Now, consider a different policy: Lay off the least-effective teachers in order to meet the budget shortfall. This policy would have enormous beneficial effects on achievement. By estimates I have done, eliminating the bottom 5 percent to 8 percent of teachers could move achievement of US students from below the average for developed countries to near the top.

We all know a few teachers are just plain bad; students in those classes would be much better off learning from a competent or superior teacher in a slightly larger class — and the students in that class would suffer little (if at all) from having one or two more classmates.

Now, back to the courts. They face less funding for schools, with groups of superintendents who will testify to the horrible educational ramifications. The courts are so used to measuring education in terms of spending that they tend to be swayed by such horror stories — without ever conceiving of reforming the way schools spend their money.

New Jersey courts will soon decide whether the cut in school appropriations was constitutional. Part of any decision depends crucially on whether the judges consider alternative policies.

New York is probably not far behind. The courts have been heavily involved in school spending, ordering vast increases in state aid in the Campaign for Fiscal Equity decision. As a result, New York state funding for schools is second in the nation, 63 percent above the national average. (The City looks like an Abbott district in terms of spending).

We will have to wait to see if the New York courts also think they should revisit school spending.

These issues have real importance.  The quality of our schools is truly vital to our nation’s future. We will have a very different economy in the mid-21st century, depending on whether our schools get better or not.

Yet judicial interventions simply haven’t been helping. They’ve consistently supported the complete status quo. They have reinforced bad operating rules like LIFO policies. They have encouraged teacher salaries that are unrelated to teacher effectiveness in the classroom. They have, in sum, rewarded school administrations for never making tough decisions.

If the courts want to help out, they should not focus on the budgetary changes. They should focus on the laws and contract provisions that inhibit the provision of high-quality education to the students of New Jersey and New York and the rest of the country.

- Eric Hanushek

This is a modified version of a commentary that first appeared in the New York Post.

Unfortunately, each of these common arguments is either terribly misleading or wrong.  Simply looking for blue sky in the test results ignores a substantial body of scientific research.  While many people want to be reassured that things are going just fine, ignoring the real message of these tests actually imperils our economic future.

Let’s start at the top.  The recently released PISA results that compare 15-year-olds around the world in reading, math, and science place U.S. students above the developed-country average in reading, at the average in science, and below average in math.  If one focuses just on reading, perhaps we are not doing so badly, even though we still trail 17 countries.  But reading is very difficult to assess accurately in the international tests.  And reading scores have proven less important than math and science for both individual and national success.  In math, we place 31^st in the world rankings.

Research has shown that international performance on these tests is very closely related to the economic growth of nations.  Does the difference between 550 points (roughly Finland) and 500 points (roughly the U.S.) make a difference?  By the historical record of growth, such a difference is consistent with one percent per year in the growth of per capita income.  If we project this out over the lifetimes of children born today, the present value of economic gains from the U.S. reaching the level of Finland would be $100 trillion!  These potential economic gains from improved schools should be compared to the huge political fights in the U.S. over a stimulus package of one trillion dollars, or one hundredth of the magnitude of the gains we are leaving on the table from ignoring the achievement in our schools.

The challenge to the U.S. is clearest when one looks at the proportion of students achieving at the advanced level in math.  Presumably our scientists, engineers, and innovators are drawn from these high performers.  Paul Peterson, Ludger Woessmann, and I assessed not only how well our best states were doing but also how well our white students and our children of college educated parents were doing in advanced skills.

The performance of U.S. students of the Class of 2009 as a whole trails 29 countries.  Sixteen countries actually produce twice the proportion of advanced math students that we do.  And there are more highly talented math students in the whole population of 18 countries than in U.S. families with a college educated parent.

Yes, the U.S. has had advantages that have covered up the poor performance of our schools.  The free and open labor and product markets of the U.S. along with the generally limited intrusion of the government and respect for individual property rights have promoted an innovative society and have attracted the brightest from abroad.  But our relative advantages in these areas are swiftly eroding as other countries emulate our economic institutions and as other countries attract their bright and well-trained students back to work at home.

The feel-good message offers solace to those who counsel maintaining the current course.  It is, however, a bad message that truly threatens our economic future.   To be sure, it will not show up very clearly for some time, maybe even a decade or two.  By then, recovery will at least be much more difficult, if not impossible.

- Eric A. Hanushek

The publication created a firestorm. The unions were apoplectic. A vocal set of commentators attacked this action from a variety of viewpoints. Nonetheless, it shows signs of spreading – to New York City and elsewhere. Since my research started this development, I believe it is useful to share my perspectives on how we should judge this development and whether we should stop its spread.

What did the Times do? It hired a professional analyst to link student gains on state achievement tests to specific teachers. These estimates, generally called value-added scores, take into account what students knew upon entry into the classroom along with other characteristics of the students and assess statistically how much the average student learned during a year. The Times then identified each teacher and published the results so that everybody, including importantly parents, could learn how effective each teacher was at raising the measured achievement of students.

What are the advantages and disadvantages of this? This approach, the focus of considerable scholarly research, correctly attempts to isolate the impact of the teacher from other factors that affect achievement – families, peers, neighborhoods, and the like. Thus, it is designed to make judgments of what the teacher contributes as opposed to the specific draw of students in the classroom. Moreover, it provides an objective measure of teacher performance.

That is not to say, however, that it is without problems.

First, it depends on the areas that are tested – typically, math and reading but not science, social studies or history.

Second, the statistical measures typically include various errors in assessing teacher performance because the tests themselves are inaccurate assessments of knowledge. Thus, the rankings from the value-added measures may be imprecise.

What are the opposing arguments? Those favoring release of this information have a simple position: Parents have a right to know the effectiveness of their child’s teacher, and policymakers should take performance into account. Those opposing this argue the problems: It is a narrow and imperfect measure of teacher effectiveness and thus should not be released because parents and others will place too much weight on it. The teachers unions in particular do not want to enter into a discussion of which teachers are not performing well, because they have generally committed to defending all teachers and such information makes their defense difficult.

As somebody deeply involved in the underlying science behind these measures, I see valid arguments on both sides.

That having been said, there has been little movement toward a more thorough evaluation system that incorporates broader measurement of teacher effectiveness. The union position is to argue that we need to develop reliable and accurate evaluation systems – but then to block any use of evaluation systems that focus on classroom performance. One needs only look to Washington, D.C., to see this. Michelle Rhee worked to develop a thoughtful and elaborate evaluation system based on classroom observations and supervisor ratings along with incorporating value-added measures of performance where possible.

This proposed evaluation system could support both large bonuses to exceptional teachers and elimination of teachers who showed that they were ineffective. The unions resisted these efforts, forcing her to institute a series of unilateral personnel actions against the most ineffective teachers. Now Rhee is gone from D.C., providing direct evidence of the difficulty that follows any attempt to evaluate performance.

I personally would not like to see personnel decisions made solely on the basis of value-added scores. But typical evaluations today are useless, because only a minuscule number of teachers are rated anything but great and because these ratings are never used in making personnel decisions. Thus, while I would like to see a broader evaluation program, the introduction of a value-added system puts information on effectiveness into play.

The importance of improving our schools is too great simply to dismiss such information as imperfect. The current system of evaluations is completely broken. And, there is virtually no chance of improving schools without paying attention to which teachers are effective and which are ineffective.

- Eric A. Hanushek

This piece originally appeared in The Sacramento Bee.

In Vancouver last winter, the United States proved its competitive spirit by winning more medals—gold, silver, and bronze—at the Winter Olympic Games than any other country, although the German member of our research team insists on pointing out that Canada and Germany both won more gold medals than the United States. But if there is some dispute about which Olympic medals to count, there is no question about American math performance: the United States does not deserve even a paper medal.

Maintaining our productivity as a nation depends importantly on developing a highly qualified cadre of scientists, engineers, entrepreneurs, and other professionals. To realize that objective requires a system of schooling that produces students with advanced math and science skills. To see how well schools in the United States do at producing high-achieving math students, we compared the percentage of U.S. students in the high-school graduating Class of 2009 with advanced skills in mathematics to percentages of similarly high achievers in other countries.

Unfortunately, we found that the percentage of students in the U.S. Class of 2009 who were highly accomplished in math is well below that of most countries with which the United States generally compares itself. No fewer than 30 of the 56 other countries that participated in the Program for International Student Assessment (PISA) math test, including most of the world’s industrialized nations, had a larger percentage of students who scored at the international equivalent of the advanced level on our own National Assessment of Educational Progress (NAEP) tests. Moreover, while the percentage of students scoring at the advanced level on NAEP varies considerably among the 50 states, not even the best state does well in international comparison. A 2005 report from the National Academy of Sciences, Rising Above the Gathering Storm, succinctly put the issue into perspective: “Although many people assume that the United States will always be a world leader in science and technology, this may not continue to be the case inasmuch as great minds and ideas exist throughout the world.”

The Demand for High Achievers

The gap between the burgeoning business demand for a highly accomplished workforce and a lagging education system has steadily widened. Even as the United States was struggling with a near 10 percent unemployment rate in the summer of 2010, businesses complained that they could not find workers with needed skills. New York Times writer Motoko Rich explained, “The problem…is a mismatch between the kind of skilled workers needed and the ranks of the unemployed.”

Skill shortages have severe consequences for a nation’s overall productivity. Two of the authors of this report have shown elsewhere that countries with students who perform at higher levels in math and science show larger rates of increase in economic productivity than do otherwise similar countries with lower-performing students (see “Education and Economic Growth,” research, Spring 2008).

Public discourse has tended to focus on the need to address low achievement, particularly among disadvantaged students. Both federal funding and the accountability elements of No Child Left Behind (NCLB) have stressed the importance of bringing every student up to a minimum level of proficiency. As great as this need may be, there is no less need to lift more students, no matter their socioeconomic background, to high levels of educational accomplishment. In 2006, the Science, Technology, Engineering, and Mathematics (STEM) Education Coalition was formed to “raise awareness in Congress, the Administration, and other organizations about the critical role that STEM education plays in enabling the U.S. to remain the economic and technological leader of the global marketplace for the 21st Century.” In the words of a National Academy of Sciences report that jump-started the coalition’s formation, the nation needs to “increase” its “talent pool by improving K–12 science and mathematics education.”

A Focus on Math

We give special attention to math performance because math appears to be the subject in which accomplishment in secondary school is particularly significant for both an individual’s and a country’s economic well-being. Existing research, though not conclusive, indicates that math skills better predict future earnings and other economic outcomes than other skills learned in high school. The American Diploma Project estimates that “in 62 percent of American jobs over the next 10 years, entry-level workers will need to be proficient in algebra, geometry, data interpretation, probability and statistics.”

There is also a technical reason for focusing our analysis on math. This subject is particularly well suited to rigorous comparisons across countries and cultures. There is a fairly clear international consensus on the math concepts and techniques that need to be mastered and on the order in which those concepts should be introduced into the curriculum. The knowledge to be learned remains the same regardless of the dominant language spoken in a culture.

Data and Methodology

Our analysis relies on test-score information from NAEP and PISA. NAEP, the National Assessment of Educational Progress, is often called the nation’s report card. It is a large, nationally representative assessment of student performance in public and private schools in mathematics, reading, and science that has been administered periodically since the early 1970s to U.S. students in 4th grade and 8th grade, and at the age of 17. PISA, the Program for International Student Assessment, is an internationally standardized assessment of student performance in mathematics, science, and reading established by the Organisation for Economic Co-operation and Development (OECD). It was administered in 2000, 2003, and 2006 to representative samples of 15-year-olds in all 30 OECD countries (which include the most developed countries of the world) as well as in many others.

We focus on performance of the international equivalent of the U.S. high-school graduating Class of 2009 at the time when this population was in the equivalent of U.S. grades 8 and 9. NAEP was administered to U.S. 8th graders in 2005, while PISA 2006 was given one year later to students at the age of 15, the year at which most American students are in 9th grade.

In 2005, NAEP tested representative samples of 8th-grade public and private school students in each of the 50 states in math, science, and reading. For each state, NAEP 2005 calculates the percentage of students who meet a set of achievement standards: a “basic” level, a “proficient” level, and an “advanced” level of achievement. The focus of this report is the top performers, the percentage of students NAEP found at the advanced level of achievement (subsequently referred to as “advanced”).

Only 6.04 percent of the students in the United States in 8th grade in 2005 scored at the advanced level in math on the NAEP. Some critics feel that the standard set by the NAEP governing board is excessively stringent. However, the 2007 Trends in International Math and Science Study (TIMSS 2007), another international test that has been administered to students throughout the world, appears to have set a standard very similar to NAEP 2005, as only 6 percent of U.S. 8th graders scored at the advanced level on that test as well.

We use the NAEP 2005 advanced standard to compare U.S. performance with that in other countries. Because U.S. students took both NAEP 2005 and PISA 2006, it is possible to find the score on PISA that is tantamount to scoring at the advanced level on NAEP, i.e., the score that will yield the same percentage of students as the percentage of U. S. students who scored at the advanced level on the NAEP.

A score on PISA 2006 of 617.1 points is equivalent to the lowest score attained by anyone in the top 6.04 percent of U.S. students in the Class of 2009. (The PISA assessment has an average score of 500 among OECD students and a standard deviation of 100.) It is assumed that both NAEP and PISA tests randomly select questions from a common universe of mathematics knowledge. Given that assumption, it may be further assumed that students who scored similarly on the two exams will have similar math knowledge, i.e., students who scored 617.1 points or better on the PISA test would have been identified at the advanced level had they taken the NAEP math test. Inasmuch as a score of 617.1 points is more than one standard deviation above the average student score on the PISA, it is clear that a group of highly accomplished students has been isolated. (For more methodological details, see sidebar.)

Because representative samples of student performance on NAEP 2005 are available for each state, it is possible to compare the percentages of students in the Class of 2009 who were at the advanced level for each state to the percentage of equally skilled students in countries from around the globe.

In short, linking the scores of the Class of 2009 on NAEP 2005 and PISA 2006 provides us with the opportunity to assess from an international vantage point how well the country as well as individual states in the United States are doing at lifting students to high levels of accomplishment.

U. S. Math Performance in World Perspective

We begin with an overall assessment of the relative percentages of young adults in the United States and other countries who have reached a very high level of mathematics achievement. It is frequently noted that the United States has a very heterogeneous population, with large numbers of immigrants. Such a diverse population, with students coming to school with varying preparation, may handicap U.S. performance relative to that of other countries. For this reason, we also examine two U.S. subgroups conventionally thought to have better preparation for school—white students and students from families where at least one parent is reported to have received a college degree—and compare the percentages of high-achieving students among them to the (total) populations abroad.

Overall results. The percentage of students in the U.S. Class of 2009 who were highly accomplished is well below that of most countries with which the United States generally compares itself. While just 6 percent of U.S. students earned at least 617.1 points on the PISA 2006 exam, 28 percent of Taiwanese students did. (See Figure 1 for these results as well as for the international rank of each U.S. state.)

Click to enlarge

It is not only Taiwan that did much, much better than the United States. At least 20 percent of students in Hong Kong, Korea, and Finland were similarly highly accomplished. Twelve other countries had more than twice the percentage of advanced students as the United States: in order of math excellence, they are Switzerland, Belgium, the Netherlands, Liechtenstein, New Zealand, the Czech Republic, Japan, Canada, Macao-China, Australia, Germany, and Austria.

The remaining countries that educate a greater proportion of their students to a high level are Slovenia, Denmark, Iceland, France, Estonia, Sweden, the United Kingdom, the Slovak Republic, Luxembourg, Hungary, Poland, Norway, Ireland and Lithuania.

The 30-country list includes virtually all the advanced industrialized nations of the world. The only OECD countries producing a smaller percentage of advanced math students than the United States are Portugal, Greece, Turkey, and Mexico. The performance levels of students in Spain and Italy are statistically indistinguishable from those of students in the United States, as are those of students in Latvia, which has subsequently joined the OECD.

State-level performance. The percentage of students scoring at the advanced level varies among the 50 states. Massachusetts, with over 11 percent of its students at the advanced level, does better than any other state, but its performance trails that of 14 countries. Its students’ achievement level is similar to that of Germany and France. Minnesota, with more than 10 percent of its students at the advanced level, ranks second among the 50 states, but it trails 16 countries and performs at the level attained by Slovenia and Denmark. New York and Texas each have a percentage of students scoring at the advanced level that is roughly comparable to the United States as a whole, Lithuania, and the Russian Federation.

Just 4.5 percent of the students in the Silicon Valley state of California are performing at a high level, a percentage roughly comparable to that of Portugal. The lowest-ranking states—West Virginia, New Mexico, and Mississippi—have a smaller percentage of the highest-performing students than Serbia or Uruguay, although they do edge out Romania, Brazil, and Kyrgyzstan.

In short, the percentages of high-achieving students in the United States—and in most of its individual states—are shockingly below those of many of the world’s leading industrialized nations. Results for many states are at a level equal to those of third-world countries. (Click the image below for an interactive map providing specific information for each state.)

Click to find specific information for each state

White students. The overall news is sobering. Some might try to comfort themselves by saying the problem is limited to large numbers of students from immigrant families, or to African American students and others who have suffered from discrimination. For example, the statement by the STEM Coalition that we “encourage more of our best and brightest students, especially those from underrepresented or disadvantaged groups, to study in STEM fields” suggests that the challenges are concentrated in nonwhite segments of the U.S. population.

Without denying that the paucity of high-achieving students within minority populations is a serious issue, let us consider the performance of white students for whom the case of discrimination cannot easily be made. Twenty-four countries have a larger percentage of highly accomplished students than the 8 percent achieving at that level among the U.S. white student population in the Class of 2009. Looking at just white students places the U.S. at a level equivalent to what all students are achieving in the United Kingdom, Hungary, and Poland. Seven percent of California’s white students are advanced, roughly the percentage for all Lithuanian students.

Children of parents with college degrees. Another possibility is that schools help students reach levels of high accomplishment if parents are providing the necessary support. To explore this possibility, we assumed that students who reported that at least one parent had graduated from college were likely to be given the kind of support that is needed for many to reach high levels of achievement. Approximately 45 percent of all U.S. students reported that at least one parent had a college degree.

The portion of students in the Class of 2009 with a college-graduate parent who are performing at the advanced level is 10.3 percent. When compared to all students in the other PISA countries, this advantaged segment of the U.S. population was outranked by students in 16 other countries. Nine percent of Illinois students with a college-educated parent scored at the advanced level, a percentage comparable to all students in France and the United Kingdom. The percentage of highly accomplished students from college-educated families in Rhode Island is just short of 6 percent, the same percentage for all students in Spain, Italy, and Latvia.

The Previous Rosy Gloss

Many casual observers may be surprised by our findings, as two previous, highly publicized studies have suggested that—even though improvement was possible—the U.S. was doing all right. This was the picture from two reports issued by Gary Phillips of the American Institutes for Research, who compared the average performance in math of 8th-grade students in each of the 50 states with the average scores of 8th-grade students in other countries. These comparisons used methods that are similar to ours to relate 2007 NAEP performance for U.S. students to both TIMSS 2003 and TIMSS 2007. His findings are more favorable to the United States than those shown by our analyses. While our study using the PISA data shows U.S. student performance in math to be below 30 other countries, Phillips found the average U.S. student to be performing better than all but 14 other countries in his 2007 report and all but 8 countries in his 2009 report. (Oddly, the 2007 report takes a much more buoyant perspective than the 2009 report, though the data suggest otherwise.) Phillips also finds that individual states do much better vis-à-vis other countries than we report.

Why do two studies that seem to be employing generally similar methodologies produce such strikingly different results?

The answer to that puzzle is actually quite simple and has little to do with the fact that Phillips compares average student performance while our study focuses on advanced students: many OECD countries, including those that had a high percentage of high-achieving students, participated in PISA 2006 (upon which our analysis is based) but did not participate in either TIMSS 2003 or TIMSS 2007, the two surveys included in the Phillips studies. In fact, 19 countries that outscored the U.S. on the PISA 2006 test did not participate in TIMSS 2003, and 22 higher-scoring countries did not participate in TIMSS 2007. As a report by the U.S. National Center for Education Statistics has explained, “Differences in the set of countries that participate in an assessment can affect how well the United States appears to do internationally when results are released.”

Put starkly, if one drops from a survey countries such as Canada, Denmark, Finland, France, Germany, and New Zealand, and includes instead such countries as Botswana, Ghana, Iran, and Lebanon, the average international performance will drop, and the United States will look better relative to the countries with which it is being compared.

Did NCLB shift the focus away from the best and the brightest?

Some attribute the comparatively small percentages of students performing at the advanced level to the focus of the 2002 federal accountability statute, No Child Left Behind, on the educational needs of very low performing students. That law mandates that every student be brought up to the level a state deems proficient, a standard that most states set well below NAEP’s proficient standard, to say nothing of the advanced level that is the focus of this report.

In order to comply with the federal law, some assert, schools are concentrating all available resources on the educationally deprived, leaving advanced students to fend for themselves. If so, then we should see a decline in the percentage of students performing at NAEP’s advanced level subsequent to the passage of the 2002 federal law. In mathematics, however, the opposite has happened. The percentage performing at the advanced level was only 3.7 percent in 1996 and 4.7 percent in the year 2000. But the percentage performing at an advanced level climbed steadily to the 7.9 percent attained in 2009.

Perhaps NCLB’s passage in 2002 dampened the prior rate of growth in the achievement of high-performing students. To ascertain whether that was the case, we compared the rate of change in the NAEP math scores of the top 10 percent of all 8th graders between 1990 and 2003 (before NCLB was fully implemented) with the rate of change after NCLB had become effective law. Between 1990 and 2003, the scores of students at the 90th percentile rose from 307 to 321, an increment of 14 points, or a growth rate of 1.0 points a year. Between 2003 and 2009, the shift upward for the 90th percentile was another 8 points, or a change of 1.3 points a year. Our results are confirmed by a more detailed study of NCLB’s impact on high-performing students conducted by economists Brian Jacob and Thomas Dee.

In short, the incapacity of American schools to bring students up to the highest level of accomplishment in mathematics is much more deepseated than anything induced by recent federal legislation.

Conclusions

The economic and technological demand for a talented, well-educated, highly skilled population has never been greater. Not only must everyday workers have a set of technical skills surpassing those needed in the past, but a cadre of highly talented professionals trained to the highest level of accomplishment is needed to foster innovation and growth. In the words of President Barack Obama, “Whether it’s improving our health or harnessing clean energy, protecting our security or succeeding in the global economy, our future depends on reaffirming America’s role as the world’s engine of scientific discovery and technological innovation. And that leadership tomorrow depends on how we educate our students today, especially in math, science, technology, and engineering.”

Unfortunately, the United States trails other industrialized countries in bringing a large proportion of its students up to the highest levels of accomplishment. This is not a story of some states doing well but being dragged down by states that perform poorly. Nor is it a story of immigrant or disadvantaged or minority students hiding the strong performance of better-prepared students. Comparatively small percentages of white students are high achievers. Only a small proportion of the children of our college-educated population is equipped to compete with students in a majority of OECD countries.

Major policy initiatives within the United States have in recent years focused on the educational needs of low-performing students. Such efforts deserve commendation, but they can leave the impression that there is no similar need to enhance the education of those students the STEM coalition has called “the best and brightest.” Yet, with rapidly advancing technologies in an increasingly integrated world economy, no one doubts the extraordinary importance of highly accomplished professionals.

Admittedly, the United States could simply ignore the needs of its own young people and continue to import highly skilled scientists and engineers who were prepared by better-performing schools abroad. But even such a heartless, irresponsible strategy relies on both the nature of immigration policies and the absence of better opportunities abroad, two things on which we might not want the future to depend. It seems much more prudent to encourage the most capable of our own people to reach high levels of academic accomplishment.

Eric A. Hanushek is senior fellow at the Hoover Institution of Stanford University. Paul E. Peterson is the director of Harvard’s Program on Education Policy and Governance and senior fellow at the Hoover Institution. Ludger Woessmann is professor of economics at the University of Munich.

But educators were encouraged in December 2008 when another respected international survey, Trends in Mathematics and Science Study (TIMSS), released results from its math testing for 2007.  It found U. S. 8^th graders to be ranked Number 9 among the 48 participating countries, and its score, at 508, was above the average for all students from participating countries.  Furthermore, there are those such as Tom Loveless at the Brookings Institution, who has claimed that TIMSS does a better job of measuring math knowledge than PISA does. (Mark Schneider took a close look at both tests in this 2009 article for Education Next.) More than one commentator took these facts to argue that the problems of the American schools had been exaggerated.

Have we unfairly maligned our schools?

To figure out why the two tests seemed to point in somewhat different directions, we decided to take a careful look at the facts. Specifically, we looked at the countries who participated in the PISA test but not TIMSS, and vice versa. As can be seen at the bottom of this post, fully 22 of the countries that outperformed us on PISA in 2006 simply did not participate in the TIMSS testing. Basically, they include a large chunk of the industrialized OECD countries that are the ordinary reference group for the United States, along with a smattering of developing countries that also do better than us in math.

It is true that students in Singapore, one of the world’s hotbeds of math knowledge, took the 2007 TIMSS but not the 2006 PISA, but otherwise the countries who took the TIMSS but not the PISA come from the developing world.  Further, the TIMSS average, calculated in 1995, was based on results that included scores from 12 developing countries.

In short, those who defend the U. S. performance by pointing to the TIMSS are making the compelling claim that the United States is just a bit better than an average score that excludes many top performers but incorporates results from many second and third world countries.

___________________________

The recent discussions in Congress, however, shed some light on this.  The discussions of teacher layoffs have led Congressman Obey to try to find money to avert any dismissals of teachers.  In order to be revenue neutral, however, he must come up with budgetary savings.  Where does he look?  Why, to reducing funds for Race to the Top, teacher incentives, and charter schools.  In other words, a simple trade is proposed:  sacrifice innovation and reform when there is a threat to maintaining the status quo of current hiring.

When push comes to shove, it is appears that it is not about the kids–it is about the adults.  More charitably, we might conclude that just slowing down the pace of innovation is appropriate in the face of the potential job losses.  But that brings us back to history.  There never appears to be a time for real reform.

What does this bonus do?  It induces many teachers to want to have a master’s degree.  (Over half of all teachers have an advanced degree now.)  Getting a master’s degree is frequently something done concurrently with a full time teaching job, so the last thing these teachers want is a challenging academic program that requires real work.  As a result, schools of education are willing to sell master’s degrees that require minimal effort.  Master’s degrees become a very profitable product.

Everybody is happy – except perhaps the students who see resources going to things that have no educational value.

If we cannot act on things that are so well-known and well-documented, how can we hope to do things that are more difficult and controversial?

Education Next: What is the evidence that inner-city schools are shortchanged on high-quality teachers?

Eric Hanushek: Inner-city schools and especially those serving the most disadvantaged students routinely display unacceptable achievement levels, ones that seal their students off from further education and from good jobs. Coupled with the general finding that effective teachers are the key to a high-quality school, it is natural to infer that the children most in need are systematically getting the poorest teachers.

Unfortunately, direct evidence on the distribution of teacher quality and its impact for disadvantaged students is hard to come by. Researcher Marguerite Roza and others have produced considerable evidence that teachers in schools serving the most-disadvantaged students have lower average salaries, reflecting in large part the movement of more-experienced teachers away from schools with a higher proportion of minority students and with lower-achieving students. There is also evidence that these schools tend to have more teachers with emergency credentials and without regular certification, although this appears to be declining over time. The problem is that these readily measured attributes of teachers have virtually nothing to do with teacher effectiveness.

Extensive research on teacher quality by me and others suggests that the only attribute of teacher effectiveness that stands out is being a rookie teacher. Teachers in their first three years do a less satisfactory job than they will with more experience. And this has an impact on schools serving highly disadvantaged populations, because the more-experienced teachers who leave these schools are generally replaced with new teachers. The net impact of this on disadvantaged schools is unclear, because there is also some evidence that the experienced teachers who leave these schools are on average not their most effective teachers.

Kati Haycock: No matter what measure of “quality” you look at, poor and minority students—and not just those in inner-city schools—are much less likely to be assigned better-qualified and more-effective teachers. Core academic classes in high-poverty secondary schools are twice as likely as those in low-poverty schools to be taught by a teacher with neither a major nor certification in the subject. The percentage of first-year teachers at high-minority schools is almost twice as high as the percentage of such teachers at low-minority schools. The list of disgraceful statistics goes on and on.

Even if we dismiss traditional measures as imperfect gauges of true teaching quality, new studies employing more-sophisticated measures reveal the same inequitable patterns. When the Tennessee Department of Education analyzed the state’s Value-Added Assessment System—which measures the impact of individual teachers on their students’ tested academic growth—it found that “low-income and minority children have the least access to the state’s most effective teachers and more access to the state’s least effective teachers.” Recently, researchers at the University of Virginia studying teaching practices and learning climate in more than 800 1st-grade classrooms were dismayed to find that lower-income and nonwhite students are much more likely than their counterparts to be placed in “lower overall quality classrooms.”

We also have clear evidence of just how damaging those inequities are. An analysis of data from Los Angeles found that the impact of individual teachers is so great that providing top-quartile teachers rather than bottom-quartile teachers for four years in a row would be enough to completely close the achievement gap between white and African American students. In fact, attending to this problem is the most important step policymakers can take to address the nation’s long-standing achievement gaps.

EN: Can we get higher-quality teachers to inner-city schools? What strategies are most likely to work? Regulation or incentives?

EH: Historically, the first policy response has been to try writing regulations. When these don’t work, the next response is generally to fine-tune the regulations. Developing regulations that ensure that local districts take appropriate action to deal with the teacher quality problem is not likely to be very successful. First, regulations work best when it is possible to measure precisely the underlying attributes that are important to success. Extensive research shows that commonly measured attributes of teachers, such as more than three or four years of experience, master’s degrees, and even state certification, are not related to effectiveness. In fact, all of the regulations that go into defining what is needed to be a fully credentialed teacher neither screen out bad teachers nor ensure that credentialed teachers are any more effective then uncredentialed teachers. Second, many union contracts in effect in inner cities vest rights to fill any teaching vacancies with senior teachers. New or reworked regulations would have to deal with collectively bargained teacher agreements.

An incentive approach must be the centerpiece of improving teacher quality in urban schools and in the most disadvantaged schools. It is necessary to reward success rather than try to regulate it. Unfortunately, we have little experience with how to structure incentives. Attempts to devise universal incentives from Washington or from state capitols are likely to be quite inefficient if not harmful.

Providing strong incentives is increasingly possible, however, as we develop better information linking teachers to student achievement, but incentives linked to so-called value-added measures are likely to be a small part of the overall answer. We need to refine the evaluation of teacher effectiveness, and we need to introduce the serious use of evaluations into the schools, evaluations that guide tenure, retention, and pay decisions.

Research that Steve Rivkin and I have done indicates that the largest variations in teacher quality are found within the typical school, and that quality variation between schools is considerably smaller than that found in any given school, including high-poverty schools. The policy implication of this is quite clear. It is not a matter of trying to swap all of the teachers in high-poverty schools with those in suburban schools. It is very much a matter of focusing on student achievement gains and of keeping those teachers who do a good job while eliminating those who are inept. For this, it is more a matter of will, combined with eliminating the rigidities that have been built into teachers’ contracts.

KH: We know it is possible to bring high-quality teachers into urban schools from recent efforts in New York City and other districts. The question is whether we will do what is necessary to provide low-income and minority students with the kind of powerful teaching they need and deserve. To solve the problem on a large scale, policymakers will need to think beyond simplistic, false dichotomies like “regulation or incentives” and embrace a robust combination of broad reforms coupled with targeted interventions.

First, we should press forward with efforts to provide education leaders with more sophisticated information on teacher effectiveness, to both maximize the impact of strategies that address distribution and to ensure cost efficiency. Education leaders need to be able to identify the strongest teachers in order to recruit and retain them, and assign them to the students who need their expertise the most. Similarly, they need to be able to identify weaker teachers in order to get them the support they need to join the ranks of effective teachers or to move them out of classrooms if they cannot improve. That is why the Obama administration is using the American Recovery and Reinvestment Act to insist that states tear down the “walls” that prevent them from linking teacher and student data and come clean on teacher evaluation systems that rate all teachers “satisfactory.”

But it will take time to develop richer and more sophisticated measures of true effectiveness. Until then, policymakers should use a combination of the best available measures to analyze teacher distribution, report on it, and act to increase equity. A study in North Carolina found that having teachers with a combination of characteristics and credentials can more than offset the gap in annual learning gains between African American students whose parents did not go to college and white students whose parents did. We need to act on the information we have available, even while we work to create more sophisticated measures.

Next, we need new policies that empower local superintendents and principals to use that information to better recruit and distribute highly effective teachers. Districts can move up timelines for teacher resignations and transfers and give principals in hard-to-staff schools first dibs on new entrants and transfers. States and districts can establish a policy of “mutual consent” that gives principals the right to choose their own teachers. States can take actions to pump up the supply of stronger teachers by using data on the effectiveness of graduates to improve teacher training programs, expanding those that produce strong teachers and shrinking or closing those that do not. States and districts can eliminate seniority-based layoffs, which should consider effectiveness instead, and make it easier to transfer or remove ineffective teachers who cannot improve.

Finally, policymakers need to make these schools much more attractive places to work, including but not limited to improving financial compensation. Effective teachers who choose to work in the most challenging schools often sacrifice pay and professional status. State leaders should reverse that relationship, offering such teachers higher pay, visible respect, strong and supportive principals who provide effective instructional leadership, and opportunities to collaborate in meaningful ways.

EN: How can we measure teacher quality on an ongoing basis?

KH: Measures of teacher quality should be based primarily on teachers’ effectiveness in promoting student learning, but should also consider evidence of classroom teaching practices known to contribute to greater student learning. All states now have at least the raw capacity to use value-added techniques to measure teachers’ contribution to their students’ academic progress. Where those data are available, they should be front and center in efforts to measure teacher quality. But since the data rely on annual standardized assessments, such analyses will not be available for all teachers. Moreover, since value-added data by themselves do not tell much about why a teacher is more or less effective or how exactly he or she can improve, such “outcome” measures can productively be coupled with new kinds of “inputs” measures, provided the two are strongly correlated.

For example, researchers at institutions such as the University of Virginia, Stanford, and Michigan State and at programs like the Teacher Advancement Program and Teach For America have developed protocols for observing classroom practices and analyzing teaching “artifacts” that produce ratings sufficiently correlated with outcomes. Typically, they use highly specific frameworks and rubrics that describe effective teaching practices, ensure that all evaluators are trained in their use, require multiple classroom observations per year, and employ quality controls to ensure reliability across evaluators. Such systems can help administrators and teachers understand why value-added scores look the way they do and how they can be improved.

Some districts are experimenting with systems that incorporate an even broader range of measures. For example, the evaluation system currently being implemented in Washington, D.C., incorporates a schoolwide value-added measure, a gauge of how much the teacher participates in and contributes to the larger school community, and measures of student growth on instruments other than standardized tests.

EH: We have devoted a lot of research to identifying the attributes of effective teachers, attributes that might be used for hiring or for policy purposes. This research has not succeeded, leading me to agree that the best way to identify a teacher’s effectiveness is to observe her classroom performance. Most other professions are assessed by performance, including that of doctors, lawyers, accountants, and so forth. Indeed, one definition of “profession” might be an occupation in which one is willing to be judged (and rewarded) according to performance.

Research suggests that we can identify effective teachers from the value added to student achievement, although there are limits to the accuracy of doing this. Moreover, Brian Jacob and Lars Lefgren, in the most recent of this research, show that principals reach many of the same conclusions about effectiveness in their evaluations; at least they seem able to distinguish effectiveness in the classroom within broad ranges, i.e., bottom, middle, or top.

The long-run hope would be that we develop both better quantitative measures of a teacher’s value added and better subjective evaluations by principals, supervisors, and peers. This approach is unlikely to satisfy a regulatory view of allocation of quality teachers, but if we are truly interested in improving student achievement, we cannot shy away from incorporating performance information of all sorts into our management decisions.

EN: All the evidence says that experience does not affect teacher quality much after the first three or four years, so should we be concerned that the more-experienced teachers leave for different locations?

EH: It is a concern if experienced teachers systematically leave the most-disadvantaged schools, because the first few years tend to be a little ragged. On the other hand, this fact by itself should not be overstated. Among all rookie teachers there is still a wide variation in skill. Take, for example, Teach For America teachers. On average, they start out looking like the typical experienced teacher from traditional training programs (even though TFA teachers will themselves improve with seasoning). More than that, the best and the worst TFA teachers or other rookies in the system are dramatically different from each other, and the difference is much larger than the performance growth typical for the first few years.

Policies that concentrate on single proxies for skill, like initial years of experience, miss the much larger differences. Yes, if we say we can do nothing about retention related to individual performance levels, it would be good to have more-experienced teachers in the disadvantaged schools. But such a focus overlooks the place where truly large changes are possible.

A policy that simply stabilized movement from these schools would not really accomplish much and might even be counterproductive if no attention were given to actual performance. On the other hand, if we made inner-city schools more attractive places to work and if we developed policies that actively reward high performance by teachers, we would probably get a bonus of lower teacher turnover in our most-disadvantaged schools.

KH: While experience in no way equals effectiveness, we still should be concerned about teacher attrition. Here’s why: high attrition rates in high-poverty schools create a “revolving door” environment with more job vacancies which, because such schools have a harder time recruiting teachers, tend to be disproportionately filled with first-year teachers. And experience does matter for inexperienced teachers. As a group, first-year teachers tend to be less effective than those with even a little more experience, and effectiveness tends to climb steeply for any given cohort of teachers until it begins to plateau after a few years. According to research by Eric Hanushek and others, disproportionate exposure to inexperienced teachers contributes to the achievement gap.

Therefore, policymakers should either seek to limit the number of rookie teachers hired to work in high-poverty and high-minority schools or ensure that beginning teachers come from programs or institutions with a proven track record of supplying teachers who are much more effective than average. Then they should track the effectiveness of beginning teachers in those schools over the first few years, offering substantial retention incentives to those who demonstrate high levels of effectiveness—not only salary incentives, but also career pathways that provide opportunities to exercise leadership while they continue to teach.

EN: If we force teachers to teach in particular schools, will they just leave for another district, or for an administrative position, or leave education altogether?

KH: We don’t know, since it’s never been tried on a large scale. More to the point, I would suggest that this is the wrong question to be asking, as nobody thinks forced reassignments are a good solution and nobody is seriously proposing it. Every once in a while, district leaders become frustrated and make noises about the possibility of forced reassignments. But no large district has done it because they know that it would be met with too much resistance and resentment.

Instead, as district leaders are discovering for themselves, a better solution lies in a creative combination of targeted incentives for teachers and policies that empower administrators and school leaders to recruit and retain effective educators.

EH: Coercion is generally costly, particularly when it violates the expectations of workers. The U.S. military found that the draft was not a good policy, even when it allowed them to get soldiers cheaply. With schools, the situation is more complicated. There are many jobs (including the all-volunteer military) where the employer can establish the right to make specific job assignments, but in general the employer must pay for that ability. Today’s urban teachers frequently have a contract that gives the more-experienced teacher certain transfer rights across schools, and changing that provision would generally require bargaining with compensation involving higher salaries or other benefits that the teachers value.

The current contractual arrangements are in many cases overly concerned with teachers’ rights and less concerned about student outcomes than is desirable. It would make sense to work toward more assignment flexibility by school districts. But, again, this may be lower priority than simply having more control over retention based on classroom effectiveness.

EN: If we pay teachers more to teach in inner-city schools, will that really attract the best teachers?

KH: Financial incentives can have a positive impact on teacher distribution, but how much of an impact depends on the size of the incentive and to whom it is being offered. Research from North Carolina suggests that smaller financial incentives can help retain teachers in hard-to-staff schools, but experience in places like Dallas and the Charlotte-Mecklenburg school system suggests that incentives need to be fairly large to convince highly effective veterans to transfer and remain there. That shouldn’t stop leaders from offering higher salaries for effective teachers who successfully take on more-challenging jobs. But the qualifiers in that sentence are important: Pay incentives should be offered only to teachers of proven effectiveness, and a portion should be in the form of bonuses contingent on continuing high performance.

Policymakers can free up resources by putting a stop to or limiting counterproductive incentives in current salary schedules. For example, they can set a ceiling on the percentage of teacher compensation districts can base on seniority, and they can stop the practice of paying teachers to earn master’s degrees, which study after study has shown to have no discernible impact on student achievement.

But higher pay alone might not be enough to solve the problem. Some districts have found that even large financial incentives, in the absence of better working conditions, fail to attract and retain strong teachers in high-need schools. The reason is simple: like any other professionals, great teachers place great value on a positive and supportive working environment characterized by strong leadership and opportunities to collaborate with colleagues.

Rather than being discouraged to know it takes more than money to attract stronger teachers to struggling schools, leaders can leverage that knowledge to devise creative solutions. For example, when recruitment bonuses failed to solve the teacher inequity problem in the Charlotte-Mecklenburg schools, leaders came up with a comprehensive “Strategic Staffing Initiative.” The district transferred high-performing principals into targeted schools, allowed them to handpick a team of strong administrators, and gave them the opportunity to recruit up to five highly effective teachers from a roster of volunteers identified and recruited by the district. Everyone who transferred received substantial financial incentives, but, just as important, all were offered the opportunity to work with a team of teachers and administrators committed to achieving success.

EH: There is a simple economic axiom that bad teachers like more money as much as good teachers. Providing higher salaries will do little to improve the quality of urban teachers or teachers of disadvantaged students unless this is coupled with a clearer judgment about effectiveness. If the objective is raising achievement, there is no real substitute for observing achievement and taking actions based on it.

School accountability systems move in this direction when the rewards to principals and teachers are linked to the growth in student learning. At that point, higher salaries, if directed toward more effective teachers and administrators, can be effective. But if higher salaries are awarded by geography and not demonstrated effectiveness, there is little reason to expect improvement.

The central message of this discussion must be that improving student outcomes in the inner city cannot be done by proxy. We must use the direct and available information on teacher effectiveness that comes from objective achievement data and subjective evaluations for both administrators and teachers to guide rewards and management decisions. We may conclude that this is too difficult—because of union contracts, traditions, or other issues. In that case, we must be willing to live with disastrous results or, alternatively, be prepared to give parents the real opportunity to choose better schools. We have a long track record of regulating that schools should “do good”; of following the current ideas, including simply paying teachers more; and of holding out for the perfect, fully tested alternative. We are left with stagnant achievement results that are especially egregious for poor, inner-city kids. More of the same will not work.

Now it is showing additional leadership by moving aggressively on issues of teacher quality.  It is poised to pass legislation that would do two things.  It would do away with teacher tenure for newly hired teachers.  And, it would require that half of teacher pay increases be based on student performance.

Who could be against these ideas?  Certainly parents and students cannot be.  But just as certainly, the teachers unions are aghast that anybody would want student outcomes to play a prominent role in teacher retention decisions.

Florida legislators recognize that teacher quality is central to student outcomes.  They also recognize that neither teacher experience nor graduate degrees bear any consistent relationship to student achievement.  This legislation is simply putting policy where the evidence is.

Florida is poised to lead the nation in crafting student policies.  No wonder the fight is being fought so hard in the Florida House.  This kind of precedent could sweep the nation.  And then where would we be left?  We would just have to make policies that were proven to support student learning.

The Secret of TSL: ?The revolutionary discovery that raises school performance
By William G. Ouchi

When I first saw the title, never having heard of TSL, I thought this might be a late-night infomercial about a new diet supplement designed to make all students attentive. Not far into the book, I discovered that TSL was Total Student Load, which, unfortunately, did not help me very much. Then to the hypothesis on the cover: The key element of a school’s organization is the number of students that a teacher regularly sees (TSL), and if this number is small (say, 80), achievement will be high.

The hypothesis is really an assertion based on a vaguely described analysis. And while it is a discernible undercurrent throughout the book, TSL is not the volume’s central feature. The book presents a series of case studies of large, and distinctly nonrandom, districts. Within those case studies, the focus is twofold: decentralization of decisionmaking and the quality of the superintendent. The book provides an in-depth look at districts that have in one way or another followed the advice given in one of Ouchi’s previous books, about the benefits of weighted student funding, whereby schools receive funds based on the make-up of their student populations, and decentralized decisionmaking. This book includes additional observations of schools where the principles of fiscal decentralization are evident.

What is good and interesting about The Secret of TSL? Ouchi traces the evolution of district policies under several high-profile leaders—Joel Klein (New York), Arne Duncan (Chicago), Arlene Ackerman (San Francisco), Rod Paige (Houston), Randy Ward (Oakland), Pat Harvey (St. Paul)—whose stories are both compelling and informative. The perspective is that of a management professor, one trained in understanding decisionmaking styles and models and the interactions of institutions and individuals. This approach is one not commonly taken by education researchers, who more often focus on what is happening in classrooms and the interactions between students and teachers. Here, an experienced observer looks at the overall structure of how education is produced. The higher-altitude view is both useful and intriguing.

The story line that emerges, perhaps unintentionally, is that the individual leaders have very different views about how to organize and run schools. No one would accuse Randy Ward of having the same style as Arlene Ackerman, even though they were for a time separated only by the Bay Bridge. Indeed, almost as an aside to the title page, the districts that are described in detail follow very different policies that lead to wholly different TSL measures.

What does not work in the book? Well, start at the beginning. There is no sense in terming TSL a “revolutionary discovery.” While TSL is calculated in each of the case studies, there is no evidence that the measure is correlated with overall district performance or district growth in achievement. In fact, the “revolutionary discovery” looks more like a required element of a standard management book aimed at the New York Times best-seller list. In the tradition of that genre, there are two numbered lists: the “five pillars” of school empowerment and the “four freedoms.” These lists largely drop out of the sky except that some of the included items appeared in Ouchi’s earlier “revolutionary” book, Making Schools Work: A Revolutionary Plan to Get Your Children the Education They Need. In actuality, the lists are not bad: choice, school empowerment, effective principals, accountability, and weighted student funding matched with control over budget, staffing, curriculum, and scheduling. But there is little explanation about how these notions are implemented, what impact might be expected, and what the trade-offs among the elements might be. In the separate case studies, the leaders sometimes pay attention to the elements on these lists, and sometimes do not, and it is hard to see that those who heed the lists do better than those who do not.

In the end, it is difficult to tell whether the story is about some gifted leaders or about decentralized authority and specific programs. At this point, the case study methodology breaks down, because it is impossible to separate structure and institutions from personality.

But, returning to TSL, the argument is compelling in an intuitive sense. How can one expect a teacher to really get to know 150 different students during a year? How can a teacher possibly assign regular and demanding homework to such large numbers if it is necessary to review and grade all the assignments?

There are, however, some crucial issues of interpretation that beg for serious empirical analysis. For example, the discussion leaves out whether TSL is expected to have an impact while all other things are held constant, such as budget, teacher expertise, curriculum, and support services, to name a few. Or, does it enhance achievement to trade some of these attributes for a smaller TSL? It would be particularly valuable to marry these organizational views with separate analyses of teacher effectiveness. Current discussions of the importance of teacher quality for achievement generally ignore such environmental features as district management and decisionmaking. Could it be that some of the observed variation in teacher quality really reflects unmeasured differences in the organizational features that Ouchi highlights in his case studies? These are testable propositions, and ones that could provide important insights into where the revolution in student achievement is most likely to occur.

Eric Hanushek is senior fellow at the Hoover Institution of Stanford University and a member of the Koret Task Force on K–12 Education.

Various people began lining up with one study or the other – largely it seems on the basis of which results they liked.  Those supporting expanded charters emphasized the New York results, while those generally disliking charters emphasized the other.

There were two major differences among the studies:  they used different evaluation methodologies, and they analyzed different sets of charter schools.  The CREDO study employed a matching approach that compared students in charter schools to a virtual student who had similar prior achievement, race, income, and so forth along with being in one of the feeder schools from which a given charter drew its students.  The New York study compared students who won a lottery for entry into each (oversubscribed) charter to students who lost the lottery.  The CREDO study looked across 15 states (which did not include New York), while the NBER study was confined to New York City schools.  Either or both of those differences could be responsible for the different results.

A new study by CREDO clears up the uncertainty.  They took their matching approach to evaluation to New York City charters, thus holding constant location.

The new CREDO results were virtually the same as the prior NBER results:  Charter schools in New York City do significantly better than the traditional public schools that feed them.  Thus, it is not methodology that drives the prior differences in results, but instead it is the fact that New York City simply is doing something different.

These results change the focus of debate.  They bring us back to considering what is it that makes some charters fly high and others fall flat.  Is it the authorizing environment?  The state of existing public schools in an area?  The role of state regulations and oversight?

It is really important to dig deeper into the underlying causes of effectiveness across the charter sector.  They will not only give us insights about how to organize charter schools but also how to manage and improve the traditional public schools.

Questions of educational adequacy and school spending have long been a point of contention in school reform. Amid the recent economic turmoil and gaping state budget shortfalls, questions of whether court-ordered funding remedies have delivered—and why they have or have not—have taken on particular import. This forum offers two sharply different takes on our experiences to date, and what lessons they offer going forward. Eric Hanushek and Alfred Lindseth are the authors of Schoolhouses, Courthouses, and Statehouses: Solving the Funding-Achievement Puzzle in America’s Public Schools (Princeton University Press, 2009), in which they propose a system of performance-based funding focused on improving student achievement. Michael Rebell is executive director of the Campaign for Educational Equity at Teachers College, Columbia University, and is the author of Courts and Kids: Pursuing Educational Equity through the State Courts (University of Chicago Press, forthcoming), in which he proposes a new functional separation of powers among the executive, legislative, and judicial branches to promote education reform and student achievement.

Education Next: Over the past four decades, many states have revised their funding of schools, through either judicial or legislative initiatives, in an effort to improve schools serving disadvantaged children. Too often, however, these actions have not yielded improved student achievement. Looking to the future, what kinds of judicial or legislative remedies are most likely to fulfill the promise of improved student outcomes?

Eric Hanushek and Al Lindseth: This question is particularly timely, as national policies on education embodied in the federal No Child Left Behind (NCLB) law are in a state of flux and likely to change under President Obama. At the same time, an economic crisis has engulfed not only our country, but most of the world, suggesting that significant increases in funding for education budgets are unlikely in the foreseeable future. The challenge is to find ways to develop a well-educated workforce that are not only more effective than those relied on in the past, but also do not depend on significant annual increases in education appropriations.

Since about 1970, the achievement levels of U.S. students on the reading and math tests of the National Assessment of Educational Progress (NAEP) have remained largely flat despite massive financial and other efforts to improve them. The problem is particularly acute for poor and minority students, with average black and Hispanic students lagging three or four grade levels behind the average white student. While lack of sufficient funding is often cited as the principal reason for low student performance, the United States already spends more on K—12 education than all but a few countries. Moreover, spending has increased dramatically over the past several decades, with today’s per-pupil expenditures almost four times, in inflation-adjusted dollars, what they were in 1960.

The underlying system, which governs how money is spent, has remained largely unchanged over that period. It is characterized by, among other things, a compensation scheme that pays teachers and administrators without regard to the results they get in the classroom; rules that make it extremely difficult to terminate unqualified teachers or assign the good ones where they are most needed; an assessment and rating system that discriminates against good teachers who are assigned to schools with significant numbers of at-risk students; a monopolistic structure that insulates public schools from competition; and numerous union and other work rules that prevent principals from effectively running their schools. It is a system more concerned with the adults and their rights than it is with ensuring the success of its students. Although some reforms have taken place in the last decade or so—the adoption of statewide standards, limited choice options, and increased accountability—they have not been sufficient to overcome the obstacles posed by the underlying system.

Given this sobering assessment, what can be done in the future to improve student achievement? The solution, we believe, lies in performance-based funding: a system of integrated education policies and funding mechanisms designed to drive and reward better performance by teachers, administrators, students, and others involved in the education process. Such a system will ensure more effective use of education dollars through better decisionmaking, eliminate perverse incentives that reward mediocrity or failure, and most important, energize and motivate those involved in the education of our young people. The essential components of a performance-based funding system cannot be ordered à la carte. These components interlock and depend on each other for their success. While various states have adopted some of these components—state-level academic standards, for example—none have implemented the integrated system we recommend, and the results have been clearly unsatisfactory.

A performance-based system of funding would contain the following nine features:

1) A focus on improving outcomes rather than on increasing inputs. States must set high and uniform achievement goals for every child to strive to meet. While every child may not reach the highest goals, high expectations will encourage children to do their best.

2) Local school administrators and teachers with the flexibility to determine how their schools can best meet high standards. Often even the most dedicated teachers and principals are hampered by severe limitations on spending and programmatic decisions by ineffective state regulations, constraints such as those that come with categorical funding, and a variety of state and local laws and contractual arrangements. The idea is to let those who are most familiar with the problems faced in the schools take the lead in deciding how to solve them.

3) Rewards for both teachers and administrators based on their success in improving student achievement. In almost every school district in the country, teachers are currently paid based solely on their years of experience and degree level, despite a consensus in the scientific community that these two factors bear little relationship to their success in improving student performance. The single-salary pay schedule—which makes it virtually impossible to pay good teachers more, to offer bonuses for teaching in hard-to-staff schools, and to pay higher salaries to teachers in shortage areas, such as math, science and special education—must go, and a pay system implemented based upon the just named considerations.

4) Greater accountability commensurate with increased authority and discretion. Teachers, schools, and principals must be held accountable for results. Just as they should be rewarded if they are successful, they must experience the consequences if they are not. Each state should adopt an accountability plan that sets clear goals as well as significant and enforceable consequences if goals are not achieved within a reasonable period.

5) Rewards and accountability based on factors within the control of the local district. Schools, teachers, and administrators should be judged and, if appropriate, rewarded based on the “value” they add during the school year, not on absolute test scores. The latter may be influenced by students’ homes and neighborhoods and may give teachers in middle-class suburban communities an advantage over those teaching in less advantaged communities. Under current practice, schools with disadvantaged students are almost always labeled “failing,” no matter how good the teachers are. Once value-added assessments are put in place, it will be possible to isolate the contributions made by the schools, teachers, and programs in raising achievement from external factors also affecting achievement and to act accordingly by following a model of continuous improvement.

6) Schooling options for parents and children who judge their school less than satisfactory. Schools must know that, if they are not successful, parents have alternatives for their children. Therefore, the finance system should also support charter and other choice schools.

7) Reasonable funding levels based on the needs of particular student enrollments and other factors outside of district control, but also discretion by local district taxpayers to augment the funding of their schools. Base funding would adjust for district poverty and external labor-market factors. Supplementation should incorporate “equalization” funds by the state to recognize differences in the ability of districts to raise funds locally when levying the same tax rate, but would permit parents and taxpayers to express directly their satisfaction with educational plans and policies.

8) Transparency incorporating value-added measures. Parents, taxpayers, and other stakeholders can then readily gauge how good a job the schools are doing.

9) A commitment to evaluating school and programmatic effectiveness. Expensive new strategies, such as large-scale class-size reduction programs, should be implemented only if they also provide for regular, independent evaluations to determine their effectiveness. Unsuccessful programs should not be allowed to continue and proliferate year after year just because they have strong sponsors.

The path to such reform will not be an easy one. While elements such as state standards, accountability measures, and value-added measures either are not controversial or are gaining acceptance, other important components, especially performance-based pay and increased choice options, are opposed by powerful forces with vested interests in the current system. Most powerful are the politically connected teachers unions. They, for example, clashed with Washington, D.C., schools chancellor Michelle Rhee over a proposal to couple higher pay with greater risk of termination because of ineffectiveness. The unions vigorously opposed those efforts, leading Rhee to move instead to improve the teaching force by terminating unqualified teachers. Unless this system is changed, it seems unlikely that outcomes will measurably improve in the district, already one of the highest funded and worst performing in the country.

The responsibility to enact and implement performance-based funding systems will fall primarily on the political branches of government, the state legislatures and governors. Although judicial remedies have played a significant role in school finance in the past, that era is drawing to a close. Beginning in the early 1970s, advocacy groups, frustrated with legislative efforts, began turning to the courts, initially to seek more equity in the allocation of education funds and later to seek vastly increased appropriations from state legislatures through “educational adequacy” lawsuits based on vaguely worded state constitutional provisions. A significant number of state courts responded positively to plaintiffs’ pleas and ordered unprecedented increases in K—12 funding in their states. Unfortunately, basic problems in the underlying systems of delivering education services were often ignored. In this sense, the courts mirrored what had been going on in the state legislatures, and the results were, not surprisingly, much the same: large amounts of money expended, but little or no improvement in student outcomes. An analysis in our recently published book examines the NAEP test-score trends in the four states that have implemented court remedies the longest, and demonstrates that, despite spending increases amounting to billions of dollars, the achievement patterns in three of them—Wyoming, New Jersey, and Kentucky—are largely unchanged from what they were in the early 1990s, before the court-ordered remedies commenced. Only in Massachusetts, where much deeper and broader reforms were instituted, has there been some improvement, although even there the state’s black students have not benefited from the remedy.

Perhaps due in part to this track record, the courts have begun to step back, opting instead to leave decisions regarding education policy and appropriations in the hands of the political branches of government, where they have traditionally resided. In the last five years, court decisions in approximately 15 states have disposed of educational adequacy cases, and, with one or two minor exceptions, the courts have either dismissed the cases or granted minimal relief. While this could change in a number of cases still pending, we believe the likelihood of significant court-ordered remedies in the foreseeable future is small.

Performance-based funding is not by itself a panacea that will solve all problems of substandard achievement or eliminate the achievement gap. Many of the problems that plague American education are beyond the control of the schools and will have to be addressed by other means. Performance-based funding will, however, put the nation’s schools back on the right track, help to raise the achievement of all students significantly, and once again make our students competitive on the world stage.

President Obama has called for increased funding to support NCLB, and Congress has provided substantial stimulus money for schools. A wise use of that money would be to underwrite transition costs in states moving to implement a performance-based funding system. For example, support for the improvement of student testing, for the development of improved databases and value-added measures, and for initial payments of expanded salaries under performance-based pay could provide important incentives for the states to move toward more logical and more effective funding systems.

Michael Rebell: The basic premise of the book and essay by Eric Hanushek and Al Lindseth—and of the question posed by Education Next—is that although “massive” amounts of money have been spent on education over the past 40 years, the results have been meager. Hanushek and Lindseth claim that states in which courts have ordered “extraordinary spending increases,” or at least the select few they have studied, have shown no improvement in student test scores. They then argue that certain “performance-based” accountability mechanisms that they recommend, rather than increased funding, should be the focus of future efforts.

I strongly dispute these premises, and I doubt that the reforms that Hanushek and Lindseth recommend are feasible, or that if enacted, they would constitute the panacea for the nation’s education ills that they imply. Extensive inequities in education funding, by which students with the greatest needs receive the fewest funds, still prevail in many parts of the United States; for that reason, state courts continue to have a critical role in ensuring meaningful educational opportunities for all children. The evidence strongly indicates that money well spent does make a significant difference in student achievement, and as Education Sector’s Kevin Carey has noted in reviewing one of Mr. Hanushek’s books: “There is little evidence that starving schools of needed funds is a catalyst for innovation, or that well-funded schools are more likely than others to be inefficient.” Moreover, although I agree that additional accountability measures are needed, continued involvement of the state courts, working in concert with the executive and legislative branches in a new, functional separation-of-powers mode, is essential for holding all parties accountable and for attaining the nation’s education goals.

Let me first put the spending issues into perspective. Hanushek and Lindseth claim that per-pupil spending in the U.S. has quadrupled since 1960. This is a gross exaggeration. According to recent analyses by Economic Policy Institute research associate Richard Rothstein, the cost of school services, when adjusted by the consumer price index, increased by 157 percent from 1967 to 2005, but when adjusted by the more relevant net services index (which omits shelter rent and medical care. the increase was only 92 percent. Moreover, these general statistics mask the fact that much of this increase has gone to special education, a sector that has dramatically expanded and substantially improved the lives of millions of students with disabilities over this time period. According to Rothstein, from 1967 to 2005 the share of educational expenditures going to regular education dropped from 80 to 55 percent and the share going to special education increased from 4 to 21 percent.

Second, for the past two decades, the United States has been committed to the historically unprecedented mission of simultaneously promoting excellence and equity in education. The standards-based reform movement seeks both to equip all of our high-school graduates to compete in the global marketplace and to narrow the achievement gap between our advantaged and disadvantaged student populations. Obviously, attaining these critical goals will require substantial resource infusions, especially for the high-need schools that historically have been treated inequitably by state education finance systems. Thus far, neither Congress, which has not even come close to fully funding the No Child Left Behind Act, nor most states, which have raised their academic standards but not their funding levels to a commensurate degree, have stepped up to the plate.

Third, Hanushek and Lindseth assert that “the United States already spends more on K—12 education than all but a few countries.” Although the U.S. is fourth among the 30 industrialized democracies that comprise the OECD (Organisation for Economic Co-operation and Development) in per-pupil spending on K—12, it is in the middle of the pack (13) in education spending as a percentage of GDP. Moreover, since, comparatively speaking, the U.S. starves health care, economic security, housing, and other areas of social welfare provision, the schools must bear an enormous burden in overcoming the impact of concentrated poverty for the poor and minority children they are committed to educating to high levels. In 2005, the childhood poverty rate in the U.S. was 21.9 percent, the highest, with the exception of Mexico, of the 24 OECD countries listed, and far higher than the 3 percent childhood poverty rate of countries like Denmark and Finland.

Given the extent of poverty in our society and the heavy burden that has been placed on the schools to alleviate its impact, it is astounding how much educational progress has been made. For example, from 1990 to 2007, black students’ scale scores increased 34 points on the NAEP 4th-grade mathematics tests (compared with a 28-point increase for whites), and the black-white achievement gap declined from 32 to 26 points during this period. Nevertheless, even greater progress can and should be made. I doubt, however, that “performance-based funding,” the solution Hanushek and Lindseth offer, will prove to be the silver bullet that will “help to raise the achievement of all students significantly, and once again make the nation’s students competitive on the world stage.”

Hanushek and Lindseth announce their performance-based funding prescriptions as if they will instantly solve the nation’s educational ills. But test-based outcomes, merit pay for teachers, rewards and sanctions, and voucher and charter alternatives have been part of the reform agenda of most states for years. Studies of each of these approaches have generally shown mixed results, and there is no strong empirical basis for dramatically expanding their use. Hanushek and Lindseth have an answer to this criticism: This is not a menu of options that can be ordered à la carte, they say. These components interlock, and they must be implemented as an “integrated system.”

Leaving aside the objections I have to many aspects of their program, full implementation of their “integrated systems approach” is clearly a pipe dream. In a democratic polity, no single reform approach can ever be fully put into effect, much less maintained, in its pure form. Policymaking for public education in a democracy inevitably is shaped by politics, and any reform proposal will inexorably be subject to compromise and modification. Although there was an unprecedented degree of bipartisan support for passage of the No Child Left Behind law in 2001, for example, that support came at a high price. As Education Next editors Rick Hess and Chester Finn recently observed, NCLB is a “Christmas tree of programs, incentives, and interventions that are more an assemblage of reform ideas than a coherent scheme. NCLB’s remedy provisions bear all the marks of concessions to various ideologies, advocates, and interest groups, with scant attention paid to how they fit together, the resources or authority they require, or whether they could be sensibly deployed through the available machinery.”

How to forge a better package of education reforms out of the positive aspects of NCLB is the main education policy challenge for the Obama administration, and how to make standards-based reform really work is the parallel problem that state education policymakers need to face. Hanushek and Lindseth’s performance-based funding proposal adds little of real value to this equation. However, the state courts’ wide experience in recent decades with fiscal equity and education adequacy litigations, which these authors roundly criticize, does provide significant possibilities for developing productive policy compromises and significantly advancing prospects for meaningful education reform.

Since 1973, when the U.S. Supreme Court held that education was not a “fundamental interest” under the federal constitution, education advocates, frustrated by continuing inequities in the funding of public education, have turned to the state courts. As Hanushek and Lindseth acknowledge, a “significant number of state courts responded positively to plaintiffs’ pleas.” In fact, during this era, cases have been filed in 45 of the 50 states, and plaintiffs have won more than 60 percent of them; since 1989, when the legal emphasis shifted from “equity” cases that seek equal funding levels for all students to “adequacy” cases that look to provide all students a basic quality education consistent with state standards, plaintiffs have prevailed in two-thirds of the final high-state-court decisions.

Hanushek and Lindseth claim that the courts have begun to step back from their support of constitutional rights in this area. But, in fact, there has been no diminution in the willingness of state supreme courts to issue strong rulings on students’ basic constitutional right to an adequate education. What has changed in recent years is that more cases have reached the remedy stage and more courts are experiencing difficulty in seeing constitutional compliance through to a successful conclusion. Some courts have cut short their remedial oversight out of frustration with the political complications and complexity of effecting meaningful change.

In other words, the adequacy movement has matured, and the courts are now grappling with many of the same implementation and compliance issues that have stymied governors and legislatures for years. The problems raised by judges in these remedial proceedings call for thoughtful responses and nuanced solutions, rather than the cavalier rejection of “judicial activism” that Hanushek and Lindseth and other opponents of adequacy articulate. (The title of a recent book that Hanushek edited and to which Lindseth contributed accuses judges of “harming our children.” This kind of hyperbole is clearly unwarranted.) As University of Wisconsin law professor Neil Komesar has insightfully pointed out, “All societal decision makers are highly imperfect.” Governors, state education departments, legislatures, and the federal Congress have been unable to solve the nation’s educational problems over the past half century, so why should anyone expect judicial interventions to achieve immediate, decisive results?

Where courts have persevered in their efforts, there have often been substantial improvements in student achievement. Hanushek and Lindseth allude to NAEP test-score trends in a few states with long-standing court orders that they claim have resulted in no improvement in student achievement in three out of four cases. The NAEP scores they focus on do not correspond in most of the cases to the relevant years in which the court orders were actually implemented; they ignore the fact that, as in Kentucky, initial increases in funding are sometimes followed by substantial decreases in later years; and their use of NAEP scores makes no sense in a state like New Jersey, where the court orders covered only a subset of the state’s students ( i.e., students in 31 poor urban school districts) and not the full statewide populations represented by NAEP scores. Recent, more finely tuned data for New Jersey, provided by Peg Goertz, a University of Pennsylvania researcher who has closely followed developments in the Garden State, indicate that from 1999 to 2007 substantial gains were made in the Abbott districts, which were the focus of the judicial remedies. For example, in 4th-grade mathematics, the achievement gaps between the Abbott districts and the rest of the state were cut by more than one-third. Similarly, Kentucky, which was near the bottom of the national rankings in virtually all performance indexes before its 1989 court decision, now ranks above the national averages in reading and science and almost at the national average in math.

Despite these gains, to fully meet our nation’s challenging goals for excellence and equity in our public school systems, clearly more needs to be done. What is required is a concerted effort by all three branches of government to bring their relative functional strengths to bear on ensuring constitutional compliance and solving the nation’s educational ills. In a forthcoming book, I propose a “successful remedies model” that is based on the extensive empirical experience that dozens of courts have had in dealing with legislatures, governors, and state education departments in crafting remedies. It is a process approach that is compatible with Hanushek and Lindseth’s performance-funding focus or any other policy perspective, or as is more likely, whatever mix of policies a state’s elected representatives choose to endorse. This process seeks to ensure that, whatever reform path state policymakers pursue, the compromise package they assemble is cohesive, adequately funded, and consistently implemented; moreover, the state should be committed to seeing the reforms through over time so that lasting results can be achieved. “Success” in implementing standards-based reforms under this model is defined not in terms of test scores in a limited number of subject areas, but broadly, in terms of providing all students a sound basic education on a sustained basis.

To achieve such success requires effective, programs and ongoing “colloquy” among the three branches of government. The courts’ role in this process is to outline in general, principled terms the expectation that the legislative and executive branches will develop challenging standards, fair and adequate funding systems, and effective programs and accountability measures, but to leave to the programs and the political branches the full responsibility for actually formulating these policies. Legislatures should make basic educational policy decisions; state education departments and local school districts should determine how best to implement educational reforms. Once the state has decided on its policy position, however, a judicial presence should be maintained to ensure that the chosen policy is fully funded, is implemented in a coherent manner, and results in substantially improved student performance, as measured by validated assessments of academic achievement and of students’ ability to function as capable citizens and workers.

Since significant compliance cannot be achieved overnight, in most cases courts will need to maintain nominal jurisdiction for a multiyear period, probably 10 to 15 years. The mere fact that judicial oversight remains in place can ensure continued adherence to implementation of stated policy goals, and actual interventions should be rare, especially if it is clearly understood that all the courts would be enforcing are the state’s own policy goals. A judicial presence is especially important to ensure that the reform process—and reasonable funding levels—are maintained in times of economic stress or recession like the present, where children’s needs and constitutional values are often given short shrift.

In short, then, my answer to the question posed by the editors of Education Next is that what is most likely to fulfill the promise of improved student outcomes in the future is not any silver bullet remedy, but rather a pragmatic process that allows courts, legislatures, state education departments, and school districts to work collaboratively to implement meaningful reforms on a sustained basis.

Hanushek and Lindseth: Notwithstanding his obfuscation, Michael Rebell’s solution is essentially more of the same. Beginning by misstating spending increases (based on incorrect data and flawed adjustments) and ignoring pertinent performance data, he rewrites the constitution of every state to give judges the major policy-setting role in a “new, functional separation of powers mode.” He further recommends that judges and legislators be guided in their efforts by a “successful remedies model” to be drawn from previous adequacy litigation—perhaps tempting if such “successful” models actually existed. Quite surprisingly, he cites New Jersey’s tortured 35-year-old Abbott litigation as an example of “success,” but neglects to mention that the state’s black students, the principal beneficiaries of the remedy, are still scoring at about the same relative levels on the NAEP tests as in 1992. In Kentucky, he relies on data for all students, which mask the fact that black students, the state’s principal minority group, have regressed compared to their peers nationally during the remedial period.

Our solution may not be a “silver bullet” for everything that ails American education, but it surely presents a better chance for our children than continuing the demonstrably failed practices of the past. In the end, Rebell basically concludes that political forces are too strong to bring about the fundamental changes we recommend, so we should just continue plowing more money into the current system. If we do, no one should be surprised in 2040 when our students are still performing, as they are now, at 1970 levels.

Rebell: If I didn’t know that Rick Hanushek was an outstanding economist and that Al Lindseth was a master litigator, I would think from some of the provocative phrases they use in their writings that they were sensationalist journalists, looking to attract readers with shocking but misleading headlines and catchphrases. They claim that I am proposing to “rewrit[e] the constitution of every state to give judges the major policy-setting role.” A detailed examination of the positions they actually take in their writings, and especially in their recent book upon which this Forum is based, indicates, however, that we agree that money—if well spent—does matter, that education finance cases have had a significant equalizing effect on state education funding formulas, and that court orders can “support legislators who want to address serious problems in education.”

The fact is that the unproven, business-model, and privatization practices they propose as education reforms have no chance of being adopted as an “integrated system,” especially in the present political climate. I would, therefore, ask Hanushek and Lindseth to stop tilting at windmills and to join with me in instituting a dialogue in major areas in which we do agree, like the fact that courts can and should hold states and school districts accountable for better performance, and that “school funding policies must recognize the underlying heterogeneity of students and their educational challenges and ensure that all schools have the means to succeed” (Hanushek and Lindseth, Schoolhouses, Courthouses, and Statehouses, page 218). That kind of conversation might help to promote real changes that might provide truly meaningful educational opportunities to all of our children.

NCLB focuses on having all students proficient in reading, math, and science.  All states had to develop rigorous learning standards and assessments of student performance, and individual schools are required to be on a path leading to universal proficiency by 2014.  Research provides both an understanding of what has and has not worked and a map for useful alterations in the law.

What Have Been the Results of NCLB?

Because test-based accountability is generally applied to entire states, it is hard to infer what might have happened in its absence.  Three separate lines of inquiry, however, provide evidence that existing accountability systems have led to larger gains than expected in a world without them.

First, comparisons of math and reading performance across states from the National Assessment of Educational Progress or NAEP – often called the nation’s report card – provides some insights.  Other things equal, states introducing accountability earlier showed larger gains on NAEP during the 1990’s.  Moreover, students in states with stronger accountability performed better. Second, by comparing students in Florida schools graded “F” on accountability and subject to increasing sanctions with almost identical schools scoring just above at “D”, David Figlio and Cecilia Rouse find positive effects of school accountability.  Finally, from results of individual state tests over time, student achievement gains tend to be larger after the introduction of NCLB than before. Although each is subject to uncertainty, the combined picture shows improved student performance after the introduction of test-based accountability.

Second, accountability, particularly after NCLB, focused attention on achievement of disadvantaged populations.  Evidence indicates that this feature has changed the dynamic within schools, yielding improvements in previously low performing.

Third, the U.S. evidence is consistent with a growing body of international evidence pointing to the value of central exit exams and more regular accountability.  Particularly where there is more autonomy in local decision making, schools facing stronger accountability pressures do better on international math and science exams.

What Changes are Needed?

At the same time, accountability is a relatively new invention, and it needs to be refined and improved.

1. Re-adjust state and federal responsibilities. NCLB has each state set learning standards, assessments, and proficiency levels independently with the federal government determining what actions should be taken when schools fail to make sufficient progress. This division appears backward.  Under NCLB, states have chosen widely different cutoffs for “proficiency.” But, in the face of national labor markets where somebody from Georgia could well end up working in Arizona, these variations make little sense. History suggests stiff opposition to a national curriculum.  But as recently seen, nothing prevents states from voluntarily joining together to develop standards and assessments.  The federal government could support and encourage this.

On the other hand, the diverse circumstances of schools indicate that centrally defined educational processes are unlikely to be effective.  The federal government is not well equipped to determine precisely how schools do their job.  Reforming NCLB could require states to develop their own plans for schools that were failing.  Indeed, recognizing the heterogeneity of schools, the U.S. Department of Education has already permitted variation in plans (“differentiated accountability”) in nine states.  Permitting local autonomy with central testing is, as noted above, a successful strategy consistent with international performance evidence.

2.  Focus accountability on learning growth. NCLB concentrates on the proportion of students below the state determined proficiency level in each year with progress determined by comparing the percentage of successive cohorts reaching proficiency.  But, schools are just one input to education.  Families, friends, and neighborhoods also exert an influence so that looking just at the overall level of a student’s achievement does not capture the school’s contribution to learning. Setting accountability in terms of individual student learning growth implies that schools are assessed much more closely to their value-added to learning. Such improvements are well-recognized with 15 states already authorized to use growth models for their accountability under NCLB.  Additionally, assessing growth across different learning levels rather than just at the proficiency threshold would eliminate incentives to ignore students already above proficiency or too far below to reach proficiency soon.

3. Improve assessments.  The current focus on more basic skills provides advantages and disadvantages.  The intent of NCLB is simply to ensure that all students will be able to participate fully in society and the labor market, but we also want to encourage and develop higher order skills.  For testing efficiency, current tests are generally designed to measure most precisely a limited range of skills.  An attractive alternative, however, is use of adaptive testing, which can improve measurement in the range of higher order skills. A set of screening questions moves the student to the relevant range of test questions – something easily done with computerized testing.  Computerization has two additional advantages.  First, it would provide immediate scoring of tests, getting around current delays in test scoring.  Second, having a large test bank would permit providing each student with a random selection of questions, minimizing any chance of cheating.  Indeed with a large test bank covering the range of relevant material, it would even be possible to make questions available beforehand with the notion that teachers could then productively teach to the test.

4.  Fix the teacher quality requirements. Research has found teacher quality to be the most important element of a good school, and this belief led the NCLB law to require all schools to have only “highly qualified teachers.”  Unfortunately, there are severe measurement problems that make prior interpretations of this requirement hollow at best and harmful at worst.  Teacher quality is not captured by typically discussed characteristics of teachers such as master’s degrees, teaching experience, or even certification – things that states typically monitor.  Requiring such things unrelated to student performance dilutes accountability and detracts from things that would make them more effective.  Fortunately, however, test-based accountability produces the student achievement data needed to assess the value-added of teachers, a more appropriate focus of policy concerns.

Conclusions

Test-based accountability is now a fixture of American education, but it has also become controversial.  Existing research indicates accountability has had a positive impact on school performance but also that it could readily be improved.  Clearly test-based accountability does not do everything, but it is a central part to almost all serious reform efforts.  Thus, improving it rather than eliminating it is the only reasonable course.

Recently, however, this trend has come to a grinding halt.  Since 2005, there have been important adequacy case decisions in over a dozen states, and in none of them have the courts required further funding increases. Several courts, when deciding new adequacy cases, have either dismissed them based on separation of powers grounds or have ruled against the plaintiffs on the merits following a trial.  Since 2005, courts in Oklahoma, Kentucky, and Oregon have dismissed such cases on motion, holding that control over school funding is the sole province of the elected branches of government.   In Texas, Missouri, South Dakota and Arizona, the courts denied motions to dismiss, but after further hearings, rejected plaintiffs’ claims regarding inadequate funding of the schools.  The one notable exception to this pattern is an October 2009 decision by the Colorado Supreme Court overturning a lower court decision which had dismissed an adequacy case on separation of powers grounds.  The higher court reversed and sent the case back for a trial on the facts.

In addition, several other courts have ended their involvement in cases initially decided in favor of the plaintiffs.  Some of these courts had retained jurisdiction for years or even decades over the “remedial” phase of the case.   The most famous of these was the Abbott case in New Jersey in which the courts have issued over a dozen funding orders since the commencement of the case almost 40 years ago.  In 2009, the New Jersey Supreme Court held that the state education funding scheme was constitutional and denied further relief to plaintiffs.   Additional long-running “adequacy” cases in Wyoming and Massachusetts have also been dismissed in recent years.

Even in those few cases that have not completely rejected plaintiffs’ claims, the relief granted has been minimal, with no significant funding mandates, unlike the pre-2005 court orders.  For example, in Alaska, the court ruled that state funding was adequate, but ordered the state to improve its monitoring of failing districts.  In South Carolina, the court rejected claims of inadequacy in the State’s K-12 system of education, faulting only its pre-K programs.

While some have tried to define these court decisions as successes, in reality, the outcomes have been extremely disappointing for advocates of increased judicial intervention.  Unlike the period from 1989 to 2005, when plaintiffs won almost every “adequacy” case that survived a motion to dismiss and went to trial, since 2005 they have not had a meaningful success in court.

The reasons for this abrupt change in the courts’ attitude are not clear, but the failure of judicial funding mandates to improve student achievement has likely been an important factor.  While the state courts have not expressly stated as much, the United States Supreme Court recently addressed the impact of funding mandates on student achievement.   In Horne v. Flores, the Court reversed a lower federal court order requiring the Arizona legislature to increase funding for programs directed at K-12 English language learners.  The Court found that the “weight of the research” indicated that structural, curricular and accountability-based reforms, “much more than court-imposed funding mandates, lead to improved educational opportunities.”  In reaching this conclusion, the Supreme Court relied upon the experience in state court “adequacy” cases, citing Schoolhouses, Courthouses, and Statehouses:  Solving the Funding-Achievement Puzzle in America’s Public Schools, a recently released book by the authors of this post.  In our book, we analyze student performance in the four states that have had the most dramatic increases in funding as a result of court orders – Kentucky, Massachusetts, New Jersey and Wyoming.   As set forth in detail in the book, Kentucky (the first of the “adequacy” rulings), New Jersey (with almost four decades of court involvement in school funding), and Wyoming (where the courts instructed the state to fund a “visionary and unsurpassed” education for its students) have each seen their school spending levels blossom under court order.  Notwithstanding these dramatic spending increases, we found that student performance has languished.  The unmistakable picture in each of these states is that during a decade or more of court funding mandates, student performance, as measured by the National Assessment of Educational Progress (commonly referred to as the “Nation’s report card”), has not measurably improved relative to other states that did not have anywhere near the same influx of new school money.

The only state where performance has significantly improved while under court order has been Massachusetts.  But here the story is more complicated, because Massachusetts combined strong non-financial remedial measures – strong standards, enhanced assessments, and strict accountability measures – with increased funding.   Importantly, although accomplished after the entry of a court order, these deeper policy changes were not specifically ordered by the court, but instead represented creative reform efforts by the political branches.

The Massachusetts result reinforces our analysis:  significant structural changes that focus incentives on improved achievement are more important than just providing additional funding.  However, most courts that have historically entered into educational policy areas other than funding have discouraged real structural changes, focusing instead on the continuation of past policies, such as class size reduction or across-the-board salary increases for teachers, which carry with them increased funding.

In conclusion, the adequacy decisions of the last five years must be taken as strong evidence that courts no longer suppose that ordering increases in school funding leads to significant gains in student performance.  This is surely reinforced by the 2009 Flores decision.

The CREDO study assessed the performance of charter schools compared to traditional public schools across 15 states and the District of Columbia.  It used an innovative matching technique to create control students in the traditional schools who were similar to those who chose charters.  The results from this study showed a number of charters (17%) doing significantly better (at the 95% level) than the traditional public schools that fed the charters, but there was an even larger group of charters (37%) doing significantly worse in terms of reading and math.  The remainder did not do significantly better or worse.  These results were greeted with mixed emotions.  The majority of researchers and policy makers were not overly surprised.  They saw that there were success stories but that further work would need to be done to ensure that more of the good charters flourished and fewer of the bad charters remained (just as the case with traditional public schools).  [Full disclosure: Macke Raymond, the lead author on the CREDO study, is my wife, so I know more about these studies than the random reader].

The study of New York City charters offered a different conclusion.  The analysis by Hoxby, Murarka, and Kang (HMK) employed a different methodology.  With sufficient popularity among parents so as to attract excess demand for enrollment, NYC charters are required to use a lottery to decide who is chosen to enroll (with exceptions for siblings and some other circumstances).  HMK traced students who were “lotteried out” into the traditional public schools and compared their subsequent performance to those who had entered the charter schools.  They found that the majority of students (86%) attended schools that had a positive effect (although it is not reported what this means for the number of schools or the statistical significance.)  Thus, in New York City the charter experience appears notably more favorable than that for the rest of the nation.

Some people believe that the results for these two studies should be the same.  If not, one of the studies must be wrong.  The media were further confused by a memo that Caroline Hoxby released that suggested an error in the statistical estimation used by CREDO and that this error would significantly bias downward the impact of charter schools.  This memo, by implication, suggested that the rest of the country might actually look like the NYC results.  The CREDO response to this points out, however, that the Hoxby memo is built upon an incorrect statement of the estimation approach by CREDO and an incorrect derivation of the statistical results – thus leaving the difference in results intact.  (Both sides of the exchange can be found here.)

In reality, expecting the results of the two studies to be the same is not the right way to look at them.  The studies ask very different questions, and there is no reason to expect them to provide the same results.  The CREDO study asks how well a typical charter school student across the sixteen separate state policy environments does compared to the counterfactual of attending a traditional public school.  The HMK study investigates how well charter school students do when attending schools popular enough with parents to be oversubscribed compared to attending a traditional NYC  public school.  Thus, the NYC study can be thought of as proof that the best charter schools, as judged by parents, can dramatically outperform the alternative traditional school.  That is important information, but it is impossible to know how to generalize it to other environments with different state laws, different union contracts, different district governance, different financing arrangements, and the like.   Just on the surface, nobody would think that it was possible to generalize from NYC (one million students) to LA (700,000 students), let alone to Kansas City (20,000 students).

Understanding the factors that make NYC charter schools perform so well relative to their traditional schools is an extraordinarily important research and policy question.    It should clearly be at the top of the research agenda.  Indeed, those charter school advocates who believe that there are important differences in state laws or that there is something special about KIPP schools already know that differences exist, even if the details are not well-understood.

Unfortunately, this is also a subtle issue that the media and the policy community have trouble understanding.  To many in the media, both studies sound like they are estimating the effectiveness of charter schools or maybe even the impact of school choice – so shouldn’t the answer be the same?

While we have learned a lot from the new studies, we still remain in a situation with an unresolved key question about what policies, laws, and incentives lead some charters to flourish and others not.

To understand the importance of the Flores ruling, it is necessary to trace the involvement of courts in school funding.  In the early 1970s, the federal courts ordered a number of states to pay school desegregation costs, but these rulings were limited in number and had little overall effect on state systems for school funding.  At the same time, litigants attempted to bring “equity cases” in federal courts designed to eliminate spending variations among school districts due to heavy reliance on the local property tax.  However, this effort failed in 1973 as a result of the Supreme Court’s ruling in San Antonio v. Rodriquez that such claims did not have a basis under federal equal protection laws.

Litigants then shifted their efforts to the state courts where they were much more successful. To date,  some 45 states have had their  funding systems challenged under the education clauses of the state constitutions. With time, as more and more states moved to equalize funding, the “equity” suits morphed into “adequacy” suits, which changed the goal to increased funding.  As such, they necessarily impinge upon state legislatures’ traditional authority to determine the level of education appropriations.    These lawsuits enjoyed considerable success in the 1990s, when a number of state courts ordered legislatures to dramatically increase school appropriations.

The underlying argument is simple:  Students are not reaching desired achievement levels so it must reflect a lack of adequate funding.  Unfortunately, the courts never asked the more relevant question:  Is increased funding the solution to improving student achievement?

When we set out to answer this question in our recent book (Schoolhouses, Courthouses, and Statehouses), we found that court orders for substantially increased school funding seldom resulted in improvement in student performance.   This was the case in Kentucky, New Jersey, and Wyoming, where billions of dollars of increased funding did not significantly improve student achievement relative to that in other states.  Only in Massachusetts, where more fundamental changes in standards, accountability, and other aspects of school policy were incorporated with increased appropriations, did students tend to do significantly better following court intervention.

The Supreme Court took notice of this analysis and applied these hard-earned lessons in Flores. Beginning with a 1992 decision, the Federal court in Arizona had ruled that the State had not taken “appropriate action to overcome language barriers that impede equal participation by its students in its instructional programs” as required under the Equal Educational Opportunities Act of 1974.  It then directed the state legislature to appropriate additional monies for ELL students, first in schools in Nogales and subsequently in all schools in Arizona.  Over the next several years, the state not only increased ELL funding but also significantly changed its ELL programs.  After achievement of ELL students improved, the State argued that the original circumstances had changed and that the State should be released from judicial supervision.  In a series of actions, the lower courts held that, even though there was improvement in student outcomes, the central issue remained whether the legislature should enact even greater increases in funding.

The Supreme Court reversed in Horne v. Flores.  It noted that the lower court decision “withdraws the authority of state and local officials to fund and implement ELL programs that best suit Nogales’ needs, and measures effective programming solely in terms of adequate incremental funding.”  After reviewing the programmatic changes made for ELL students in Nogales, the Court reached the conclusion that “the weight of research suggests these types of local reforms, much more than court imposed funding mandates, lead to improved educational opportunities.”

The Supreme Court’s decision forcefully makes a set of extraordinarily important points.  First, educational opportunity is better defined in terms of student outcomes.   Second, pedagogical and administrative reforms are often more important than court-ordered funding mandates, which it found had not been very successful.  And, third, such judicial funding decisions inappropriately intrude upon the power of states and localities to set their own public priorities and to make appropriate decisions.

While U.S. Supreme Court decisions on a federal statute do not necessarily bind state courts,   its well-argued position should be influential.  State courts, previously intervening significantly into state educational policy making, pushed up spending without commensurate results in student performance. Most recently, increasing numbers of state courts have themselves become skeptical about the appropriateness of intervening into school policy making and setting of appropriations.  And, today there are few state cases currently active, up from a large number always active over the past two decades.  The Flores decision almost certainly will reinforce and strengthen this desirable trend.

For more on this topic by Eric Hanushek, please see “Many Schools Are Still Inadequate: Now What?” in the Fall 2009 issue of Education Next.

According to the latest international math and science assessment conducted by the Organisation of Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) (see Figure 1), the United States remains a long distance from that target. Rather than worrying about the consequences, some have begun to question what all the fuss was about. Education researcher Gerald Bracey, for example, has argued that no one has “provided any data on the relationship between the economy’s health and the performance of schools. Our long economic boom suggests there isn’t one—or that our schools are better than the critics claim.”

Truth be told, the Bracey critique is not entirely misplaced. Most commentators rely more on the commonsense understanding that countries must have good schools to succeed economically rather than presenting conclusive empirical evidence that connects what students learn in school to what subsequently happens in a nation’s economy. Even economists, the people who think the most systematically about the way in which “human capital” affects a nation’s economic future, have skirted the heart of the question by looking only at “school attainment,” namely the average number of years students remain in school.

Using average years of schooling as an indicator of a country’s human capital has at least two major drawbacks. First and foremost, the approach assumes that students in diverse school systems around the world receive the same educational benefits from a year of schooling. A year of schooling in Papua New Guinea and a year of schooling in Japan are treated as equally productive. Second, this measure does not account for learning that takes place outside the classroom—within families, among peers, or via the Internet, for example.

A more direct measure of a country’s human capital is the performance of students on tests in math and science, something that might be called the average level of “cognitive skills” among those entering a country’s work force. At one time, internationally comparable information on student performance was not available for a sufficient number of countries over a long enough period of time to allow for systematic study, which is why economists relied upon the less informative measures of school attainment. Now that test-score data for many countries over an extended period of time are readily available, it is possible to supplement measures of educational attainment with these more direct measures of cognitive skills.

In a series of studies conducted over several years, the four of us have explored the role of both school attainment and cognitive skills in economic growth. Beginning in the mid-1960s, international agencies started conducting tests of students’ performance in mathematics and science at various grade levels. We used performance on 12 of these standardized tests as rough measures of the average level of cognitive skill in a given country. With this information, we could assess how human capital relates to differences in economic growth for 50 countries from 1960 to 2000, more countries over a longer period of time than any previous study. We were also able to pay close attention to institutional factors that influence economic growth, such as openness of the economy and protection of property rights.

What we discovered gives credence to the concerns expressed in A Nation at Risk. The level of cognitive skills of a nation’s students has a large effect on its subsequent economic growth rate. Increasing the average number of years of schooling attained by the labor force boosts the economy only when increased levels of school attainment also boost cognitive skills. In other words, it is not enough simply to spend more time in school; something has to be learned there.

We also discovered that the size of the impact of cognitive skills depends on whether a nation’s economy is open to outside trade and other external influences. The more open the economy, the more important it is that a country’s students are acquiring high levels of cognitive skills. As the world becomes increasingly interdependent or “flat,” to use New York Times columnist Thomas Friedman’s familiar terminology, enhancing human capital will become increasingly critical. As the world continues to change, the United States can ill afford to rest easily on its past accomplishments.

Measuring Cognitive Skills

Reaching these conclusions required a multistep analysis. The first step was to use the 12 PISA and other international math and science assessments, dating back to 1964, to construct an index of cognitive skill levels for a large sample of countries at various points in time. Because the number of countries participating in the 12 test administrations changed from one administration to the next, and because testing agencies have made no attempt to link their results to one another, we needed to develop comparable scores for each test. This required a norm against which each test could be calibrated. Fortunately, we could construct that norm by using information from tests in the United States, the country that has had the earliest, most sophisticated, and most comprehensive system of testing. The United States has participated in all of the international tests since 1964, and it has also maintained a separate longitudinal testing system of its own, the National Assessment of Educational Progress (NAEP). With that information in hand, it was possible to calibrate scores on each of the separate international tests to one another via the connection of those tests to the NAEP. To obtain further precision, we used the variation in scores across a subset of the more-advanced developed countries to obtain an estimate of the spread in scores across countries. By following these two steps, we were able to aggregate all available scores for each country into measures of average cognitive skill levels for each country.

The 50 countries for which we were able to develop a comparable measure of cognitive skill levels include the 30 democracies that have market economies and have been accepted as members of the OECD, most of which are at a relatively high level of economic development. The other 20 countries are at lower levels of economic development. In Figure 2, you can identify top performers like Finland and Japan, average performers such as the United States and Germany, and low performers that include Albania, the Philippines, and South Africa.

Impact on Economic Growth

We wanted to use this new information to compare the economic benefits of higher levels of just school attainment with the benefits of higher levels of cognitive skills. We therefore took measures of average educational attainment and average cognitive skill levels for as many countries as possible and examined their relationship to the average annual growth rate in the country’s gross domestic product (GDP) per capita from 1960 through 2000.

First, we looked just at the impact of average school attainment on the economic growth rate. (An adjustment was made for the initial level of GDP because it is “easier” to grow if you are starting out at a lower level; that is, it is easier to copy more productive technologies than to initiate progress on your own.) When we performed this analysis, we found, as other economists before us, that when the average number of years of schooling in a country was higher, the economy grew at a higher annual rate over subsequent decades. Specifically, we found that, across the 50 countries, each additional year of average schooling in a country increased the average 40-year growth rate in GDP by about 0.37 percentage points.

That may not seem like much, but consider the fact that since World War II, the world economic growth rate has been around 2 to 3 percent of GDP annually. Lifting it by 0.37 percentage points is a boost to annual growth rates of more than 10 percent of what would otherwise have occurred, a significant amount.

But the impact of improved cognitive skills, as measured by the performance of students on math and science tests, is considerably larger. When we performed the analysis again, this time also including the average test-score performance of a country in our model, we found that countries with higher test scores experienced far higher growth rates. If one country’s test-score performance was 0.5 standard deviations higher than another country during the 1960s—a little less than the current difference in the scores between such top-performing countries as Finland and Hong Kong and the United States—the first country’s growth rate was, on average, one full percentage point higher annually over the following 40-year period than the second country’s growth rate. Further, once the impact of higher levels of cognitive skills are taken into account, the significance for economic growth of school attainment, i.e., additional years of schooling, dwindles to nothing (see Figure 3). A country benefits from asking its students to remain in school for a longer period of time only if the students are learning something as a consequence.

Another indication of the importance of education quality to economic growth lies in our ability to explain global variation in GDP growth. When we tried to account for economic growth with information only about school attainment levels and the level of a country’s GDP in 1960, we were able to explain only one-quarter of the differences we saw among countries. But when we also included cognitive skills in our statistical models of economic growth, we were able to attribute nearly three-quarters of the differences among countries to these three factors. In other words, higher levels of cognitive skill appear to play a major role in explaining international differences in economic growth.

Of course, the initial level of economic development, schooling attainment, and cognitive skills are not the only factors that affect economic growth. Could it be that some other factor we have overlooked is responsible for the close connection between test scores and economic growth?

Other economic research has identified two additional factors that affect a country’s economic growth rate: the security of its property rights and its openness to international trade. When those two factors are taken into account, the positive effect of cognitive skills on annual economic growth becomes somewhat smaller, but is still 0.63 percentage points per half of a standard deviation of test scores. This is the best available estimate of the size of the impact of cognitive skills on economic growth.

Other commonly discussed determinants of economic growth are fertility and geography. However, when we took into account the total fertility rate and common geographical proxies, such as latitude or the fraction of the land area of a country that is located in the tropics, neither of these additional variables was significantly associated with economic growth. Once again, the strong effect of cognitive skills remained clear.

We performed a variety of additional tests to assess the validity of these basic results. For example, we estimated the relationships over shorter periods of time, used different subsets of international tests, and compared smaller groups of the 50 countries.

One of our tests was particularly interesting. We thought it possible that the effect of cognitive skills could be the result of the presence in our sample of East Asian countries, most of which have both high levels of cognitive skill and rapidly growing economies. To see whether the inclusion of those countries in our study influenced our results, we excluded them from one of our models. The impact of cognitive skill remained very powerful, albeit diminished.

We also looked at cognitive skills as measured in the 1960s through the mid-1980s to see what their impact was on growth between 1980 and 2000, ensuring that the cognitive skills themselves were not caused by the economic growth. Again, our basic findings remained intact. Finally, we looked at whether a country’s estimated cognitive skills affected the earnings of immigrants working in the United States. Higher home country cognitive skills translated into higher earnings if the immigrants were educated in their homeland but not if educated in the United States.

Our commonsense understanding of the importance of good schools can thus be documented quite precisely. A highly skilled work force can raise economic growth by about two-thirds of a percentage point every year.

More Rocket Scientists or Basic Skills for All?

To gain additional insight into the relationship between cognitive skills and economic growth, we examined the separate impact of improvements at different levels of a nation’s distribution of skills. Loosely speaking, is it a few “rocket scientists” at the very top of the distribution who spur economic growth, or is it “education for all” that is needed?

To address this question, we measured the share of students in each country who reach a threshold of basic competency in mathematics and science, as well as the share of students who perform at very high levels. To estimate the importance of basic competency, we identified the share of students performing at least at a very basic level, that is, no more than one standard deviation below the international average of all OECD countries. In the average OECD country in our study, 89 percent of the students achieved at least at this very basic level. The share of students with at least basic skills ranged widely among countries, from as low as 18 percent in Peru to 97 percent in the Netherlands and Japan. To show a country’s ability to develop a large cadre of high-performing students, we identified the share of students performing at very high levels—at or above one standard deviation over the OECD average. On average across all countries, 6 percent of students performed at that high level. Once again, countries varied enormously in this respect, the percentage ranging from as low as 0.1 percent in Colombia and Morocco to 18 percent in Singapore and Korea and 22 percent in Taiwan.

Which is more important for growth—having a substantial cadre of high performers or bringing everyone up to a basic level of performance? The answer, it seems, is not one or the other but both! When we estimated the importance of each within the same model, we found each of them to be separately important to economic growth. That is, both the performance of countries in ensuring that almost all students achieve at basic levels and their performance in producing high-achieving students seem to matter.

The reasons that a substantial cadre of highly skilled citizens and near-universal basic skills matter are not difficult to imagine. Even if a country is simply making use of new technologies developed elsewhere, as is often the case in developing parts of the world, the more workers that have at least basic skills, the easier it will be for them to make use of those new technologies. Some workers need a high level of skill so they can help adapt the new technologies to their countries’ particular situation. In countries on the technological frontier, substantial numbers of scientists, engineers, and other innovators are obviously needed. But so is a labor force that has the basic skills needed to survive in a technologically driven economy.

But even if the results seem intuitively correct, they should be taken as suggestive rather than definitive, because the two measures of cognitive skills are closely related to one another and our models have difficulty in separating out the precise impact of each individually. Most countries that have a high percentage of students with very high cognitive skills also are ones in which basic skills are near universal. Conversely, countries with a substantial percentage of students lacking even basic skills tend to be those that have only a small percentage of highly skilled students. Still, that pattern is not a perfect one, so we are able, at least tentatively, to identify the impact of each type of human capital, and we are quite confident that we can recommend that countries both concentrate resources on their “best and brightest” and make sure that “no child is left behind.”

The Impact of Becoming a World Leader

What would it mean for economic growth, then, if a country like the United States, currently performing somewhat below the average of OECD countries, managed to increase its performance by 50 points (or 0.5 standard deviations) so that it would score alongside the world leaders? (On average on the PISA 2006 math and science exams, countries such as Canada and Korea scored about 50 points higher than the U.S., Hong Kong and Taiwan about 60 points higher, and Finland as many as 74 points higher.) That increase of 50 test points is exactly what George H. W. Bush and the nation’s governors together promised in 1989 the United States would achieve by the year 2000.

Unfortunately, no such gains were realized. But had the promise been fulfilled by 2000, our results suggest that GDP would by 2015 be 4.5 percent greater than in the absence of any such gains (see Figure 4). That 4.5 percent increment in GDP is equal to the total the U.S. currently spends on K–12 education. In other words, had that money effectively raised cognitive skills by the 50 test points that would have brought the United States close to world leadership, the economic returns to the country would probably have been enough to cover the entire cost of education in 2015 and after.

Figure 4 shows that the benefits of successful reform grow even more vivid when we look farther out. Over 75 years, even a reform that takes effect in 20 years (instead of the governors’ 10 years) yields a real GDP that is 36 percent higher than it would be if there was no change in the level of cognitive skills.

None of this is meant to suggest that schooling is the only factor contributing to a society’s cognitive skill development. Family, individual ability, and health combine with school quality to determine a student’s level of achievement. Yet there is every reason to believe that the single best route to higher levels of cognitive skill is strengthening a country’s education system. After all, most people think that is the system’s primary purpose.

An American Exception?

The United States has never done well on international assessments of student achievement. Instead, its level of cognitive skills is only about average among the developed countries. Yet the country’s GDP growth rate has been higher than average over the past century. If cognitive skills are so important to economic growth, how can we explain the puzzling case of the U.S.?

Part of the answer is that the United States may be resting on its historic record as the world’s leader in educational attainment. In addition, the United States has other advantages, some of which are entirely separate and apart from the quality of its schooling. The U.S. maintains generally freer labor and product markets than most countries in the world. There is less government regulation of firms, and trade unions are less powerful than in many other countries. Put more broadly, the U.S. has generally less intrusion of government in the operation of the economy, including lower tax rates and minimal government production through nationalized industries. Taken together, these characteristics of the U.S. economy encourage investment, permit the rapid development of new products and activities by firms, and allow U.S. workers to adjust to new opportunities.

Those economic institutions seem to matter on their own and in conjunction with cognitive skills. Our analyses suggest that the value of a high-quality education system is substantially diminished in closed economies. We estimate that the effect of a one-standard-deviation improvement in cognitive skills on annual economic growth is 0.9 percentage points per year in closed economies, identified by heavy restrictions on international trade, but 2.5 percentage points in open economies. It may be that rich human capital combines with a laissez-faire economy to foster robust economic growth.

It is also the case that, over the 20th century, the expansion of the U.S. education system outpaced the rest of the world. The U.S. pushed to open secondary schools to all citizens. Higher education expanded with the development of land grant universities, the GI Bill, and direct grants and loans to students. The extraordinary U.S. higher-education system is a powerful engine of technological progress and economic growth in the U.S. not accounted for in our analysis. By most evaluations, U.S. colleges and universities rank at the very top in the world.

Although the strengths of the U.S. economy and its higher-education system offer some hope for the future, the situation at the K–12 level should spark concerns about the long-term outlook for the U.S. economy, which could eventually have an impact on the higher-education system as well. The U.S. higher-education system may also be challenged by improvements in higher education across the world. Other countries are doing more to secure property rights and open their economies, which will enable them to make better use of their human capital. Most obviously, the historic advantage of the U.S. in school attainment has come to an end, as half of the OECD countries now exceed the U.S. in the average number of years of education their citizens receive. Those trends could easily accelerate in the coming decades.

Not Just a Matter of Money

Our evidence of a clear, strong relationship between cognitive skills and economic growth should encourage continued reform efforts. Improvements in mathematics performance called for by No Child Left Behind would matter, contrary to what critics sometimes suggest. Yet reformers should bear in mind that money alone will not yield the necessary improvements. Many expensive attempts around the world to improve schooling have failed to yield actual improvements in student achievement.

Economic growth flows only from reforms that bring actual improvements in cognitive skills. Identifying what works and how to implement it on a society-wide scale remains a challenge, not only for the U.S. but also for many nations across the globe. But, if we are to remain economically competitive, we need to solve the puzzle of our schools and meet the governors’ challenge. We should not, simply because we have failed to meet them in the past, decide that the goals were not legitimate or important.

Eric A. Hanushek is a senior fellow at the Hoover Institution of Stanford University. Dean T. Jamison is professor of health economics in the School of Medicine at the University of California, San Francisco. Eliot A. Jamison is an investment professional at Babcock & Brown. Ludger Woessmann is professor of economics at the University of Munich and heads the Department of Human Capital and Innovation of the Ifo Institute for Economic Research. The opinions expressed in this article are those of the authors and do not necessarily reflect those of their employers.

Checked by Eric A. Hanushek

Lawsuits aimed at compelling legislatures to increase school funding have been filed in some 42 states. Courts have found for the plaintiffs in more than half of the cases on the grounds that schools are not “adequately” funded (see Figure 1). These decisions have, in effect, changed the way education appropriations are made, moving decisionmaking from legislatures to the courts. Instead of flowing from the political process, determinations of adequate appropriations come from judges who are informed by paid consultants. Recently, adequacy plaintiffs have suffered some serious setbacks (see legal beat). Undaunted, they soldier on.

In the state of Washington, adequacy plaintiffs filed a new lawsuit in early 2007 that is expected to rely heavily on a report prepared at the request of a gubernatorial-appointed commission, Washington Learns. This report, “An Evidence-Based Approach to School Finance Adequacy in Washington,” claims to present scientific evidence of exactly what needs to be done to bring every child to proficiency as defined under state and federal law. The advance, if true, would go far beyond this specific court case and could revolutionize American education. For if, indeed, we now know how to create an effective educational system, and only the funds are lacking, then the country’s education problems can be solved.

The analysts who purport to have assembled this knowledge are led by two professors, Lawrence Picus of the University of Southern California and Allan Odden of the University of Wisconsin. The two formed a consulting group known as Picus and Associates and have become increasingly popular among groups seeking to expand school spending, be they plaintiffs in funding lawsuits, teachers unions, or state departments of education. The Washington Learns commission asked Picus and Associates to recommend policy changes that will place the state’s education system on a sound footing. Specifically, Picus and Odden answer the question, “What are the high-impact education programs and strategies that will allow every school to provide each Washington student with the opportunity to learn at or above proficiency on  state standards as measured by the Washington Assessment of Student Learning, with proficiency standards calibrated over time to those of NAEP [National Assessment of Educational Progress], or even the performance of students in other countries?”

Even if only the state of Washington were getting precise, scientific answers to such critical questions, the work of the Picus-Odden team would command the attention of national policymakers. But the consulting group has already established a national reputation for its ability to ascertain, scientifically, what needs to be done in education—and precisely how much it costs to do it—through prior studies along much the same lines prepared for policymakers in Kentucky, Arkansas, Arizona, and Wyoming.

Of course, the evidence base does not change very rapidly, as is evident from the various reports, which were carried out between 2003 and 2006. The 2006 study conducted for Washington Learns has an extensive bibliography, some 260 entries. But, since the production of the cost study for Kentucky in 2003, only 30 new references were added (including the obligatory reference to Thomas Friedman’s The World Is Flat). So similar are the studies that at times it seems the copy function of the Microsoft word processor deserves to be listed among the authors.

The ease with which one report can build on another does not seem to translate into efficiencies in the consulting group’s operations, at least as reflected in the fees charged. According to available records, the Kentucky study, conducted in 2003, was executed for $349,000. Arkansas’s original study, conducted the same year, cost about the same initially but rose to over twice that amount ($800,000) when the authors accepted a commission to ascertain whether districts used their extra money in a way consistent with the consultants’ evidence-based policies. Wyoming, a small but rich state, was asked to pay $1,260,000 in 2005 for a calibration of its finance formula along evidence-based lines and a subsequent implementation study. Washington, in 2006, managed to squeeze the price back down to the total Arkansas figure, although Washington could get only the original evidence-based analysis without the follow-up.

Even the Wyoming deal is a bargain, however, if the study can answer the question posed by the Washington Learns commission. After all, we spend some $500 billion nationally on K–12 education, and even small improvements applied to the nation’s schools could quickly cover the study costs.

The Picus-Odden Miracle

The frequency with which education policy initiatives of the past, though based on high hopes, have yielded disappointing results when implemented in the field has led to rather low expectations. As a general rule, in education discussions a policy is considered successful if an evaluation has shown it to have a statistically significant positive effect on student outcomes. Translated, there must be a high degree of certainty that positive results were not simply the result of chance. But just finding that some policy is likely to improve student outcomes does not mean that the improvement will reach the high levels sought by Washington Learns, or by others with similar views about what students should know. The research would have to provide evidence about the magnitude of improvements in achievement that can be expected, and these improvements would have to be large.

Such evidence is precisely what Picus and Odden purport to provide for their fees. They have combed the research evidence to provide rather precise, and remarkable, predictions about the achievement effects of programs whose power has apparently escaped the attention of almost all other researchers.

Picus and Odden convey the magnitude of achievement gains that can be expected from their evidence-based policies through a unit of measurement known as effect size. Effect size is the change in standard deviations of achievement that can be expected, according to the research, from the introduction of a given policy. In itself, that step is perfectly acceptable, as the unit is widely used in education research.

Discussion of effect sizes and standard deviations is something most policymakers, even when introduced to the concepts in their undergraduate statistics course, would rather avoid. But some heuristics will help to understand the essence of effect sizes and make clear the import of the Picus and Odden evidence. The National Assessment of Educational Progress (NAEP) measures achievement in different grades and attempts to put it on a common scale. One full standard deviation (an effect size of 1.0) is roughly equal to the average difference in test score performance between a 4th grader and an 8th grader. In other words, it is a big effect, as the typical 8th grader has learned quite a bit since 4th grade.

By this perspective, any education strategy that in a single year can raise average achievement of a large aggregate of students by one full standard deviation must be taken very seriously. Pursued systematically, it could eliminate the persistent ethnic test-score gap (which is about one full standard deviation) or could vault the math and science performance of U.S. students beyond counterparts in Korea, Singapore, and Japan (who are about one-half of a standard deviation ahead now).

Picus and Odden identify strategies they claim can do that, and much more. They provide “scientific evidence” to support the claim that a specific set of policies can shift average student performance upward by three to six standard deviations, an extraordinary gain. The policies they identify include providing a year of full-day kindergarten, reducing class size to 15 students through grade 3, using multi-age classrooms, hiring classroom coaches, employing one-to-one tutoring for disadvantaged students, getting half of the students eligible for free and reduced-price lunch to attend summer school, embedding technology within the classroom, creating a gifted and talented program for the top 5 percent of all students, and accelerating instruction for the 2 percent of students capable of benefiting from it (see Figure 2). The range in claimed impact reflects the fact that they sometimes admit to uncertainty about the exact effect size from a specific program.

Most Americans would be extraordinarily satisfied with average gains of one full standard deviation for a school or district. Picus and Odden claim to be able to do that three or possibly even six times over for all students in Washington. After their policies are fully implemented in Washington, Albert Einstein, were he not participating in these programs, would find himself achieving at or below the state average.

This can all happen within one year of application of these policies, the consultants say. But they would not give these programs just a single year. They would apply them, where appropriate, across all years of schooling. (Full-day kindergarten, for example, happens just once for each student.) If one then assumes a cumulative impact from giving students not just a single application but continuing treatment through grade 12, the gains reach astronomical proportions, somewhere in the range of 23 to 57 standard deviations.

The Truth behind the Numbers

This, of course, is the stuff of science fiction novels, not research-based school policies. How does a well-funded study, conducted by scholars of national reputation, reach such startling conclusions? The procedure is roughly as follows:

1) Find a study, preferably one that has some surface credibility, that shows that a particular intervention had a certain effect on a particular group of students.

2) Ignore all the studies of that intervention that show a smaller effect or no effect at all.

3) Interpret the study as identifying a true causal relationship, not just a correlation or association.

4) Finally, assume that the conditions that produced the very large effect can be perfectly replicated throughout the state of Washington.

Take full-day kindergarten, for example, which Picus-Odden estimate to have by itself an impact of 0.77 standard deviations on student achievement for advantaged and disadvantaged students alike. (In NAEP terms, this by itself would be equivalent to three full years of later schooling.) Picus and Odden cite a 1997 meta-analysis by John Fusaro that shows such an impact. But they disregard Fusaro’s own strong warning: “A seductive conclusion from these results is that attendance at full-day kindergartens causes students to achieve at a higher level than attendance at half-day kindergartens. It is imperative, however, that we strenuously resist succumbing to such a seduction.” Meanwhile, Picus and Odden ignore a large body of literature that shows little impact on advantaged students and smaller impacts on disadvantaged ones, to say nothing of the empirical reality that the 56 percent of students currently attending schools that have full-day kindergarten do not surpass the remaining 44 percent attending schools without full-day kindergarten by anything like a 0.77 margin.  Note, for example, that black students and disadvantaged students are currently more likely to attend schools with full-day kindergarten than more advantaged students.

Or take summer school, which Picus and Odden estimate would have an effect size of 0.45 standard deviations. This policy recommendation is apparently based on a single study in 2000 of the Voyager summer learning program, although they note that a major meta-analysis suggests widely varying effect sizes from the evaluations of different studies. Note also that in Odden’s peer review in 2004 of William Driscoll’s and Howard Fleeter’s Ohio study of the costs of bringing all students to proficiency in math and reading in order to comply with NCLB, he castigates the study’s authors, who called for expanded summer school, because they “reference no research to support this assertion, when in fact most research shows that summer school as typically administered has little if any impact on learning.”

These patterns are repeated when one goes to the other “evidence-based” recommendations of Picus and Odden, including class size reduction and professional development. Their estimate of the benefits of professional development comes directly from the professional association representing those who supply professional development. And so on. There is little reason to believe that the effect sizes identified in their work indicate what can be expected from implementing any policy on a broad scale.

The approach of Picus and Odden to policies is simple: if a program shows a large positive effect in one study, it should immediately be implemented across the state. Indeed, they assert in public hearings that adopting anything less than the complete set of recommended programs would constitute an inadequate program, and that they would testify to the inadequacy in court.

Are Costs Important?

The primary purpose of reviewing the evidence on programs is to establish the cost of providing a new and improved (adequate) education. The various programs suggested by Picus and Odden have very different price tags associated with them. They make it hard to tell from their report what prices might go with each of the programs, because they bury the costs within the staffing of each prototypical school. It is, nonetheless, relatively easy to obtain reasonable cost estimates for each program.

The basic building blocks for calculating the cost per pupil of the various policies Picus and Odden propose are the approximate average expenditure of $7,800 per pupil and average teacher compensation (salary plus benefits) of $60,000 for the state of Washington. We can first translate these into the cost per recipient for each program based on resource demands and then take into account the proportion of all students who receive the program. The results show wide variations in costs. For example, full-day kindergarten would increase average spending in the state by $154 to $300 per student, while the K–3 class size reduction would increase average spending by $410 to $800 per student. Some programs have no obvious costs. For example, multi-age classrooms might reasonably be taken as free. Similarly, changes in curriculum do not in general have significant added costs (past, say, an initial teacher-training period). Other programs, such as skipping grades, would actually save money, since students would spend 12 rather than 13 years in the system.

Once program costs are separated, one can immediately see the variation that exists and can make judgments about where money is better (more efficiently) spent. A simple cost calculation gives the improvements in student achievement (measured again in standard deviations) that could, by the Picus and Odden estimates of benefits, be expected for a $100 addition to spending per pupil from each of the separate programs. By their low-end estimates of benefits (which total to just three standard deviations), each $100 spent on classroom coaches would be expected to yield at least a 0.25 standard deviations gain in achievement, very similar to the expected gain for full-day kindergarten. Their class-size reduction proposal would yield only one-sixth that gain, or 0.04 standard deviations, an effect very similar to that for one-to-one tutoring.

Using the upper range of their effect size estimates, $100 spent on classroom coaches would yield a gain of over one-half standard deviations in student achievement, and one-to-one tutoring would yield a one-quarter standard deviations improvement. According to their estimates, some of their favored programs (such as classroom coaches) are more than 10 times as cost efficient as others, such as class size reduction for K–3.

Picus and Odden contend that all programs, regardless of cost, must be simultaneously undertaken. But it is clear that the programs they identify have very different expected returns on spending. Their method of distributing costs through their prototypical schools provides no information on the relative efficiency of investing in the various components. Nor does it say anything about the costs of improving outcomes if done efficiently. Unless there are unlimited funds to spend on educational programs, it would not make sense to put the money into all the programs without regard to cost.

What Are States Paying For?

Cost estimates are an important component in the politics of court and legislative deliberations on schools. The adequacy debates are typically motivated by obvious and real shortfalls in the achievement of a state’s students, but a combination of naive concerned citizens and self-interested parties invariably pushes to translate these debates into a simple dollar figure. Such translation is salient for courts and legislatures and both simplifies and focuses the issue for the media.

What Picus and Odden provide in their reports is essentially a selective review of the published literature on program effects. Why do different states and organizations pay ever-increasing amounts to see this research review when Google would bring up the most recent version immediately and without expense? The answer is simple. Clients want a bottom-line statement about how much spending would provide an adequate education, and they want this cost estimate attached to their specific state. Few people care about the “studies” on which consultants base their reports, or even their validity, because nobody really expects schools to implement these specific programs if given extra funding. Clients simply want a requisite amount of scientific aura around the number that will become the rallying flag for political and legal actions.

Summing the added cost of the separate programs suggested by Picus and Odden, I estimate that the overall plan, if fully applied, would increase average spending in Washington by $1,760 to $2,760 per student, or 23 to 35 percent. This estimate of the increased spending necessary to achieve “adequacy” is very similar to the percentage increases they have recommended to other states, and numbers like these will presumably become part of the headlines surrounding the new court case.

But pity the poor states that actually implement the Picus and Odden plan. They are sure to be disappointed by the results, and most taxpayers (those who do not work for the schools) will be noticeably poorer.

Eric A. Hanushek is a senior fellow at the Hoover Institution, Stanford University, and a member of its Koret Task Force on K–12 Education.

What Do Test Scores in Texas Tell Us?

by Stephen P. Klein, Laura S. Hamilton, Daniel F. McCaffrey, and Brian M. Stecher

RAND Corporation, 2000.

Just two weeks before the presidential election, yet another team of RAND researchers released a short paper that seemingly contradicted Grissmer et al.’s celebration of Texas’s achievement gains on the NAEP. RAND II found only small NAEP achievement gains in Texas, similar to those nationwide and contrasting sharply with “soaring” scores on the Texas Assessment of Academic Skills (TAAS). These disparities, the authors suggested, point to potentially serious flaws in Texas’s state-run testing program.

The direct conflicts of RAND I and RAND II underscore the fact that RAND is a collection of franchisees. The parent company attempts to maintain some degree of quality control but ultimately is not able fully to adjudicate quality—particularly, one suspects, when the answers are fuzzy and when the sponsor pressures are high.

RAND II presents two separate analyses that, taken together, seem to undermine Texas students’ spectacular gains on the TAAS. First, Texas students showed substantially more improvement on the TAAS than they did on the NAEP during the 1990s. Second, in a sample of 20 schools that the authors had collected for other purposes, the expected negative relationship between a student’s TAAS score and his eligibility for the federal school lunch program, a common measure of disadvantage, didn’t arise on the TAAS. This latter finding led RAND II to conclude not just that the TAAS is a poor instrument but also that high-stakes testing leads to the artificial inflating of scores through “teaching to the test,” especially for disadvantaged students.

It should not be particularly surprising that student performance improved more dramatically on a test that was aligned with a particular state’s curriculum (the TAAS) than on a more generic test of subject matter (the NAEP). Thus, while the question of the TAAS test’s validity is an important one, the simple evidence presented in RAND II falls very short of yielding any solid answers.

Likewise, the fact that data on 20 schools show a peculiar relationship with any variable is unremarkable. After all, even if the authors attempted to draw a representative sample—which they did not—the idiosyncrasies of such a small sample would preclude any ability to generalize. Indeed, a simple plot or a formal statistical analysis of TAAS scores across all Texas schools reveals a clear, and expected, strong negative relationship between students’ scores and their eligibility for subsidized school lunches.

The point of clearest conflict with RAND I is the consideration of NAEP performance. RAND I—not as focused on the relationship between its statistics and presidential campaigns—considered all seven NAEP tests given between 1990 and 1996 and attempted to adjust for differences in the students’ backgrounds. The result was high marks for Texas’s performance improvements on the NAEP. RAND II, by contrast, ignored student background, placed more weight on a different subset of test results (including the 1998 results, which were not included in RAND I), used somewhat different approaches, and concluded that there was nothing special about performance in Texas.

What lessons might we take away from the RAND I vs. RAND II debate?

• Analyses of small amounts of imperfect data can yield widely different conclusions. Such analyses should be heavily discounted.

• Consideration of a study’s quality tends to get lost in the ensuing policy discussion. Neither RAND study holds up to a modicum of scrutiny.

• The desire for publicity apparently pushes some researchers to prepackage their own sound bites. The PR blitzes that accompanied both RAND I and RAND II undermined any public discussion of what turns out to be relatively impotent research designs.

• Journalists tend to judge a study’s quality—particularly a complicated statistical study—by its conclusions and by an undue emphasis on the study’s source rather than the strength of its analysis. RAND’s undeniable history of producing solid research doesn’t mean that every study under the RAND imprimatur deserves unquestioned repeating.

The result is a distorted and unhealthy policy discussion.

by David W. Grissmer, Ann Flanagan, Jennifer Kawata, and Stephanie Williamson

RAND Corporation, 2000.

In the summer of 2000, perfectly timed to shape the election debate over education reform, came a new RAND study that claimed to contradict the conventional research wisdom on the connection between school expenditures and class size on the one hand and student achievement on the other. “Our results certainly challenge the traditional view of public education as ‘unreformable,’” the study’s director, David Grissmer, said in an accompanying press release. “But the achievement of disadvantaged students is still substantially affected by inadequate resources. Stronger federal compensatory programs are required to address this inequity.” While academic studies usually retire to footnote-land, a well-orchestrated PR blitz pushed the RAND report to the front pages. It even earned prominent campaign mentions: Both presidential candidates commandeered the study’s findings to their own ends—Al Gore to support his proposal to lower class sizes, George W. Bush to trumpet Texas’s accountability system.

A trusted name like RAND lent instant credibility to the study’s results—so much credibility that the major newspapers reported the findings without even a question mark. This, combined with the lack of statistical expertise among journalists and the crushing deadlines under which they work, allowed RAND to sculpt the dissemination of its results with a carefully worded press release that pumped its most provocative yet methodologically flawed conclusions. “The education reforms of the 1980s and 1990s seem to be working,” the release began. It went on to highlight the report’s finding that “[d]ifferences in state scores for students with similar families can be explained, in part, by per-pupil expenditures and how these funds are allocated.” In particular, RAND reported that, other things being equal, National Assessment of Educational Progress (NAEP) scores in math are higher in states that have:

• higher per-pupil expenditures
• lower pupil-teacher ratios in the early grades
• higher percentages of teachers reporting that they have adequate resources
• more children in public prekindergarten programs
• lower teacher turnover

These highlights were asserted without qualification or doubt, without any mention of weaknesses in the data or the analysis. RAND was notably less charitable with results that accorded with past research findings. The press release at least mentioned the study’s finding that “having a higher percentage of teachers with master’s degrees and extensive teaching experience appears to have comparatively little effect on student achievement across states. Higher salaries also showed little effect.” But here the authors were sure to qualify their findings, carefully emphasizing that “salary differences may have more important achievement effects within states than between states.” The authors quickly rushed past these less popular findings to boldly propose specific policy interventions: “To raise achievement scores, the most efficient and effective use of education dollars is to target states with higher proportions of minority and disadvantaged students with funding for lower pupil-teacher ratios, more widespread prekindergarten efforts, and more adequate teaching resources.” In short, any reader of the news release—or the articles it generated—might have reasonably concluded that RAND, the highly respected think tank, had overturned years of research (including this author’s).

What research does the RAND study purport to contradict? Between 1960 and 1995, per-pupil spending in the United States (in constant 1996–97 dollars) grew dramatically, from $2,122 to $6,434, a threefold increase. This trend cannot be explained by the country’s increased commitment to disabled students, which at most accounts for just 20 percent of the increase. At the same time that costs were rising, the student-to-teacher ratio fell by about a third, from 26:1 to 17:1. Nevertheless, despite our greatly enhanced commitments to public education—and despite the fact that children are growing up in better-educated and smaller families than ever before—student performance during this period, as measured by NAEP test scores for high school seniors in math and reading, moved hardly a hair’s breadth. Complementing these overall trends are more than 400 studies that have searched for a connection between spending and achievement in particular schools, districts, and, occasionally, states. In general, these studies have been unable to detect any consistent, positive relationship between increased resources and student learning.

This is not to say that schools don’t matter. The best of these studies, so-called value-added studies that concentrate on the determinants of growth in achievement across individual classrooms, find that differences in teacher quality have a profound impact. But they also find that teacher quality is not closely related to school resources. The only studies that consistently find positive effects of resources are those that rely on student performance and school data averaged across all students and schools in a state.These aggregate studies, of which the RAND study is one, rely on limited data and are prone to serious statistical shortcomings, so they have been heavily discounted in the past. Undaunted, RAND’s researchers argue that their results should lead to a reinterpretation of three decades’ research.

However much they might protest, RAND’s researchers for the most part have only confirmed what has been known all along. In fact, the RAND study is startling in its conformity to conventional wisdom. RAND’s best model for estimating the impact of spending increases on student performance yields an estimate that an additional $1,000 per student—a $50 billion annual increase nationally—would yield a rise in performance of about two percentile points (just 0.05 standard deviations), a trivial impact (see Figure 1). Moreover, the RAND study repeats the finding that teachers’ salaries, experience, and whether or not they hold a master’s degree bear little or no relationship to student performance.

High Costs, Low Returns (Figure 1) Though RAND said resources were inadequate, it actually found that huge increases in spending would raise test scores by only a trivial amount. National cost of obtaining a 2 percentile increase in NAEP performance using RAND cost estimates*

*The hardcopy of Education Matters erroneously refers to Normal Curve Equivalents (NCE) instead of percentiles in the text and presents cost estimates for changing NCE scores by two points in Figure 1. NCEs are a transformed version of percentile scores that follow a normal distribution with a mean of 50 and a standard deviation of 20.

What about the study’s most celebrated finding, on the impact of class size? The study found that class size, as measured by a state’s average pupil-to-teacher ratio, has a minuscule impact on the performance of the average student. At best, the RAND study is just another in a long list of reports that have demonstrated the minimal impact of school resources on the typical student’s performance. RAND attempts to distance itself from the conventional research wisdom by declaring that “money, if spent appropriately, is productive.” But who would be surprised by such a tautology? Only if we are told exactly which expenditures are productive can the study give much guidance. But the RAND study’s data are too weak and its methodology too flawed to support the specific policy recommendations its authors make.

A Sow’s Ear

Be skeptical when a research analyst tells you he has fashioned a silk purse out of the proverbial sow’s ear. Consider the limitations of the data with which RAND was working. The study’s sample consisted of 44 independent observations—the states that voluntarily participated in one or more of seven NAEP tests that were administered from 1990 to 1996. Moreover, the number of states participating in any one test varied from 35 to 44. Tests were given in 8th grade math in 1990, 1992, and 1996; in 4th grade math in 1992 and 1996; and in 4th grade reading in 1992 and 1994. Although RAND attempted a variety of analytical methods, its general approach was to estimate the impact of family background and measures of school resources on average student performance on as many of these tests as were administered in a given state.

The NAEP tests themselves have certain advantages.They have been carefully designed, the same test is given in all states, and they allow for comparisons from one time period to the next. Schools have few incentives to score high on the NAEP, leaving little chance that much “cheating” or “teaching to the test” goes on. It is troublesome that, when asked, a sizable number of schools exercise their right to refuse to participate in NAEP testing. Despite this drawback, though, the NAEP remains one of the best available measures of average student performance in most states.

But RAND’s analysis of the NAEP scores is another matter. First of all, 44 observations is a very small sample, so drawing any strong, statistically valid conclusions is at best difficult, at worst misleading. Moreover, data collected at the state level are marvelously imprecise. These aggregate data ignore the enormous differences within a state—implicitly assuming that the past three decades of legal challenges to the inequitable distribution of resources among well-to-do and poor school districts are groundless. When all these differences are averaged away so that it is impossible to identify their importance, how can we possibly have high quality data that trump all previous research on the subject?

While the measure of student performance with which RAND was working was adequate, not much else was. RAND attempted to control for the family background of the students taking the test, but the only information on family background available to RAND was census figures on the average statewide education and income of school-age families in 1990. RAND attempted to adjust these data to the actual years the students took the tests by assuming that these factors change precisely with changes in the racial composition of test-takers. But, of course, one cannot assume that the education and income of students of different racial groups change at the same rate in all 44 states. And RAND did not have any information from individual students; throughout its analysis it refers to average results across the state. So, from the very beginning, RAND was forced to work with an imprecise measure of the characteristics of students who actually took the tests.

Similarly, RAND used statewide averages as its measure of school resources, an extremely imprecise indicator of the actual resources being spent on particular students who attend specific schools. RAND also relied on statewide averages of teachers’ impressions of whether their school supplies were adequate, statewide averages of prekindergarten attendance, and statewide averages of class size. These averages obviously mask wide disparities within a state.

The RAND researchers insist that their study is superior because they factored in the average school resources for all the years that students were in school, a measure they find superior to studies that look only at current resources being spent on a student. This may be a worthy research innovation, though the average school resources available to a student from one year to the next do not change dramatically—unless the student moves, something that happens with surprising frequency. In 1995, 6 percent of the school-age population lived in a different state than they had in 1990; another 2.5 percent had been living outside the United States in 1990. These percentages vary widely among regions of the country. In the mountain states, as many as 15 percent of students had lived elsewhere in 1990. None of this movement was taken into account by the RAND study.

Where Are the Reforms?

These weaknesses in the data were exacerbated once RAND tried to glean specific policy recommendations from its findings. For instance, RAND says that we should reduce class size in states with higher shares of disadvantaged students. RAND, however, doesn’t ever look at whether or not disadvantaged students are in large classes, because they have averaged across all students in the state.

RAND also says that students perform better when teachers think that their supplies are adequate. This finding is plausible. If teachers have adequate materials, one would expect them to be more effective. But it suffers from the chicken-and-egg problem: We can’t be sure whether high-performing students make teachers feel better about their supplies, or whether the supplies themselves have a causal impact. And, of course, this subjective question means wildly different things to teachers in different schools and states.

Much the same can be said for the finding that low teacher mobility leads to higher student performance. Do high levels of teacher mobility lower student performance, or does low performance increase the chances that teachers will move on? One simply cannot tell from the kind of data with which RAND was working.

The authors admit that it would be preferable to have data from the schools that students actually attended. But they claim that using statewide data allows them to consider the fact that the states, not local school boards, are the ultimate political entities responsible for public education within their boundaries. Only by looking at states as a whole can one incorporate the panoply of state policies that may influence school achievement, RAND says. To be sure, statewide analyses can provide accurate estimates of the impact of school resources—but only if the analyst includes within the statistical model all the factors that affect student performance and, in the standard linear regression model generally favored by RAND, if these factors have a constant, additive effect on student achievement. In other words, if the same amount of class size reduction has similar effects on those originally in very large classes and those originally in quite small classes, and if all other factors in the model work in the same constant, additive manner, then relying on state-level data can provide unbiased statistical estimates. But RAND itself argues that the impacts of resources on student performance are anything but constant and additive. Witness its conclusions on class size, where it finds that class-size reduction has its greatest effect in states with high shares of disadvantaged children. Witness also its finding that it is particularly important to reduce class sizes in states that begin with high average pupil-teacher ratios.

Finally, while the motivation of the entire study was to investigate the role and effect of different state policies, the only policies RAND’s researchers actually built into their main statistical models were differences in per-pupil spending, student-teacher ratios, and other resource variables. Except in an ad hoc fashion, RAND overlooked state efforts to establish accountability in the form of standards and testing and the wide variance in teacher certification requirements. The researchers themselves claim that these policies are important—in fact, they even suggest that such policies explain why Texas students perform better than California’s—yet they didn’t include variations in these policies in the models they constructed, except by creating “fixed effects” models that have so few independent observations that their results can’t survive rigorous statistical tests. In other words, RAND’s analysis failed to include the precise variables that the study itself claims are key.

Overturned?

The RAND study’s authors want to convince people that they have identified the most effective interventions and that outcomes are improving as a result of past reforms. If true, the authors argue, then there is no need to consider more fundamental changes in the education system’s structure or incentives. In order to make this case, the authors must prove that most earlier studies of the impact of school resources on student achievement should be disregarded.

Most scholars believe that studies that look at the impact of resources available to individual schools and specific school districts should be given the heaviest weight because they are the most precise. These studies are also the least likely to find that per-pupil expenditures, teacher pay, or class sizes make a difference.

The studies most likely to find that school resources have a positive effect rely on statewide data, like RAND’s. In this sense, RAND simply repeats an already well-known finding: that if you rely on imprecise statewide data and if you ignore all other aspects of state educational policy, you will often find that average statewide school spending and class size have at least a minor effect on student performance. But as mentioned previously, these studies have a serious methodological limitation: They rely on average results obtained from large, heterogeneous units that differ from one another in many ways other than the amounts they spend on schools.

RAND claims that only by looking at statewide data can you include the impact of statewide policies. Yet statewide studies have not yet found a way of including information about these policies in their statistical analyses. As a result, it is difficult to place more weight on these findings than on those that look at individual schools and school districts.

Again, this is not to say that schools don’t matter. On the contrary, value-added studies find that teacher quality has a major impact on student performance. If we could find ways of keeping good teachers in the classroom—perhaps by giving these successful teachers the additional compensation it would take to encourage them to make teaching a lifelong career—then we could probably boost student performance significantly.

But the authors of the RAND study take exception to value-added research. They claim that value-added studies that measure gains from one point in time to the next fail to account for the fact that “two students can have pretest scores and similar schooling conditions during a grade and still emerge with different posttest scores influenced by different earlier schooling conditions.” Put simply, Suzie may learn more than Johnny in 3rd grade not because Suzie had the better teacher that year but because she may have had a better education the previous year, even though this was not reflected in her 2nd grade test score. Since value-added studies usually don’t incorporate a student’s entire educational history, their results, according to the RAND study, may be biased in some unknown direction.

RAND, however, doesn’t provide any persuasive evidence that this is the case either in its own study or from other studies. Of course, one cannot rule out the possibility that gains in a particular year may somehow be influenced by events in the past. But RAND’s critique of value-added studies comes back to haunt its own research. If its critique is valid, then RAND’s own results are just as flawed as the results of the studies RAND criticizes. If earlier school conditions are important and affect the impact of current resources on student achievement, then one cannot assume constant, additive effects across all students in the state—the RAND researchers’ own methodology. Instead it is necessary to know the specific paths of resources to the individual students in the state and to incorporate that information into the statistical analysis. In other words, the very arguments the authors use to make the case for the superiority of their estimates over the hundreds of previous estimates again undermine their own analysis.

The RAND researchers also try to bolster their methodology by referring to the Project STAR (Student-Teacher Achievement Ratio) experiment, which involved a substantial reduction in class size (from an average of 24 students to an average of 16 students) in Tennessee. The study has received a great deal of attention, in part because it is one of the few evaluations of school resources based on random assignment of students to test policy effects while controlling for other conditions, a method that is generally thought to be a high-quality research design. However, the findings from the study are often misunderstood and misinterpreted, and RAND’s scholars have only added to the confusion.

In essence, the Tennessee study shows that students in substantially smaller classes in their first year of schooling (whether kindergarten or 1st grade) perform better than those remaining in classes of larger size. No similar benefits were observed for students in older grades, however. Those in the smaller kindergarten classes maintained the same higher achievement level that they had realized in kindergarten.

The STAR study, while methodologically superior to the RAND study, has its own limitations. The principle of random assignment was potentially compromised in several ways, and no student test information was obtained before assigning students to “control” and “experimental” groups. As a result, it is unclear how much the study, as implemented, deviated from a random-assignment design. Since almost all the gains from small-class assignment were registered in the initial year, it is possible that even these small “gains” were apparent rather than real.

But even if the STAR study doesn’t suffer from these implementation flaws—without baseline data we’ll never know one way or another—the study is not open to the inferences made by the RAND researchers. First, RAND assumed that the STAR study demonstrates that class-size reduction is effective in multiple grades when in fact it demonstrates, at most, that a very large reduction in class size has positive effects only in the first year of schooling. After that, the initial effects only manage to survive—they do not continue to increase even when the student remains in much smaller classes. Yet RAND uses these results to justify its policy recommendation to lower class size throughout the elementary school years.

Second, the RAND authors try to validate their own problematic methodology by claiming that their estimates of the effects of class size reduction are essentially the same as those obtained from the STAR study. But assessing the validity of studies by their answers violates all scientific principles. Generally speaking, a study’s validity depends on the scientific merits of its methodology, not the results it obtains. And even if one were to accept RAND’s claim to validity by virtue of its match with the results of another study, this claim applies only to the class-size findings.

Conclusions

RAND’s claims to have overturned conventional research wisdom are highly problematic. The report draws sweeping conclusions from average statewide data for just 44 states. The analysis of these data is subject to significant analytical error. The authors leave out of the statistical equations factors that they themselves insist are of critical importance. Claiming that only state-level analysis can take state policies into account, the researchers then leave key state policies out of their most crucial equations.

Worse, the interpretation of the results far exceeds the normal bounds of inference, thereby suggesting that the authors had a policy commitment that shaped their handling of the material.

But let’s take the RAND study at its word. If we do, we would conclude that, in general, education expenditures have little effect on student performance, that increasing teacher pay yields no effect, that the effects of class-size reduction depend very much on the state in which it is implemented, that monies should be set aside so that teachers who say they need them have more materials. The study also asserts that the strong accountability systems in Texas and North Carolina led to particularly spectacular student achievement gains in the early to mid-1990s. This is not necessarily a bad policy agenda. But one can hardly cite the RAND study as scientific evidence that it is the correct one. The conclusions reached by the RAND authors are based more on their personal sense of plausibility than on results from high-quality data subject to properly specified statistical equations.

–Eric A. Hanushek is a senior fellow at the Hoover Institution at Stanford University and a research associate of the National Bureau of Economic Research.

The early 1990s saw the height of the East Asian miracle. The economies of Japan, South Korea, Thailand, Malaysia, and other countries of the region were expanding at rates that dwarfed those of the United States and the mostly European nations of the Organization for Economic Cooperation and Development (OECD). The so-called “Asian tigers” were projected to surpass the U.S. economy in the not-so-distant future. In the national soul-searching that ensued, new attention was focused on the U.S. education system. Our poor academic performance vis-á-vis the countries of East Asia and indeed most of the developed world became a source of deep concern. The touchstone A Nation at Risk report expressed fear of a deteriorating education system leading to an erosion in our global standing. Thus many educators breathed a sigh of relief when the East Asian countries lost their luster. The U.S. economy entered a record-breaking period of high employment and recession-free growth during the 1990s, while Japan stagnated terribly-thereby validating, supposedly, our educational performance and approach.

Take testing critic Alfie Kohn, who writes, “As proof of the inadequacy of U.S. schools, many writers and public officials pointed to the sputtering condition of the U.S. economy. As far as I know, none of them subsequently apologized for offering a mistaken and unfair attack on our educational system once the economy recovered, nor did anyone credit teachers for the turnaround.” Or consider Gerald Bracey, a prominent defender of the public schools. Criticizing those before and after A Nation at Risk who have urged education reform in the interest of maintaining economic growth, he wrote in a recent Washington Post essay, “None of these fine gentlemen provided any data on the relationship between the economy’s health and the performance of schools. Our long economic boom suggests there isn’t one-or that our schools are better than the critics claim.”

Thus the sterling recent performance of the U.S. economy has become a convenient rhetorical tool for those who maintain that the education system isn’t in need of any serious reform. But these critics have woefully misunderstood the nature of economic growth and its link to educational performance. The confusion owes partly to the language and perspectives of the A Nation at Risk report itself. Written in 1983, on the heels of the stagflation of the 1970s and a recession caused by the Federal Reserve’s attempts to curb inflation, it implied that the current state of the economy could be traced directly to the performance of the education system.

Such a perspective fails to distinguish between economic growth and the business cycle. Economic growth is a long-term concept. It depends on past investments in physical capital, like industrial plants and machinery; human capital, the economist’s term for workers’ education and skills; and the pace of technological innovation. Now, growth may slow in the short term because of a downturn in the business cycle. But these short-term fluctuations in the unemployment rate, inflation, and economic growth from quarter to quarter or even year to year should not be confused with the economy’s ability to grow in the long term. And they certainly bear no relation to the current state of the education system.

That unemployment is lower on any given day in the United States than in Japan or Korea says virtually nothing about the relative quality of their schools. It might instead say something about the quality of current fiscal and monetary policies or about the extent of labor market and trade barriers across countries. It might even bear some relationship to the human capital investments made in past periods-when the current range of workers in the labor force was attending schools and investing in skills. Most workers in the economy were educated years and even decades in the past-and they are the ones that have the greatest impact on current levels of productivity and growth. Concerns about the current performance of U.S. schools reflect concerns about the potential for economic growth in the future-when today’s elementary, middle, and high schoolers become tomorrow’s engineers and scientists.

Skilled labor is becoming more and more valued in today’s economy. This is reflected in the compensation that skilled workers receive and the subsequent distribution of income in the economy. The gap between the skilled and the unskilled continues to grow. In turn, the economy’s long-term health is dependent on having a skilled labor force. The education system is central to the development of skills and human capital, a fact long recognized by parents, policymakers, and educators. During the past century, the United States led the world in the expansion of its education system, contributing to the dominant position of the United States in the world economy. Nonetheless, there is reason to be concerned about the future. The evidence suggests that the American K-12 education system is falling behind those of other developed nations. As a result, it is unclear whether we will be able to count on the education system to fuel future U.S. economic growth. As economic growth is crucial to our well-being, this is a matter we should take very seriously.

Human Capital and Economic Growth

Economic growth determines how much improvement will occur in a society’s overall standard of living. The effect of differences in growth rates on economic well-being is easy to see. If gross domestic product (GDP) per capita were to grow at 1 percent each year for 50 years, it would increase from $34,950 in 2000 to $57,480 in the year 2050-more than a 50 percent increase over the period. However, if it were to grow at 2 percent per year, it would reach $94,000 in 2050! Small differences in growth rates have huge implications for the income and wealth of society.
In turn, a society’s ability to develop human capital is crucial to its ability to grow. Human capital consists of the skills possessed by individuals and, in the aggregate, by the labor force as a whole. It is the result of a variety of investments made by individuals and institutions-in formal schooling, workplace training, life experience, and so on. In other words, formal schooling is not the only way to develop human capital-but it is a critical component. And schooling, as William Easterly emphasizes and I discuss below, is not sufficient to ensure growth, but it certainly plays a large role in a society like that of the United States, where the other preconditions for growth are in place.

Educational Quality

Concerns about the current performance of U.S. schools reflect concerns about the potential for economic growth in the future-when today’s elementary, middle, and high schoolers become tomorrow’s engineers and scientists. Photograph by Comstock Images.

Much of the early empirical work on human capital concentrated on the role of school attainment-that is, the quantity of schooling. This focus was natural. The revolution in the United States during the 20th century was universal schooling. Moreover, quantity of schooling is easily measured, and long-term data on years attained are readily available. Today, however, policy concerns revolve around quality issues much more than quantity issues. The completion rates for high school and college have been roughly constant for a quarter of a century. In addition, measures like completion rates and universal elementary education don’t say much about the kind of education that is being offered. Developing skills useful in the labor market and for economic growth requires rigorous training in high-quality schools. Schools that amount to no more than free day care or points of religious indoctrination won’t contribute to economic growth.

The current economic position of the United States is largely the result of its strong and steady growth during the 20th century. Strangely, in the period after World War II, economists did not pay as much attention to economic growth as they did to macroeconomic fluctuations in an attempt to tame the business cycle. In the past 15 years, economists have returned to questions of economic growth. A variety of models and ideas have been developed to explain differences in growth rates across countries; the importance of human capital is invariably a component.

The typical study finds that quantity of schooling is highly related to economic growth rates. But, again, quantity of schooling is a very crude measure of the knowledge and cognitive skills of people. It is unlikely that what is learned during the 6th grade in a rural hut in a developing country equals what is learned in an American 6th grade. Yet that is what empirical analyses implicitly assume when they focus exclusively on differences in average years of schooling across countries.

Recently, Dennis Kimko and I have delved into issues of educational quality. We incorporated the information developed during four decades of international testing on the varying mathematics and science performance of nations around the world. Our research has found a solid link between differences in school quality and differences in economic growth.

In 1963 and 1964, the International Association for the Evaluation of Education Achievement (IEA) administered the first of a series of mathematics tests to a voluntary group of countries. These assessments faced a number of challenges: developing a test that provided a fair comparison across countries with different school structures, curricula, and languages; creating comparable groups of testing participants across countries; and persuading countries to participate. The first tests did not document or even address these issues in any depth. However, these tests did prove that such testing was feasible and set in motion a process to expand and improve on the undertaking. Subsequent testing, sponsored by the IEA and others, has included both math and science and has expanded the group of countries tested. In each, the general model has been to develop a common assessment instrument for different age groups of students and to attempt to obtain a representative group of students taking the tests in each country.

Our analysis was very straightforward. We combined all of the available earlier test scores into a single composite measure of quality and introduced it into statistical models that explain differences in growth rates across nations during the period 1960 to 1990. (We excluded results from the 1995 Third International Math and Science Study and subsequent tests because they were obtained outside the analytical period of interest.) The underlying objective was to obtain a measure of quality for the labor force during the period for which we have measurements of economic growth. The basic statistical models, which include the level of income, the quantity of schooling, and population growth rates, explain a substantial portion of the variation in economic growth. Significantly, the quality of the labor force as measured by math and science scores proved to be extremely important.

Worldwide, we found that a difference in test performance of one standard deviation was related to a 1 percent difference in the annual growth rate of per-capita GDP. The impact of such a difference in growth rates is very large. As we saw earlier, 1 percent higher growth-say, growth of 2 percent versus 1 percent per year-over a 50-year period yields incomes that are 64 percent higher. Moreover, adjusting the data for other factors that are potentially related to growth, including aspects of international trade, private and public investment, and political instability, leaves the effect of having a quality labor force unchanged.

A common concern in analyses like this one is that schooling might not be the actual cause of growth but may just reflect other attributes of the economy that are beneficial to growth. For example, East Asian countries consistently score high on the international tests, and they also had extraordinarily high growth during the 1960 to 1990 period. It may be that other aspects of these East Asian economies have driven their growth and that the statistical analysis of labor-force quality is simply picking out these countries. But if the East Asian countries are excluded from the analysis, a strong-albeit slightly smaller-relationship is still observed between test performance and economic growth.

Another concern might be that other factors affecting growth, such as efficient market organizations, are also associated with efficient and productive schools-so that again the test measures are really a proxy for other attributes of the country. To investigate this, we concentrated on immigrants to the United States who received their education in their home countries. We found that immigrants who were schooled in countries that have higher scores on the international math and science examinations earn more in the United States. This analysis makes allowance for any differences in school attainment, experience in the labor market, or being a native English- language speaker. In other words, skill differences as measured by the international tests are clearly rewarded in the U.S. labor market, reinforcing the validity of the tests as a measure of individual skills and productivity.

Finally, the observed relationships could simply reflect reverse causality. In other words, countries that are growing rapidly have the resources to improve their schools. In this case, better student performance is the result of growth, not its cause. As a simple test of this, we investigated whether the international math and science test scores were systematically related to the resources devoted to the schools in the years before the tests. They were not. If anything, we found relatively better performance in those countries that spent less on their schools.

In sum, the relationship between math and science skills on the one hand and productivity and growth on the other comes through clearly when investigated in a systematic manner across countries. This finding underscores the importance of high-quality schooling to future well-being.

Explaining the U.S. Economy

Photograph by David Muir/Masterfile.

In this context, the United States presents a difficult conundrum. In the international exams of math and science that have taken place since 1970, the United States has been at best in the middle of the pack, at worst well below average. At the same time it has become the world’s economic superpower. How to reconcile these diverging trends?

The answer is that the quality of the labor force is just one aspect of the economy that contributes to economic growth. Expanding education in a developing economy, as Easterly argues, is unlikely to foster much growth if the economy fails to simultaneously acquire the market structures and legal and governance systems that are necessary for a high-performing economy. The United States has an abundance of these attributes, and they appear to compensate for the shortcomings of its education system.

Almost certainly the most important factor sustaining the growth of the U.S. economy is the openness and fluidity of its markets. The U.S. maintains generally freer labor and product markets than most countries in the world. The government generally lathers less regulation on firms, and trade unions are less powerful than those in many other countries. More broadly, the U.S. government intrudes less in the economy-not only with less regulation, but also with lower tax rates and minimal government production through nationalized industries. These factors encourage investment, permit the rapid development of new products, and allow U.S. workers to adjust to new opportunities. While identifying the precise importance of these factors is difficult, a variety of analyses suggest that such market differences could be very important explanations for differences in growth rates. These favorable institutional conditions have in some ways compensated for the deficits of our education system.

The United States has also been saved by the expansion of opportunities for higher education for its citizens. During the 20th century, the expansion of the education system in the United States outpaced the rest of the world. The United States pushed to open secondary schools to all citizens. With this also came a move to expand higher education with the development of land grant universities, the G.I. bill, and direct grants and loans to students. Compared with other nations of the world, the U.S. labor force has been better educated, even after accounting for the lesser achievement of its graduates. In other words, more schooling with less learning each year has yielded more human capital than found in other nations that have less schooling but where students learn more in each of those years.

This historical approach, however, appears on the verge of reaching its limits. Other nations of the world, both developed and developing, have rapidly expanded their schooling systems, and many now surpass the United States. In a comparison of secondary-school completion rates in 1999, the United States trailed a large number of other countries and fell just slightly below the OECD average completion rate. The United States gains some by having rates of college attendance above the typical OECD country. Nonetheless, U.S. students are not likely to complete more schooling than those in a significant number of other developed and developing countries. Thus, going into the future, the United States appears unlikely to continue dominating others in human capital unless it can improve on the quality dimension.

Still, it is not just college completion, but also the quality of the college experience that has been our saving grace. The analysis of growth rates across countries emphasized the quality of the elementary and secondary schools of the United States. It did not include any measures of the quality of U.S. colleges. By most evaluations, U.S. colleges and universities rank at the very top in the world. While there are no direct measures of the quality of colleges across countries, there is indirect evidence. Foreign students by all accounts are not tempted to emigrate to the United States to attend elementary and secondary schools-except perhaps if they see this as a way of gaining entry into the country. They do emigrate in large numbers to attend U.S. colleges and universities, however. They even tend to pay full, unsubsidized tuition at U.S. colleges, which a much smaller share of American citizens does.

A number of the economic models emphasize the importance of scientists and engineers as a key ingredient of growth. By these views, the technically trained college students who contribute to invention and to the development of new products provide a special element in the growth equation. Here again the United States appears to have the best programs. If this view is correct, U.S. higher education may continue to provide a noticeable advantage over other countries.

But the raw material for U.S. colleges is the graduates of our elementary and secondary schools. As has been noted frequently, the lack of preparation of our students necessitates extensive remedial education at the postsecondary level, detracting from the ability of colleges and universities to be most effective. On this count, there is yet another troubling aspect to U.S. academic performance. In international comparisons, U.S. students start out doing well in elementary grades and then fade by the end of high school. Figure 1 shows the slippage that occurs over time in comparison with other countries participating in the TIMSS math and science testing. To the extent that performance at the end of secondary schooling is most important-because it represents the skills students have as they enter college, because it sets the stage for science and engineering skills, or because it is important in its own right for workers in the labor force-schools in the United States are not keeping up with the preparation of students.

Source: Third International Mathematics and Science Study

Just a Jeremiad?

Observers like Bracey, Kohn, and others, dead set against any fundamental changes in the nation’s schools, usually work hard to dismiss the TIMSS results as evidence of any structural weakness in the U.S. education system. At least one objection they raise is interesting enough to consider. If innovation is key to economic growth, the argument goes, then an education system that encourages creativity and questioning rather than drill and memorization may give a nation a competitive advantage. This argument is usually trucked out as a defense against standardized testing, which is supposed to suppress our students’ natural curiosity. Bracey writes, “We should think more than twice before we tinker too much with an educational system that encourages questioning. We won’t benefit from one that idolizes high test scores. It could put our very competitiveness as a nation at risk.” However, none of these critics has ever produced any evidence that creativity is lessened when students improve their math and science skills. Nor do they speak to the costs placed on those individuals who neither reap rewards for exceptional creativity nor have the skills necessary to perform in the modern economy.

Bracey also suggests that we shouldn’t worry about the TIMSS results because they don’t seem to have any effect on our economic competitiveness. He cites as evidence the fact that a nation’s place on the Current Competitiveness Index developed by the World Economic Forum is not perfectly correlated with the TIMSS results, and the United States ranks high on the index. His explanation for our high ranking? All the reasons suggested above: the higher quantity of education in the United States, greater college attendance, retention of our scientists and engineers (while attracting foreign immigrants), and greater innovative capacity. It’s true that all of these factors have probably compensated for our educational deficiencies. But then just imagine how the U.S. economy might perform if its education system was top-notch! And what happens when we lose our advantage in any one of these factors-like when other nations create higher education systems like ours?

In February 1989, in an unprecedented meeting of the nation’s governors with President George H.W. Bush, an ambitious set of goals was set for America’s schools. One goal was that by the year 2000, “U.S. students will be first in the world in mathematics and science achievement.” By 1997, as it became evident that this goal wasn’t going to be met, President Clinton returned in his State of the Union speech to the old model of substituting quantity for quality: “We must make the 13th and 14th years of education-at least two years of college-just as universal in America by the 21st century as a high-school education is today.” It may make a better sound bite simply to offer two more years of schooling, but the best way to cement our competitiveness in the future is to ensure that every student gets a solid education during the first 12 years of school life.

-Eric Hanushek is a senior fellow at the Hoover Institution, Stanford University.

The concerns about school quality expressed in A Nation at Risk reflected declining trends in performance among U.S. students and their mediocre standing relative to students in other nations. America’s failure to address these concerns has led to substantial losses for individuals and for society as a whole. The workers who failed to acquire essential skills can attest to the fact that their earnings have not kept up with those of the typical worker. And the aggregate effects are even more dramatic.

By some estimates, today’s entire K-12 education budget could be funded by the “reform dividend” that might have been expected from improving math and science achievement in response to the calls of the 1983 report. To wit, improvements in the schools would have boosted U.S. economic growth, and the annual windfall by 2002 would have exceeded total K-12 spending for that year.

Quality Matters

Much of the research on the economic impact of education has properly concentrated on the role of school attainment-that is, the quantity of schooling. This focus is natural. The revolution in the United States during the 20th century was the universal provision of a basic education. Moreover, years of schooling are easily measured, and data on years attained, both over time and across individuals, are readily available.

Yet today’s policy concerns revolve around issues of quality much more than of quantity. Completion rates for high school and college have been roughly constant for a quarter of a century in the United States, while the rest of the industrialized world has largely caught up on measures of school attainment. Risk was more concerned with the fact that U.S. students had fallen behind their peers in nations like Japan, the Netherlands, and France on international exams in math and science (see Figure 4, page 44, in Paul E. Peterson, “Ticket to Nowhere“). These concerns sparked the standards and accountability movement, which seeks to define what students should learn and tests to see whether they have mastered the material.

In good part because of the impact of the Risk report, it is now generally recognized that students’ cognitive skills are a crucial dimension of education quality. But it has not, until recently, been clear just how important differences in cognitive skill are for the long-term well-being of a nation’s economy. Fortunately, data are now available that allow one to estimate the connection between cognitive skill and the economy. The conclusions of this emerging body of research are clear: education quality, as measured by test scores, is positively related to the earnings of individuals, national productivity, and economic growth.

Individual Earnings

Just as most parents believe, economists have clearly shown that a student’s achievement in school directly affects his or her earnings later in life, after allowing for differences in the quantity of schooling, experience in the labor force, and a variety of other factors that also influence earnings. Students who do well in school also tend to go on for further schooling, which provides an additional boost to their earnings. As is well known, the economic benefits of a college education have risen dramatically during the past quarter century, and substantial evidence shows that students with good grades or high scores on achievement tests tend to pursue more education.

These facts are part of the reason that so much attention has been paid to the schools as an agency of equal opportunity, ultimately helping to reduce inequities in the distribution of income. Long before the Risk report, the War on Poverty saw schools as key to reducing racial and other disparities in economic opportunity. Through schooling it was hoped that family poverty would not be transferred to the next generation: high-quality school investments would make up for deficits originating in the home.

Today, many believe that the continuing difference between the earnings of black and white workers is due in good part to differences in their educational achievement, as measured by tests of cognitive ability. These continuing differences are especially worrisome, given the fact that the importance of education for the acquisition of well-paying jobs continues to grow, increasing the disparities in income between those with college degrees and those with less than a high-school diploma. Only if skill levels can be enhanced within high schools will many of the more disadvantaged in society have access to the college education that is crucial in a society where high-level skills are fundamental to success.

Economic Growth

No less important is the overall relationship between the quality of the labor force, as measured by tests of cognitive skill, and economic growth. Economic growth rates determine how much improvement will occur in society’s overall standard of living. Moreover, the education of each individual has the possibility of making others better off (in addition to the increased earnings the individual receives). For instance, a more educated society may have higher rates of invention; may make everyone more productive by virtue of the fact that firms are better able to introduce new, more sophisticated production methods; and may lead to the more rapid introduction of new technologies. These “externalities” that make everyone better off provide still another reason for taking measures that will enhance the quality-not just the quantity-of schooling.

Recent work in which Dennis Kimko and I have been engaged has looked closely at the size of the impact of labor force quality, as measured by tests of cognitive ability, on the economic growth of countries. Our study has drawn on information about the mathematics and science performance of students in many countries during the past four decades. In making our estimates, we take into account differences between countries in their level of income, the average number of years students are in school, and population growth rates. We find that a difference of one standard deviation in test performance is related to a 1 percent difference in annual growth rates of per-capita gross domestic product (GDP). This suggests that school quality has a great impact on economic productivity and growth. To some, 1 percent may not appear to be a large number. But a 1 percent increase each year in the growth rate of a country soon compounds to a very large number. Consider the United States at the beginning of the 21st century, for example. In the year 2000, GDP per capita was $34,950. An annual growth rate of 1 percent raises average income to no less than $57,480 in 2050-more than a 50 percent increase over the period. Quite simply, small differences in growth rates have huge implications for the income and wealth of society.

If education has such a dramatic impact on a country’s economic productivity and growth, what are the implications of a less-than-adequate education system for economic growth today and in the future? Can the U.S. economy continue to lead the world when the performance of its students on international tests in math and science has been mediocre at best? Some believe it can, that strengths in other areas compensate for America’s educational deficiencies. The United States, for example, has more open and competitive markets and less intrusive government regulation than do the economies of many other industrialized societies. In addition, its system of higher education is the envy of the world. But these assets only mean that the United States may well have enjoyed even greater economic productivity had it enhanced the quality of its K-12 schools. Had the reform movement sparked by the Risk report led to real improvements in academic achievement, it would have had a dramatic impact on the already strong economy.

Squandered Potential

Consider a hypothetical scenario in which schools instituted truly effective reform in math and science instruction at the time of the Risk report. Had the reforms translated into achievement gains of 0.12 standard deviations a year for the remainder of the decade, with performance constant thereafter, scores of graduates would be one standard deviation higher going into the 1990s and the future. This would have required a Herculean effort, but was within the bounds of expectations. Recall that a 1989 meeting of the nation’s governors set a goal of making U.S. students’ performance in mathematics and science first in the world by 2000. An improvement of the more modest magnitude considered above would have put U.S. student performance in line with that of students in several European countries, but it still would not be at the top of the world rankings.

Such a path of improvement would not have had an immediately discernible effect on the economy, because new graduates are always a small portion of the labor force. However, the impact would mount over time. Figure 1 plots the potential GDP from 1990 to 2002 and adds an estimate of what the school reform sketched above would have implied for the economy. If past relationships between quality and growth held, GDP in the United States would have been more than 4 percent higher than was realized in 2002. The area between the two trend lines shows the “reform dividend,” which totals to $2.5 trillion in the two decades after the release of Risk. With close to a $10.5 trillion economy, the unrealized gain for 2002 alone would have amounted to $450 billion, or more than the nation’s total annual expenditure on K-12 public education.

Measuring Quality

A segment of the education policy community has argued against the current testing regime-either because it does not measure attributes they think are important or because the test outcomes are irrelevant. Nevertheless, the evidence reviewed above demonstrates that differences in performance on existing tests have significant implications for both individual and aggregate success. This is not to say that existing tests are the best possible. It just shows that the existing tests measure something real, something real enough that it has important ramifications for individuals and the economy.
That the acquisition of cognitive skill as measured by tests is important does not mean other aspects of education are unimportant. In fact, some research suggests that other dimensions of individual skill also influence economic performance. For instance, to the extent that aggregate growth is fueled by invention, creativity is likely to be important, and this may differ from measured cognitive skills. Currently it is in vogue to argue that schools must do more than simply teach reading, math, and science. Of course this is true. But such arguments do not deny that cognitive skills are important, and they do not say what should be done if one wants to enhance these other, currently unmeasured areas.

The question for the United States is how to create policies that boost achievement and thus economic growth. It would be easy, if we could improve quality simply by spending more or by reducing class size. But, unfortunately, evidence from both the United States and other countries shows that more school resources and smaller classes do not have much of an effect on how much a student learns in school, as measured by tests of achievement. The international math and science scores so important for growth rates are not related to variations in spending on education or other standard measures of school resources, such as pupil-teacher ratios. Similarly, within countries that participated in the 1995 Third International Mathematics and Science Study, there is no systematic relationship between resources and student performance. Consequently, the policies following Risk failed in large part because they concentrated on simply adding more resources to the pot.

In other words, we need to look for ways other than mere increases in expenditure or reductions in class size if we are going to enhance the quality of our education system. A large body of evidence suggests that differences in quality between schools affect how students learn, but it will take creative policies to tap this potential. What students learn in school impacts their earnings later in life, their productivity in the work force, and, ultimately, the country’s rate of growth. Over time, the cumulative impact of a high-quality education system can be dramatic.

-Eric A. Hanushek is a senior fellow at the Hoover Institution, Stanford University.

“Make-or-Break Exams Grow, But Big Study Doubts Value” intoned a front-page New York Times headline in December 2002. The article continued, “Rigorous testing that decides whether students graduate, teachers win bonuses, and schools are shuttered … does little to improve achievement and may actually worsen academic performance and dropout rates, according to the largest study ever on the issue.” Thus a deeply flawed study was catapulted to national prominence. More important, its conclusions were opposite those found through rigorous scientific studies.

The report in question, authored by Arizona State University researchers Audrey Amrein and David Berliner, purported to examine student-performance trends on national exams in states where legislators have attached “high stakes” to test scores. High-stakes testing has become a lightning rod as more and more states adopt accountability measures in response to the mandates of the federal No Child Left Behind Act. While it is crucial to analyze and debate the wisdom of such policies, the discussion must be informed by evidence of the highest quality. The controversial nature of high-stakes testing has led to the hurried release and dissemination of research that lacks scientific rigor, of which the Amrein and Berliner study is one of the more egregious examples.

This says much about the standards for research in education today. The situation is so contentious that in 2000 the National Research Council found it necessary to convene a panel to decide which scientific principles should apply to educational research—the kind of question that other fields of social science settled long ago. In the case at hand, Amrein and Berliner trumpet the fact that their report was reviewed by a panel of four scholars based at other schools of education, yet this should only be a source of greater concern. Sharing a paper with sympathetic colleagues is no substitute for a system of blind peer review—a bedrock principle of scientific research.

Here we closely examine Amrein and Berliner’s underlying data and methodology. Our results are astonishing: if basic statistical techniques are applied to their data, it reverses nearly every one of their conclusions. Later we also present the results of separate research on accountability that we conducted for a June 2002 Federal Reserve Bank of Boston conference. Rigorous analysis reveals that accountability policies have had a positive impact on test scores during the past decade.

The Unscientific Method

Amrein and Berliner identified 28 states where test scores are used to determine various consequences, such as bonuses for teachers, the promotion of students, or allowing children to transfer out of a failing school. These stakes go beyond less controversial accountability measures such as publishing test scores in the newspaper. The states range from Georgia and Minnesota—where the only penalty is experienced by students who fail a high-school graduation exam—to North Carolina and Texas, where the authors found a total of six stakes each, stakes that affect both schools and students.

Once Amrein and Berliner identified the high-stakes states, they looked at changes in the average scores students earned on the National Assessment of Educational Progress (NAEP). Choosing this test as a basis for considering the impact of high-stakes tests on students in the 4th and 8th grades (ages 9 and 13, respectively) is a sensible idea, because the validity and reliability of NAEP, often called the “nation’s report card,” are well accepted. It is a test for which students cannot easily be prepped and, since the performance of individual school districts, schools, or students is not reported, there is little incentive to cheat or even to prepare for the test. It also provides a neutral standard for assessing the effects of state policies. But if the Arizona State team’s decision to look at NAEP scores was correct, less can be said for their other analytical choices.

Amrein and Berliner’s basic strategy was to look at how each high-stakes state’s scores changed with the introduction of accountability and to compare this with the national trend. If the state’s gains exceeded the national gains, they deemed that an increase in scores. If the state’s gains trailed the national gains, they deemed that a decrease. But whenever the rate at which students were excluded from the NAEP because of a disability or lack of language proficiency moved in the same direction as that state’s NAEP scores (in other words, an increase in test scores coupled with an increase in test exclusions), Amrein and Berliner declared the results contaminated and simply tossed out the state as inconclusive. (At least that is what they claimed to do; in fact, they applied the rule inconsistently.)

As a result, their conclusions are based on only a fraction of the high-stakes states. For instance, they recorded positive or negative results on the NAEP 4th-grade math test for just 12 of the 26 states with stakes for K-8 students (as noted earlier, two of the states, Georgia and Minnesota had only a high-school graduation exam and thus were not used for this analysis). Amrein and Berliner found that 4th-grade math scores increased at a slower rate than the national average in 8 of the 12 states, faster in just 4. Yet they write this up in a highly misleading fashion, claiming that “67 percent of the states posted overall decreases in NAEP math grade 4 performance as compared to the nation after high-stakes tests were implemented.” Actually, Amrein and Berliner witnessed gains slower than the national average in just 8 of 26 high-stakes states, or 31 percent.

Amrein and Berliner’s misleading reporting practices took on new importance when the media dutifully broadcast their results as they were written. Consider the article in Education Week, which reported, “Movement in elementary-school reading scores was evenly split—better than the national average in half the states, worse in the other half.” In fact, Berliner and Amrein based their conclusions in 4th-grade reading on just ten states, five of which they recorded as gaining against the national average, five of which as losing. So less than a fifth of the high-stakes states saw decreases against the national average in reading, not “half.” At the 8th-grade level in math, Amrein and Berliner were able to look at only eight states, five of which gained against the national average, three of which lost. Here, again, Amrein and Berliner wrongly reported this as “63 percent of the states posted increases in NAEP math grade 8 performance as compared to the nation after high-stakes tests were implemented.”

All of this ignores the truly fatal flaw of Amrein and Berliner’s methods: their point of comparison. If one wants to assess the effect of high-stakes testing, the obvious comparison is between states that adopted accountability systems and those that did not. Amrein and Berliner’s decision instead to compare the gains in high-stakes states with the national average violates a basic principle of social-science research. The national gain on NAEP incorporates any gains in high-stakes states, so Amrein and Berliner’s strategy is akin to a medical trial where the treatment group receives the full dose of a medication while the control group receives a half-dose. It would not be surprising to find that the full dose was not dramatically more effective. The real question is whether the full dose is more effective than no medication at all.

On Their Terms

Amrein and Berliner concluded, as announced in their press release, “High-stakes tests may inhibit the academic achievement of students, not foster their academic growth.” Let’s take a look at their evidence in more detail.

Before doing so, however, we need to be clear: we are not in any way endorsing Amrein and Berliner’s analytical approach. We return below to discuss the results from a more scientific study of accountability. But using their approach in a systematic manner will at least reveal the degree to which their decisions about what information to include and to exclude distorted the facts and thereby confused the debate over accountability.

An initial problem with their analysis is that Amrein and Berliner disregarded the magnitude of any changes in test scores. By simply listing the results as “Increase,” “Decrease,” or “Unclear” (in cases where exclusion rates rose), Amrein and Berliner discarded rich information. They converted useful continuous data (test scores) into hollow binary data (test scores went up or down). In a purely hypothetical example, say six of the high-stakes states gained 20 percent, while the other 20 gained 2 percent each and the no-accountability states made no gains whatsoever—yielding a national average gain of 3 percent. Amrein and Berliner’s approach would supposedly demonstrate the failure of accountability: just six states beat the national average, while 20 were below the average. In fact, ignoring any complications from test exclusions, Amrein and Berliner would report this as something like, “Just 23 percent of states posted gains on NAEP higher than the national average after high stakes were introduced.” The right approach is to compare the average gains of high-stakes states with those of no-accountability states.

When this is done, the analysis yields starkly different results than Amrein and Berliner report. Table 1 compares the math gains among 4th and 8th graders in the same way as Amrein and Berliner—by following different cohorts as they reach 4th or 8th grade in different years. In other words, they compared the 4th graders of 1996 with the 4th graders of 2000, two completely different cohorts of students. For each of the comparisons, data were available for 34-36 states, 18-20 of which were part of the high-stakes group, due to the varying participation of states in the NAEP testing program. For either the 1992-2000 period or the 1996-2000 period, the average gain in math among high-stakes states noticeably exceeded that of the no-accountability states. The differences in performance were statistically significant at conventional levels, meaning that we can be highly confident that they are not just chance occurrences. (By contrast, Amrein and Berliner did no significance testing whatsoever, neglecting one of the oldest and most basic tools of social-science research.)

Amrein and Berliner might object that we have included states where students were excluded from tests at higher rates after accountability reforms were introduced, possibly contaminating the results. Amrein and Berliner’s solution was just to toss these states out, no matter how small the change in exclusion rate or how large the change in achievement. As Table 1 shows, we instead adjusted the achievement gains for observed changes in exclusion rates. And the results barely changed: high-stakes states still significantly outperformed no-accountability states across the board. In fact, the changes in test-participation rates were not statistically different in high-stakes states from those in other states, indicating that this was not even remotely as influential a factor as Amrein and Berliner declared it to be.

Scientific quality is determined not only by the overall methodology, but also by the care and precision of any measurements. To assess the latter, let’s focus on the eight states where Amrein and Berliner concluded that 4th-grade math scores decreased following the introduction of high-stakes testing. Consider Table 2. Three of the eight states—New Mexico, Oklahoma, and West Virginia—adopted high-stakes testing during the 1980s. However, NAEP scores at the state level became available only during the 1990s. For these states, Amrein and Berliner lacked the “before” data for their “before and after” analytical strategy, but went ahead and labeled their scores as “decreasing” anyway. The other five “decreasing” states all experienced greater gains than no-accountability states during the time that they introduced high-stakes testing; New York even beat the national average gain in every time period. And this is the group of states that Amrein and Berliner identify as being harmed by accountability! Not a single one provides evidence of harm following the introduction of high-stakes testing.

Even where before-and-after data were available, Amrein and Berliner did not always use the data from the NAEP tests immediately preceding and following the adoption of high stakes. In several cases, they apparently chose an interval that began after the state’s accountability system came on-line—an “after-after” comparison. These procedures yielded results that reflected negatively on accountability, but they have no scientific justification. To see this, consider the table on p. 52 and try to think of a consistent rule that justifies Amrein and Berliner’s decision to place both Maryland and Missouri in the “decreasing” category.

In short, Amrein and Berliner used scientifically inappropriate methods and applied them in an even shoddier manner. Simply taking Amrein and Berliner’s approach and applying it correctly to all of the data on NAEP achievement reverses their conclusions. Again, these simple comparisons are not the best way to examine these questions, but the results of even these crude analyses confirm the findings from the more sophisticated approach we describe below: greater accountability is accompanied by improved student performance.

Amrein and Berliner also used trends on the SAT, the ACT, and Advanced Placement (AP) exams to assess the effectiveness of minimum-competency exams in the 18 states where students must pass such tests in order to graduate from high school. This comparison suffers from all the same problems as the NAEP comparison and more. For example, does anyone believe that nothing else has changed in North Carolina since the introduction of a graduation test in 1980? Amrein and Berliner’s simplistic trend analysis attributes all subsequent changes in graduation rates and dropouts to the introduction of this high-stakes exam. Nonetheless, because these discussions are less directly related to the current state accountability debates and these data are more difficult to interpret than NAEP scores, we do not pursue them.

Results of Rigorous Analysis

Assessing the impact of state accountability systems is clearly complicated. In many states, these systems are quite young; in 1996, just ten states had active accountability systems. Moreover, states differ in many ways other than their accountability provisions—ways that can make it difficult to isolate the impact of high-stakes testing. They also change in different ways over time, adopting new accountability provisions and other legislation at different times and being influenced by shifting demographics at different rates. This does not make gathering evidence about the effects of accountability impossible. It simply reinforces the need to apply stringent scientific methods to the analysis.

Here we report results from our own analysis of state accountability systems using NAEP data. These results were reviewed at a high-profile conference and were subject to a blind peer review for publication in a Brookings Institution volume, No Child Left Behind? The Politics and Practice of Accountability, which is slated for release this fall.

NAEP tested 4th graders in mathematics in 1992 and 1996 and 8th graders four years after each of these assessments, in 1996 and 2000. As noted earlier, whereas Amrein and Berliner simply compared the test scores of 4th graders in one year with those of a different set of 4th graders four years later, we measured students’ growth in achievement between the 4th and 8th grades. In other words, we compared 4th graders’ math achievement in 1996 with their performance four years later, when they were 8th graders. The same exact students were not tested in each grade, but the two samples are at least representative of the same cohort of students. We also adjusted the data to account for changes in state spending on education and for parents’ educational levels, which provides controls for simultaneous changes in state policies or differences in demographics that might confound the analysis of how accountability systems influenced student achievement. Amrein and Berliner used no statistical controls at all.

Our analysis focuses on state testing and accountability systems that impose consequences on schools rather than on students. These are the most relevant policies for evaluating the potential impact of the No Child Left Behind Act. Our statistical analysis includes all states that have relevant NAEP data, and we explicitly allow for the timing of states’ introduction of their accountability systems.

Figure 1 summarizes our findings in mathematics. The typical student progressing from grade 4 in 1996 to grade 8 in 2000 in a state with a consequential accountability system could expect to see a 1.6 percent increase in his NAEP proficiency score (calibrated to the appropriate learning standards for each grade). By contrast, the typical student in a state with no accountability system could expect to experience only a 0.7 percent gain in mathematics proficiency, a statistically significant difference. Students in states with “report card” systems, where scores are publicly reported but no consequences are attached to performance, fell in the middle: they could expect to gain 1.2 percent in achievement between grades 4 and 8, over and above what they would normally learn from grade to grade. In short, states with high-stakes and even low-stakes systems for schools performed significantly better on NAEP than states with no stakes at all.

We are not the only ones reporting positive effects of accountability. In a forthcoming paper, Stanford University economists Martin Carnoy and Susanna Loeb conducted a similar analysis but expanded it to include testing policies that impose high stakes on students. They found that NAEP performance increased in states with high-stakes systems compared with states that had not yet attached consequences to schools’ test scores. Carnoy and Loeb also investigated the impact of accountability on student retention and high-school graduation rates and demonstrated that there is no discernible negative effect on either outcome.

Other rigorous studies have been carried out of accountability systems within states and school districts. As opposed to the Amrein-Berliner study, they have been vetted at scientific conferences and are being peer reviewed according to normal scientific practice. The Brookings Institution volume is one example. The accumulated literature generally supports two conclusions. First, student performance on the available measures, usually state tests, improves after accountability reforms are introduced. Second, other short-run changes—such as students’ being excluded from taking the tests at greater rates, or explicit cheating—are observed. In other words, some unintended consequences often tend to accompany the introduction of accountability, although there is little evidence that these influences continue over time.

Schools may exclude low-performing students from taking the test in an attempt to “game” the system—to increase their performance artificially by removing scores that bring down their averages. We looked at differences among the states in terms of their placement rates into special education—often one way to exclude students from state tests—and at whether these differences were related to the introduction of state accountability systems. From 1995 to 2000, the time when many state accountability systems were coming on-line, we found no evidence that special-education placement increased in reaction to the introduction of accountability. Special-education placements did increase nationally, just not in any systematic way suggestive of a relationship to state accountability.

No Accountability for Research

That a study of such dubious scientific quality could make the front page of the nation’s most respected newspaper is disturbing, but perhaps not so unusual. In the contentious environment of K-12 education, the media too often gives attention to findings that are relevant to policy regardless of their scientific merit. This discussion shows that education studies vary so much in their scientific rigor that one cannot just review them based on press releases and the sensationalism of the reported results.

Reporters need not be experts in statistical analysis any more than they must be fully versed in biochemistry or investment-banking regulations. But when a report is commissioned by an organization like the Great Lakes Center for Education Research and Practice, a Midwestern group sponsored by six state affiliates of the National Education Association, it would seem to call for a reasonable dose of skepticism. Why not bring in some outside expertise to review such a report before heralding its arrival? There will definitely be further opportunities for review. After all, the Arizona State shop promises that this is just the first of many annual reports on the impact of high-stakes testing.

The media is not alone. Resources at the state and federal levels must be committed to evaluating the quality of research and disseminating evidence of effective practices to schools and the public. The No Child Left Behind Act’s emphasis on research-based practices, the creation of the federal Institute of Education Sciences, and efforts such as the What Works Clearinghouse, which will review and disseminate research findings, are important developments in this regard. State policymakers must also devote resources to evaluating their programs and synthesizing available research. Identifying effective reforms using rigorous evaluative techniques is a crucial task, especially since improving the education system is likely to have a greater economic impact than any of the medical breakthroughs of the past decade.

We also do not mean to suggest that the book has been closed on accountability. It appears that high-stakes states performed better than no-accountability states during the 1990s, but there is still much to be learned. For instance, there is uncertainty about the best way to translate test scores into overall school ratings. Also, states have yet to design accountability systems that directly link test-score performance to appropriate incentives. The vast majority of state accountability systems simply report the average scores for each school, sometimes disaggregating by racial and ethnic groups. However, average scores are highly dependent on socioeconomic factors outside the control of schools. States—and researchers—must become adept at discerning the components that make up the scores and how they can be influenced by high-stakes regimes. Measuring the gains that students make over time would provide a better measure of school performance and serve as a proper basis for reward or sanction, but such value-added techniques need some work before they can serve as reliable performance measures. There are other issues as well. Nonetheless, the evidence points in the direction of refining accountability systems rather than scrapping them altogether.

-Margaret E. Raymond is the director of CREDO, an education policy research group at the Hoover Institution. Eric A. Hanushek is a senior fellow at the Hoover Institution.

Experienced teachers are, on average, more effective at raising student performance than those in their early years of teaching. This gives rise to the concern that too many teachers leave the profession after less than a full career and that too many leave troubled inner-city schools for suburban ones. Until now, the roots of these problems have not been well understood. In particular, it is not known whether teachers leave schools with high concentrations of disadvantaged and low-achieving populations for financial reasons or because of the working conditions associated with serving these students. Nor are there good estimates of what kinds of salary increases would need to be offered to slow the turnover among teachers.

The chief obstacle to resolving these issues has been the difficulty of separating the effects of teachers’ salary levels from their working conditions and preferences. The outstanding suburban school that retains most of its teachers is likely to be attractive on a number of levels: the pay is good, students are high performing, and parents are supportive. Since all three factors help in attracting and retaining teachers, it becomes difficult to calculate the degree to which each factor separately affects a teacher’s decision to return to that school the following year. Conversely, the school that has disadvantaged and low-performing students may suffer high rates of teacher turnover, but sorting out the causes of turnover is difficult. Doing so requires detailed information for enough teachers and students to allow analysts to distinguish statistically among the various factors that affect teachers’ decisions.

Fortunately, important parts of the necessary information are now available for elementary schools in the state of Texas for the years 1993 through 1996. Working in cooperation with the Texas Education Agency, the University of Texas at Dallas’s Texas Schools Project has combined various data sets to create a database of key characteristics of both teachers and students during this period in all Texas public schools. This information includes the race, ethnicity, and gender of both students and teachers; students’ eligibility for a subsidized lunch; and students’ performance on the Texas Assessment of Academic Skills (TAAS), a criterion-reference test administered each spring to students in grades 3 through 8. The database also contains annual information about the teachers: their years of experience, their education and salary levels, the grades and subjects they teach, and the size of their classes.

Our analysis of these data reveals that teachers transfer from one school to another–or exit the Texas public school system altogether–more as a reaction to the characteristics of their students than in response to better salaries in other schools. This tends to leave disadvantaged, low-achieving students with relatively inexperienced teachers. Because teachers appear so unresponsive to salary levels, it would take enormous across-the-board increases to stem these flows. Indeed, the results suggest that policymakers ought to consider selective pay increases, preferably keyed to quality, for work in inner-city schools, together with efforts to improve the working conditions in these schools.

Reasons for Leaving

Teachers decide whether to remain at a school for a multiplicity of reasons, which can be divided into four main categories: 1) characteristics of the job, including salary and working conditions; 2) alternative job opportunities; 3) teachers’ own job and family preferences; and 4) school districts’ personnel policies. Although we were not able to look at the ways in which all of these factors affect teachers’ decisions with respect to their employment situation, we were able to examine directly the impact of salary and certain working conditions. We were also able to draw some reasonable inferences about how family considerations and alternative job opportunities influence their decisions by examining how teachers’ choices differ by gender and experience.

Admittedly, working conditions is a broad concept that can cover everything from class size to discipline problems to student achievement levels. Though we do not have data on every aspect of teachers’ working conditions, we do know certain characteristics of their students that many believe affect the teaching conditions at a school: the percentage of low-income students at the school (as estimated by the percentage eligible for a subsidized lunch), the shares of students who are African-American or Hispanic, average student test scores, and class sizes. Whether these characteristics directly affect teachers’ decisionmaking or indicate other less tangible factors (such as the disciplinary climate or bureaucratic environment at the school) cannot be determined.

When looking at the impact of working conditions on retention rates, one needs to take into account other factors that may affect teachers’ employment choices. Some teachers possess skills that are considered valuable in the marketplace. For instance, math and science teachers may find more competition for their services in the private sector than an English teacher would. However, our study focuses on elementary teachers, who tend to have similar educational backgrounds and similar opportunities outside the education system. As a result, differences in opportunities among teachers of different subjects should not be very important for this analysis.

A more important consideration is that many teachers may wish to remain at a particular location for other than job-related reasons, perhaps out of a desire to live near their hometown or near their spouse’s workplace. Consequently the availability of jobs in the locality may be an important determinant of the probability of exiting a school, and the control for any systematic differences across regions within Texas.

Retention rates can also be affected by the number of years a teacher has spent in a particular location. The more years working in a particular district, the more costly it becomes to leave, simply because pay, responsibilities, and job opportunities are often tied directly to experience within the same school district. The financial attractiveness of moving elsewhere also attenuates with the passage of time. Because many districts credit a transferring teacher with only a limited number of years of experience, teachers may have to take a salary cut if they switch school districts. In general, switching careers grows costlier with age and experience. One must give up the higher salary that comes with experience within a particular field, and the time to accumulate gains from any change in job or career grows shorter as one ages. For this reason, our analysis takes into account the number of years teachers have held their jobs by comparing only teachers with similar levels of experience.

Other relevant differences among teachers may arise from their family circumstances, such as the job opportunities of a spouse or a desire to stay home with young children or to enjoy the benefits of home ownership. For example, many female teachers who leave teaching do so in order to leave the labor market altogether, often for family reasons. We unfortunately lack information on family structure, sources of income other than salary, the location or type of housing, and whether and where a spouse works. However, we are able to look separately at teachers grouped by gender, giving us an opportunity to assess the extent to which female and male teachers are influenced by different considerations.

Ethnicity may also affect decisionmaking. Teachers may prefer to teach in schools where they share the ethnic characteristics of students, or they may find it easier to obtain a position if administrators prefer instructors who have certain ethnic characteristics. To ascertain whether ethnic background affects teachers’ decisionmaking, we also look separately at white, African-American, and Hispanic teachers.

One limitation of our study is that we do not have direct information on school districts’ hiring and retention practices. Districts have options when hiring, and the willingness of a teacher to leave a position will depend on the availability of an attractive position elsewhere. Although few teachers are involuntarily separated from their jobs, we do not know whether a job change is determined primarily by a teacher’s decision or by that of the employer, and the circumstances undoubtedly affect both opportunities and the range of choices a teacher will consider. Our lack of information about employer-initiated moves may lead to an underestimate of the improvements in pay and working conditions achieved by teachers who move voluntarily, but the size of this underestimate is probably not very large.

Movement between and within Districts

Nationwide, approximately one-fifth of all teachers decide to leave the school at which they are teaching each year. The pattern in Texas is roughly the same as in the nation as a whole. On average, in each year between 1993 and 1996, more than 18 percent of Texas teachers decided not to remain at the school at which they were teaching. More than 6 percent changed schools within their districts, another 5 percent switched from one district to another, and 7 percent left Texas public schools altogether.

Let’s look first at the changes in salary typically experienced by teachers moving to a new district. Instead of relying on salary data reported for each individual teacher, we calculate district average salaries for teachers in each of their first ten years of experience during the period from 1993 to 1996. These averages are based on regular pay for teachers without advanced degrees and exclude extra pay for coaching or other activities. (The latter is not an important part of compensation: more than 85 percent of teachers receive no extra pay, and the median extra pay for those who do receive it is about $1,000 per year.) We use these averages to characterize the salary schedule of each district and then estimate the potential salary change resulting from a move, given the experience level of each teacher. For example, the salary change for a teacher who switches districts after four years of teaching is assumed to equal the average salary of fifth-year teachers in the new district minus the salary for that level of experience in the old district.

On average, teachers who move between districts after no more than two years at a school improve their salaries, though just barely. Male teachers gain 1.2 percent in salary, while women gain 0.7 percent. Even these small gains begin to disappear for teachers with more experience. Overall, the average annual salary gain among all teachers with less than ten years’ experience is 0.4 percent of annual salary, or roughly $100. Women with three to nine years of experience who decide to change districts actually take, on average, a small pay cut. In short, most teachers moving between districts do not receive substantially better pay in their new jobs.

The picture for working conditions is quite different. There is strong evidence that teachers moving between districts have the opportunity to teach higher-achieving, higher-income, nonminority students. The findings for achievement are the clearest and most consistent. The average job switcher moving from one district to another moved to a district whose average achievement was 0.07 standard deviations higher on the TAAS than the district the teacher left. (The difference is three percentile points on a 100-point scale.) The shares of the district’s students who were African-American, Hispanic, or low income also declined significantly for movers. On average, the districts to which teachers moved had 2 percentage points fewer African-American students and 4.4 percentage points fewer Hispanic students than the districts they had left. The percentage of low-income students in movers’ districts fell by more than 6 percent.

These patterns were even more pronounced for teachers who moved from urban to suburban districts. The salaries of such teachers actually declined by 0.7 percent, on average, as a result of their moves. Meanwhile, the average achievement in the new districts increased by 0.35 standard deviations (14 percentile points), and the shares of African-American and Hispanic students decreased by 14 and 20 percentage points, respectively. Teachers who moved between different suburban districts experienced similar, albeit smaller, changes in student characteristics. Student achievement in their new districts was one-tenth of a standard deviation higher, while the percentages of African-American, Hispanic, and economically disadvantaged students all declined.

We can gain further insight into the factors associated with teacher mobility by examining the pre- and post-move school characteristics for teachers moving to a new school within the same district. These results confirm that teachers who move between schools within urban districts typically arrive at a school with higher average student achievement (0.11 standard deviations) and a smaller percentage of minority and low-income students. In other words, those who choose to change schools within districts appear to follow the same attributes, seeking out schools with fewer academically and economically disadvantaged students. These patterns are also consistent with the notion that new teachers are often placed in the most difficult teaching situations and that senior teachers can often choose comfier positions within the system.

Important differences emerge, however, when we separate teachers by their own ethnic background. African-American teachers tend to move to schools with higher percentages of African-American enrollment than their previous schools, regardless of whether they change districts or simply move to a new school in the same district. However, the average change in the percentage of Hispanic students for teachers of Hispanic descent is not much different from the changes experienced by teachers as a whole. The typical gap in average test scores between their current and former school is also much smaller for African-American and Hispanic teachers who have switched schools.

It is not clear whether these ethnic differences are the result of teachers’ preferences or of the job opportunities available to them. It could be that African-American teachers prefer to work at a school near where they live. If so, then residential segregation by race may lead to the selection of schools with more African-American students. Or teachers may simply prefer to teach students of a similar ethnic background. Alternatively, job opportunities for African-American teachers may be more extensive in schools with higher proportions of African-American students.

All this movement of teachers among schools obviously affects the composition of the teaching force at particular schools. Since exiting rates are smaller at schools with more advantaged students, these schools also enjoy more experienced teachers. The pattern is particularly striking when schools are grouped according to their average level of student achievement. As Figure 1 shows, almost 20 percent of teachers in schools in the bottom quartile of student achievement leave their schools each year, while in the top-quartile schools only 15 percent leave. The driving force of this relationship is not teachers’ leaving urban districts for suburban ones; on the contrary, most of the difference in leaving rates between these types of schools is caused by teachers moving to new schools within their original district. Since teachers with fewer than two years of experience tend to be less effective than more experienced teachers, existing mobility patterns in Texas are likely to adversely affect the achievement of disadvantaged students.

Salaries and Student Demographics

The analysis to this point has not disentangled the effects of salaries from the effects of the working conditions associated with students of varying achievement and family backgrounds. To identify more precisely the independent effects of the multiple factors affecting teachers’ choices, we use regression analysis to estimate the separate effects of salary differences and school characteristics on the probability that a teacher will leave a school district in a given year, holding constant a variety of other factors, including class size and the type of community (urban, suburban, or rural) in which the district is located. We also compare the impact of salaries and school characteristics on the probability of switching to another district with their impact on the probability of leaving teaching altogether.

The results of this analysis confirm that teachers are more likely to leave districts with low average achievement scores. Ethnic composition of the student body is also an important determinant both of the probability of leaving the public schools entirely and of switching from one school district to another. White teachers, regardless of their teaching experience, will tend to move to schools with fewer African-American and Hispanic students. Less experienced white teachers are also more likely to leave the public schools altogether if they come from schools with higher concentrations of African-American and Hispanic students. For African-American and Hispanic teachers, the reactions to varying concentrations of African-American students are almost exactly the opposite.

The differential effect of the ethnic composition of the student body for white and African-American teachers could reflect personnel policies that prefer minority teachers in schools with higher concentrations of minority students. But teachers’ own preferences may be even more important, as suggested by the fact that the decision to leave the Texas public schools altogether–a decision much more closely related to the individual teacher’s preferences than to the district–is influenced in the same way by the schools’ ethnic composition.

If the ethnic composition of the school is the most important factor affecting teachers’ decisions to change jobs, financial considerations are also relevant, especially when it comes to a decision by a somewhat less experienced male teacher to move from one district to another. For male teachers with fewer than three years of experience, the estimated change in the probability of switching districts for a 10 percent increase in salary is 2.6 percentage points; for men with three to five years of experience, the estimated change for a salary increase of the same magnitude is 3.4 percentage points; for still more experienced male teachers, financial effects trail off, down to essentially zero for those with more than 20 years of experience.

The results indicate that higher salaries significantly reduce the probability that male teachers will leave a district. The magnitude of the effect is largest for those early in their career. By contrast, the effects of salary difference for more experienced women teachers are significantly smaller. While females in their first five years of teaching are somewhat responsive to salary differences, such differentials have no observable effects on those with six or more years of experience.

In short, the financial impact on the decisions of female teachers is less than half that for men. Because they represent the vast majority of elementary teachers, women’s unresponsiveness to financial differentials is important to the subsequent policy discussion.

Policy Implications

The results presented above confirm the difficulty that schools serving academically disadvantaged students have in retaining teachers, particularly those early in their careers. Teaching lower-achieving students–whether because teachers find it more difficult or less rewarding–is a strong factor in decisions to leave Texas public schools, and the magnitude of the effect holds across the full range of teachers’ experience levels. There is also strong evidence that a higher rate of minority enrollment increases the probability that white teachers will leave a school. By contrast, increases in the shares of African-American and Hispanic students reduce the probability that African-American and Hispanic teachers will leave.

Given these findings, a key question is how to reduce the flows out of low-achieving, high-minority schools and out of the teaching profession altogether. One oft-proposed solution is to provide teachers with “combat pay”–salary increments designed to encourage them to remain at a tough school. But how large would the increase need to be in order to neutralize the effects of difficult working conditions? Let’s consider this closely.

The situation is complicated by the fact that most elementary-school teachers in Texas are white females (only 20 percent are African-American or Hispanic, while only 14 percent are male). As noted earlier, female teachers are less responsive to increases in salary, meaning that the bonus required to keep them at a school will be larger than for males. In addition, white teachers are the most likely to exit low-achieving, high-minority schools, meaning that it will take even larger increases to retain them. If the teaching corps looked much different–say, if the teachers in urban elementary schools were mostly African-American and Hispanic males–the costs of the “combat pay” solution would be lower.

Based on our findings of what causes teachers to leave their schools, we calculated the salary increases that would be necessary to offset the effects of difficult working conditions in large urban versus suburban schools. These calculations, performed separately for white male and female teachers in their early careers, are shown in Figure 2. The findings suggest that truly large boosts in salary would be needed, particularly for women. Female teachers in large urban school districts would require a 25 percent initial increase in compensation, rising to more than 40 percent when they reach three to five years of experience. Moreover, this is only in the “typical” urban school. For the neediest or most troubled schools in urban areas, even the differentials calculated in Figure 2 would probably not be sufficient to stem the high levels of turnover in such schools.

Not only would across-the-board salary increases of 25 to 40 percent for teachers in urban areas be an enormously expensive reform, but it would be difficult to target such a solution, since teachers typically negotiate salary schedules that apply to all the teachers in the district, not just to those in the most disadvantaged schools. Similarly, even if targeted to the most disadvantaged schools, any increases in salaries would almost certainly go to new and middle-career teachers alike, even though our results indicate that salary differentials are nearly irrelevant for women teachers with ten or more years of experience.

As a result, improving the working conditions of teachers may prove both more effective and more realistic. Unfortunately, at this time, we do not fully understand the working conditions that are most important. But, to the extent that other characteristics of schools where disadvantaged students are found–such as safety and disciplinary problems, more bureaucratic rules, poor leadership, greater student turnover, or a greater distance to work–are important elements, improving these working conditions could mitigate the turnover problem we have identified. And these improvements might have their own effect on student performance.

Finally, it is important to note that this study focuses solely on how many teachers move among schools and out of teaching. We have not examined the quality of the teachers who move from one district to another or leave teaching altogether. The actual cost of improving the quality of instruction depends crucially on whether good teachers, not just experienced teachers, are being retained. Salary policies that are guided just by the characteristics of the students in a school will retain both the good and the bad teachers.

We do know from our other work that differences in teacher quality are more significant than the differences arising from having inexperienced teachers. Therefore, an approach with more appeal might be simply to accept the fact that there may be greater turnover in schools serving a larger disadvantaged population, but then to concentrate much more attention and resources on the quality dimension. While we do not have much experience with such policies, they seem like the most feasible way to deal with the problems of schools serving low-income and minority students.

-Eric A. Hanushek is a senior fellow at the Hoover Institution, and Steven G. Rivkin is an associate professor of economics at Amherst College. The late John F. Kain was a professor of economics and political economy at the University of Texas at Dallas.

“The New York Adequacy Study: Determining the Cost of Providing All Children in New York an Adequate Education,” American Institutes for Research and Management Analysis and Planning (March 2004).

“Resource Adequacy Study for the New York State Commission on Education Reform,” Standard & Poor’s School Evaluation Service (March 2004).

“Report and Recommendations of the Judicial Referees,” in Campaign for Fiscal Equity, Inc., et al., Plaintiffs, against The State ofNew York, et al., Defendants (November 2004).

Most people who read the headlines last February were stunned to learn that New York City schools were being shortchanged by $5.6 billion per year, or more than $5,000 per student. The 43 percent court-ordered budget increase, from around $13 billion in operating expenditures to something approaching $19 billion (not including some $9 billion over five years for building improvements), is the largest school finance “adequacy” judgment ever awarded.

Of course, most people do not have a good grasp on either the economics or the performance of New York City schools. If they did, they would be even more stunned by the declared shortfall.

Figure 1 shows the recent history of spending in New York City, now nearly $13,000 per student per year, which is than 50 percent above the national average and pulling away.

SOURCE: New York State Education Department; National Center for Education Statistics

The city does, by any standard, face huge education problems. Indeed, despite a drastic restructuring of the school bureaucracy, implemented by Mayor Michael Bloomberg beginning in 2002 (see Forum), and despite the heavy infusions of cash shown described in Figure 1, Gotham’s academic outcomes remain poor. On the 2003 National Assessment of Educational Progress (NAEP) tests, 46 percent of the city’s students scored “below basic” in mathematics, and 38 percent were below that low threshold in reading (compared with 33 and 28 percent for the nation, respectively). On the state exams that can be tracked over time, New York City has had mixed results—improvement in some areas but declines elsewhere.

But the discrepancy between years of budget increases and years of mediocre academic outcomes did not deter New York State Supreme Court Judge Leland DeGrasse from deciding that the problem could be solved by an annual addition of $5.6 billion.

The very process of budget determination implicit in such judicial appropriations gives the first indication that something is fundamentally haywire. Ordinarily, courts have nothing to do with expenditures. That is a matter for the political branches, not the courts, to decide—a constitutional arrangement that led that great New Yorker, Alexander Hamilton, to declare the judiciary the weakest branch. In New York, as in all other states, the normal appropriations process begins with the governor’s creating the budget recommendations for education and other state services. The legislature, subject to gubernatorial veto, appropriates the funds. But such constitutional proprieties were set aside when Judge DeGrasse—with no previous education expertise and no relevant staff support and without considering the impact on other areas of expenditure—intervened to establish the level of education appropriations for New York City. Suddenly the weakest branch had declared itself the boss.

Given the fundamental constitutional conflict involved, this judicial decision will probably be in and out of the courts and legislature for some time. To get some hint of the future, one may look no farther than neighboring New Jersey, where the courts have retained control over the financing of several city school districts for decades.

Nonetheless, it is informative to investigate what is behind the DeGrasse appropriations, because New York is only the leading edge of a national movement. In more than two-thirds of the states, teacher unions, school districts, and other interested parties have filed similar lawsuits that seek judgments resembling the stunning result handed down in New York.

The DeGrasse judgment is the result of a decade-long political and legal struggle (described by New York Daily Newsreporter Joe Williams in this journal earlier this year: “The Legal Cash Machine,” Education Next, Summer 2005). Several groups, led by the Campaign for Fiscal Equity (CFE), a nonprofit legal advocacy organization, filed suit in 1993 claiming that New York State was depriving New York City public school students of their constitutional rights to a “sound basic education,” a standard that had been prescribed in 1982 by the state’s highest court (in New York, the Court of Appeals). Despite its name, the lead plaintiff in the 1993 complaint, CFE, did not argue that the state’s financing arrangements were inequitable, but that the funds given to New York City were not “adequate” for a sound basic education. From his Manhattan courtroom, Judge DeGrasse sided with the plaintiffs. The decision was ultimately upheld by the Court of Appeals, which remanded the case to DeGrasse to ensure that the Constitution was served; hence his appropriations figure.

But the interesting question, ignored in all the righteous hoopla over the court decision, is: Where did Judge DeGrasse get that $5.6 billion figure? Why not $10 billion? Or just $1 billion? How much does a sound basic education cost?

The Inexact Science of Costing Out

The paternity of the $5.6 billion figure is easily traced to the plaintiffs in the case, whose expertise was treated as authoritative, despite their obvious vested interest in the outcome. The Campaign for Fiscal Equity had commissioned a costing-out study by a consortium of two consulting firms, the American Institutes for Research (AIR) and Management Analysis and Planning, Inc. (MAP). Both firms claimed to have the analytical capacity to determine objectively the funding schools need to perform adequately. The consortium, known as AIR/MAP, made the extraordinary claim in its November 2002 proposal that its study would answer the question, “What does it actually cost to provide the resources that each school needs to allow its students to meet the achievement levels specified in the Regents Learning Standards?”

The following year AIR/MAP submitted its final costing-out analysis to its client, the plaintiff, who then submitted the document to the court by way of the panel of three referees appointed by Judge DeGrasse to assist in fashioning an appropriate remedy. These referees were a Fordham Law School dean and two retired New York judges, none with any particular expertise in school finance. After an intensive and expensive period (the three referees submitted combined bills in excess of $350,000 for their part-time work over the course of four months), they issued a 57-page report accepting the essential elements of the AIR/MAP document that CFE had submitted to the court. The referees recommended that funding of New York City schools be ramped up an additional $5.6 billion a year within four years; that new studies be undertaken every four years to find out how much, if any, additional funding would be required; that $9.2 billion be spent for capital projects spread over the following five years; and that another study be conducted after five years to see if additional spending was required.

Both Judge DeGrasse and the mainstream New York City media covering the story treated the referees’ report as authoritative. Little attention was given to the other studies reviewed by the referees that recommended quite different levels of expenditure. One might have thought the referees would give at least equal consideration to the report submitted by the New York State Commission on Education Reform appointed by Governor George Pataki. Known as the Zarb Commission, after its chairman, Frank G. Zarb, a former chairman of NASDAQ, the commission estimated that the city needed $1.9 billion to provide an adequate education. Meanwhile, the City of New York, eager to get as much state money as possible, proposed additional spending of $5.4 billion, an amount that resembled the AIR/MAP recommendation. It added the caveat that none of this funding should come from the city. Not to be left out, the New York State Board of Regents calculated its own figure, $3.8 billion. Even Standard & Poor’s jumped in, with an independent study that included 16 different estimates for the resource gap, ranging from as high as $7.3 billion to as low as $1.9 billion, depending on achievement targets, regional cost adjustments, and cost effectiveness of districts. The Zarb Commission, in fact, used the lowest of S&P’s estimates as the basis for its own recommendation.

This range of estimates underscores the arbitrary nature of any number the court would order the legislature to spend. Even the plaintiff’s own consultant, AIR/MAP, admitted that its “‘costing out’ methods are not based on an exact science.” Far from being an exact science, the method they chose, as we shall see, was profoundly subjective, a matter of judgment by and for self-interested parties.

Aligning Professional Judgment and Self-Interest

The AIR/MAP study relied on the “professional judgment” method in its costing-out analysis. The consultants brought together multiple panels of school personnel and asked them to design a program that would ensure that all New York City students could get a sound basic education and determine the resources needed to deliver the program. But these program designers, 56 in all, were also service providers whose pay, working conditions, and other funds were directly dependent on the resources put into the system. Such a procedure is akin to asking Martha Stewart how much you should pay for her to decorate her own house. When someone else is to pay, and Martha is to enjoy, one can only expect the sky to be the limit.

Admittedly, not all 56 panelists worked within the New York City school system. But all except one, a retired employee, were currently working somewhere within the New York State school system. Since the panelists were asked to cost out programs statewide (presumably in anticipation that any financing changes would spill over to districts outside New York City), the conflict of interest could hardly be more direct, unless the panelists had been paid for their labors in proportion to the amount they recommended.

These arguments are not against professional judgment per se, but against its misuse in this case. There is a big difference between asking professional educators to make education decisions and resource allocations within the constraints of a fixed budget and asking them to determine what that budget should be. The former endeavor is what they traditionally do, exactly where the professional judgment of an administrator might be helpful, just as it would be useful to have Martha Stewart’s decorating opinion. But that opinion is solicited after a fixed budget has been set. Asking the professional educators to determine the budget only guarantees solutions that retain the basic organization of the current system, including the existing incentive structure. After all, it is a structure that the participants have accepted and to which they have grown accustomed.

Notably, the AIR/MAP approach did not consider any ways of reconfiguring the education system so as to make it more efficient. Instead, it assumed that existing arrangements were fixed and made their best guess as to how much more money that system might need to get the job done. Not surprisingly, the professionals’ recommendations included such nostrums as paying employees (themselves) more and giving them less work to do (reducing class size). The notion that the city’s current stable of teachers should be paid more is particularly ironic, given that much of the plaintiffs’ evidence at trial was devoted to documenting their shortcomings. Moreover, research has shown that any of these steps would cost a fortune, far beyond any reasonable expectations of achieving adequate performance levels. The professional judgment panels paid such research no attention whatsoever.

Substituting Self-Interested Judgment for Data

The AIR/MAP analytic approach ignores ample evidence from New York indicating the absence of a clear connection between performance and expenditure. Take, for example, the percentage of students in a district who obtain a Regents’ diploma, a key measure of education quality in New York. Districts that are higher performing by this indicator actually spend, on average, no more than the lower performing districts (after adjustment for differences in family income, special-education placements, and the percentage of students who are of limited English proficiency). Thus the normal operations of districts in the state give no indication that increasing expenditure alone would necessarily enhance student achievement.

Now consider New York City itself. The judicial referees call for a 43 percent increase in spending. Between 1998 and 2003, as Figure 1 shows, expenditures in New York City increased by almost exactly that amount, 44 percent, an increase that surpassed the rate of increase for the state as a whole and for the nation. If money is the answer, this history should help foretell the results of the next infusion. But as Figures 2a through 3b demonstrate, student passing rates in reading and math for New York City students have remained barely above 50 percent—in fact, have worsened in 8th-grade reading. Whatever small gains have occurred, they hardly support the conclusion that spending increases constitute the solution to the city’s inadequate schools. Perhaps these numbers led AIR/MAP to qualify their findings so dramatically as to undermine the validity of their study:

The success of schools also depends on other individuals and institutions to provide the health, intellectual stimulus, and family support on which public school systems can build. Schools cannot and do not perform their role in a vacuum. Furthermore, schools’ success depends on effective allocation of resources and implementation of programs in school districts.

If more resources are not sufficient, what is the evidence that they are necessary? Are there reasons to believe that the next 40-plus-percent spending increase will have a greater impact than the last? Or should we expect the next quadrennial costing-out study to call for yet another 40-plus-percent increase in spending to meet the achievement goals?

SOURCE: New York State Education Department

SOURCE: New York State Education Department

S&P’s Successful Schools Model

The Standard & Poor’s study relied on  the “successful schools” method, focusing on observed costs for a set of New York districts that obtain good student outcomes. Even after allowing for the cost of educating students with special needs, S&P’s analysis showed a wide dispersion across school districts in the spending observed to achieve equivalent outcomes. The lower-spending half of successful districts spent 50 percent less than the higher-spending districts, proving that many good schools do quite well with much less than other schools. Recognizing this, the Zarb Commission went with the average expenditures of the lower-spending half of the successful districts.

The definition of success is particularly relevant to understanding the synthesis of the different approaches, since, as noted, the full S&P analysis considered a variety of possible definitions of “successful schools.” The Zarb Commission relied on the set of school districts meeting the Regents’ operational definition of an adequate education: 80 percent of their 4th graders passed the math and English exams and passed five of the high-school graduation tests. This definition of the objective of an adequate education was consistent with the court’s decision on how to interpret the requirement of a sound basic education.

Curiously, however, AIR/MAP defined a sound basic education quite differently. It determined that a successful school district was one in which all students meet the full Regents Learning Standards, a much higher bar that moved the 80 percent pass rate to 100 percent. That measure was explicitly rejected in the New York Court of Appeals decision, which the referees were being asked to implement.

Meeting more stringent standards should clearly cost more than meeting the lesser standards. Yet the referees, by carefully selecting and modifying components of the S&P study, were pleased that they could extract similar estimates of adequate funding requirements from the various studies. They state, “This relative convergence of costing-out results derived from three different methods—the successful school district method used in the State’s costing-out analysis, the professional judgment method used in plaintiffs’ costing-out analysis, and the City’s detailed planning method—provides comfort that our $5.63 billion costing-out recommendation to the Court is indeed sound.”

If the costing-out studies have any validity, the cost of achieving very different outcomes should not be the same.

Why Worry about Efficiency?

The most basic problem is the absence of a scientific method in the application of the costing-out models. The reasonable scientific question is, “What level of funding would be required to achieve a given level of student performance?” In fact, there is no evidence to suggest that the methodology used in any of the existing costing-out approaches, including the two considered here, is capable of answering that question.

The existing analyses never consider the minimum cost, or efficient level of spending, needed to achieve the desired outcome. Instead, they are fixated on identifying any policies that might lead to an improvement in performance, almost without regard to the magnitude of gains or cost. The focus on minimal required spending is a necessary ingredient, because without this restriction the question of cost is completely arbitrary (and thus beyond science). Actual spending to achieve an outcome can obviously range anywhere from the efficient level to infinity. But none of the available methodologies focuses on the efficient spending required for any given performance level.

Moreover, locking in the current technology (through professional judgment or successful schools) can at best produce marginal changes in outcomes. Overcoming the deficits illustrated in Figures 2a through 3b will require more dramatic improvements.

Consider again the AIR/MAP analysis. There is, first, no demonstration that the schools that employ the panel members are using their funds in a particularly effective manner or that their experiences indicate they have the data to answer the “level of funding” question. Second, there is no way to replicate the wish lists of the specific panels, because they are based solely on personal opinions of the selected panelists and not on any data about school operations.

More important, the specific approach of AIR/MAP for combining the judgments of the separate professional judgment panels led directly to costing out the “italic”>maximum, not minimum, recommended resource use to achieve the Regents Learning Standards. Thus, ignoring whether the choices would conceivably lead to the desired outcomes, the methodology necessarily produced a biased answer, albeit one that suited the interests of the clients.

The referees seemed unconstrained by any of this logic, however. The state, using S&P’s estimates, had suggested that it was reasonable to concentrate on the spending patterns of the most efficient of the successful schools, those that did well with lower expenditure, and thus excluded the top half of the spending distribution in its calculations. But when the referees attempted to reconcile the state’s recommendation of $1.9 billion with the AIR/MAP estimates of more than five billion dollars, they insisted on adding in all the high-spending districts, even when such districts did not produce better academic outcomes. Thus they forced on S&P an inefficiency standard that, on its face, violates the premise of the successful schools model. After all, the referees reasoned, “there was no evidence whatsoever indicating that the higher spending districts … were in fact inefficient.” In other words, spending more to achieve the same outcomes should not be construed as being inefficient. One might then ask, What would indicate inefficiency?

Perhaps, however, the top-spending districts are using the money for some unmeasured reason. If so, this would only magnify the analytical problem, for if the top-spending districts are not comparable, then their spending level does not indicate what would happen if funds were added to a typical district. It would not reflect the causal effect of added funds on student outcomes, but rather the effects of unknown underlying differences between the districts. But, again, neither AIR/MAP nor the referees made any use of historical data, so no consideration of variations in spending across districts entered their deliberations.

Furthermore, in neither the successful schools nor the professional judgment methodologies is there a sense that the results of the successful districts could be reproduced without instituting a host of reforms (unmentioned by the referees) to ensure that the extra money led to better schools. In fact, the multiplicity of high-spending/low-achievement districts would seem to indicate that money is decidedly not the measure of a good school, that the approach fails on fundamental grounds of science.

To avoid the dead end that both logic and the facts create for costing-out proponents, the referees use a clever bit of language throughout their report. They calculate the amountof annual funding required “to provide all New York City school children the opportunity for a sound basic education”. They never say that the spending they propose will achieve the desired results. Such a statement, or rather, such an omission, clearly suggests that the referees and Judge DeGrasse are not interested in improving student outcomes as much as they are in equalizing opportunities for inefficiency. Unfortunately, doubling the dosage of an ineffective pill seldom provides an effective cure.

Just Send the Money

The courts, of course, do not condone wasting funds. In fact, court judgments about school finance frequently contain explicit notes cautioning that the funds will lead to improvements only if they are used effectively. Such tautological statements seldom recognize that New York City (and other states under judgments) have no history of spending funds effectively.

At the same time, the objective is a serious one. The education problems in New York City (and a number of other jurisdictions that face court financing challenges) are real and important. Many people would indeed be willing to put more money into New York City schools (or any poorly performing school for that matter) if they had any reason to believe that students’ achievement would improve significantly.

Unfortunately, addressing these problems by simply augmenting the current system, which has virtually nonexistent performance incentives, will not solve the problems. At such a critical juncture, students and taxpayers alike deserve an approach that embraces the best of what we already know about investments in public schooling that work. This is not ensured by any of the legal proceedings to date.

In the end, the big difficulty with the costing-out exercise is that it purports to provide something that cannot currently be provided: a scientific assessment of what spending is needed to bring about dramatic improvements in student performance. By their very nature such studies provide little information about the costs of achieving improvements efficiently. They contain nary a word about changing the reward structure for teachers (other than paying everybody more). They avoid any consideration of accountability systems based on student outcomes. And they lack any appropriate empirical basis.

Asking the courts or, more precisely, outside consultants to provide a scientific answer to the question of how much should be spent on schools is irresponsible. Decisions on how much to spend on education are not scientific questions, and theycannot be answered with methods that effectively rule out all discussion of reforms that might make the school system more efficient.

Even the weak statement from the New York Court of Appeals that new accountability should accompany added funding was met with indifference by the judicial referees, who accepted the thrust of Mayor Bloomberg’s testimony when he appeared before them: he is already accountable through the electoral system, so just send the money.

-Eric A. Hanushek is a senior fellow at the Hoover Institution, Stanford University, and a member of its Koret Task Force on K–12 Education.