Policymakers and reform advocates alike have rallied around the movement toward a national curriculum, suggesting that this will break the stagnation in achievement of U.S. students.  But there is little evidence that confusion about what we should teach has been a real inhibition to student achievement.  In fact, the existing evidence suggests just the opposite:  there is no relationship between the learning standards of the states and student performance.

To be sure, it is a real problem when students in one state learn very different things than those in other states, and in particular when students from some states lack the skills needed for our modern economy.  We really do have a national labor market, and significant numbers of our population end up living and working in a state different than that where they were born and went to school.  The presumption behind having national standards (whether voluntary or coerced) is that having a clearer and more consistent statement of learning objectives across states would tend to lessen the problem of heterogeneous skills that students bring to the labor market.  Again, however, the fundamental problem is lack of minimal skills and not the heterogeneity of skills per se.

Experience provides little support for the argument that just more clearly declaring what we want children to learn will have much impact.   In arguing for focusing on standards, proponents of national standards conventionally point to Massachusetts:  strong standards and top results.  But it is useful to expand thinking from just Massachusetts to include California, a second state noted for its high learning standards.  Indeed, some have argued that both states would have to lower their standards in order to fit into the structure of the Common Core.  But California balances Massachusetts:  strong standards and bottom results.

In order to see the issue more broadly, it is possible to compare state-by-state measures of learning standards to student outcomes.  There are different independent ratings of the quality of the learning standards currently existing for each state, and these can be combined with assessments of student performance from the National Assessment of Educational Progress (NAEP).  The most comprehensive rating of state standards is probably that of Education Week.   Education Week developed a comprehensive grading across grade-specific standards, testing, and the accountability that goes with them in each state.  This ranking provides aggregate grades for each state.  (Another widely acknowledged rating of state standards by subject is produced by the Fordham Institute.  These competing rankings are correlated with those of Education Week, though not perfectly, and it really makes no difference for the analysis which we use.)

The figure below shows how the ranking of standards compares to NAEP scores – here the 8^th grade math scores.  (The specific NAEP assessment for grade and subject has no influence on the overall conclusions).

As can be seen, the better the state standards the worse the students tend to do.  But, of course, this does not imply that we should move toward weaker standards.  The real conclusion is that state standards have little to do with student performance.

In other words, what really matters is what is actually taught in the classroom.  Simply setting a different goal – even if backed by intensive professional development, new textbooks, and the like – has not historically had much influence as we look across state outcomes.

There are a number of refinements that one can think about for this analysis, but they do not change the answer.  This conclusion holds even under more sophisticated analysis, as demonstrated quite conclusively by Tom Loveless of the Brookings Institution.  Indeed his analysis helps to frame the entire debate.

The continuing emphasis on Common Core standards, including the debates about the legality of them, is often interpreted as indicating that the Common Core is a really big deal in school reform.  The data suggest otherwise.

The one possible complementary gain from the move to national standards is that the assessments of performance might become better.  It is widely recognized that the current tests used to judge outcomes within individual states tend to be quite weak.  (This concern about tests is not leveled at NAEP, which was used in the comparisons above, but instead applies to the tests states use for accountability purposes).  If the new standards lead to better tests – something that might come out of the two testing consortia funded by the U.S. Education Department – we might have the basis for improved school policies.  But that is also not certain and cannot be used as a primary justification for the focus on Common Core standards.

One interpretation of the emphasis on developing the Common Core curriculum is that these debates provide a convenient distraction from potentially more intractable fights over bigger reform ideas like teacher evaluations, expanded school choice, or improved accountability systems.    While I am not against having better learning standards, I believe that we cannot be distracted from more fundamental reform of our schools.  The future economic well-being of the U.S. is dependent on improving the achievement and skills of today’s students.

-Eric Hanushek

Then came the meteoric shock of consequential accountability, and student test scores (on the National Assessment of Educational Progress and state exams, too) started to take off. For some subgroups of students, math and reading skills improved by two or three grade levels since just the mid 1990s.

Yet we all know the downsides of the narrow focus on reading and math scores in grades three through eight and once in high school. This regimen puts enormous pressure on schools to ignore or exclude other important subjects (art, music, history, even science). It penalizes schools with an educational strategy that succeeds in the long term but doesn’t produce sky-high scores now. (I’m thinking of Waldorf schools, for instance, such as the preschool my son attends.) And it undervalues other important contributions that schools make, such as to students’ character development and social skills.

When it comes to evaluating teachers, there’s wide agreement that we need to look at student achievement results—but not exclusively. Teaching is a very human act; evaluating good teaching takes human judgment—and the teacher’s role in the school’s life, and her students’ lives, goes beyond measurable academic gains. Thus the interest in regular observations by principals and/or master teachers. These folks can pick up on nuances missed by the value-added data—plus can provide actionable feedback to instructors so that they can improve their craft. (Harrison School District Two in Colorado has one of the best plans in this regard.)

So why do we assume, when it comes to evaluating schools, that we must look at numbers alone? Sure, there have been calls to build additional indicators, beyond test scores, into school grading systems. These might include graduation rates, student or teacher attendance rates, results from student surveys, AP course-taking or exam-passing rates, etc. Our own recent paper on model state accountability systems offers quite a few ideas along these lines. This is all well and good.

But it’s not enough. It still assumes that we can take discrete bits of data and spit out a credible assessment of organizations as complex as schools. That’s not the way it works in businesses, famous for their “bottom lines.” Fund managers don’t just look at the profit and loss statements for the companies in which they invest. They send analysts to go visit with the team, hear about their strategy, kick the tires, talk to insiders, find out what’s really going on. Their assessment starts with the numbers, but it doesn’t end there.

So it should be with school accountability systems. The best ones today take various data points and turn them into user-friendly letter grades, easily understandable by educators, parents, and taxpayers alike. So far so good. Why not add a human component to the process, via school inspectors like those in England? (See this excellent Education Sector paper, by my friend Craig Jerald, for background on how that works.)

Imagine: At least once a year (more would be better) a group of inspectors visits a school. (These would be professionals on contract with the state department of education—typically retired teachers and principals. In the case of charter schools, authorizers would be involved, too.) They would mostly look for two things:

• Evidence that the school is achieving important outcomes that may not be captured by the state accountability system. For example, the school’s administrators might show them test score data from a computer adaptive exam like NWEA’s that demonstrates progress for individual kids (especially those well above or below grade level) that isn’t picked up by the less-sensitive state test. Or perhaps a high school has compelling data about its graduates’ college matriculation and graduation rates that put its mediocre test scores in a different light.
• Indications that the school’s culture and instructional program are inculcating valuable attributes in their students. This is to guard against the “testing factory” phenomenon. Is the school offering a well-balanced curriculum (and extra-curriculars), or engaging in test-prep for weeks on end? Is it focused on teaching “non-cognitive” skills and attributes, such leadership, perseverance, and teamwork? Character traits like empathy, honesty, and courage?

The school visits should not be exercises in excuse-making. This isn’t about lowering expectations because of difficulties particular communities face, or delaying needed changes because the school’s educators appear to be “trying hard.” Rather, it’s a chance to round out the picture generated by the state’s (inevitably) incomplete accountability report.

So here’s how it would work: The state would develop school grades based on a variety of indicators, as it does now. Then those grades could be raised or lowered based on the findings of the school inspectors. (Generally just a letter-grade, but sometimes more.) Grades would go up because of evidence of strong outcomes not captured by the state accountability system; grades would go down because of evidence of unhealthy curricular narrowing.

Such a system would remain imperfect. Human judgment would introduce subjectivity and error into the process. Inspectors might face pressure (maybe even bribes) to raise schools’ grades. And it would be expensive—at least as compared to the testing-and-accountability systems we have now. These issues would need to be addressed.

Still, it’s worth it. To the extent that school grades (and consequences linked to them) drive policy and behavior, we ought to make sure that those grades are informed by more than just numbers. The correct response to the unintended consequences of accountability isn’t to end accountability, but to make it work better. That could have positive consequences for many years to come.

-Michael Petrilli

This post originally appeared on the Fordham Institute’s Flypaper blog.

• Its content-and-skill expectations for grades K-12 in English and math are, by almost everyone’s reckoning, about as rigorous as the best state-specific academic standards and superior to most.
• It was developed outside the federal government, voluntarily by states, using private dollars. (The related assessments are another matter.) And both standards and assessments remain voluntary for states.
• It opens the way, for the first time, to comparing student, school and district performance across the land on a credible, common metric—and gauging their achievement against that of youngsters in other countries on our shrinking and ever-more-competitive planet.
• Besides comparability, it brings the possibility that families moving around our highly mobile society will be able to enroll their kids seamlessly in schools that are teaching the same things at the same grade levels.

Ever since it landed, however, the Common Core has been the object of ceaseless attacks from multiple directions. The number of zealous assailants is small and, for a time, it all looked like a tempest in a highly visible teapot. That may yet turn out to be the case. But the attacks are growing fiercer; some recent recruits to the attack squad are people who tend to get taken seriously; and anything can happen in an election year. Remember the classic Peter Sellers movie, The Mouse That Roared? The Duchy of Grand Fenwick ended up triumphing over the United States of America. As you may recall, that happened in large part because the U.S. government contributed to its own defeat. In the present case, something similar could well transpire. Please read on.

Before examining the assaults, however, let’s remind ourselves what the Common Core is not. It is no guarantee of stronger student achievement or school performance. Huge challenges await any (serious) academic standards on the implementation, assessment and accountability fronts. To get traction in classrooms, states that adopt these standards (and all but four say they’re doing so) must take pains with curriculum, teacher preparation, assessment, accountability and more. To yield real rigor (and comparability), the currently-under-development assessments must avoid numerous pitfalls and incorporate hard-to-achieve consensus on genuinely challenging issues (such as where to set the “cut score”).

In and of themselves, academic standards merely describe the end point to be reached and the major stops en route. They don’t get you there. But it’s far better to have an education destination worth reaching, i.e. rigorous standards set forth with sufficient specificity, clarity, and rich content to provide real guidance to curriculum designers, classroom teachers, test developers and more. Few states have managed to do that on their own.

To be sure, other states could simply copy the best of those that already exist. But that’s more or less what the Common Core is: an amalgam of good standards put together by people who know a lot—and care a lot—about both content and skills.

So why the nonstop attacks against it? As best I can tell, they arise from six objections and fears.

First, a few earnest critics are convinced that the standards are substantively flawed, that the algebra sequence (or grade level) is wrong, the English standards don’t contain enough literature, the emphasis on “math facts” isn’t as strong as it should be, etc. This sort of thing has accompanied every past set of standards of every sort, and it’s perfectly legitimate. Insofar as such criticisms are warranted, the Common Core can be revised, states can add standards of their own, and jurisdictions that find the common version truly unsatisfactory can change their minds about using it at all.

Second, the Common Core will be difficult and expensive to implement. Many organizations are working hard to help states surmount these genuine challenges. Many philanthropists are kicking money into the effort. And some groups (Fordham included) are trying to cost it all out. Nobody denies that doing this right will be hard and costly (though some of those costs are already embedded in state and district budgets.) Of course, those who think the country is doing OK today have every reason to shirk that challenge and stick with what they’re used to.

Third, the Common Core won’t make any difference in student achievement—but may cause a politically-unacceptable level of student failure. As noted above, standards per se do not boost achievement. (Of course, standards per se don’t carry costs or failure rates, either. They don’t, by themselves, do much of anything!) And failure rates will worsen only if (a) the new assessments are truly rigorous and (b) schools neglect preparing their pupils to pass them.

Fourth, states have done as well, or better, on their own, and switching over to the Common Core will just mess them up. This criticism mostly emanates from Massachusetts, which has done a commendable job on its own and where the decision to adopt the Common Core was truly conflicted. Other states that prefer to go it alone, mostly notably Texas and Virginia, have simply declined to adopt the Common Core. Others are free to exit from it (though doing so would, for some, violate commitments they made in their Race to the Top proposals.)

Fifth, “national” is not the right way to do anything in American education. We retain a deep (if, in my view, unwarranted) affection for “local control” in this realm and constitutional responsibility for education is undeniably vested in the states. Some folks dread the prospect of a “national curriculum.” (Some simply mistrust the Gates Foundation, which has bankrolled much of this work.) Others are incapable (perhaps willfully so) of seeing any distinction between “national” and “federal”, though we seem to have no difficulty making that distinction elsewhere in education. (E.g. National Governors Association, S.A.T., A.P., ACT.)

Sixth, and closely related to the blurring of national with federal is the expectation that Uncle Sam won’t be able to keep his hands off the Common Core—which means the whole enterprise will be politicized, corrupted and turned from national/voluntary into federal/coercive. This is probably the strongest objection to the Common Core and, alas, it’s probably the most valid, thanks in large measure to our over-zealous Education Secretary and the President he serves.

Let’s face it. Three major actions by the Obama administration have tended to envelop the Common Core in a cozy federal embrace, as have some ill-advised (but probably intentional) remarks by Messrs. Duncan and Obama that imply greater coziness to follow.

There was the fiscal “incentive” in Race to the Top for states to adopt the Common Core as evidence of their seriousness about raising academic standards.

Then there’s today’s “incentive,” built into the NCLB waiver process, for states to adopt the Common Core as exactly the same sort of evidence.

(In both cases, strictly speaking, states could supply other evidence. But there’s a lot of winking going on.)

The third federal entanglement was the Education Department’s grants to two consortia of states to develop new Common Core-aligned assessments, which came with various requirements and strings set by Secretary Duncan’s team.

This trifecta of actual events is problematic in its own right, not because the federal government is evil but because Washington has become so partisan and politicized and because of angst and suspicion that linger from failed efforts during the 1990’s to generate national standards and tests via federal action.

What’s truly energized the Common Core’s enemies, however, has been a series of ex cathedra comments by President Obama and Secretary Duncan. Most recently, the Education Secretary excoriated South Carolina for even contemplating a withdrawal from the Common Core. Previously, the President indicated that state eligibility for Title I dollars, post-ESEA reauthorization, would hinge on adoption of the Common Core. Talking with the governors about NCLB waivers earlier this week, he stated that “if you’re willing to set, higher, more honest standards then we will give you more flexibility to meet those standards.” I don’t know whether he winked. But everybody knew what standards he was talking about.

It will, of course, be ironic as well as unfortunate if the Common Core ends up in the dustbin of history as a result of actions and comments by its supporters. But in March 2012 there can be little doubt that the strongest weapons in the arsenal of its enemies are those that they have supplied.

-Chester E. Finn, Jr.

This post originally appeared on the Fordham Institute’s Flypaper blog.

He has more to say about a larger strategy for boosting teacher quality in “An Effective Teacher in Every Classroom,” which appeared in the Summer 2010 issue of Ed Next.

He also authored “Valuing Teachers: How Much is a Good Teacher Worth?” which appeared in the Summer 2011 issue of Ed Next.

Rather, I think the reluctance to contribute [to a debate in support of Common Core] is due to hubris, impatience to focus on implementation, political naivete, and disdain for what they see as mean-spirited carping….

There are long rows of argument and persuasion still to be hoed. And, if you’re eager to overhaul what gets taught in forty-odd states serving forty million or more students, that’s probably as it should be. If Common Core-ites don’t have the patience or stomach for that task, they should let us know now–and save everyone a whole lot of grief.

The notion that Common Core proponents needn’t make their case is an affront to democratic values.

Well, Ed Next managed to find someone to argue for and against the quality of Common Core standards, producing a really excellent and illuminating exchange.  W. Stephen Wilson took the pro side and Ze’ev Wurman was on the con side.  I would encourage you to read the entire debate yourself, but here is my takeaway:  They were mostly in agreement about the quality of Common Core.  Both seemed to agree that Common Core was better than the standards previously in place in most states but worse than in a non-trivial number of other states.  They also agreed that Common Core standards are significantly weaker than the ones in most high-achieving countries.

So if they agree that Common Core is sort of mediocre, why does Wilson support them while Wurman oppose them?  Wilson sees the improvement on the standards of 30 or more states to be substantial progress.  He sees this as a first step toward developing stronger national standards that would be comparable to those of our overseas competitors and better than all previously existing state standards.

Wurman sees Common Core as significantly lowering the bar relative to several previously existing state standards, including very large states like California.  More importantly, he sees Common Core as the end of progress in improving standards rather than the beginning.  Once put in place, he sees no incentive for anyone to toughen national standards since no state will be competing to offer a more rigorous education in order to attract residents and businesses.  He also sees national standards as more easily captured and dummied-down by teachers unions and other entrenched interests who would prefer to have their members (and students) jump over a lower bar.

-Jay P. Greene

UPDATE — Stephen Wilson contacted me over at the Jay P. Greene blog to object to the description of his views as supporting the adoption of Common Core. He thinks Common Core math standards are much better than those that previously existed in 30 states but still lagging those in other states and high achieving countries. And he generally has no opinion on whether universal adoption of Common Core would represent progress or not or is desirable or not.

It appears that I was wrong. The Ed Next forum was more a discussion among critics than a debate between a supporter and opponent.

So we are back to Rick’s original complaint. We still don’t have anyone who was willing to debate in favor of the national adoption of Common Core based on the quality of the standards.

It’s pretty pathetic that supporters of Common Core couldn’t produce anyone to take the “pro” side of this debate. And it’s even more pathetic that supporters are determined to cram Common Core down our throats without feeling the need to intellectually defend it.

Call this the test score hypothesis. It explains reformers’ enthusiasm for test-based accountability; for “college and career-ready standards”; for teacher evaluations based, in significant part, on student outcomes; for “data-based instruction”; and for much of the rest of the modern-day reform agenda. After all, if reading, math, and science knowledge and skills are so directly linked to the life chances of individual kids, and of the livelihood of the country as a whole, why not get the education system focused like a laser on them?

But is this hypothesis correct? Is stronger academic performance related to better life outcomes for kids and better economic outcomes for nations?

In a word: yes. As Kevin Carey noted recently, the big Chetty et al study didn’t just demonstrate the importance of teacher effectiveness. It also offered strong support for the Test Score Hypothesis.

If you believe standardized tests are worthless or highly flawed or deeply inadequate or even troublingly limited in accuracy and scope–and many reasonable people believe these things–then you could dismiss or downplay value-added measures of teacher effectiveness, by definition….But now the CFR study says that teachers who are unusually good at helping students score high on standardized tests today aren’t just unusually good at helping students score high on standardized tests tomorrow. They also have an unusual effect on the likelihood of students going to college, going to a good college, earning a good living, living in a nice place, and saving for retirement. In other words, whatever the limitations of standardized tests may be, test-based value-added scores do, in fact, provide valuable information about the things most people care most about.

Then there’s the international evidence. As Eric Hanushek has been arguing vociferously for years, there’s a direct link between academic achievement (as measured by math and science tests) and a country’s economic growth.

The level of cognitive skills of a nation’s students has a large effect on its subsequent economic growth rate. Increasing the average number of years of schooling attained by the labor force boosts the economy only when increased levels of school attainment also boost cognitive skills. In other words, it is not enough simply to spend more time in school; something has to be learned there.

Hanushek further argues that the only way to solve our country’s long term fiscal challenge is to grow our way out of it. If we could indeed boost the cognitive skills of our students, even by a little, our structural deficit would go away.

So student achievement matters a lot. But does it matter the most? It’s hard to make the case anymore that test scores are irrelevant. But what remains unknown is whether reading, math, and science are the most important things that schools could be teaching. As Dana Goldstein noted back in December,

I’ve been struck again and again by the newness of the idea that schooling is primarily a matter of academic achievement…. It is only really since “A Nation at Risk” that we’ve had a national dialogue about academic excellence for every child. This is a much-needed development in American culture, but its discontents are numerous: A lack of attention paid to the civic, social, and artistic benefits of schooling, and the ways in which children are (ideally) shaped as moral, cultured, socially-responsible people by their teachers and school communities. 

We might all want schools to walk and chew gum at the same time—to boost “academic achievement” while also developing “moral, cultured, socially-responsible people.” But our policies—especially school-level accountability and test-based teacher evaluations—focus on academic achievement alone.

The nagging question then—the “known unknown”—is whether other stuff matters more—both to kids’ life chances and to the country’s economic success. What if, for instance, “social and emotional intelligence”—knowing how to relate to others—is more important than many reformers have been willing to acknowledge? What if these interpersonal skills are what help lift poor kids out of poverty and enable economies to succeed? Or other “soft skills” and attributes like grit, perseverance, industriousness, the ability to delay gratification, and so forth?

In that case, is it smart to push Head Start centers to focus overwhelmingly on pre-literacy and pre-numeracy skills (as many of us have)? Is it wise to cut time for recess, to trim extracurriculars, or to push for the maximum amount of homework, to be completed by solitary would-be scholars? Does it make sense to ask teachers to obsess about student achievement over everything else?

The private school sector, which many reformers admire, is not so conflicted. Every high-end school boasts about its commitment to the “whole child,” to kids’ intellectual, emotional, social, and physical development. These schools would never consider their graduates to be well-educated without an appreciation for the arts, participation in sports, a commitment to community service, and the development of strong character. And judging by the admissions policies of the nation’s great universities, our elite higher education institutions hold this holistic view, too. Are these non-academic attributes just “extras”—luxuries that schools serving poor or working class kids just can’t afford? Or are they as essential as academics, for everyone?

Reading, math, and science matter a lot, but they are almost certainly not enough. That is why we must tread carefully when designing next-generation school accountability and teacher evaluation systems. If we accidentally create incentives for schools and teachers to focus solely on academic achievement and ignore the rest, we could be making our children and our nation less competitive, not more so. Let us proceed with care.

-Mike Petrilli

This blog entry originally appeared on the Fordham Institute’s Flypaper blog.

• It worked!

As Mark Schneider shows in his recent paper for Fordham—and as Eric Hanushek and others demonstrated before him—poor, minority, and low-achieving students made huge progress in math, and sizable progress in reading, during the late 1990s and early 2000s. Their most recent scores on the National Assessment of Educational Progress indicate all-time highs for most grades and subjects. These students are typically performing two grade levels ahead of where their peers were fifteen years ago in math, and are reading at least one grade level higher. So how to explain these historic gains? While we can’t draw causal conclusions from NAEP, we can make educated guesses. What’s clear is that states that adopted “consequential accountability” in the nineties saw big test-score jumps, and the late-adopter states saw similar progress once No Child Left Behind kicked into action. So, while other factors could have been in play, too (such as efforts to reduce class size or the cessation of the crack-cocaine epidemic), there’s a pretty good case that testing and accountability succeeded in spurring higher student achievement, at least at the bottom of the performance spectrum.

• But it couldn’t work forever. As Schneider argues, the test-score gains sparked by NCLB-style accountability appear to have hit a plateau. We’re back to anemic progress in most grades and subjects, particularly in the states (like Texas) that embraced testing and accountability first. That shouldn’t be too surprising. While the initial pressure (and shame) provided by consequential accountability appears to have changed behavior at the district and school level, after a while being called a “failing school” loses its sting. Furthermore, holding “schools” accountable has rarely equaled holding individuals accountable—real-live teachers and principals who might lose their jobs. Once it became clear that NCLB was all bark and no bite, schools could return to the status quo ante.
• The trade-offs are real. The good news is that we’ve seen enormous progress for our lowest-achieving students. The bad news is that we’ve seen languid progress for our highest achievers. The good news is that math scores are way up and, to a lesser degree, reading scores are up, too (especially for poor and minority kids). The bad news is that history and science have been squeezed out of the elementary school curriculum, particularly in high-poverty schools. Whether these trade-offs were worth it depends on your point of view. Personally, I’d prefer a policy that aims for more balance: achievement gains across the performance spectrum, not just at the bottom; and a more holistic view of what it means for students to be well educated. Literacy and numeracy are (obviously) not enough.
• Pet ideas from both parties crashed and burned. The Democrats gave the country the “white elephant” gift of the “highly qualified teachers” mandate, a policy that succeeded in turning the nation’s teachers against NCLB from the very beginning; managed to tie up myriad schools (including charters) in all manner of red tape; and gravely threatened Teach For America, one of the most promising reforms of the NCLB era. From the Republicans we got “supplemental educational services,” a.k.a. free tutoring. This was more of an impulse than a fleshed-out idea. It was never clear whether SES was meant to be a sanction for failing districts (if you don’t improve your test scores, we’ll take some of your Title I money away from you); a serious effort at parental choice; or a way to “extend” learning time for needy kids. Regardless, its entire design was predicated on cooperation from school districts, which were responsible for facilitating the flow of funds away from their coffers and into the hands of nonprofit and for-profit providers. As my Italian grandmother would have said, “Fatta chance.
• It’s time for something new. On this point, virtually everybody agrees. But what should the next phase of education reform entail? The contours are now taking shape. First, there’s agreement that, for accountability to be real, it has to be placed upon real-live people, not just amorphous “schools.” That means, first and foremost, holding teachers accountable for their performance. Thus the interest in: more sophisticated teacher-evaluation systems, tenure reform, performance pay, and all the rest. Second, there’s broad consensus that we need to balance the “tough love” approach of accountability with the “helping hand” of capacity-building: Providing teachers with tools like a coherent curriculum—linked to the new Common Core standards—so they don’t have to make it all up on their own. And third, we can all glimpse the promise of digital learning, if technology can be harnessed effectively and if the political and governance roadblocks can be removed. But what’s the appropriate (and politically feasible) federal role in all of this? In all of these reforms, Uncle Sam’s involvement will be—and should be—minimal. The political thirst for aggressive federal involvement in education has been quenched, and the dollars to fund it spent. Plus these “next wave” reforms require nuance, care, and thoughtfulness to get them right—attributes not associated with Uncle Sam. In other words, reform will continue, but the federal government will lead from behind. As well it should.

Happy birthday, No Child Left Behind. And here’s hoping that you don’t make it to eleven.

-Michael Petrilli

This post also appeared on Flypaper.

Many educators and elected officials, including more than a few members of Congress, regard “No Child Left Behind,” the well-known moniker of George W. Bush’s 2001 education act, as a discredited “brand.” Indeed, the very acronym NCLB is about to be tossed into the dustbin of history in favor of its progenitor, ESEA (the Elementary and Secondary Education Act), or perhaps some new title yet to be devised on Capitol Hill. There are many reasons why NCLB has been discredited, including, to quote Kevin Carey, the “apocalyptic language out there, that standards and tests have ruined American public education, driven the best teachers out of the classroom, etc., etc.”

Yet, as the data presented below demonstrate, NCLB—and the accountability movement it embodied, codified, and symbolized—contributed to a major change in the performance level of American students in math. The data also suggest, however, that the accountability movement has likely reached a point of diminishing (or perhaps even no) returns. While moving on from NCLB is probably essential to produce further growth in student performance, “consequential accountability” was an important and meaningful education reform and ought not be dismissed as a failed initiative.

Debates over the effects and effectiveness of NCLB almost always revolve around national and state scores on the National Assessment of Educational Progress (NAEP). Not surprisingly, the release in November 2011 of the newest NAEP Mathematics and Reading Report Cards set off a new round of discussion about the impact of NCLB and accountability more generally. Given the ongoing fights surrounding the overdue reauthorization of ESEA/NCLB, the debate over the effects of accountability is more important now than ever.

Remember that NCLB’s system of consequential accountability (in which schools face cascading penalties for failure, e.g., replacement of the school’s principal, reconstitution, closure, etc.) was built upon the experience of many states that had already developed such systems before 2001. There is considerable agreement that states adopting consequential accountability before NCLB experienced more rapid growth in their test scores relative to non-adopting states. However, as Hanushek and Raymond note, as NCLB took hold, all states became “effectively consequential accountability states.” Perhaps not surprisingly, after NCLB, states that were new to the accountability regime experienced faster growth on NAEP assessments than states that had introduced their own accountability regimes before 2001.

The Case of Texas

Texas was one of the first states in the nation to adopt strict and consequential accountability. The Texas experience was fundamental to the framing of NCLB, as George W. Bush took the lessons and practices of Texas along with him when he moved from Austin to Washington. Thus, looking at the growth in NAEP scores in Texas relative to changes in the nation as a whole allows us to tease out some lessons about the effects of accountability on student performance and to speculate about the effectiveness of accountability past, present, and future.

As we look at these data, we should remember that, while NAEP is rightfully viewed as the “gold standard” of assessments, it is not the ideal instrument for detailed statements of cause and effect. We should further keep in mind one of the prime maxims of statistics: Correlation is not causation.

The Remarkable Growth in NAEP Math Scores

It is well known that, as measured by NAEP, American students have improved substantially in math (more in fourth grade than in eighth) and little in reading over the last two decades. Separate and apart from overall averages, there has been continuing concern for the level of skills among racial/ethnic minorities as well as concern for the effects of accountability on low- versus high-performing students (specifically, whether or not NCLB placed so much attention on low-performing students that high-performing students were neglected and suffered as a result). Looking at trends in Texas versus the nation presents some insights into these issues.

Fourth-Grade Mathematics

Consider Figure 1, which graphs the average scale scores on NAEP’s math assessment for fourth-grade students in Texas and in the United States as a whole. The growth in the performance of these students is nothing short of remarkable. Using the very rough rule of thumb that a 10-point change in NAEP scores equals about one year of learning, in 2011 our fourth graders are about two years ahead of where they were in 1992. But, as the figure shows, Texas and the nation marked their peaks of achievement at two distinct points in time.

In 1992, students in Texas were performing at the same level as the students in the nation. In the 1993-94 school year, Texas introduced its system of consequential accountability and, by the time of the next NAEP assessment in 1996, Texas fourth-graders had surpassed their peers nationwide. Between 1992 and 2000, math scores across the nation began to creep up; during the same period, a growing number of states began to adopt accountability systems.

By 2003, NCLB had turned every state into a consequential accountability state, and the rate of increase nationwide in math scores between 2000 and 2007 was remarkable. While Texas students continued to outperform the nation as a whole through 2007, the sharp uptick in national performance after 2000 narrowed the Texas lead substantially. Indeed, the last two assessments, in 2009 and 2011, show no significant difference between fourth graders in Texas and fourth graders nationwide.

We return to these overall patterns later, but first we turn to the performance of three groups of students who served as particular focal points of NCLB and the accountability movement more generally: blacks (Figure 2), Hispanics (Figure 3), and low-performing students (Figure 4), defined here by the cut score identifying those students performing at NAEP’s 10th percentile.

At the beginning of the series in 1992, black and Hispanic fourth-grade students in Texas scored slightly higher than their nationwide peers, while those low-performing students at the 10th percentile in Texas achieved at the same level as those at the 10th percentile nationally.

Between 1992 and 2000, the scores of Texas students in all three groups increased faster than those of their peers nationwide, with the size of the gap between student in Texas and the nation widening to well over 10 points for each group. Between 2000 and 2003, nationwide, the gains for students in each group increased dramatically but then slowed substantially. Gains among Texas fourth graders were sustained over a longer period of time, but also show evidence of little growth since 2005, with Hispanic and the lowest-performing students actually scoring lower in the latest assessments than in 2007.

The growth in fourth-grade math achievement represents one of the most significant success stories in contemporary American education. Again, the reader is reminded that, while correlation is not causation, the introduction of consequential accountability in Texas and then across the nation coincided with impressive spikes in the performance of students in fourth-grade math, and in particular among the students of most concern to NCLB and the accountability movement more generally.

Eighth-Grade Mathematics

NAEP test results for eighth-grade math represent a somewhat weaker reflection of this striking pattern (Figure 5). The first NAEP eighth-grade math assessment was in 1990, at which time Texas eighth graders lagged the nation by 5 points. That gap disappeared by 2000. By 2005, as the strong fourth-grade performers moved into the eighth grade and as the Texas system of consequential accountability continued to gain traction, Texas eighth graders moved past their national peers, producing a gap of 6 points. Whether eighth-grade test scores can continue to grow, given the flattening scores at the fourth grade, is something that remains to be seen.

Among black and Hispanic eighth graders, Texas students started at about the same place as their national peers in 1990. Over time, however, they experienced steady growth in performance, producing a widening gap with the nation. Indeed, the size of the gap for black students (in favor of Texas) has increased from 6 or 7 points before 2000 to 10 points in the last three assessments (Figure 6). The size of the gaps in favor of Hispanic students in Texas has been somewhat more variable, and was not statistically significant before 2000 (Figure 7). But this gap has grown to over 10 points in the last three assessments. Similarly, the cut score defining the lowest 10th percentile has risen more rapidly in Texas than in the nation as a whole (Figure 8), becoming statistically significant in 2000 and almost doubling in size from 2000 (7 points) to the latest assessment in 2011 (13 points).

High-Performing Students

A frequent criticism of the accountability movement and NCLB was that the focus on racial and ethnic minorities and on the lowest-performing students led to a neglect of the nation’s highest-performing youngsters.

Here we define high-performing students as those performing at NAEP’s 90th percentile. Fourth-grade math scores for these students both in Texas and in the nation display sharp increases since 1992 (Figure 9). The cut score for the top performers nationwide stood at 259 in 1992 and steadily rose to 276 in 2011, a gain of 17 points. The highest-performing fourth graders in Texas saw a correspondingly large jump in cut scores from 256 in 1992 to 273 in 2005. (Interestingly, half of that gain occurred between the assessments immediately preceding and following implementation of the state’s accountability system in 1993-94). Since 2005, however, there has been no statistically significant change in cut score for those Texas youngsters, although the national cut score for high performers has continued to rise—producing a statistically significant difference (to the disadvantage of Texas) in the two most recent administrations of NAEP.

Eighth-grade math scores among the highest performers also improved substantially over the period, gaining 14 points nationally and 17 points in Texas (Figure 10). The sharpest gains for these high-performing eighth graders in Texas were between 2000 and 2005, building on the improvement made in math by Texas fourth graders four years earlier. Gains continued thereafter at somewhat slower rates, likely reflecting the slower growth in fourth-grade math skills.

The growth in NAEP scores of the highest-performing students in Texas and the nation essentially mirrors the gains made by student groups that were focal to the policy goals of NCLB. Whatever changes more directly focused on specific target populations apparently spilled over to affect the performance of high performers as well. And just as we saw evidence of diminishing effectiveness in recent years for average, minority, and low-performing students, there is evidence that the spillover effects of accountability on high-performing students are also wearing thin. The recent absence of growth in Texas fourth-grade math skills among these high-performing students may portend the end of a remarkable period of growth among the highest performers in the second-largest state in the union.

The Disappointing Case of Reading Scores

The improvements in NAEP math scores were an unquestionable success for America’s fourth and eighth graders and even more so for students in Texas. However, neither the nation as a whole nor Texas has done nearly as well improving students’ reading skills. Figure 11 shows no significant difference between the reading scores of fourth-grade students in Texas and in the nation as a whole, except in 2003, and minimal improvement across the board. And Texas’s eighth graders have significantly lagged the nation since 2003: by 2 points in 2007 and by 4 to 5 points in every other assessment between 2003 and 2011 (Figure 12).

Accountability and NCLB Were a Success, But…

In 1972, Stephen Jay Gould and Niles Eldridge proposed a theory of evolutionary change that emphasized what they termed “punctuated equilibrium.” Their core insight was that complex systems will exist in long periods of stasis. Rather than coming in small incremental steps, change is often characterized by abrupt radical transformations caused by events external to the existing system. Perhaps the most dramatic example is the relatively sudden disappearance of dinosaurs associated with a meteor crashing into the Earth and changing the climate. As a result, the dinosaurs’ long reign was replaced by a new equilibrium dominated by mammals.

In 1993, political scientists Frank Baumgartner and Bryan Jones introduced this theory to the study of public policy, and it has since become a common lens through which to view change in social systems. Baumgartner and Jones argued that policy generally changes only incrementally, until some event, such as change in the party control of government or sizable shifts in public opinion, lead to large policy alterations. In their approach, large changes in external conditions (what Baumgartner and Jones term an “exogenous shock”) are often needed to produce change in complex social and political systems.

The pattern of test scores in Texas and the nation suggest that consequential accountability—adopted early by Texas, then by more states, and finally by the nation as a whole—was a shock to the U.S. school system that altered the ecosystem and led to a different outcome than had existed before. Over a relatively short period, math performance in fourth and eighth grade abruptly shifted to higher levels of performance. For example, between 2000 and 2005—the five years spanning the introduction of accountability via NCLB—the average math scale score nationwide at the fourth grade rose by 12 points, roughly a year of learning. In the same period, the average scale score for black fourth graders rose by 18 points, for Hispanic students by 17 points, and the cut score defining the 10th percentile of performance increased by 16 points. The corresponding changes among eighth-grade math scores are small only in comparison: 6 points nationwide, 11 points for black students, 10 points for Hispanic students, and 8 points for those students at the 10th percentile.

To be sure, an important lingering issue is the absence of growth in reading scores in Texas and in the nation as a whole. Many have argued that the foundation for reading, compared to math, is far more dependent on what happens early in children’s lives—before they enroll in school—and that improving reading skills is therefore much harder to accomplish. Whatever the explanation, clearly the absence of growth reflects a failure of the accountability “meteor” to affect reading levels in a fundamental way.

There is once final pattern to note: As would be expected when viewed through the punctuated- equilibrium lens, once the disruption of consequential accountability has wrung all changes out of the system, a new stasis should take hold. Indeed, Texas, an early adopter, led the nation to higher scores and seems to be ahead of the nation in reaching a new plateau where changes are minimal compared to what came in response to the introduction of an accountability system. The nation, which lagged Texas in adopting accountability, now seems to be entering a period of little change in test scores.

In the 1990s and early 2000s, accountability was an exogenous shock that produced radical gains in math if not in reading. But we now need a new shock to prevent a prolonged period of stasis and stagnation. Scanning the heavens for the next meteor, the most likely candidates to come crashing into the school ecosystem are the Common Core and the better measurement of teacher performance. If the United States is lucky, one or both of these shocks will produce yet another major uptick in math scores. If we are really lucky, these shocks will produce upticks in reading and other subject areas as well.

Mark Schneider, a former commissioner of the National Center for Education Statistics, is a vice president at American Institutes for Research and visiting scholar at the American Enterprise Institute. This article was commissioned and also published by the Fordham Institute.

Checked by Eric A. Hanushek

The No Child Left Behind Act of 2001 (NCLB) was scheduled for reauthorization in 2007, and its future has in recent months garnered renewed attention. Yet so far, Congress has found it impossible to reach sufficient consensus to update the legislation, as competing groups want to a) keep all the essential features of the current law as a way of maintaining the pressure on schools to teach all students, b) modify the federal law by moving to a value-added or some alternative testing and accountability system, or c) eliminate federal testing and accountability requirements altogether, reverting to the days when the compensatory education law was simply a framework for distributing federal funds to school districts. Critics of NCLB’s testing and accountability requirements have a litany of complaints: The tests are inaccurate, schools and teachers should not be responsible for the test performance of unprepared or unmotivated students, the measure of school inadequacy used under NCLB is misleading, the tests narrow the curriculum to what is being tested, and burdens imposed upon teachers and administrators are excessively onerous.

But in all the acrimonious discussion surrounding NCLB, surprisingly little attention has been given to the actual impact of that legislation and other accountability systems on student performance. Now a reputable body, a committee set up by the National Research Council (NRC), the research arm of the National Academy of Sciences, has reached a conclusion on this matter. In its report, Incentives and Test-Based Accountability in Education, the committee says that NCLB and state accountability systems have been so ineffective at lifting student achievement that accountability as we know it should probably be dropped by federal and state governments alike. Further, the committee objects to state laws that require students to pass an examination for a high school diploma. There is no evidence that such tests boost student achievement, the committee says, and some students, about 2 percent, are not getting their diplomas because they can’t—or think they can’t—pass the test. The headline of the May 2011 NRC press release is frank and bold in the way committee reports seldom are: “Current test-based incentive programs have not consistently raised student achievement in U.S.; Improved approaches should be developed and evaluated.”

Needless to say, the report can be expected to play an important role in the continuing debate over NCLB. Upon its initial release, the report captured top billing, appearing on Education Week’s front page. Certainly, the NRC intends for the report to influence the NCLB conversation, rushing a draft version to the media five months before the completed report was available to the public.

Unfortunately, the NRC’s strongly worded conclusions are only weakly supported by scientific evidence, despite the fact that NRC’s stated mission is “to improve government decision making and public policy, increase public understanding, and promote the acquisition and dissemination of knowledge.”

The Report

Reports from the NRC are generally treated as highly credible. The NRC convenes panels of outside experts who volunteer their time to provide consensus opinions on issues of policy significance. And this particular panel includes a number of especially qualified researchers (see sidebar). The committee chair, Michael Hout, is a member of the National Academy of Sciences; 7 of the 17 panel members have named professorships; 2 are deans (of law and education schools); and a majority have published articles about testing, accountability, or incentives.

When it comes to gathering together the general literature, both theoretical and empirical, on the use of incentives in various contexts, the committee’s work is solidly constructed. But this strong scientific discussion of theory and empirical analysis of incentives and accountability breaks down when it comes to the committee’s core purpose: evaluating accountability regimes in education that employ incentives and tests.

The report comes to two policy conclusions: NCLB and state accountability systems have proven ineffective and state-required high-school exams are counterproductive. The unequivocal presentation of the conclusions is clearly designed to leave little doubt in the minds of policymakers. When the underlying evidence is examined, however, it becomes apparent that neither conclusion is warranted. Instead of weighing the full evidence before it in the neutral manner expected of an NRC committee, the panel selectively uses available evidence and then twists it into bizarre, one might say biased, conclusions.

Selecting Evidence

To get a grasp of the bias that motivated the report’s authors, consider how its first conclusion is phrased:

Test-based incentive programs, as designed and implemented in the programs that have been carefully studied, have not increased student achievement enough to bring the United States close to the levels of the highest achieving countries.

Note especially that the conclusion does not say that there is no evidence that testing and accountability work. It says that testing and accountability, by themselves, cannot lift the United States to the level of accomplishment reached by the world’s highest-achieving countries, an extraordinary standard for evaluating a policy innovation. To catch up to the leading countries would require gains of at least half of a standard deviation, or roughly two years of learning (see “Are U.S. Students Ready to Compete?” features, Fall 2011). No individual reform on the public agenda—neither merit pay, class size reduction, salary jumps for teachers, nor Race to the Top—can claim or even hope for anything close to that level of impact. The appropriate question is not whether testing and accountability is a panacea, but whether it has proven worthwhile.

By that more appropriate standard of judgment, the committee’s own data indicate that testing and accountability have proven effective, if not quite the spectacular success promised by those who enacted NCLB into law. The committee report tells us that the average estimated impact of these interventions is 0.08 standard deviations of student achievement. In other words, the average student in a state without accountability would have performed at the 53rd percentile of achievement had that student been in a state with an accountability system, all other things being equal.

That estimate may well be too low. The report states that “our literature review is limited to studies that allow us to draw causal conclusions about the overall effects of incentive policies and programs,” and then it goes on to describe several types of studies that would be excluded by this criterion. Where does the 0.08 come from? The committee considers a review from 2008 of 14 studies, and 4 studies conducted after that review. The review presents an average impact of 0.08. The NRC committee apparently felt no need to look any further and ignored the fact that a majority of the 14 studies would not come close to meeting its standard of enabling a “causal conclusion.” The committee determines that one of the more recent studies also supports an estimate of 0.08, although that study’s authors prefer estimates that are much higher. The 14 earlier studies and the 4 later ones produce a wide distribution of estimated impacts, but the committee makes no attempt to investigate whether the unusual estimates suggest circumstances under which accountability seems particularly effective (or ineffective). The committee chooses to emphasize the studies with negative findings (10 percent) while downplaying a number of those that have positive findings (90 percent). Thus the NRC mantra, repeated with slightly different wording throughout the report: “Despite using them for several decades, policymakers and educators do not yet know how to use test-based incentives to consistently generate positive effects on achievement and to improve education.” Apparently, the inconsistent results heralded in the press release reflect the 10 percent of studies that differed from the overwhelming majority.

Small Gains Add Up

Let us put this concern aside and consider the increment in student performance of 0.08 standard deviations of individual achievement that the committee presents as its best estimate. Is that so small an effect that it cannot justify continuation of testing and accountability? Consider that this is the average effect of a program that has been implemented on a national scale, affecting students across the country. We are hard pressed to come up with any other education program working at scale that has produced such results. Moreover, these average gains are the result of accountability systems that many people believe have important flaws. Even larger gains might be expected if those flaws could be corrected, as many experts, though not the NRC panel, have suggested.

The estimated benefits from a 0.08 standard deviation gain in student performance vastly outweigh its estimated costs. The cost of designing, administering, grading, and reporting the results from statewide examinations have been estimated at between $20 and $50 per pupil, a trivial sum considering that per-pupil education expenditures in the United States run above $12,000 annually. Most reforms—including class size reduction, merit pay, across-the-board raises for teachers, in-service training programs, or the scaling up of charter schools—would cost many, many times as much. For these innovations to have the same kick for every dollar invested, results would have to be improbably large.

The NRC, instead of considering these actual costs, suggests that implicit costs in the form of narrowed curricula are the most important, but it provides no evidence for its view.

What might the economic impact of a 0.08 standard deviation improvement in average achievement nationwide be? Along with University of Munich professor Ludger Woessmann, I have estimated the impact on U.S. Gross Domestic Product (GDP) of higher levels of student achievement. These estimates project the historical pattern of growth to determine the result of gains in student achievement, calculate the additions to GDP over the next 80 years, and discount them back to today so that they are comparable to other current investments. A 0.08 improvement has a present value of some $14 trillion, very close to the current $15 trillion level of our entire GDP, and equivalent to $45,000 for every man, woman, and child in the U.S. today. In other words, an inexpensive program that affects every student nationwide can, over the long run, have a very large impact, even if its average effect seems at first glance to be quite small. Indeed, if we figured testing cost $100 per student each year for the next 80 years and we tested all students rather than the limited grades tested now, the rate of return on the investment would be 9,189 percent. Google investors would be envious.

Several omissions from the report are also noteworthy. The report gives only passing attention to the positive impact of NCLB on the education of the most disadvantaged students, a consequence of the requirement to report performance by specific subgroups (e.g., racial and ethnic groups and the economically disadvantaged). The NRC report’s main reference to this feature of current accountability systems is that consideration of subgroup performance has added analytical difficulties because of the smaller samples.

Perhaps more telling, this panel of experts on testing and incentives makes absolutely no effort to describe how accountability programs could be improved. Being good researchers themselves, they do favor continued research on testing, however, and provide recommendations on what research should be done, which not surprisingly matches their own interests and expertise.

Lower the Bar?

The report also addresses a second, widely used accountability policy: high-school exit exams that hold students responsible for meeting a set of content standards. The report’s second conclusion reads,

The evidence we have reviewed suggests that high school exit exam programs, as currently implemented in the United States, decrease the rate of high school graduation without increasing achievement.

The panel strongly suggests that states that impose an exit exam should repeal this requirement. To understand this conclusion, it is necessary to understand the exams themselves and to evaluate the evidence behind the committee’s conclusion.

Currently, more than half of the states require that students pass a test of some sort to obtain a normal diploma (see Figure 1), and virtually all of these current requirements have been put in place since 2000. The tests almost always cover English and math, but many states add science and history. Test difficulty varies by state, but the modal level is grade 10. Although that standard may seem low, it is considerably more stringent than the standards that existed prior to 1990, when no state had a test reaching even the 9th-grade level. The current tests are not as high a barrier to high school graduation as they are often alleged to be, as a student may generally take the exam multiple times in order to achieve a passing score. And in all but three states (South Carolina, Tennessee, and Texas), students can either appeal the test result, if they feel the score misrepresents their accomplishments, or obtain a diploma by some alternative path.

The motivations for administering exit exams are to create incentives for students to apply themselves to the task of learning and to set uniform (minimum) quality standards for the state’s schools. Such content standards provide guidelines to schools about what to teach. They also indicate to colleges and universities what knowledge and abilities a graduate can be expected to possess. And they give similar information to prospective employers.

According to the best available evidence (discussed below), perhaps 2 percent of students are induced to drop out of school either because of failure to pass the exam or because of fear of not being able to pass the exam. Implicitly, the committee assumes this consequence does considerable harm to the affected students, given the substantial economic rewards that accrue, on average, from receiving a high school diploma. But average effects do not necessarily apply to the 2 percent on the border line between graduating and failing to graduate from high school. The impact for this particular group of students is likely to be much less, unless you make the bizarre assumption that it is only the diploma—not what the student learns—that affects job prospects and future income. The people who are induced to drop out because they cannot pass a 10th-grade exam would most likely be near the bottom of the earnings distribution of graduates were they to be handed a diploma. The economic impact on these students will be much lower than the average difference between graduate and dropout.

Perhaps the best argument against exit exams is simple: If a student shows up for school for 12-plus years and cannot pass a 10th-grade exam, it must be the school’s fault, and it would be unfair to hold the student responsible. This argument, interestingly enough, is the precise opposite of one of the primary arguments against the testing and accountability provisions of NCLB: We should not hold schools responsible for low achievement, because achievement is affected by student motivation and family background characteristics beyond the school’s control. Taken together, the arguments embedded in the committee’s two conclusions imply that nobody—not schools, not teachers, not even students themselves—bears responsibility for low student achievement.

Interestingly, the committee’s conclusion with respect to exit exams does not pick up on the full report’s emphasis on the importance of the design features of incentive systems, which include warnings that tests aimed at ensuring minimum competency may lower expectations, and concerns about both the potential narrowing of the curriculum and the tendency for score inflation on a known test. Instead, the presumed problem is the inherent unfairness of denying a diploma to a student who has met the attendance and course distribution requirements for a diploma.

If the main objective is to maximize high school graduation, there are many ways to do that. We could eliminate all exams, even those administered by teachers. We could loosen up course requirements. We could offer the diploma after 10 or 11 years of schooling, instead of 12. Of course, nobody is willing to take such steps, even though class exams, course requirements, and the inclusion of the 12th grade of schooling all have negative impacts on graduation rates. So why then does the NRC promote the idea of eliminating a 10th-grade-level examination as a requirement for high school graduation on the narrow basis that a few students will, as a result, not earn the degree? Is the NRC also against the movement of many states toward increasing the required amount of math or moving to college and career-ready standards?

The Data Shuffle

Let’s examine the evidence the committee supplies for its exit exam conclusion. The report marshals three studies that explore the issue: two on dropouts and one on achievement. Evaluating the impact of exit exams on achievement is inherently difficult. Because the exams apply to everybody in a state at the same time, it is not possible to compare students of the same age within the same state to find out the impact of exams. It is possible, however, to look at different cohorts of students, for example, those who attended school before the exam was in place and those who attended after, and to compare these to similar cohorts in other states where no such change in policy took place. In conducting this type of study, one must rule out other differences, such as those in family background or those in state education policies that might also affect student performance over time. Even when these challenges are met, one cannot be entirely sure of the results, as exit exams may influence student and school performance even before they come into effect, if teachers and students know that they will soon be introduced, which is usually the case.

The committee tosses out every exit-exam study (save three) that has ever been conducted on the grounds that it is not possible “to draw causal conclusions about the overall effects of test-based incentives” (that is, the very same criteria the committee ignored in considering school-level accountability). Some of the excluded studies use the well-regarded quasi-experimental technique known as regression discontinuity analysis. In the committee’s view, “Such regression discontinuity studies provide interesting causal information about the effect of being above or below the threshold, but they do not provide information about the overall effect of implementing an incentives program.” That criticism is odd, since the impact of an exit examination is of special interest for exactly those students on the cusp of adequate levels of achievement. While these excluded studies are not really appropriate for studying achievement, they tend to show little impact of exit exams on dropout behavior or graduation outcomes.

The committee relies for its conclusion regarding exit examinations exclusively on a 2009 study by Eric Grodsky, John Robert Warren, and Demetra Kalogrides. Because of the significance of this piece of research for the committee project as a whole, it is worth considering in some depth. The Grodsky team identified trends in student achievement in each state that administers an exit examination by drawing on data provided by the long-term trend assessments of the National Assessment of Educational Progress (NAEP). The long-term NAEP, begun in the late-1960s and continued with testing every few years, was designed to provide consistent score information to judge achievement of the nation as a whole. It was not designed to be used to evaluate the schools of any particular state or district. As a result, NAEP never collected in its long-term trend assessment a representative sample of students for any specific state, and the median number of tested students in each state was very small.

Grodsky et al. pretend that the NAEP provides them with just that: a representative sample of students for each state. They assume that the average performance of students in each state on the long-term NAEP provides an accurate measure of the average performance of students in that state, thereby violating the first principle of statistical sampling.

They then merge the information with information on the timing of the adoption of an exit exam by a state between 1971 and 2004. The study includes observations of math and reading achievement at 9 and 10 different points in time, respectively. The researchers report results for achievement of 13-year-olds and 17-year-olds separately, acknowledging that there are limitations to using either cohort. Thirteen-year-olds may be too young to detect the impact of exit exams, while the sample of 17-year-olds suffers from the noninclusion of school dropouts.

The Grodsky analysis encounters a further difficulty. For the most part, the researchers consider only the very early years, when exit exams were first introduced, a time when the exams were set at a very low level of difficulty, below that of a 9th-grade student. Only 1 percent of the observations included in their analysis are for states that had an exit exam rated at the 9th-grade level or higher, as most current examinations are.

Not only does the Grodsky team rely on inadequate data, but the analysis itself is flawed. Any attempt to see the effects of state tests should compare the changes that occur in the states that introduce them with changes in the states that do not. But the Grodsky study effectively tosses out all the information available for the 27 states that do not have an exit examination before 2004. As important, the analysis does not consider any measures of state policies except for exit exams, implying that any other policy changes for the three decades between 1971 and 2004 are either irrelevant for student performance or are not correlated with the introduction and use of exit exams.

The central finding is that exit exams do not have a statistically significant effect on test scores. But this insignificance could arise because of any or all of the above-mentioned problems rather than the absence of an effect of exit exams, as the NRC committee wants us to presume.

The committee’s estimate of the effects of exit exams on school dropout rates is less controversial. It relies on two quite reliable studies, although they are not without limitations: they study the effects of specific exit exams, which may not generalize to other arrangements. The studies indicate that perhaps 2 percent of potential high-school graduates would have received the diploma had it not been for the exit exams.

The committee touts the possibility of alternative incentives to exit exams: “Several experiments with providing incentives for graduation in the form of rewards, while keeping graduation standards constant, suggest that such incentives might be used to increase high school completion.” The key of course is just what the phrase “while keeping graduation standards constant” means. The idea behind exit exams is to ensure a minimum level of quality, as distinct from meeting the course completion requirements. Moreover, the report never makes the case that exit exams and other potential incentive programs are mutually exclusive. In principle, nobody would argue against employing other incentive programs as long as they were worth the expense and, as the committee says elsewhere, do not introduce perverse incentives of one kind or another.

The Takeaway

The NRC clearly wants to enter into the current debate about the reauthorization of NCLB. And the NRC has an unmistakable opinion: its report concludes that current test-based incentive programs that hold schools and students accountable should be abandoned. The report committee then offers three recommendations: more research, more research, and more research. But if one looks at the evidence and science behind the NRC conclusions, it becomes clear that the nation would be ill advised to give credence to the implications for either NCLB or high-school exit exams that are highlighted in the press release issued along with this report.

The framing of policy in the NRC report is simple: “The small or nonexistent benefits that have been demonstrated to date suggest that incentives need to be carefully designed and combined with other elements of the educational system to be effective.” Nobody would oppose careful design of incentives. Nobody would oppose evaluating the intended and unintended outcomes of incentives. And nobody would oppose combining carefully designed incentives with “other elements of the educational system to [make them] effective.”

The NRC is careful to offer no guidance on how NCLB or state exit exams might be modified to make them more effective. And the NRC is very careful not to offer any guidance on “other elements of the educational system.” The message that comes through is clear: keep working on test development, but never use tests for any incentive or policy purposes.

A better takeaway message might be, “Never rely on the conclusions of this NRC report for any policy purpose.”

Eric Hanushek is senior fellow at the Hoover Institution of Stanford University and member of the Koret Task Force on K–12 Education.

American education has problems, almost everyone is willing to concede, but many think those problems are mostly concentrated in our large urban school districts. In the elite suburbs, where wealthy and politically influential people tend to live, the schools are assumed to be world-class.

Unfortunately, what everyone knows is wrong. Even the most elite suburban school districts often produce results that are mediocre when compared with those of our international peers. Our best school districts may look excellent alongside large urban districts, the comparison state accountability systems encourage, but that measure provides false comfort. America’s elite suburban students are increasingly competing with students outside the United States for economic opportunities, and a meaningful assessment of student achievement requires a global, not a local, comparison.

We developed the Global Report Card (GRC) to facilitate such a comparison. The GRC enables users to compare academic achievement in math and reading between 2004 and 2007 for virtually every public school district in the United States with the average achievement in a set of 25 other countries with developed economies that might be considered our economic peers and sometime competitors. The main results are reported as percentiles of a distribution, which indicates how the average student in a district performs relative to students throughout the advanced industrialized world. A percentile of 60 means that the average student in a district is achieving better than 59.9 percent of the students in our global comparison group. (Readers can find all of the results of the Global Report Card at http://globalreportcard.org. The web site contains a full description of the method by which we calculated the results. For a summary, see the methodology sidebar.)

For the purposes of this article, we focus on the 2007 math results, although the GRC contains information for both math and reading between 2004 and 2007. We focus on 2007 because it is the most recent data set, and we focus on math because it is the subject that provides the best comparison across countries and is most closely correlated with economic growth. Readers should feel free to consult the GRC web site to find reading results as well as results for other years.

The Example of Beverly Hills

It is critically important to compare exclusive suburban districts against the performance of students in other developed countries, as these districts are generally thought to be high-performing. The most wealthy and politically powerful families have often sought refuge from the ills of our education system by moving to suburban school districts. Problems exist in large urban districts and in low-income rural areas, elites often concede, but they have convinced themselves that at least their own children are receiving an excellent education in their affluent suburban districts.

Unfortunately, student achievement in many affluent suburban districts is worse than parents may think, especially when compared with student achievement in other developed countries. Take for example Beverly Hills, California. The city has a median family income of $102,611 as of 2000, which places it among the top 100 wealthiest places in the United States with at least 1,000 households. The Beverly Hills population is 85.1 percent white, 7.1 percent Asian, and only 1.8 percent black and 4.6 percent Hispanic. The city is virtually synonymous with luxury. A long-running television show featured the wealth and advantages of Beverly Hills high-school students (as well as their overly dramatic personal lives). If Beverly Hills is not the refuge from the ills of the education system that elite families are seeking, it’s not clear what would be.

But when we look at the Global Report Card results for the Beverly Hills Unified School District, we don’t see top-notch performance. The math achievement of the average student in Beverly Hills is at the 53rd percentile relative to our international comparison group. That is, one of our most elite districts produces students with math achievement that is no better than that of the typical student in the average developed country. If Beverly Hills were relocated to Canada, it would be at the 46th percentile in math achievement, a below-average district. If the city were in Singapore, the average student in Beverly Hills would only be at the 34th percentile in math performance.

Of course, people don’t think of Beverly Hills as a school district with mediocre student achievement. This is partly because people assume that affluent suburbs must be high achieving and partly because state accountability results inflate achievement by comparing affluent suburban school districts with large urban ones. According to California’s state accountability results, the average student in Beverly Hills is at the 76th percentile in math achievement relative to other students in the state. But outperforming students in Los Angeles, which is only at the 20th percentile in math relative to a global comparison group, should provide little comfort to Beverly Hills parents.

Los Angeles Unified is not the main source of competitors for Beverly Hills students, so the state accountability system encourages the wrong comparison. If Beverly Hills graduates are to have the kinds of jobs and lifestyles that their parents hope for them, they will have to compete with students from Canada, Singapore, and everywhere else. Beverly Hills students have to be toward the top of achievement globally if they expect to get top jobs and earn top incomes.

Results from Affluent Suburbs Nationwide

We can repeat the story of Beverly Hills all across the country. Affluent suburban districts may be outperforming their large urban neighbors, but they fail to achieve near the top of international comparisons (see Figure 1). White Plains, New York, in suburban Westchester County, is only at the 39th percentile in math relative to our global comparison group. Grosse Point, Michigan, outside of Detroit, is at the 56th percentile. Evanston, Illinois, the home of Northwestern University outside of Chicago, is at the 48th percentile in math. The average student in Montgomery County, Maryland, where many of the national government leaders send their children to school, is at the 50th percentile in math relative to students in other developed countries. The average student in Fairfax, Virginia, another suburban refuge for government leaders, is at the 49th percentile. Shaker Heights, Ohio, outside of Cleveland, is at the 50th percentile in math. The average student in Lower Merion, Pennsylvania, near Philadelphia, is at the 66th percentile. Ladue, Missouri, a wealthy suburb of St. Louis, is at the 62nd percentile. And the average student in Plano, Texas, near Dallas, is at the 64th percentile in math relative to our global comparison group.

All of these communities are among the wealthiest in the United States. All are overwhelmingly white in their population. All of them are thought of as refuges from the dysfunction of our public school system. But the sad reality is that in none of them is the average student in the upper third of math achievement relative to students in other developed countries. Most of them are barely keeping pace with the average student in other developed countries, despite the fact that the comparison is to all students in the other countries, some of which have a per-capita gross domestic product that is almost half that of the United States. In short, many of what we imagine as our best school districts are mediocre compared with the education systems serving students in other developed countries.

Pockets of Excellence

While many affluent suburban districts have lower achievement than we might expect, some districts are producing very high achievement even when compared with that of students in other developed countries. For example, the average student in the Pelham school district in Massachusetts is at the 95th percentile in math. That means that if we were to relocate Pelham to another developed country in our comparison group, the average student in Pelham would outperform 95 percent of the students in math. That’s very impressive.

Of course, Pelham is a small district that is home to Amherst College, among other institutions of higher learning, and serves a rather select group of students. But not all college-town school districts are equally high achieving. As we have already seen, Evanston, Illinois, is at the 48th percentile in math in a global comparison. Palo Alto, California, the home of Stanford University, is at the 64th percentile. And the average student in Ann Arbor, Michigan, home to the University of Michigan, is at the 58th percentile in math relative to students in other developed countries. So, the 95th percentile math achievement in Pelham is outstanding, even for college towns.

Spring Lake, New Jersey, has a similarly impressive record of having the average student at the 91st percentile in math. It is a very small and affluent community on the New Jersey shore that has somehow escaped the influence of Snooki and The Situation. Waconda, Kansas, a small rural community, also is at the 91st percentile. Highland Park, Texas, an affluent community near Dallas, is at the 88th percentile.

Interestingly, of the top 20 U.S. public-school districts in math achievement, 7 are charter schools (some states treat charter schools as separate public-school districts). And most of the 13 traditional districts remaining are in rural communities rather than in a large suburban “refuge” from urban education ills.

Pools of Failure

In total, only 820 of the 13,636 public-school districts for which we have 2007 math results had average student achievement that would be among the top third of student performance in other developed countries. That is, 94 percent of all U.S. school districts have average math achievement below the 67th percentile. There aren’t that many truly excellent districts out there.

Of the 13,636 districts, 9,339, or 68 percent, have average student math achievement that is below the 50th percentile compared with that of the average student in other developed countries. Most of our large school districts are well below the 50th percentile. This is especially alarming, because these lower-performing large districts comprise a much greater share of the total student population than do the relatively small higher-performing districts.

The average student in the Washington, D.C., school district is at the 11th percentile in math relative to students in other developed countries. In Detroit, the average student is at the 12th percentile. In Milwaukee, the average student is at the 16th percentile. Cleveland is at the 18th percentile. The average student in Baltimore is at the 19th percentile in math relative to students in other developed countries. In Los Angeles, the average student is at the 20th percentile. The average student in Chicago is at the 21st percentile in math. Atlanta is at the 23rd percentile. The average student in New York City is at the 32nd percentile in math. And in Miami-Dade County, the average student is at the 33rd percentile in math.

Not 1 of the largest 20 school districts is above the 50th percentile in math relative to other developed countries. Those districts contain almost 5.2 million students or more than 10 percent of the country’s schoolchildren. The rare and small pockets of excellence in charter schools and rural communities are overwhelmed by large pools of failure.

Previous Research

The Global Report Card is not the first analysis to compare the performance of U.S. students to international peers. Eric A. Hanushek, Paul E. Peterson, and Ludger Woessmann (see “Teaching Math to the Talented,” features, Winter 2011) used a very similar method to compare the performance of students in each state to students in other countries and arrived at similarly gloomy conclusions. Using state NAEP results for 8th-grade students and PISA results for 15-year-olds internationally, the researchers focused on the percentage of students performing at an advanced level in math. In almost every state, they found that we had far fewer advanced students than most of the countries taking PISA. They also narrowed the comparison to white students in the U.S. and to students whose parents had a college education to show that even advantaged students in the U.S. failed to achieve at an advanced level in math relative to their international peers. More recently, Hanushek et al. updated their analysis to examine the percentage of students in each state and across countries performing at the proficient level in math and reading.  The results were similarly disappointing.

The main difference between the GRC and the Hanushek et al. analyses is that in our study we push the comparison down to the district level. By focusing on white students and children of college-educated parents, Hanushek et al. clearly mean to convey that even students in elite suburban districts have mediocre achievement. Our contribution with the GRC is to name the districts so that people do not indulge the fantasy that their suburb’s record is somehow different from the disappointing performance of others with advantaged students in their state.

There are other important differences between the GRC and the Hanushek et al. analyses. We incorporate test results for U.S. students in all available grades (typically grades 3 through 8 and grade 10) rather than focusing on the grade closest to the 15-year-olds in the PISA sample. We could have focused only on 8th-grade results, as Hanushek et al. did, but in doing so we would have greatly reduced the number of test results on which we were doing the calculations for school districts. We preferred to gain precision in estimating the achievement in each district by increasing our sample size rather than restricting the sample to 8th graders in order to gain comparability in the age of the students under review.

The GRC analysis also differs from those of Hanushek et al. in that the latter focus on students performing at the advanced or proficient level, while we focused on the average student performance in both math and reading. Hanushek et al. concentrated on advanced or proficient performance because they were trying to compare our best students with the best abroad to show that even our best are mediocre. We did the same by highlighting the results for elite suburban school districts. Focusing on the average also avoids any dispute about how “advanced” or “proficient” are defined across different tests.

Gary Phillips at the American Institutes for Research has also conducted a series of analyses comparing state achievement on NAEP to international performance on a different international test, the Trends in International Mathematics and Science Study (TIMSS). Phillips arrives at somewhat less gloomy conclusions about U.S. performance, but that is because the countries included in TIMSS differ from those covered by PISA. Hanushek et al. rightly note that PISA provides a much more appropriate comparison for the U.S.: “Put starkly, if one drops from a survey countries such as Canada, Denmark, Finland, France, Germany, and New Zealand, and includes instead such countries as Botswana, Ghana, Iran, and Lebanon, the average international performance will drop, and the United States will look better relative to the countries with which it is being compared.”

This has sparked a debate among researchers about whether TIMSS or PISA provides a better set of countries against which we should compare the U.S. The Global Report Card circumvents this dispute by developing its own set of countries against which we compare U.S. students. The comparisons provided by TIMSS and PISA depend on which countries decide to take each test each time it is administered. And PISA scales its scores against the results for members of the OECD, which excludes countries like Singapore while including countries like Mexico. Our comparison group depends on PISA results, but it is also based on objective criteria, like per-capita GDP, to identify a set of developed economies that can be reasonably compared with that of the U.S. Our comparison group is a significant improvement on the self-selection of countries that choose to take a test as well as an improvement upon arbitrary membership in an organization like the OECD.

No Refuge

The elites, the wealthy families that have a disproportionate influence on politics, clearly recognize the dysfunction of large urban school districts and have sought refuge in affluent suburban districts for their own children. But the reality is that there are relatively few pockets of excellence to which these families can flee.

In four states, there is not a single traditional district with average student achievement above the 50th percentile in math. In 17 states, there is not a single traditional district with average achievement in the upper third relative to our global comparison group. And apart from charter school districts,  in over half of the states, there are no more than three traditional districts in which the average achievement would be in the upper third.

The elites in those states have almost nowhere to find an excellent public education for their children. But state accountability systems and the desire to rationalize the lack of quality options have encouraged the elites to compare their affluent suburban districts to the large urban ones in their state. These inappropriate comparisons have falsely reassured them that their own school districts are doing well.

This false reassurance has also perhaps undermined the desire among the elites to engage in dramatic education reform. As long as the elites hold onto the belief that their own school districts are excellent, they have little desire to push for the kind of significant systemic reforms that might improve their districts as well as the large urban districts. They may wish the urban districts well and hope matters improve, but their taste for bold reform is limited by a false contentment with their own situation.

But the elites should not take comfort from the stronger performance of affluent suburban districts relative to large urban districts. As the Global Report Card reveals, even our best public-school districts are mediocre when compared with the achievement of students in a set of countries with developed economies.

Of course, the Global Report Card does not isolate the extent to which schools add or detract from student performance. Factors from student backgrounds, including their parents, communities, and individual characteristics, have a strong influence on achievement. But the GRC does tell us about the end result for student achievement of all of these factors, schools included. And that end result, even in our best districts, is generally disappointing.

Jay P. Greene is professor of education reform at the University of Arkansas and a fellow at the George W. Bush Institute. Josh B. McGee is vice president for public accountability initiatives at the Laura and John Arnold Foundation.

LAUSD is being sued by a group that includes Alice Callaghan, a member of the Episcopalian clergy and the manager of Las Familias del Pueblo, a community center for the poor and homeless in downtown Los Angeles. Back in 1996, Callaghan organized 70 Spanish-speaking immigrant parents, who boycotted the Ninth Street Elementary School — calling for an end to failed bilingual-education methods and instead demanding that the school system teach the children of immigrant garment workers academic English as soon as possible.

Callaghan and this different group of parents are suing to enforce the Stull Act. The law goes back four decades and says that the board of trustees of each school district shall evaluate teachers, at least in part, by their student’s performance on the state’s standards-based tests. The law says “shall,” not “may.” It is mandatory that each district do this.

(The law is named for its sponsor, now-deceased Republican Assemblyman John Stull of San Diego, who received bipartisan support at the time for this statutory requirement that teachers be held accountable for the academic achievement of their pupils.)

The attorneys for the plaintiffs are Kyle Kirwan, a prominent Los Angeles litigator, and Scott Witlin, both partners at the law firm of Barnes & Thornburg. Their request for a court order was drafted in consultation with EdVoice, a Sacramento-based education-advocacy group. Before going to court, the plaintiffs sent a letter on Oct. 26 asking the district to comply. The letter stresses that for years the district has engaged in wanton lawlessness. In the letter, the plaintiffs’ attorneys say that the district “refuses to implement the Stull Act in complete abdication of its responsibility to its students, their parents, and the taxpayers of the district.”

The letter says that the district has never evaluated the teachers using student test scores, and, as a consequence, has never told teachers where they stood and counseled them on how to improve in terms of increasing their students’ learning – all of which are required by the law. “In short, the district has never complied with the Stull Act.”

The letter also points to the involvement of the teachers’ union United Teachers Los Angeles (UTLA) in this lawbreaking. Compliance with the law, the letter says, has been “deliberately evaded” through a series of “complicitous” collective-bargaining agreements between the LAUSD and UTLA, at the expense of students — who deserve effective teachers.

Specifically, the district has been pretending that it can avoid compliance with the Stull Act by making collective-bargaining agreements with the teachers’ union that overrule a statute (the Stull Act) passed by the state legislature. It doesn’t work that way. Valid contracts are written under and within the law, not in violation of the law. The lawsuit seeks to end this make-believe in the service of lawbreaking.

In their Nov. 1 petition for a court order, the plaintiffs’ attorneys say that the UTLA has treated the public school system in Los Angeles as “a taxpayer-funded jobs and entitlement program” for adults, even when a teacher‘s performance would be considered “demonstrably unsatisfactory” when judged by pupil results.

The petition described how the teachers’ union adopted a strategy of “stonewalling” when it came to putting the Stull Act into effect. “In collusion with the District‘s governing boards and superintendents,” the petition says, the teachers’ union has blocked lawful evaluation of teachers and the “corrective action” needed to ensure that students get effective teachers.

As a consequence, “the adults‘ collective employment and political interests” are turning the children’s opportunity for learning while in school “on its head” and instead the system is providing job guarantees to teachers as well as “preserving the political power of the Board and the Superintendent.” All of this comes at the expense of children — particularly the “socio-economically disadvantaged.”

These shenanigans by the district and the union have been presented to the public in a way that is designed to pull the wool over people’s eyes: “The result has been a perversion of the evaluation system and a knowing effort to deceive the public using educational jargon.”

Witlin, one of the attorneys, told education policy analyst and blogger RiShawn Biddle: “The school district is supposed to exist for the benefit of the children and not for the adults.”

The teacher evaluation program that is in place in Los Angeles, according to the petition, “does not comply with the Stull Act” and “perpetuates a fraud on the community” by letting teachers get high evaluation ratings whether or not their students are learning the material listed in the curriculum-content standards.

The petition cites damning statements from LAUSD Superintendent John Deasy in which he condemns his own evaluation program for teachers. For example, he recently said: “I would argue that nobody has told me that the current system of evaluation, which is performance review, helps anybody. It is fundamentally useless. It does not actually help you get better at [your] work and it doesn‘t tell you how well you’re doing.”

Superintendent Deasy also stated: “One would have to argue: ‘So … there are schools where 3 percent of the students are proficient at math and 100 percent of the teachers are at the top rating performance.’ That doesn‘t make sense to me whatsoever. And it doesn‘t make sense because the rating performance does not actually help teachers get better.”

In terms of what actually happens, the district is condemned out its own mouth.

Back on March 13, 2011, retired Los Angeles school district teacher Doug Lasken and I wrote an opinion column for the San Francisco Chronicle about non-compliance with the Stull Act in Los Angeles and other California districts – so I could not be happier about this lawsuit, which may finally bring some justice for Los Angeles schoolchildren after years of the district’s deliberate dodging of the law. Success in Los Angeles will mean that districts across California will have to begin evaluating teachers properly and getting struggling employees the extra help they need to become effective teachers.

LAUSD has been negotiating with UTLA to try to put in place a pilot program with three percent of district teachers, who would be evaluated in part on student performance on the state’s standards-based tests. But these negotiations are deadlocked because of the refusal of UTLA to even study the idea of complying with the law.

The plaintiffs in this case reject the proposed pilot program, which has no guarantee of ever having meaningful evaluations that actually count, even for the volunteer participants in the pilot. They point out that LAUSD has a record of “years of non-compliance” with the Stull Act and that there is no reason to believe that the pilot would even expand to the other 97 percent of teachers. “Sadly, the District has abdicated its duty to the children.” The plaintiffs demand instead that LAUSD comply with the Stull Act as soon as practically possible “in its entirety.”

-Bill Evers

Last night was fun for the kids, but today is every education wonk’s favorite holiday: NAEP release day! Kevin Carey is already out with some savvy analysis; let me add some thoughts on the trends in reading.

The big news is that we finally eked out some statistically significant progress in 8th-grade reading. This goal has eluded us before, and has led commentators such as E.D. Hirsch to note that we’re not doing enough to build kids’ content knowledge and vocabulary. Initiatives like Reading First might have helped our youngsters to decode, goes the argument, but that’s not enough to create strong readers, especially as kids get older.

That’s still true, I think, but the NAEP results might indicate that those decoding skills are nothing to scoff at. The middle schoolers who took the NAEP last spring were in first grade in 2004–the heyday of Reading First implementation. It’s possible that scientifically-based reading instruction got them off to a better start as readers, and that head-start has been maintained through elementary and middle school. I can’t prove it (it’s NAEP–no one can prove anything!) but it’s a hypothesis worth exploring. Furthermore, the 8th graders who made the greatest progress since the early 2000s were the lowest-achievers–the very population Reading First was designed to help.

What’s disappointing is that 4th-grade reading results have held steady since 2007–after a big bump up (across all achievement levels) from 2005-2007. This one-time bump might be credited to Reading First (again, stress on “might”). But only a few states have continued making progress; in the last two years only Alabama, Hawaii, Maryland, and Massachusetts have done so. I can’t quite explain Maryland and Hawaii (OK, maybe they DO deserve Race to the Top funds, after all) but Alabama and Massachusetts have some of the most aggressive policies in place to promote research-based reading instruction. And it shows.

Again, these are just guesses. The big question going forward, it seems to me, is whether any of our reform efforts are like to lead to another big bump in test scores anytime soon. Large-scale initiatives like “accountability” and “parental choice” set the context for improvement, but they have rather indirect impacts on achievement. More focused instructional and teacher quality strategies–like implementing the Common Core standards or improving teaching through better evaluation systems–are more likely to result in big gains, it seems to me. Neither of those will be in full force until around 2013 or 2014. So I would expect more steady-state on NAEP scores for the time being, with our next big chance for major gains coming in 2015.

Those are my thoughts. What about you?

-Mike Petrilli

This post also appeared on Flypaper.

I must admit to a bias: I am a strong advocate of national standards, was intimately involved with their first iteration in the 1980s, and am delighted to witness their partial resurrection in a new guise.  As Robert Rothman observes, the new Common Core standards in English language arts and mathematics are not top-down driven reforms (one of the difficulties of the first national standards initiatives) or bottom-up efforts, which have suffered in the past when the states, with the singular exception of Massachusetts, tended to water down their individual attempts to the detriment, rather than the amelioration, of American public education.

The Common Core standards, to quote directly from the book’s introduction, “set expectations for student learning at every grade level,” and the book “describes the development process, the states’ adoption decisions, and early steps by states to implement them.”  Most important, the book explains in depth the content of the standards, what they expect of students, and how the assessment of student results is going to be carried out.

While all of this activity, strenuous and complex as it is, may seem to the educational neophyte to be more theoretical than practical, the fact of the matter is that within six months of the standards being issued in 2010, 43 states and the District of Columbia had adopted them.  Furthermore, they are designed to be “all or nothing.”  (It is difficult if not impossible to adopt some of them.)  They are written with every student in mind, rather than for the gifted few.  Their potential for transforming what is taught and raising the level of academic achievement nationally is truly extraordinary.

Why am I guilty of such unbridled optimism?  First of all, a great deal was learned from the pre-Common Core efforts. The first version of the national standards in the 1980s was vastly too ambitious.  Second, current federal education policy is very favorably disposed towards the common core initiative. Third, international comparisons with other highly developed countries, once shunned, are now fashionable.  They reveal that, no matter how the tests are framed, America is in the middle of the pack, well behind the likes of Finland, Singapore and Japan, in what we traditionally expect of our high school graduates. Fourth, other organizations are in the process of developing common core standards in science (to be released in 2012).  Assuming that they are of the same high quality as their 2010 counterparts, people may be emboldened to do the same for the other basic subjects, and thus escape from the current tendency to narrow the curriculum to the point of no return, a concern of particular moment to Diane Ravitch.  Fifth, unlike the situation in the 1980s, the charter school movement has matured to the point that its growth can provide a nationwide institutional context to pilot the teaching strategies appropriate to implementing the common core strategies.

Finally, there is a significant movement to align teacher education with these new standards.  Ross Perot once said to me “All teacher colleges ought to be torched.”  Such single-minded excoriation may be over the top, but there is no question that the new three R’s of teacher recruitment, retention and renewal are integral to any genuine education renaissance, and are indispensable to the implementation of the common core standards. Let’s hope that all of this really happens.  Robert Rothman certainly thinks there is a good chance it will.  After all, all these favorable circumstances are referenced in this informative volume.

-A. Graham Down

More book reviews by Graham Down can be found here.

For nearly a decade, the No Child Left Behind Act (NCLB) has focused the attention of policymakers and researchers squarely on the achievement of low-performing students, with some apparent success. The math and reading scores on the National Assessment of Educational Progress of the nation’s lowest-achieving 10 percent of 4th and 8th graders have risen sharply since 2000, continuing a trend that began in the 1990s. Yet some may wonder about the potential cost of this focus on higher-achieving students, for whom improvements over the same time period have been modest. Among the questions related to this debate is whether additional programs and resources should be devoted to students on the higher end of the spectrum, those considered gifted.

Three million students in the United States are classified as gifted, yet little is known about the effectiveness of traditional gifted and talented (G&T) programs. In theory, G&T programs might help high-achieving students because they group them with other high achievers and typically offer specially trained teachers and a more advanced curriculum. While previous research indicates that ability grouping is in fact correlated with higher achievement, these findings could be misleading if students placed in high-ability classrooms were likely to be successful for reasons that researchers are unable to measure, such as stronger motivation. To our knowledge, no existing studies offer convincing evidence on the causal effect of G&T programs on student achievement.

Our research begins to fill this gap with two studies of the G&T programs available to high-achieving middle-school students in a large urban school district in the southwestern United States which, to preserve anonymity we shall refer to as LUSD. Since 2007, all 5th-grade students in LUSD have been evaluated to determine eligibility for gifted and talented programs starting in 6th grade. Those students who are deemed eligible often are grouped in classes with other gifted students. They are also permitted to apply for admission to two middle schools that have oversubscribed magnet G&T programs.

The two studies use different methods to ask distinct but closely related questions. The first exploits the fact that eligibility for G&T programming in LUSD is determined by a well-defined cutoff in students’ evaluation scores. By comparing students who score just above the cutoff to those who score just below, the study provides evidence on the effect of enrollment in a G&T program on achievement for those students on the margin of eligibility. The second study takes advantage of the randomized lotteries that determine admission to the district’s two premier magnet G&T programs. By comparing students who win the lottery and attend the magnet G&T schools to those who lose the lottery and attend other “neighborhood” programs, the research provides evidence on whether the magnet G&T programs provide any additional benefits.

The results of both studies will be discouraging for those hopeful that current G&T programs provide a means to accelerate the progress of our most capable students. The first shows that barely eligible students who participated in LUSD’s G&T curriculum for all of 6th grade and half of 7th grade exhibit no significant improvement in test scores across a range of subjects, despite their being surrounded by higher-achieving peers and taking more advanced courses. The lottery study corroborates these results, as students admitted to the G&T magnet schools show little improvement in test scores by 7th grade, despite having higher-achieving peers and being taught by more effective teachers. The lone exception is in science, where students admitted to G&T magnet schools performed at substantially higher levels.

It is difficult to know what accounts for these puzzling results. Our best guess, which we discuss in detail below, is that being placed with higher-achieving peers is not all that it is cracked up to be. Students admitted to both types of G&T programs suffer a large drop in their relative rank in terms of grades within their classes, which could have adverse consequences that offset any benefits of improvements in their educational environment. But we are getting ahead of ourselves. Let’s first take a closer look at the programs and the evidence on their effects.

Gifted Students in LUSD

LUSD is a large school district, with more than 200,000 students. The district is heavily minority and very low income; the minority population is more heavily Hispanic than African American. All LUSD students are evaluated for placement in middle-school G&T programs during 5th grade, including those who participated in the district’s G&T program in elementary school. In order to be deemed eligible for the middle school G&T program, a student must meet the eligibility criteria set forth in the “gifted and talented identification matrix.” The matrix converts scores on standardized tests—the Stanford Achievement Test for English-speaking students and the Aprenda exam for Spanish-speaking students with limited English proficiency—scores on the Naglieri Nonverbal Ability Test (NNAT), average course grades, teacher recommendations, and indicators for socioeconomic status into an overall index score.

While all students who meet these requirements qualify for the G&T program, not all end up being classified as G&T, because parents are allowed to opt out. Some students also enroll in the program initially but later withdraw. Schools in LUSD have a monetary incentive for attracting gifted students, as LUSD provides a funding boost of 12 percent over the average allotment for a regular student.

Gifted students in LUSD are far less likely to be economically disadvantaged and more likely to be white or Asian than other students in the district. They also perform at far higher levels on the Stanford Achievement Tests, which the district administers annually in five subjects: math, reading, language, social science, and science. Their advantage in math and reading test scores in 5th grade is roughly 0.7 of a standard deviation, which amounts to well over two years of academic progress (see Figure 1). By the time the same students have reached 7th grade, these gaps have widened to 1.5 standard deviations in math and 1.25 standard deviations in reading. While this pattern suggests that the students enrolled in the district’s G&T programs learn at a faster rate between 5th and 7th grade, it does not necessarily mean that the G&T programs are the cause. It is to that question we now turn.

Effects on Barely Eligible Students

Our first study examines the effects of participation in a G&T program on students who were just barely eligible to participate based on their overall index scores. We focus on students who were evaluated for G&T eligibility as 5th graders in the spring of 2008 for whom we are able to observe outcomes as 7th graders in the 2009–10 school year. Our outcome measures include Stanford Achievement Test scores and attendance rates, both of which are drawn from administrative data provided by the district. After restricting the sample to students near the G&T eligibility cutoff, we are able to examine these outcomes for roughly 2,600 students.

The method used in the study, known as regression discontinuity analysis, takes advantage of the fact that the district uses a strict numerical cutoff in the index score assigned to students as 5th graders in order to determine their eligibility to participate in the G&T program the following year. Because the students are unable to precisely manipulate their index scores, those scoring just below the eligibility cutoff should be very similar to those scoring just above the cutoff. We can therefore attribute any differences in student outcomes on either side of the cutoff to the effect of having being deemed eligible.

As noted above, not all eligible students end up participating in G&T programs due to factors such as a parent’s decision to opt out. Similarly, some students who do not initially qualify later become eligible through an appeals process that allows parents to submit an alternative standardized test score or through additional evaluations conducted in 6th grade. As a result, we use standard statistical techniques to account for the fact that the cutoff our regression discontinuity analysis exploits is “fuzzy” rather than sharp. This allows us to provide evidence on the effects of actual participation in the G&T program, not simply eligibility for it.

Before looking at student outcomes, we first used the same method to confirm that participation in the district’s standard G&T programs led to measurable differences in students’ educational experiences. Clearly, it did. The average achievement of the peers in G&T students’ classrooms were between 0.25 and 0.33 of a standard deviation higher in each core academic subject. Participation in the G&T program also increased the number of advanced courses in which students enrolled in 6th and 7th grade. We found no evidence, however, that the teachers to whom students in the G&T program were assigned were any more effective, as measured by their impact on student test scores.

Did these improvements in peer characteristics and curricular rigor translate into improved outcomes? Our results indicate that they did not (see Figure 2). Our estimates of the effects of G&T participation for barely eligible students are close to zero in all five subjects and are sufficiently precise to allow us to rule out with 90 percent confidence effects as small as 0.04 standard deviations (sd) in math, 0.07 sd in reading, 0.12 sd in language, 0.10 sd in social studies, and 0.19 sd in science. We also looked at the impact of G&T participation for specific student subgroups defined by gender, race/ethnicity, socioeconomic status, and whether the students had been classified as gifted in elementary school. We found little evidence of differential impacts for students in any of these groups.

The Effects of G&T Magnet Programs

Why does the G&T program in LUSD not yield benefits for students on the margin of eligibility? One reason could be that the qualification boundary is set so low that such students are not able to take advantage of the programs’ purported benefits. Our second analysis, which uses experimental research methods to study the effects of enrollment in the district’s G&T magnet programs, is intended to shed light on this concern.

LUSD has 41 middle schools, of which 8 have G&T magnet programs, and 2 of these are oversubscribed. As a result, the district uses lotteries to determine which students will be admitted as 6th graders. Our analysis compares the performance of students who win the lottery and attend one of the G&T magnet programs to those who lose the lottery and either attend a neighborhood G&T program in the district, a magnet school based on a different specialty, or a charter school. Because the lottery is random, any differences in outcomes between lottery winners and losers can be attributed to the effect of enrolling in the G&T magnet program rather than one of these alternatives. Moreover, the results of this analysis will apply to the entire population of students who chose to apply.

Our lottery analysis is based on the sample of LUSD 5th-grade students determined to be eligible for G&T programs in 2007–08 who applied for admission to one of the two middle schools with an oversubscribed G&T magnet program. This group includes 542 students, 394 of whom were offered admission and 148 of whom were not. We find no statistically significant differences in the observed characteristics of lottery winners and losers, suggesting that the lotteries were in fact conducted in a random way.

The students in the lottery differ both academically and demographically from the students who were included in the regression discontinuity study. Not only do the lottery students have higher test scores than students at the eligibility cutoff, but their test scores exceed those of the average G&T student in the district. Lottery participants are also less likely to be on subsidized lunch, and less likely to be minority.

Of the 542 lottery participants, only 440 students, including 331 winners (84 percent) and 109 losers (74 percent), remain in LUSD by 7th grade. Fortunately, the observed characteristics of lottery winners and losers who remain in the district continue to be very similar. Even so, when analyzing the data we control for students’ demographic characteristics and prior achievement, and use weights designed to make the final sample comparable in terms of its observed characteristics to the set of students that initially applied for the lottery.

One disadvantage of this second study is that the lottery losers have a range of alternative experiences and most participate in standard G&T programs, so the comparison group’s educational experience is less clear than it was in the regression discontinuity analysis. Nonetheless, our data confirm that students admitted to the G&T magnet schools with lotteries seem to have experienced large improvements in their educational environment. Winning the lottery increased the average achievement of students’ classroom peers by as much as a full standard deviation in some subjects. And in contrast to the G&T program as a whole, students admitted by lottery to G&T magnet program were assigned to more effective teachers.

Turning to student outcomes, however, our results provide little evidence that attending a G&T magnet program leads to improvements in student achievement (see Figure 3). The one exception is science test scores, for which we estimate a positive effect of 0.28 standard deviations. Due to the relatively small sample sizes, all of the effects are imprecisely estimated and do not allow us to definitively rule out reasonably large positive effects. Even so, the estimated effects for math, reading, and social studies are negative, and the estimated effect for language is effectively zero.

Discussion

It is difficult to understand why we find little evidence that G&T programs positively affect achievement. A common concern with studies of high-achieving students is that the available achievement measures may not be well suited to discern improvements for this group. This would be particularly worrisome if we were using a state accountability exam targeted toward low-achieving students, but it is less of an issue with the Stanford Achievement Tests. Indeed, we found little evidence of students performing near the maximum levels on these tests in either the regression discontinuity or lottery samples. Although it is possible that the additional course material taught in G&T classes is poorly aligned with topics covered in the achievement test, research documenting the benefits of being placed with higher-ability peers suggests that we should see improvements, even if that were the case.

The effect of being placed in a higher-ability classroom may not necessarily be positive, however, especially for a marginal G&T student. In particular, the drop in ranking relative to one’s peers may have a negative effect: a marginal G&T student is likely to go from being near the top of the regular class to being near the bottom of the G&T class. Even students in the middle of the G&T distribution are likely to experience a loss of ranking in the magnet G&T schools as compared to their neighborhood schools. It may be that students are demoralized by the drop in their relative rankings or that teachers provide more resources to students at the top of the class.

Substantial evidence from educational psychology indicates that students who are placed in higher-achieving groups can suffer psychological harm. A commonly used measure is a student’s “self-concept,” how a student perceives her abilities relative to an objective measure such as achievement. A 1995 study by Herbert Marsh and colleagues compared G&T students to observably similar students in mixed G&T and non-G&T classes and found that G&T students show declines in their math and reading self-concept. More recent research has documented lower self-concept and greater test anxiety among gifted students in ability-segregated classrooms.

Although we do not have direct evidence on student confidence, we can make use of student course grades and rank within the class to probe for evidence consistent with this kind of effect. We evaluate the impact of G&T program enrollment in the regression discontinuity study and of attending a G&T magnet in the lottery analysis. In both cases, we find clear reductions in student grades. For the regression discontinuity sample, grades fall by a statistically significant 4 points out of 100 (3 points changes a grade from a B+ to a B, for example) in math and by 2 to 3 points in other subjects, although these effects are not statistically significant for 7th grade. For the lottery analysis, the grade reductions are even more dramatic, with drops of 7 points in math, 8 in science, and 4 in social studies.

It is also useful to consider how students’ rankings within their peer groups differ by treatment status, as this provides a direct measure of how a student may perceive his position in the overall distribution of student ability. We assume that students mostly compare themselves to their schoolmates who take the same courses in the same grade. Thus, we rank students within each school, grade, and course by their final course grades and then convert these rankings to percentiles. The rankings based on 7th-grade courses exhibit notable drops when students cross the G&T eligibility threshold. Controlling for race, gender, economic disadvantage, LEP (Limited English Proficiency), and prior gifted status, marginal G&T students have a relative rank in 7th grade that is 13 to 21 percentiles lower than similar students who were not admitted. Attending a premier G&T magnet in 7th grade generates a nearly 30 percentile ranking drop in all four of the courses examined.

In short, the necessary conditions are clearly met for a drop in relative ranking to play a role in offsetting the expected positive impact of more rigorous courses, more effective teachers, and higher-achieving peers. The possibility that G&T students are subject to such a mechanism suggests potential constraints on the benefits of programs that provide more similar peers and an increase in traditional education inputs.

One should not conclude from the lack of achievement results, however, that the G&T programs should be scuttled. Our analysis occurs in a district with a large number of relatively high-quality magnet programs, and thus the alternatives to the G&T programs may be strong. There may also be benefits that we are not able to capture, such as impacts on SAT scores, graduation rates, and college attendance. Further, our study examines a G&T program in one district. Certainly, districts vary in the approaches they take to educating gifted students, so it may be that similar studies of programs in other districts would yield different results. Nonetheless, this study does raise questions about the efficacy of G&T programs and the traditional model of ability-segregated classrooms.

Sa Bui is a doctoral candidate in economics at the University of Houston, where Steven Craig is professor of economics and Scott Imberman is assistant professor of economics.

In “Accountability’s End?“, Petrilli divides accountability supporters into four distinct groups, which I found accurate. His categories also make it a lot easier to explain why teachers have so much trouble explaining that we are not against all accountability, so much as we have seen the idea of accountability misused for political reasons. We also have ongoing experience with its unintended consequences.

The truth is that while many teachers disagree strongly with some of the groups described in Petrilli’s article, we are in agreement with others. Opinions among teachers vary, so I won’t claim to speak for all of us, but here are my reactions to each group as Petrilli describes them.

The Tough Lovers: This group wants to make sure that teachers are not unduly shielded from a tough economy and that only hard, competent workers stay on school payrolls. Personally, I don’t mind. I’ve never had a problem with being expected to do a good job – few teachers I’ve ever met have a problem with that. Then again, I have a smart and fair-minded principal who isn’t likely to bully me over some comment I’ve made at a faculty meeting. Not all teachers are so lucky. More so than in the private sector, teachers let go for poor performance – or perceived poor performance – will likely have their careers permanently destroyed. It would be nice to see increased “tough-love” in HR departments balanced out with options for good teachers who get stuck in tough situations.

The World is Flatters: This group supports things like STEM and the Common Core standards as a means to make American education more cohesive and keep us competitive with other countries. I have no major disagreements with this group, and have yet to hear from a teacher who does.

The Tight-Loosers: This group favors results-based accountability as a means to cut back on traditional regulation. On paper, the idea of using some type of end results as a means of giving teachers more autonomy sounds great. For example, I’d love English teachers to be able to read more novels instead of giving expensive, time-consuming, relatively useless bi-weekly assessments. At the same time, teachers get uneasy about the “tight-looser” camp because we’ve seen firsthand that standardized tests don’t tell us everything accountability hawks say they do. Plus, increased emphasis on test results has yet to be matched with more autonomy in most public schools – instead they are the justification for things like replacing novels with bi-weekly assessments. For now, I’d describe the way it plays out as the “tight-tighter” approach.

The Poverty Warriors: This group claims that test-based accountability will keep schools and teachers from shortchanging poor, minority students. It is generally arguments from this group – however well intentioned – that frustrate me the most. Teachers at low-income schools often choose to work there in spite of problems known to impact student achievement. The “poverty warrior” rhetoric has recast these teachers as lazy, racist conspirators against poor kids. It is disingenuous and unfair to suggest that non-teachers in clean, well-decorated offices with all the copy paper they could ever ask for somehow care more about poor kids than teachers who get up at 5AM and break up hallway fights and work with these kids every day.

Teachers have also seen how many accountability measures – even well-meaning ones – have unintended consequences that undermine their stated goals. When we bring this up, however, even if we are really only arguing with one of the four groups above, we get slapped with the label of being against everything all four groups stand for, and thus treated as lazy, against what’s best for children, unrealistic about what kids need to know, and un-caring.

If we had those four characteristics, why, exactly, would we have chosen this profession?

While I don’t agree with every statement in Petrilli’s article, I still hope a lot of people read it. It is definitely a starting point for a more thoughtful discussion of this issue.

-Roxanna Elden

Roxanna Elden is a National Board Certified Teacher in Miami, and the author of See Me After Class: Advice for Teachers by Teachers.

I was prepared for a rant against all things reform when I started reading the New York Times Q & A interview with Maria Velez-Clarke, the principal of the Children’s Workshop School in Manhattan’s East Village, about the school’s C-grade from the City.  The school is “one of several small schools,” said the Times intro, “started in the 1990s by people who had worked at the widely praised Central Park East School.”

Central Park East?  The school started by Deborah Meier, current scourge of standardized tests, charters, accountability, and just about everything associated with Michael Bloomberg and Joel Klein, who initiatiated the school report cards program?  (See the Bridging Differences blog Meier shares with Diane Ravitch and this wonderful 1994 profile of Meier and her hugely successful Central Park East experiment written by veteran NYC educator Sy Fliegal.)  Children’s Workshop offers ballet and yoga, for heaven’s sake!

Instead of a progressive principal complaining about Gotham’s new accountability system squishing her student’s creative impulses, however, we hear an 18-year veteran school leader who was shocked by the C grade the school received in 2010 and determined to do something about it:

I shared it with absolutely no one because it was so devastating to me. I took it home. I sat with my husband and I said, “My God, do you know what this is going to do to morale?” And he looked at me and he said, “O.K., you have the weekend: have a pity party and then move on.”

Velez-Clark actually went to the Transit Museum and bought “C” buttons (for the C train), brought them to school, sat down with her staff and said, “O.K., now what do we have to do here in order to get off the C train and get on the B train?”

She then took her staff on a weekend retreat, where they reviewed every child’s test scores.  And what is most interesting about the school’s response is that Velez-Clark seems unafraid to admit that she has learned something that may be good for her students.

When she and her teachers began to dig into the test scores, for instance, they discovered

….a correlation between attendance and a child’s score. So we worked on attendance. I didn’t always send a note home before, but now sometimes if a child is absent too much, I have to send a letter home saying “this could lead to A.C.S. [Administration for Children's Services] coming to visit your house” or “your child is at risk of being held over because of attendance.”

The school didn’t stop being progressive, it simply integrated cultural history into its ballet lessons and nutrition and science into its yoga classes.

This is how it’s supposed to work.  Congratulations to the staff and children of Children’s Workshop for their B grade this year – and showing that it’s okay to do well in tests.

–Peter Meyer

This post also appears on Flypaper

A  more complete integration of testing, accountability, and teaching would be superior to dealing with the integrity of testing in isolation.   Let’s put the tests out in the sun instead of trying to lock them up in more and more secure rooms.

The use of student outcome measures for accountability is now firmly entrenched and is not about to go away.  But a variety of complaints about the current testing system exist.   First, the tests tend to narrow the teaching to just what is expected on the tests – with excessive teaching to the test and drilling on  practice tests.  Second, the tests are too easy for students in some schools and too hard for others, either wasting the time of some or frustrating others.  Third, using tests for accountability purposes encourages cheating and other ways to evade scrutiny.

To address these different issues, we need to think differently about aims and means.  Here is a brief proposal to deal with all of the problems.  It starts with developing a large item bank of test questions of varying difficulty.  Imagine 1,500 questions for fourth grade math that cover the entire scope of appropriate material from basic to advanced topics.  Next, make all of the test items – not just sample items – publicly available and encourage teachers to teach to the test, because the items cover the full range of the desired curriculum.  Making the items public will also ensure the quality of the test items.  One could invite feedback ratings or open sourcing to provide a path to improving the questions over time.  Then, move to computerized adaptive testing, where answers to an initial set of questions move the student to easier or more difficult items based on responses.  This testing permits accurate assessments at varying levels while lessening test burden from excessive questions that provide little information on individual student performance.  Such assessments would not be limited to minimally proficient levels that are the focus of today’s tests, and thus they could provide useful information to districts that find current testing too easy.  Students would be given a random selection of questions, and the answers would go directly into the computer – bypassing the erasure checks, the comparison of responses with other students, and the like.

This proposal actually follows the current testing by the FAA of knowledge needed to obtain a private pilot license.  While there are commercial books on these tests, replete with questions and answers, the efficient way to prepare for the tests is simply to learn the underlying concepts.  It is not to attempt to memorize the answers, because it is easy to confuse such an attempt.

What are the potential problems?  Some say developing test items would make this too costly, but remember that it is only necessary to have one item bank, not the continually changed banks of today.  Some worry about ensuring that sufficient computers are available in all schools, but with all of the digital devices currently in use, surely there are a range of possibilities to deliver the tests effectively and efficiently.  There is the problem that the testing companies would not particularly like the proposal.   They find they are happy with mindlessly developing slightly different variants of existing tests for different states, years, and administrations.  But, maybe there are more productive ways for them to enter into the process.

The proposed system would yield quick and reliable feedback on student achievement, would deal with the various cheating and gaming issues, and would more effectively define what students should know than the currently available standards.

-Eric A. Hanushek

Eric Hanushek is a senior fellow at the Hoover Institution of Stanford University and member of the Koret Task Force on K-12 Education.

On August 17, 2011 Paul Peterson discussed the findings of this study in a free online webinar. An archived recording of this webinar can be found here.

At a time of persistent unemployment, especially among the less skilled, many wonder whether our schools are adequately preparing students for the 21st-century global economy. Despite high unemployment rates, firms are experiencing shortages of educated workers, outsourcing professional-level work to workers abroad, and competing for the limited number of employment visas set aside for highly skilled immigrants. As President Barack Obama said in his 2011 State of the Union address, “We know what it takes to compete for the jobs and industries of our time. We need to out-innovate, out-educate, and out-build the rest of the world.”

The challenge is particularly great in math, science, and engineering. According to Internet entrepreneur Vinton Cerf, “America simply is not producing enough of our own innovators, and the cause is twofold—a deteriorating K–12 education system and a national culture that does not emphasize the importance of education and the value of engineering and science.” To address the issue, the Science, Technology, Engineering, and Math (STEM) Education Coalition was formed in 2006 to “raise awareness in Congress, the Administration, and other organizations about the critical role that STEM education plays in enabling the U.S. to remain the economic and technological leader of the global marketplace.” Tales of shortages of educated talent appear regularly in the media. According to a CBS News report, 22 percent of American businesses say they are ready to hire if they can find people with the right skills. As one factory owner put it, “It’s hard to fill these jobs because they require people who are good at math, good with their hands, and willing to work on a factory floor.” According to a Bureau of Labor Statistics report, of the 30 occupations projected to grow the most rapidly over the next decade, nearly half are professional jobs that require at least a college degree. On the basis of these projections, McKinsey’s Global Institute estimates that over the next few years there will be a gap of nearly 2 million workers with the necessary analytical and technical skills.

In this paper we view the proficiency of U.S. students from a global perspective. Although we provide information on performances in both reading and mathematics, our emphasis is on student proficiency in mathematics, the subject many feel to be of greatest concern.

Student Proficiency on NAEP

At one time it was left to teachers and administrators to decide exactly what level of math proficiency should be expected of students. But, increasingly, states, and the federal government itself, have established proficiency levels that students are asked to reach. A national proficiency standard was set by the board that governs the National Assessment of Educational Progress (NAEP), which is administered by the U.S. Department of Education and generally known as the nation’s report card.

In 2007, just 32 percent of 8th graders in public and private schools in the United States performed at or above the NAEP proficiency standard in mathematics, and 31 percent performed at or above that level in reading. When more than two-thirds of students fail to reach a proficiency bar, it raises serious questions. Are U.S. schools failing to teach their students adequately? Or has NAEP set its proficiency bar at a level beyond the normal reach of a student in 8th grade?

One way of tackling such questions is to take an international perspective. Are other countries able to lift a higher percentage—or even a majority—of their students to or above the NAEP proficiency bar? Another approach is to look at differences among states. What percentage of students in each state is performing at a proficient level? How does each state compare to students in other countries?

In this article, we report results from our second study of student achievement in global perspective conducted for Harvard’s Program on Education Policy and Governance (PEPG). In our 2010 PEPG report, we compared the percentage of U.S. public and private school students in the high-school graduating Class of 2009 who were performing at the advanced level in mathematics with rates of similar performance among their peers around the world (see “Teaching Math to the Talented,” features, Winter 2011). The current study continues this work by reporting proficiency rates in both mathematics and reading for the most recent cohort for which data are available, the high-school graduating Class of 2011.

Comparing U.S. Students with Peers in Other Countries

If the NAEP exams are the nation’s report card, the world’s report card is assembled by the Organization for Economic Co-operation and Development (OECD), which administers the Program for International Student Assessment (PISA) to representative samples of 15-year-old students in 65 of the world’s school systems (which, to simplify the presentation, we shall refer to as countries; Hong Kong, Macao, and Shanghai are not independent nations but are nonetheless included in PISA reports). Since its launch in 2000, the PISA test has emerged as the yardstick by which countries measure changes in their performance over time and the level of their performance relative to that of other countries.

Since the United States participates in the PISA examinations, it is possible to make direct comparisons between the average performance of U.S. students and that of their peers elsewhere. But to compare the percentages of students deemed proficient in math or reading, one must ascertain the PISA equivalent of the NAEP standard of proficiency. To obtain that information, we perform a crosswalk between NAEP and PISA. The crosswalk is made possible by the fact that representative (but separate) samples of the high-school graduating Class of 2011 took the NAEP and PISA math and reading examinations. NAEP tests were taken in 2007 when the Class of 2011 was in 8th grade and PISA tested 15-year-olds in 2009, most of whom are members of the Class of 2011. Given that NAEP identified 32 percent of U.S. 8th-grade students as proficient in math, the PISA equivalent is estimated by calculating the minimum score reached by the top-performing 32 percent of U.S. students participating in the 2009 PISA test. (See methodological sidebar for further details.)

What It Means to Be Proficient

According to the National Center for Education Statistics (NCES), which administers NAEP, the determination of proficiency in any given subject at a particular grade level “was the result of a comprehensive national process [which took into account]…what hundreds of educators, curriculum experts, policymakers, and members of the general public thought the assessment should test. After the completion of the framework, the NAEP [subject] Committee worked with measurement specialists to create the assessment questions and scoring criteria.” In other words, NAEP’s concept of proficiency is not based on any objective criterion, but reflects a consensus on what should be known by students who have reached a certain educational stage. NAEP says that 8th graders, if proficient, “understand the connections between fractions, percents, decimals, and other mathematical topics such as algebra and functions.”

PISA does not set a proficiency standard. Instead, it sets different levels of performance, ranging from one (the lowest) to six (the highest). A student who is at the proficiency level in math set by NAEP performs moderately above proficiency  level three on the PISA. (See sidebar for a statement of the 8th-grade proficiency standard and sample questions from PISA and NAEP that proficient students are expected to pass.)

Crossing the Proficiency Bar

Given that definition of math proficiency, U.S. students in the Class of 2011, with a 32 percent proficiency rate, came in 32nd among the nations that participated in PISA. Performance levels among the countries ranked 23rd to 31st are not significantly different from that of the U.S. in a statistical sense, yet 22 countries do significantly outperform the United States in the share of students reaching the proficiency level in math. Six countries plus Shanghai and Hong Kong had majorities of students performing at least at the proficiency level, while the United States had less than one-third. For example, 58 percent of Korean students and 56 percent of Finnish students performed at or above a proficient level. Other countries in which a majority—or near majority—of students performed at or above the proficiency level included Switzerland, Japan, Canada, and the Netherlands. Many other nations also had math proficiency rates well above that of the United States, including Germany (45 percent), Australia (44 percent), and France (39 percent). Figure 1 presents a detailed listing of the scores of all participating countries as well as the performance of the individual states within the United States.

Shanghai topped the list with a 75 percent math proficiency rate, well over twice the 32 percent rate of the United States. However, Shanghai students are from a prosperous metropolitan area within China, so their performance is more appropriately compared to Massachusetts and Minnesota, which are similarly favored and are the top performers among the U.S. states. When this comparison is made, Shanghai still performs at a distinctly higher level. Only a little more than half (51 percent) of Massachusetts students are proficient in math, while Minnesota, the runner-up state, has a math proficiency rate of just 43 percent.

Only four additional states—Vermont, North Dakota, New Jersey, and Kansas—have a math proficiency rate above 40 percent. Some of the country’s largest and richest states score below the average for the United States as a whole, including New York (30 percent), Missouri (30 percent), Michigan (29 percent), Florida (27 percent), and California (24 percent).

Click image to enlarge

Proficiency in Reading

According to NAEP, students proficient in reading “should be able to make and support inferences about a text, connect parts of a text, and analyze text features.” According to PISA, students at level four, a level of performance set very close to the NAEP proficiency level, should be “capable of difficult reading tasks, such as locating embedded information, construing meaning from nuances of languages critically evaluating a text.” (See sidebar for more specific definitions and sample questions.)

Click image to enlarge

The U.S. proficiency rate in reading, at 31 percent, compares reasonably well to those of most European countries other than Finland. It takes 17th place among the nations of the world, and only the top 10 countries on PISA outperform the United States by a statistically significant amount. In Korea, 47 percent of the students are proficient in reading. Other countries that outrank the United States include Finland (46 percent), Singapore,  New Zealand, and Japan (42 percent), Canada (41 percent), Australia (38 percent), and Belgium (37 percent).

Within the United States, Massachusetts is again the leader, with 43 percent of 8th-grade students performing at the NAEP proficiency level in reading. Shanghai students perform at a higher level, however, with 56 percent of its young people proficient in reading. Within the United States, Vermont is a close second to its neighbor to the south, with 42 percent proficiency. New Jersey and Montana come next, both with 39 percent of the students identified as proficient in reading. The District of Columbia, the nation’s worst, are at the level achieved in Turkey and Bulgaria, while the one-eighth of our students living in California are similar to those in Slovakia and Spain. (See Figure 2 for the international ranking of all states.)

Click image to enlarge

Ethnic Groups

The percentage proficient in the United States varies considerably among students from different racial and ethnic backgrounds. While 42 percent of white students were identified as proficient in math, only 11 percent of African American students, 15 percent of Hispanic students, and 16 percent of Native Americans were so identified. Fifty percent of students with an ethnic background from Asia and the Pacific Islands, however, were proficient in math, placing them at a level comparable to students in Belgium, Canada, and Japan.

In reading, 40 percent of white students and 41 percent of those from Asia and the Pacific Islands were identified as proficient. Only 13 percent of African American students, 5 percent of Hispanic students, and 18 percent of Native American students were so identified.

Given the disparate performances among students from various cultural backgrounds, it may be worth inquiring as to whether differences between the United States and other countries are due to the presence of a substantial minority population within the United States. To examine that question, we compare U.S. white students to all students in other countries. We do this not because we think this is the right comparison, but simply to consider the oft-expressed claim that education problems in the United States are confined to certain segments within the minority community.

While the 42 percent math efficiency rate for U.S. white students is considerably higher than that of African American and Hispanic students, they are still surpassed by all students in 16 other countries. White students in the United States trail well behind all students in Korea, Japan, Finland, Germany, Belgium, and Canada.

White students in Massachusetts outperform their peers in other states; 58 percent are at or above the math proficiency level. Maryland, New Jersey, and Texas are the other states in which a majority of white students is proficient in math. Given recent school-related political conflicts in Wisconsin, it is of interest that only 42 percent of that state’s white students are proficient in math, a rate no better than the nation as a whole.  (Results for all states are presented in the unabridged version of the paper.)

In reading, the picture looks better. As we mentioned above, only 40 percent of white students are proficient, but that proficiency rate would place the United States at 9th in the world. Its proficiency rate does not differ significantly (in a statistical sense) from that for all students in Canada, Japan, and New Zealand, but white students trail in reading by a significant margin all students in Shanghai, Korea, Finland, Hong Kong, and Singapore. In no state is a majority of white students proficient, although Massachusetts comes close with a 49 percent rate. The four states with the next highest levels of reading proficiency among white students are New Jersey, Connecticut, Maryland, and Colorado.

Are the Proficiency Standards the Same for Math as for Reading?

Has NAEP set a lower proficiency standard in math than in reading? If so, is the math standard too low or the reading bar too high?

At first glance it would seem that the standard is set at pretty much the same level. After all, 32 percent of U.S. students are deemed proficient in math and 31 percent are deemed proficient in reading.

But that coincidence is quite misleading. When compared to peers abroad, the U.S. Class of 2011 performed respectably in reading, trailing only 10 other nations by a statistically significant amount. Admittedly, the U.S. trails Korea by 16 percentage points, but it’s only 10 percentage points behind Canada. Meanwhile, U.S. performance in math significantly trails that of 22 countries. Korean performance is 26 percentage points higher than that of the United States, while Canadian performance is 18 percentage points higher. Judged by international standards, U.S. 8th graders are clearly doing worse in math than in reading, despite the fact that NAEP reports similar percentages proficient in the two subjects.

A direct comparison of NAEP’s proficiency standard with PISA’s proficiency levels three and four also indicates that a lower NAEP bar has been set in math than in reading. To meet NAEP’s standards currently, one needs to perform near the fourth level on PISA’s reading exam, but only modestly above the third level on its math exam.

Clearly, the experts set an 8th-grade math proficiency standard at a level lower than the one set in reading. Perhaps this is an indication that American society as a whole, including the experts who design NAEP standards, set lower expectations for students in math than in reading. If so, it is a sign that low performance in mathematics within the United States may be deeply rooted in the nation’s culture. Those who are setting the common core standards under discussion might well take note of this.

Of course, it could be argued that the math proficiency standard is correct but the reading standard has been set too high. In no country in the world does a majority of the students reach the NAEP proficiency bar set in 8th-grade reading.

What Does It Mean?

Many have concluded that the productivity of the U.S. economy could be greatly enhanced if a higher percentage of U.S. students were proficient in mathematics. As Michael Brown, Nobel Prize winner in medicine, has declared, “If America is to maintain our high standard of living, we must continue to innovate…. Math and science are the engines of innovation. With these engines we can lead the world.”

But others have argued that the overall past success of the U.S. economy suggests that high-school math performance is not that critical for sustained growth in economic productivity. After all, U.S. students trailed their peers in the very first international survey undertaken nearly 50 years ago. That is the wrong message to take away however. Other factors contributed to the relatively high rate of growth in economic productivity during the last half of the 20th century, including the openness of the country’s markets, respect for property rights, low levels of political corruption, and limited intrusion of government into the operations of the marketplace. The United States, moreover, has always benefited from the in-migration of talent from abroad.

Furthermore, the United States has historically had far higher levels of educational attainment than other countries, with many more students graduating from high school, continuing on to college, and earning an advanced degree. It appears that in the past the country made up for low quality in elementary and high school by educating students for longer periods of time.

As we proceed into the 21st century, none of these factors remains as favorable to the United States. While other countries are lifting restrictions on market operations, the opposite has been occurring within the United States. The U.S. has also placed sharp limits on the numbers of talented workers that can be legally admitted into the country. Our higher education system, though still perceived to be the best in the world, is recruiting an ever-increasing proportion of its faculty and students from outside the country. Meanwhile, educational attainment rates among U.S. citizens now trail the industrial-world average.

Even if some of these trends can be reversed, that hardly gainsays the desirability of enhancing the mathematical skills of the U.S. student population, especially at a time when the nation’s growth in productivity is badly trailing growth rates in China, India, Brazil, and many smaller Asian countries. Eric Hanushek and Ludger Woessmann have shown elsewhere that student performance on international tests such as those we consider here is closely related to long-term economic growth (see “Education and Economic Growth,” research, Spring 2008). Assuming past economic patterns continue, the country could enjoy a remarkable increment in its annual GDP growth per capita by enhancing the math proficiency of U.S. students. Increasing the percentage of proficient students to the levels attained in Canada and Korea would increase the annual U.S. growth rate by 0.9 percentage points and 1.3 percentage points, respectively. Since current average annual growth rates hover between 2 and 3 percentage points, that increment would lift growth rates by between 30 and 50 percent.

When translated into dollar terms, these magnitudes become staggering. If one calculates these percentage increases as national income projections over an 80-year period (providing for a 20-year delay before any school reform is completed and the newly proficient students begin their working careers), a back-of-the-envelope calculation suggests gains of nothing less than $75 trillion over the period. That averages out to around a trillion dollars a year. Even if you tweak these numbers a bit in one direction or another to account for various uncertainties, you reach the same bottom line: Those who say that student math performance does not matter are clearly wrong.

Given the integration of the world economy, a global perspective is needed for assessing the performance of U.S. schools, districts, and states. High-school graduates in each and every state compete for jobs with graduates from all over the world. Charles Vest, president of the National Academy of Engineering and president emeritus at Massachusetts Institute of Technology, has warned, “America faces many challenges…but the enemy I fear most is complacency. We are about to be hit by the full force of global competition. If we continue to ignore the obvious task at hand while others beat us at our own game, our children and grandchildren will pay the price. We must now establish a sense of urgency.”

Paul E. Peterson is the director of Harvard’s Program on Education Policy and Governance and senior fellow at the Hoover Institution. Ludger Woessmann is professor of economics at the University of Munich. Eric A. Hanushek is senior fellow at the Hoover Institution of Stanford University. Carlos X. Lastra-Anadón is a research fellow at the Program on Education Policy and Governance at Harvard University. An unabridged version of this paper is available here.

The legitimacy of test score increases in District of Columbia Public Schools (DCPS),  in particular those at Crosby S. Noyes Education Campus, are the focus of the latest installment in USA Today’s “Testing the System,” a multi-part series exploring the extent and causes of cheating — by teachers, principals and schools — on standardized tests. At the heart of the allegations are multiple erasures — presumably adults correcting test answer sheets — that were detected by the test scanning computers that grade the multiple choice tests. In one classroom at Noyes, USA Today reports, seventh-graders averaged 12.7 wrong-to-right erasures per student. Perhaps the students were lucky, but statisticians say they would have been more likely to pick winning lottery ticket numbers than to make that many wrong-to-right erasures by chance.

If the USA Today allegations are true, then the adults who changed students’ answers did much more than just cheat on a test. They also cheated those students, by allowing them — and their families — to think that they had learned material they clearly hadn’t.

But these deceptions are not new. For decades, less was expected from students attending schools in poor communities. Expectations were even lower for children with disabilities or in special education programs. It was virtually impossible to learn how these children were doing. Because scores were reported only as averages, it was easy to mask the fact that entire groups of students were not learning. Educators and policymakers knew–but the absence of hard data made it easy to turn a blind eye.

The federal No Child Left Behind Act (NCLB), which mandated yearly testing and public reporting of schools’ results in grades 3-8 and once in high school, was written in part to respond to these issues. Lawmakers wanted to ensure that test results would be comparable from student to student and create common standards for all students, regardless of their backgrounds. That’s why the law requires that tests align with state content standards and that students be assessed at their official grade level. It’s also why the law requires that schools report results for smaller groups of students–those who don’t speak English, for example, or those with disabilities–separately.

So, before we heed reactionary calls to end standardized testing, it’s important to remember the decades of willful neglect prior to NCLB. Most notably, the implementation of standards-based reform, first in the states in the 1990s and then by the federal government under NCLB in 2002, spurred an unprecedented focus on the deficiencies of schools that serve poor and minority students–students who were long ignored and whose outcomes were mostly hidden from view.

Still, the law is almost a decade old and its flaws are increasingly clear. Few people would defend the quality of most state tests and the low bar that they set to proclaim a student “proficient.” Part of the solution is a better assessment system. The Obama Administration agrees and is investing $350 million in two different consortia of states to develop these assessments. Ideally, improved assessment practices will show us not only what students are learning but also how they are learning and why they may or may not be gaining particular skills or knowledge. We also need to continue explorations of data of all types (not just test scores), building on, for example, important research that’s helping us develop early warning indicators to prevent students from dropping out.

We need voices and ideas from many places to continue to improve our understanding of how well students are learning in our public schools. But there’s no excuse for cheating, whether it’s done by children or adults. And let’s remember that the system before NCLB also cheated children, denying them a chance to get a good education. Ignoring the past and taking us back to those days–however you score it–is definitely a wrong answer.

-Bill Tucker

(Post cross-posted from the Huffington Post.)

Fischel’s basic argument is that our educational institutions have largely evolved in response to consumer demands.  That is, the consolidation of one-room schoolhouses into larger districts, the development of schools with separate grades, the September to June calendar, and the relatively common curriculum across the country all came into being because families wanted those measures.  And in a highly mobile society, even more than a century ago, people often preferred to move to areas with schools that had these desired features.  In the competitive market between communities, school districts had to cater to this consumer demand.  All of this resulted in a remarkable amount of standardization and uniformity across the country on basic features of K-12 education.

Hearing Fischel’s argument made me think about how ill-conceived the nationalization effort led by Gates, Fordham, the AFT, and the US Department of Education really is.  Most of the important elements of American education are already standardized.  No central government authority had to tell school districts to divide their schools into grades or start in the Fall and end in the Spring. Even details of the curriculum, like teaching long division in 4th grade or Romeo and Juliet in 9th grade, are remarkably consistent from place to place without the national government ordering schools to do so.

Schools arrived at these arrangements through a gradual process of market competition and adaptation.  Parents didn’t want to move from one district to another only to discover that their children would be repeating what they had already been taught or were  inadequately prepared for what was going to be taught.  To attract mobile families, districts informally and naturally began to coordinate what they taught in each grade.  Of course, not everything is synced, but the items that are most important to consumers often are.

That’s how standardization in market settings works and we have a lot of positive experience with this in industry.  VHS became the standard medium for home entertainment because the market gravitated to it, not because some government authority mandated it.  If we followed the logic of Gates-Fordham-AFT-USDOE we would want some government-backed committee to decide on the best format and provide government subsidies only to those companies that complied.

Instead of ending up with VHS, they may well have imposed Betamax on the country, even though market competition would have shown that approach to be inferior.  Sony was the industry leader and if a government-backed committee were in charge they almost certainly would have had the most influence.  The Fordham folks might want to keep this in mind.  A government-backed committee is almost certain to prefer what the AFT wants over what Fordham may envision since the teacher unions are like Sony except only 100 times more powerful.

Even worse, once government-enforced standardization occurs it becomes extremely difficult to change.  If we had a government-backed panel decide on Betamax, we may have been stuck with that format for decades.  We almost certainly would have stifled the innovation that led to DVDs and now Blue-Ray.  Once Sony had entrenched their format, what incentive would they have had to change it?

Similarly, once the Gates-Fordham-AFT-USDOE coalition settles on the details of nationalizing standards, curriculum, and testing, it will become extremely difficult to change anything about education.  Terry Moe and Paul Peterson’s dreams of technology-based instruction may never leave the dream stage because it may fail to comply with certain provisions of the national regime.  If I were the AFT, I’d almost certainly insert those details into the regime to prevent the reductions that technology may bring to the need for teaching labor.  No one should be naive enough to think the Edublob won’t figure out how to use nationalization to block that and other threatening innovations.

I’m also sure that Bill Gates would have preferred being able to get a government-backed committee to enshrine Microsoft-DOS or Windows forever.  But thanks to market competition we have Google innovating with cloud computing.  And I’d bet that Google would love to get government backing for their approach if they could.  Dominant companies almost always favor government regulation.

So I understand why the AFT, USDOE, and Gates favor the current effort to nationalize education.  The mystery to me is why Fordham is protecting the right-flank of this movement or why some conservative governors have gone along.  Don’t they realize that it will enshrine arrangements that favor the teacher unions and are bad for kids?

-Jay P. Greene

In fact, the long report (it is nearly 4,000 words) about how the New York State education department struggled with, and ultimately flubbed, the testing challenge is less a narrative of education failure than a cautionary tale about relying on big government (and the attendant bureaucracy) to run our schools. (See USA Today today:  apparently, six in ten Americans say the government has too much power, and nearly half agree with this alarming statement: “The federal government poses an immediate threat to the rights and freedom of ordinary citizens.”)

“State long ignored red flags on test scores” is the headline in today’s Times.

[E]vidence had been mounting for some time [that the state’s tests] had serious flaws. The fast rise and even faster fall of New York’s passing rates resulted from the effect of policies, decisions and missed red flags that stretched back more than 10 years and were laid out in correspondence and in interviews with city and state education officials, administrators and testing experts.

Paul Peterson and Rick Hess raised red flags long ago, with their research comparing the assessment systems of the states. (See Johnny Can Read…in Some States.) And then there was the classic, to me, front page Times story, from 2002:  The Elderly Man and the Sea? Test Sanitizes Literary Texts, telling the story of the Brooklyn mother who caught the New York State Education Department “cleaning up” (i.e. changing) the texts of famous writers for the state’s ELA tests.

What is amazing to me is that no one goes to jail for this kind of thing. We have become so numb to the news of school failure—and so blinded by the blame-the-victim excuses—that we greet stories like Medina’s with an institutional yawn instead of an arrest warrant.

Part of the problem is that these debates are like TV wrestling: it has some entertainment value.  We can easily get caught up in the testing narrative and miss the essential question: what do we want our children to know? In fact, Medina’s story mentions “curriculum” just once, and then only in an oblique reference to “the citywide math and English curriculums” in New York City.

And that is the second doleful takeaway from this depressing story: that we have handed over our schools to test-makers. Please don’t misunderstand. Tests—and assessments—are critical to learning, but they are the thermometer not the weather. The jobs of educators these days seem to be to trick kids and teachers rather than provide a measure of knowledge.

“In many cases you could not write an unpredictable question no matter how hard you tried,” the Times quotes Daniel Koretz, who “specializes in assessments systems,” according to the Times and who “oversaw the study of New York’s tests that led to the state’s conclusion that they had become too easy to pass.”

We are supposed to have “unpredictable” questions?

This is a system very much lost in the weeds.

There is some hope in the Common Core State Standards Initiative, but until we put the horse in front of the cart and have rigorous, detailed, and broad curricula for our schools and districts, we will continue with a largely useless argument about who sits where in the carriage.

The data used to determine the grades in Figure 1 are available here.

Much ado has been made about setting high standards over the past year. In his first major address on education policy, given just two months after he took the oath of office, President Barack Obama put the issue on the national agenda. They ought “to stop lowballing expectations for our kids,” he said, adding that “the solution to low test scores is not lowering standards—it’s tougher, clearer standards.” In March 2010, Secretary of Education Arne Duncan accused educators of having “lowered the bar” so they could meet the requirements set by the federal education law, No Child Left Behind (NCLB), which requires that all students be proficient in reading and math by the year 2014.

Current conversations about creating a common national standard largely focus on the substantive curriculum to be taught at various grade levels. Even more important, we submit, is each state’s expectations for student performance with respect to the curriculum, as expressed through its proficiency standard. Curricula can be perfectly designed, but if the proficiency bar is set very low, little is accomplished by setting the content standards in the first place.

To see whether states are setting proficiency bars in such a way that they are “lowballing expectations” and have “lowered the bar” for students in 4th- and 8th-grade reading and math, Education Next has used information from the recently released 2009 National Assessment of Educational Progress (NAEP) to evaluate empirically the proficiency standards each state has established. This report is the fourth in a series in which we periodically assess the rigor of these standards (see “Johnny Can Read…in Some States,” features, Summer 2005; “Keeping an Eye on State Standards,” features, Summer 2006; and “Few States Set World-Class Standards,” check the facts, Summer 2008).

The 2009 NAEP tests in reading and math were given to a representative sample of students in 4th- and 8th-grade in each state. NAEP, called “the nation’s report card,” is managed by the Department of Education’s National Center for Education Statistics and is currently the “gold standard” of assessments. Its proficiency standard is roughly equivalent to the international standard established by those industrialized nations that are members of the Organisation for Economic Co-operation and Development (OECD). If a state identifies no higher a percentage of students as being proficient on its own tests than NAEP does, then the state can be said to have set its standards at a world-class level. To ascertain objectively whether state standards are high or low, and whether they are rising or falling, we compare the percentage of students deemed proficient by each state with the percentage proficient as measured by NAEP. The state assessment data used in this report consist of those compiled in 2009 by the 50 states and the District of Columbia.

States have strong incentives not to set world-class standards. If they do, more of their schools will be identified as failing under NCLB rules, and states will then be required to take corrective actions to bring students’ performance up to the higher standard. As a result, the temptation for states to “lowball expectations” is substantial. Perhaps for this reason, a sharp disparity between NAEP standards and the standards in most states has been identified in all of our previous reports. In 2009, the situation improved in reading, but deteriorated further in math.

Every state, for both reading and math (with the exception of Massachusetts for math), deems more students “proficient” on its own assessments than NAEP does. The average difference is a startling 37 percentage points. In Figure 1, we provide a uniform ranking of the rigor of state standards using the same A to F scale used to grade students (see sidebar for the specifics on the methodology we used).

Racing to the Top?

Ironically, Tennessee received an F and had the lowest standards of all states, despite the fact that it is one of the two winners in the first phase of the bitterly contested Race to the Top (RttT) competition sponsored by the Obama administration’s Department of Education. Indeed, Tennessee has had the lowest standards of all states since 2003. Based on its own tests and standards, the state claimed in 2009 that over 90 percent of its 4th-grade students were proficient in math, whereas NAEP tests revealed that only 28 percent were performing at a proficient level. Results in 4th-grade reading and at the 8th-grade level are much the same. With such divergence, the concept of “standard” has lost all meaning. It’s as if a yardstick can be 36 inches long in most of the world, but 3 inches long in Tennessee.

Delaware, the other RttT First Phase winner, also had below-average standards, for which we awarded a grade of C- and ranked it 36th of the 50 states and the District of Columbia. Delaware claimed that 77 percent of its 4th-grade students were proficient in math, when NAEP shows that only 36 percent were. In 8th-grade reading, Delaware said 81 percent of its students were proficient, but NAEP put the figure at 31 percent.

From these findings one might conclude that the Obama administration is having a huge policy impact by getting states like Tennessee and Delaware to set standards they have been unwilling to establish in the past. But Tennessee earned almost full marks (98 percent) on the section of the competition (weighted a substantial 14 percent of all possible points) devoted to “adopting standards and assessments,” even though its standards have remained extremely low ever since the federal accountability law took hold. The proof will be in the pudding. If Tennessee and Delaware and other states now shift their standards dramatically upward, RttT will win over those who think it is performance, rather than promises, that should be rewarded.

Disparities in State Standards

Despite the incentive to lowball expectations, five states—Hawaii, Massachusetts, Missouri, New Mexico, and Washington—have set their standards at or close to the world-class level, earning them an A. Notice that we award grades purely for the expected standard for performance, not actual proficiency. New Mexico earned the same mark as Massachusetts, even though only about one-quarter of its students are proficient, while half of Massachusetts students score at that level. The two deserve equal grades, however, because both are rigorous in their expectations. Another eight states—Colorado, Maine, Minnesota, Montana, New Hampshire, New Jersey, Rhode Island, and Vermont—earned a B for their standards.

President Obama is undoubtedly correct, however, in suggesting that many states are “lowballing expectations.” Of the remaining 38 states, 27 earned a C, and 8—Arizona, Idaho, Illinois, Maryland, Michigan, New York, Texas, and Virginia—a D. Three states—Alabama, Nebraska, and Tennessee—had such low standards that we awarded them an F. All of the states that earned grades of F have been ranked D or below in all three of our previous reports. This suggests that once a standard, however low, has been set, it tends to persist—another reason to be concerned about promises from Delaware and Tennessee.

Changes in Standards

Secretary of Education Duncan is not altogether correct in suggesting that educators are lowering the bar, however. Figure 2 shows that in 2009 the differences between state and NAEP standards shrank by 0.08 standard deviations as compared to the average for the three prior surveys. This is a reversal of the trend of declining standards we observed between 2003 and 2007. Eight states improved the overall rigor of their assessments by a full letter grade or more since 2007: Georgia, Mississippi, Montana, New Jersey, New Mexico, Oklahoma, Utah, and West Virginia. By contrast, we gave just four states—Alaska, California, New York, and South Carolina—grades that were at least a full letter grade worse than they received in 2007.

The reversal in the overall trend is, however, driven wholly by an improvement in the rigor of reading assessments, which set expectations that are higher by 0.49 standard deviations in 4th grade and by 0.26 standard deviations in 8th grade. As a matter of fact, 17 states increased the rigor of their 4th-grade reading assessments by a whole letter grade since 2007, and 17 states did the same for 8th grade. But math standards have slipped by 0.12 standard deviations in 4th grade and by 0.31 in 8th grade. This means that at least some of the state-reported improvements in mathematics proficiency are misleading.

Converging on a De Facto National Standard?

Most changes to standards, as we noted, have been fairly small: only 12 states have made changes to their standards that alter their standing by a whole letter grade. But since our last report two states, Hawaii and South Carolina, have made major alterations to state assessments. The results of these moves have been at odds: while Hawaii’s increased alignment with NAEP raised its grade from a B+ in 2007 to an A, South Carolina dropped from an A to a C-.

States nonetheless seem to be continuing their trajectory of convergence toward standards of similar rigor in math (which, given the slipping standards noted above, constitutes a downward convergence), but are more divergent in reading since 2007, particularly in 4th grade. If the convergence of math standards were to continue, we could gradually attain something like a national standard. But it would take a great deal of national patience to achieve a national standard by convergence creep.

In this report, as in previous ones, we assess the rigor of standards that states set. This is an important task, as it reminds states that whether students have or have not learned cannot be a matter of how the test is designed and where the “proficiency line” is drawn. Rather, setting high standards for proficiency is the first step in the journey toward actually improving the learning of a high percentage of students. According to NAEP, less than one-third of students are proficient in reading and a similar proportion in math nationwide. For the sake of the children of this country, we should be doing much better than that.

Paul E. Peterson is professor of government at Harvard University, senior fellow at the Hoover Institution, and editor-in-chief at Education Next. Carlos Xabel Lastra-Anadón is a research fellow at the Program on Education Policy and Governance at Harvard University.

Everyone knows the early drafts have been the object of much discord. How confident can we be that what will emerge from these tussles and dust-ups will be coherent, complete and sufficiently demanding without being overwrought? If this national standards endeavor were a new drug for fighting swine flu or breast cancer, the FDA would subject it to rigorous long-term “field trials” to determine both its safety and its efficacy before releasing it for widespread use. Yet the Education Department, the White House, the Gates Foundation, the National Center on Education and the Economy and plenty of other parties are sounding and acting as if these standards and assessments had already proven themselves. The high command at Gates seems to assume that all of American K-12 education is going to be reconfigured around them. Secretary Duncan asserts that only states pledging their troth to the Common Core State Standards Initiative (CCSSI) should be eligible for Race to the Top funding. Yesterday, the President declared that future Title I funding for a state should hinge on whether it has embraced the new standards and assessments. And more.

A little humility would seem to us to be in order. If these standards and assessments end up representing a huge improvement over those in use in most states today, then much that’s good may reasonably follow from their installation and use. But what if they don’t? And even if they do, what about those (few) states that have done a creditable job on their own and for which CCSSI may represent either a lateral move or a step backward? In any case, would it not be prudent to appraise their safety and efficacy before demanding that they become the center of America’s new education universe?

“CRTC scandal stuns the state”

“Cheating details revealed”

“Atlanta board calls for cheating probe”

“CRTC” stands for “criterion-referenced competency tests,” and they are administered to students in grades 3-8 to gauge learning.  The problem is signaled in the first few paragraphs of one of the stories:

“One a late June day two years ago, two DeKalb County school administrators panicked.  A few dozen of their elementary school students had just finished high-stakes summer retests–exams first taken in spring but not passed.  With just a glance at the answer sheets, Atherton Elementary Principal James Berry and Assistant Principal Doretha Alexander saw they were in trouble.

“We cannot not make AYP,’ Alexander said. Not making AYP, or adequate yearly progress, meant not meeting a required federal benchmark.  These students, all fifth-graders, also faced being held back if they did not pass.

“‘Okay,’ Barry answered.  He pulled a pencil from a cup on Alexander’s desk.  ’I want you to call the answers to me.’  With that, he began to erase the students’ answers.”

That’s one scene, and it occurred a while ago.  But official investigations have enlarged the problem, and the general picture is this: fully 191 schools in the state of Georgia, 10 percent of the total number of elementary and middle schools, are up for investigation for altering test answer sheets, the story reported.

The next day’s story put the count at one in five Georgia public schools.  More than half of those schools had at least one classroom that displayed abnormal numbers of wrong answers changed to right answers.  In one elementary school classroom, 4th-grade math tests showed an average of 27 answers changed from wrong to right (out of 70 total answers).  In one middle school in Atlanta, nearly 90 percent of classrooms came up suspicious.

The extent of the scandal remains to be seen, and its impact is long-term.  What happens to kids whose tests were flagged, but who might have done much of the answer-changing themselves in the course of taking them?  What about the fate of the governor’s proposal to tie teacher pay to student performance (in other words, is this too strong an incentive to cheating)?  Who is going to investigate all the individual cases and measure out relative culpability and punishments?

This is going to take awhile.

The warm greetings given by both Republicans and Democrats on the committee reflect Duncan’s reputation as a centrist in the ideologically fraught battles over education reform. He has received national attention for moves favored by reformers, such as opening 75 new schools operated by outside groups and staffed by non-union teachers; introducing a pay-for-performance plan that will eventually be in 40 Chicago schools; and working with organizations, including The New Teacher Project, Teach For America, and New Leaders for New Schools, that recruit talented educators through alternatives to the traditional education-school route.

At the same time, Duncan maintained at least a cordial working relationship with the Chicago Teachers Union, and both the National Education Association (NEA) and the American Federation of Teachers (AFT) backed his nomination. He supported the No Child Left Behind law (NCLB), but also called for dramatic increases in spending to help schools meet the law’s targets, and additional flexibility for districts like his own. In nominating Duncan, Obama said, “We share a deep pragmatism about how to go about this. If pay-for-performance works and we can work with teachers so it doesn’t feel like it’s being imposed upon them…then that’s something that we should explore. If charter schools work, try that. You know, let’s not be clouded by ideology when it comes to figuring out what helps our kids.”

Given the strong union support for the Obama presidency, there was great speculation within education circles throughout the fall as to whether the new president would turn out to be a reformer—willing to challenge existing practices and the teachers unions in order to achieve dramatic changes in schools—or play it politically safe by backing programs that brought only marginal changes. A sharp divide among Democrats was in full view at the party’s national convention in Denver, where urban mayors and educators, gathered at a forum sponsored by Democrats for Education Reform (DFER), challenged the dominant role of teachers unions in shaping policy. Newark mayor Cory Booker told those assembled, “We have to understand that as Democrats we have been wrong on education, and it’s time to get it right.”

Even before the national convention, conflicts between the unions and Democratic reformers were intensifying. At a New York fundraiser in 2007, Obama reportedly made a similar point. According to Joe Williams, DFER’s executive director, Obama incriminated the teachers unions when the director of a Harlem charter school asked the then candidate why Democrats threw up so many obstacles.

Williams explained, “We’re at this point where the nation wants to change education more than the unions and the unions are going to have to decide if they’re going to be part of the change or be left out of it entirely.”

Two manifestoes issued during the Democratic primaries laid out competing philosophies on improving student achievement that were intended to influence the eventual Democratic nominee. A “Broader, Bolder Approach to Education,” a letter issued by the liberal Economic Policy Institute, signed by national leaders across much of the political spectrum, and endorsed by the AFT, argued that improving schools alone would not close achievement gaps between disadvantaged and advantaged students. It called on policymakers to provide preschool, afterschool programs, and summer school, and take steps to improve students’ health and social development. Another letter, issued by a coalition called the Education Equality Project, advocated addressing school system failures through greater accountability, school choice, and changes in compensation that would promote teacher quality. Those who signed on to the project, a diverse group of leaders in education, philanthropy, and public service, vowed to “challenge politicians, public officials, educators, union leaders and anybody else who stands in the way of necessary change.”

Obama has allies in both camps. Arne Duncan was one of only a handful who signed both statements. Yet in his confirmation hearing, Duncan left little doubt that the administration wants to make systemic changes.

“We must do dramatically better,” Duncan told the Senate committee. “We must continue to innovate. We must build upon what works. We must stop doing what doesn’t work. And we have to continue to challenge the status quo.”

Advisors to Obama say the rhetorical distinction was overdrawn and that the thrust of the president’s strategy is to make progress without causing further polarization. His education platform reflected that approach. Like many Democrats, he wants to spend more money: on helping students attend college; early childhood care and education; and improving teaching through mentoring and professional development for both principals and teachers. He has criticized NCLB for encouraging teaching solely focused on preparing students to pass tests. But in line with many Republicans and more conservative Democrats, Obama, like Duncan, supports school choice, charter schools, performance-based pay, and alternatives to education schools for teacher preparation (see sidebar). He and his opponent, Senator John McCain, both praised the work of Washington, D.C. schools chancellor Michelle Rhee, who has fought the local union as well as the AFT over tenure and teacher pay.

Economic Stimulus

Widespread agreement that only a massive stimulus package could rescue the U.S. economy presented the new administration with the opportunity to placate both sides of the Democratic divide. The unions and their allies would get a massive infusion of federal funds into the schools that would help offset state and local budget cuts. And this would give Obama cover to push for tougher reforms down the road.

House Democrats, after negotiations with Obama’s team, in mid-January proposed a stimulus package of $825 billion that included between $120 billion and $140 billion for public schools and colleges. Most of the money would have few strings attached.

The spending package would boost federal spending on Title I programs for low-income students and for special education, distributing the money according to current formulas. It would also provide at least $39 billion to offset state cuts in education budgets and $20 billion for capital improvements at schools and colleges. About $15 billion would be available to states as bonuses for efforts such as ensuring that low-performing schools and districts have effective teachers and that the performance of English-language learners and special education students is properly assessed (see Figure 1). One Obama aide said similar incentives would be incorporated into education programs to be introduced later in the spring.

The stimulus package also proposed to boost funding for the Teacher Incentive Fund (TIF), a Bush-era program that provides financial incentives to teachers and principals who raise overall student achievement and close achievement gaps. After Democrats took control of Congress in 2006, they zeroed out funding for TIF but restored $100 million for the following year. In his last budget, Bush requested $200 million for the program, the same amount Obama’s team has proposed.

Thirty-six states plus the District of Columbia already have local or statewide teacher compensation systems that add some sort of financial incentive to the standard step-and-column pay plan, according to the NEA. Former NEA president Reg Weaver cautioned that “while we can be open to alternatives, we should always oppose politically motivated, quick fixes designed to weaken the voice of teachers and the effectiveness of education employees. If they want to talk about changing the way we’re paid, they need to do that with us, not to us.”

In Obama’s platform, he agreed that such plans should be developed in consultation with teachers. Among the promising models is a voluntary pay-for-performance program in place in districts in a dozen states, funded in part by TIF, and implemented by Duncan in Chicago. The Teacher Advancement Program (TAP) provides teachers with professional support, helps them to use data in instruction, holds them accountable for results, and provides bonuses. Teachers in 10 Chicago schools voted to participate in TAP starting in the fall of 2007, and bonuses totaling $340,000 were given out the following year for improved test scores at 9 of the schools. “This is a landmark event for Chicago’s schools—recognizing and rewarding educators for exemplary work and compensating them accordingly,” Duncan said at the time.

The scale of the proposed spending on education is stunning, more than doubling the federal contribution. Of course, even an increase of that magnitude would leave the feds as the junior investors in public education, their contribution dwarfed by current state and local spending. But the funds proposed to offset cuts in state funding would mean that, for the first time, the federal government would be directly covering the cost of basic school operations. That kind of money could buy a lot of goodwill, especially if it helps states avoid laying off thousands of teachers. By December 2008, 19 states had cut K–12 education spending, according to the Center on Budget and Policy Priorities, a liberal research group. Even with the infusion of federal support proposed so far, states may have to make further cuts in their education budgets if the economy does not improve quickly. States spend between one-third and one-half of their budgets on elementary and secondary education, and the revenue available to state and local governments is shrinking fast. By January 2008, states had reported deficits of $350 billion. “If the economy doesn’t get better, schools are in trouble,” said Jack Jennings, founder and CEO of the Center on Education Policy. “For the sake of the schools it’s important that Obama pay attention to the economy.”

Even if the economy recovers and the stimulus package goes through intact, some observers question whether the proposed spending will do enough to address persistent disparities in achievement. Despite past federal support directed toward the needs of low-income students, African American 4th and 8th graders did not make measurable progress on the National Assessment of Educational Progress between 2005 and 2007. “Is the stimulus going to benefit kids in ways that are palpable and real and that improve achievement?” asked Dianne Piche of the Citizens Commission on Civil Rights. As the House was passing its version of the stimulus package (see Figure 1), Frederick Hess of the American Enterprise Institute noted that most of the money simply gave states dollars to keep intact the programs of the past: “It’s like an alcoholic at the end of the night when the bars close and the solution is to open the bar for another hour,” he told a New York Times reporter.

No Child Left Behind

The pressing economic issues, as well as difficult politics, will likely push reauthorization of NCLB into 2010 or even 2011. California Democrat Representative George Miller, who was one of four members of Congress who worked with the first Bush administration on the original NCLB, wants to see it revised and reauthorized. Yet Miller acknowledged to the Washington Post that “at the end of the day, it may be the most tainted brand in America.”

NCLB has been a great success in the sense that no one disagrees with its goals: accountability for results, addressing issues of teacher quality, putting a spotlight on the learning of all students, and better targeting of funds to districts serving the most disadvantaged students. Still, its detractors argue that the law has had unfortunate side effects: too much time spent teaching to narrow tests, schools focused on boosting the scores of students who are just below the proficiency threshold, and some states lowering their standards to reduce the number of schools missing their achievement targets.

“We’ve learned over the past five to 10 years that we have to align curriculum, align standards, and align tests with professional development,” Jennings said. “We’ve also learned that it is very, very hard to do. We’ve also learned that if we really set certain goals…teachers will pay attention to those students who are just below the goal and not pay attention to those who are further down or further up.”

Obama spoke during his campaign at length about the ins and outs of testing and decried teaching to the test. Rather than abandon the testing in NCLB, he has said he wants to invest in improving assessments, so that they measure a broader range of skills than just the basics.

The battle fought over reauthorization of NCLB in 2007 offers a preview of the challenges the Obama team will face. In a speech at the National Press Club outlining his priorities for reauthorizing the law, Representative Miller said, “Throughout our schools and communities, the American people have a very strong sense that the No Child Left Behind Act is not fair, not flexible, and is not funded. And they are not wrong.”

Hope for Reform
Despite the challenges, many in Washington are hopeful that public schools may in fact improve under an Obama administration. Although he cannot ignore the unions that form a key part of the party’s constituency, Obama owes less to them than did past Democratic presidents. The unions did not support him in the primaries and, because he raised so much money on his own, Obama was not as dependent on their money as others have been. Of course, he is hugely popular with teachers, and the staggering amount of money he appears to be willing to spend on education will only make him more so.

In addition, the leaders of the two unions at least appear more willing to be flexible on some long-standing issues. AFT president Randi Weingarten has said several times that “nothing is off the table” except vouchers. Not that much is known about Dennis Van Roekel, the Arizona math teacher who became president of the NEA last summer (see “Same Old, Same Old,” features, Winter 2009). But he was among those who supported Bob Chase, an earlier NEA president, when he tried to get the union to endorse what he called the “new unionism.” Chase wanted the union to experiment with new forms of performance pay and peer review of teacher performance, but the rank-and-file members nationally were reluctant to go along. It remains unclear how far Weingarten and Van Roekel will be able to push their members now to accept changes in compensation, evaluation, tenure, and so on.

Weingarten finds it “very sad” and frustrating that unions are always blamed for opposing reforms. “There’s a lot of demonizing and blame-mongering going on in education and it’s ridiculous…because it just creates excuses,” she said. “It says to me that they don’t think anything can be done because they are looking for the fall guy rather than helping all kids achieve.”

Weingarten expressed hope that Obama would push for more rigorous standards, better curricula, more valid assessments, and investments in helping teachers improve. “You can’t buy it by putting money out there and saying to teachers, ‘if you don’t do it, you’re fired,’” she said, referring to her opposition to Michelle Rhee in Washington, D.C. “We have the responsibility…to recruit and support and retain teachers if they’re doing a good job, and if not, to counsel them out of the profession.”

But Kate Walsh, president of the National Council on Teacher Quality, counters that the unions have resisted that course of action. “I think the unions are up against the wall,” she said. “The whole movement toward the notion that teachers don’t have a basic right to be in the classroom unless they are effective is proving so powerful as an idea that they’re weakened because they’ve run away from it rather than embrace it.”

It is well known that one of the strongest threads in the narrative of Obama’s journey from his childhood to the White House is educational opportunity (see “The Early Education of Our Next President,” features, Fall 2008). Schooled first in Indonesia, he returned to Hawaii because his mother wanted him to get a better education. There, his maternal grandmother and grandfather enrolled him in the private Punahou School, where he studied with the island’s elite. Then, it was on to Occidental College, Columbia University, and Harvard Law School.

“I wasn’t born with a lot of advantages, but I was given love and support, and an education that put me on a pathway to success,” Obama said during a major campaign speech on education last September in Dayton, Ohio. “The reason Michelle and I are where we are today is because this country we love gave us the chance at an education. And the reason that I’m running for president is to give every single American that same chance.”

Joe Williams believes that all of those factors, as well as Obama’s personal commitment to improving education, create a real opportunity to bring about systemic, long-lasting changes. “Everyone says they support the goals of NCLB and if that’s real, then he can use his bully pulpit to say that we’ll do in education the equivalent of saying we’ll put a man on the moon in 10 years.

“He can say that we will make sure that every kid who starts the race will cross the finish line and it will give everyone goose bumps and start a new type of discussion about what the game is. But it only has the potential to change the game if he treats it as an opportunity to wipe the slate clean and inspire people to think very big about what is possible,” adds Williams. “Obama is the only person I’ve seen in the last 20 years who may be up to that job.”

“His vision of education is as a foundation not just of the economy but of a society in which people take care of each other,” explained Stanford University education professor Linda Darling-Hammond, who advised Obama during the campaign and handled education policy for the president-elect’s transition team, in remarks delivered in November 2007 at a National Academy of Education event. “I think we can make great strides in a very short time.”

Although some may worry about the cost of all of the new programs, Darling-Hammond views the amount Obama wants to spend on education as a relatively small part of the overall bailout and recovery package, which could exceed $1.5 trillion.

In his speech last September in Dayton, Obama assured his audience, “We can do it all.”

Richard Lee Colvin is a longtime education journalist and director of the Hechinger Institute on Education and the Media, Teachers College, Columbia University.

The quality of our public schools has been measured in innumerable ways, and stakeholders may draw on any number of sources for rankings to support a particular agenda. Each winter, Education Week issues Quality Counts as a magazine supplement to its weekly newspaper. Report cards track and compare state education policies and outcomes in six areas: chance-for-success; K–12 achievement; standards, assessments, and accountability; transitions and alignment; the teaching profession; and school finance. For example, the grade for transitions and alignment is based on 14 indicators related to “early-childhood education, college readiness, and economy and workforce,”  while the school finance indicators measure spending patterns and resource distribution. Through these report cards, Education Week purports to “offer a comprehensive state-by-state analysis of key indicators of student success.”

The Quality Counts rankings are eagerly anticipated, thoroughly perused, and widely quoted. After the 2009 rankings were released, the Maryland State Department of Education issued a press release touting the state’s place at “the top of the list in Education Week’s tally, just ahead of Massachusetts.” Florida governor Charlie Crist celebrated the news that Education Week’s Quality Counts rated Florida’s schools 10th in the nation, based on its average rating across the six categories that comprise the analysis. Are Florida’s schools among the nation’s best? It depends on what you measure. By November of 2009, two lawsuits had been filed in Florida claiming the state was failing to provide high-quality education to its students. The plaintiffs claimed the state has low graduation rates, frequent school violence, and low levels of education spending and teacher pay compared to other states.

The rankings are also frequently misunderstood. Among the most widely cited of the Quality Counts ranking schemes is the Chance-for-Success Index (CFSI), which attempts to measure a state’s capacity for helping young people succeed. Here’s what Education Week’s Editorial Projects in Education (EPE) Research Center has to say about the index:

The Chance-for-Success Index captures the critical role that education plays at all stages of an individual’s life, with a particular focus on state-to-state differences in opportunities. While early foundations and the returns in the labor market from a quality education are important elements of success, we find that the school years consistently trump those factors. In every state, indicators associated with participation and performance in formal schooling constitute the largest source of points awarded in this category, and help explain much of the disparity between the highest- and lowest-ranked states.

The CFSI’s stated aim is to show the role that education plays as a student moves from childhood through the formal K–12 system and into the workforce, but then the rest of the description is fairly ambiguous. Many states nonetheless interpret the index as a simple measure of school quality. Maryland came in fifth in 2009, with a B+. The Maryland schools’ press release cited above reported that the state “ranked among the nation’s leaders in ‘Chance for Success,’ which looks at how well graduates achieve beyond high school.” Of course, some states choose not to emphasize their CFSI score. For example, the New Mexico education department’s January 2009 press release led with its number-two rank and A grade for transition and alignment policies and buried in the middle its 51st-place CFSI grade of D+.

Does CFSI measure the school system’s contributions to achievement beyond high school? It’s hard to say. Most of its components, described as “key facets of education spanning stages from childhood to adulthood,” are a grab bag of demographic characteristics. The index combines indicators related to family background, wealth, education levels, and employment with schooling measures, including kindergarten enrollment and selected National Assessment of Educational Progress (NAEP) test scores. The 13 components of success are identified in the sidebar. Not all of these have a clear relationship to postsecondary success, and several are beyond the control of state policymakers.

Consider the parental employment indicator and its role in an index that is updated annually or even every other year. Short-run trends in parental employment may not have any impact on the overall quality of a state’s education system; even the direction of possible influence is unclear. Parents who see how difficult it is to get and retain employment without education may stress the value of school completion, but it is also conceivable that underemployed parents may seek to accelerate their children’s entry into the labor force, even at the expense of their education. A similar problem exists with annual income: many factors outside of education quality influence the vitality of a state economy. Even if strong gains in public education are realized, it will be years before the effects are reflected in adults’ annual income. Income trends over the next few years will have little or nothing to do with current levels of education quality.

A Different Approach

Absent a sound theory of action, it is easy to go on a data spree. As seen in the CFSI, the more the merrier. As an experiment, we reconstructed the Chance-for-Success Index. First, we selected a clear standard for our index: we defined “success” as the percentage of young adults, aged 18 to 24, who are productively engaged in postsecondary endeavors (pursuing a college degree, active military service, or full-time employment). We limited the indicators to only those factors for which a reasonable empirical base of evidence shows an association between the indicator and our definition of success and that are plausibly under the control of education policymakers. Five indicators have a clear bearing on education outcomes: preschool enrollment, kindergarten enrollment, 4th-grade reading, 8th-grade mathematics, and high school graduation. Using the same source data as the 2009 CFSI and giving each factor equal weight, we computed new averages for each state and compared the new rankings to the originals.

Our results show marked divergence from the CFSI rankings (see Table 1). Only Maryland (5th) and Arizona (43rd) retained their rankings, although four of the top five stayed within that band. Looking down the list, however, 34 states moved 3 or more places, 21 shifted by 5 or more places, and 13 states moved by 8 or more places. Does our revised index precisely rank states’ public education systems? Probably not. The ideal index would be one that measured how well states and schools did, given their demography. Still, this exercise shows how sensitive the CFSI is to the choice of indicators.

Removing family background characteristics from the index changes states’ rankings substantially. The states that drop the most in the revised rankings are Hawaii, Rhode Island, Indiana, Alaska, Nebraska, and North Dakota. The states that gain the most are Florida, Texas, Maine, Idaho, Arkansas, and Mississippi, mostly poor, rural states.

Is the CFSI largely a measure of parental education? We looked at where the states would fall if we ranked them by individual family background variables. The variable that by itself provides a ranking with the closest fit to the CFSI is percentage of children with at least one parent with a postsecondary degree (parent education). Ranked by that measure alone, only 8 states would move by 8 or more places from their positions in the CFSI. Indicators of family income and adult education levels also produce rankings similar to the CFSI. Ranking states by either the percentage of children in families with incomes at least 200 percent of poverty level (the family income indicator) or the percentage of adults (25–64) with a 2- or 4-year postsecondary degree (adult educational attainment), only 15 states would move 8 or more places.

Measuring the Measure

Report cards must meet a number of conditions if they are to be reliable. First, they need to clearly define the condition or result being examined. None of the descriptions provided by the CFSI editors accomplish this—they never reveal exactly what they take the “chance for success” to be, asserting only that some states provide better opportunities than others. Explained the EPE Research Center’s director, “a child’s life prospects depend greatly on where he or she lives.”

Second, the indicators that are employed should have direct and proven association with the outcome being measured. The CFSI’s current approach mixes inputs such as demographics with outcomes like academic results to arrive at a single score. The result is a tautology: success is the sum of the parts; the parts are by default the components of success.

The editors of Quality Counts gather and report a variety of measures that reflect current education and policy performance across all 50 states and the District of Columbia and, through comparison, encourage states to take actions that would lead to improvements in their ratings. Nowhere do the Quality Counts editors show how or why the Chance-for-Success Index is a good predictor of success. Instead, they provide statistics that divert attention away from the things that actually do matter, such as high-quality teaching, a good range of school options, and success in early elementary schools. There is risk in including variables that have no real impact on the result being studied. States may view the results as motivators to improvement, and ineffective indicators may lead to ineffective attention and investment. Narrowing the scope of the Chance-for-Success Index to factors both causally related to school achievement and under the control of state education officials or school districts would improve its value and deliver the right signals to states.

CREDO at Stanford University supports education organizations and policymakers in using research and program evaluation to assess the performance of education initiatives. The team is led by Margaret Raymond and includes Kenneth Surratt, Devora Davis, Edward Cremata, Emily Peltason, Meghan Cotter Mazzola, Kathleen Dickey, and Rosemary Brock.

A major debate among education reformers over how best to reduce the achievement gap broke out during the 2008 presidential campaign. Most advocates on both sides backed Barack Obama, but they urged him to pursue different policies. The Education Equality Project (EEP) supported a continuation of accountability and other school-focused reforms. The coalition for A Broader, Bolder Approach to Education claimed that the greatest gains could be achieved by addressing health, housing, and other social ills (see “Straddling the Democratic Divide,” features, Spring 2009).

A close look at recent changes in education in the state of Florida sheds light on that debate. One finds in this southern state, a closing of the achievement gap that has eluded allegedly more progressive states. When it comes to education progress, Florida is a star performer. Moreover, its success has come in spite of a challenging student demographic profile and relatively modest resources.

Let us begin with a basic demographic fact often cited by those in the Broader, Bolder camp. Over the past 20 years, the schools of Florida, California, Texas, and New Mexico have all seen rapid growth in their Hispanic populations. Compared to other groups, Hispanic students underperform academically, drop out of school in higher numbers, and attend college in lower numbers. A straight projection of the recent past into the future looks bleak for these students and their educational outcomes.

But demography need not be destiny. Over the past decade, Florida has succeeded in improving student achievement despite its demographic profile. Low-income students (those eligible for free or reduced-price lunch) make up almost half of Florida’s K–12 student body. Florida has a “majority minority” mix of students, with non-Hispanic white students making up 48.3 percent of the total, African Americans about 24 percent, and Hispanics 25 percent. But the educational situation is not as bleak as those statistics might imply: both minority groups have recently made academic strides forward.

Florida has managed to realize such gains although the state’s per-student funding is below the national average. More than making up for its fiscal limitations, the state, led by former governor Jeb Bush, implemented a series of school reforms that together appear to have had dramatic consequences for student performance. Upon taking office in 1999, the governor pursued a multipronged strategy of education reform: an emphasis on reading, standards and accountability for public schools, and new choice options for students. The bulk of the reforms passed in his first year in office. Subsequently, those initial measures were buttressed by additional innovations, including the curtailing of social promotion for students who failed to learn to read in the early elementary grades.

Academic Achievement

Prior to the introduction of those innovations, Florida’s educational record was little short of abysmal. Among the 43 states that in 1998 were gathering information on their students’ performance on the National Assessment of Educational Progress (NAEP), Florida had the fifth-lowest 4th-grade reading scores. But over the next decade those scores moved sharply upward so that by 2007, Florida’s scores were tied for 8th highest among all the states. As the state moved up the leader board, Florida students, on average, were making strikingly larger gains on NAEP exams than the average student nationwide (see Figures 1 and 2). Nor were gains occurring only in reading. Fourth-grade math scores were climbing at an even faster rate. In 8th grade, reading and math gains in Florida were less impressive but they still outpaced the nation.

Those statewide trends could have been masking a widening of the achievement gap between whites and minorities. But exactly the opposite was happening. Far from lagging behind, Florida’s minority students were doing much to drive the overall rise in test-score performance. In the decade after the education reforms began, the average NAEP reading score for Hispanic 4th graders in Florida rose steeply so that by 2007 scores were higher than the average NAEP reading scores of all students (regardless of ethnicity) in 15 states (see Figure 3).

One might think that rising scores among Hispanic students reflect their families’ movement up the income ladder into the middle class. But even Florida’s low-income Hispanic students scored, on average, equal to or higher than nine statewide averages for all students (regardless of income). Average scores of all Florida low-income students, regardless of ethnicity, tied or exceeded the statewide average for all students in seven states. Many of these differences are small and thus within the margin of NAEP sampling error, so should be thought of statistically as ties. The margin of error cuts both ways, however, as the 2007 statewide averages for all students in Georgia, Illinois, Michigan, North Carolina, Rhode Island, and Texas all fell within two points of Florida’s average for Hispanics on the 4th-grade reading exam. Indeed, the national average for all students falls within this same narrow margin.

Comparison of trends in Florida with those in California is particularly intriguing. Both are large states with growing Hispanic populations. California’s median family income is 12 percent higher than Florida’s, meaning families have more resources to devote to their children’s education. But California has largely eschewed the kinds of accountability and choice reforms that Florida adopted.

The consequences for students range between noteworthy and startling, as shown in Figure 4. The chart displays the differences in the NAEP gains 4th- and 8th-grade students made in the two states. As they did in 4th-grade reading, Florida’s Hispanic students outperformed California students in 4th-grade math and 8th-grade reading; they tied the California average in 8th-grade mathematics.

Explaining Florida’s Success

Not everyone thinks that something remarkable has been happening in Florida. The state’s success is only an apparent one, says Boston College education professor Walter Haney. He discounts Florida’s progress on 4th-grade NAEP scores, on the grounds that Florida’s worst-performing readers repeat 3rd grade and thus are not included in the 4th-grade NAEP. Without the low-performing 4th graders who have been held back for a year, the average scores of the remainder jump upward.

Haney’s critique is worth considering. One of the pillars of the Florida accountability reforms has been the policy, introduced in 2003, of not promoting 3rd graders unless they perform at a minimally acceptable level on the reading portion of the Florida Comprehensive Assessment Test (FCAT).

But a more careful look at the data shows that Florida’s NAEP scores were rising before implementation of the retention policy. Between 1992 and 1998, Florida’s average NAEP score in 4th-grade reading dropped by two points. Between 1998 and 2002 (before the social promotion policy affected 4th grade), however, it increased by eight points. One of the reasons for lower retention rates was improved 3rd-grade performance on the state’s examination. In 2002, 27 percent of 3rd graders scored at the lowest level on the reading portion of FCAT, but by 2008 only 16 percent did so, a 40 percent reduction in the pool of students eligible for retention. This helps explain why actual retention rates declined by 40 percent between 2002 and 2007.

One would expect, if Haney’s interpretation is correct, to see an upward spike in 4th-grade test scores in 2003 followed by a steady decline in test performance in subsequent years. But in fact the trend line shows no such spike and decline, only a steady movement upward.

Perhaps Florida’s gains are only apparent for another reason: its low starting point in 1998. Is it possible that gains are realized most easily when scores are initially very low? On this question, opinion is quite divided. Some think gains are more easily realized if students are already accomplished, while others think those with high scores have neared a ceiling, making it difficult to raise their scores further. However that issue is settled in principle, it cannot account for the fact that Florida made striking gains while states with equally low scores did not. For example, on the 1998 4th-grade reading test Florida was near the bottom, with Arizona, California, Hawaii, Mississippi, Louisiana, and New Mexico. In 2007, all those states but Florida were still clustered near the bottom.

But if the Florida achievement gains are genuine, and not imaginary, they might still be attributed to factors over which schools have little or no control, for example, demographic changes in the state. Such is the claim of those who say that demography is destiny. Were demographic change the best explanation, however, student performance in Florida would be worse than ever. According to the National Center for Education Statistics (NCES), 45 percent of Florida children attending public schools in 1998 were of minority background. By 2005, that percentage had climbed to just over 50 percent. Similarly, the percentage from low-income backgrounds (eligible for free or reduced-price lunch) rose from 43 to 45 percent between 1998 and 2007.

Another plausible explanation for the Florida success story is the 2002 passage of an amendment to the state’s constitution mandating universal preschool education for all those who would like to participate. But however valuable the program may prove to be, it cannot explain the gains in achievement observed thus far. The amendment did not require implementation until 2005, and none of the students participating in the Florida early childhood program had reached the 4th grade by 2007, the most recent year for which NAEP data are available.

Much the same can be said for a second constitutional amendment—the class-size reduction amendment—approved by Florida voters in 2002. As in the case of preschool, there is some research evidence that suggests class-size reduction can yield significant gains in student achievement in the early grades. Florida state law now mandates no more than 18 students per classroom in grades K through 3, and no more than 22 students in grades 4 through 8. But the constitutional amendment is being implemented slowly. Through the 2008–09 school year, administrators have been considered in compliance if their schoolwide average class sizes were under the constitutional limits. According to the state department of education, from 2002 to 2008, average class sizes in the early grades were reduced from roughly 23 to 16 students in pre-K to grade 3 and from 24 to 18 students in grades 4 to 8. Still, if class-size reductions had any effect on achievement gains between 1998 and 2007, it could only have been toward the end of the period.

Nor can the gains in education be easily attributed to changes in public school funding. Florida’s average spending per pupil rose from $7,183 in 1998–99 to $7,683 in 2004–05, in constant dollars. This was less than half the increase in the national average over the same period.

One can pretty much rule out these possible explanations for the Florida success story. The gains are not an artifact of the elimination of social promotion in 3rd grade or of the ease with which low test scores can be lifted. Nor can they be attributed to demographic change, the introduction of preschool education or class-size reduction, or greater per pupil expenditure. One must look elsewhere for an explanation. The most likely remaining candidate is school-focused reforms, which have the vigorous support of the EEP side of the education reform debate.

The Florida Reforms

Over the past decade, Florida has introduced a comprehensive program of school reform that has five main points: school accountability, literacy enhancement, student accountability, teacher quality, and school choice. Together, the reforms created a system that appears to have focused teachers and students on the task of learning in a way that has yielded the dividends we have highlighted above.

School Accountability.  In 1999, the state legislature enacted a law that required students in grades 3 through 10 to take annual tests in reading and mathematics, known as the Florida Comprehensive Assessment Test or FCAT. The assessment had two distinctive features lacking in most other accountability systems, including the one prescribed by the federal law, No Child Left Behind (NCLB). First, it gave each school in the state a very clear grade of A to F based on the results from the test and offered a specific fiscal incentive to schools to try to reach as high a grade as possible. Bonuses were given for obtaining an A or raising one’s grade from one year to the next. Conversely, schools receiving an F grade twice over a four-year period were asked to carry out a variety of reforms. The law offered students at “double F” schools the opportunity to attend private schools until a court decision disallowed the practice in 2006.

Beginning in 2002, the accountability system included measures of student progress from one year to the next, a feature not incorporated into NCLB. That gave schools with low-performing students an opportunity to raise their grades without imposing upon them the extremely difficult task of matching the performances of schools whose student body enjoyed a preferred demographic portfolio. Clear, realistic incentives to improve were made available to schools across the state.

Focus on Literacy.  Along with its accountability system, Florida in 2002 introduced a statewide program known as “Just Read, Florida!” The effort created new academies to train teachers in reading instruction and provided for the hiring of 2,000 additional reading coaches. Teachers in grades K–3 took mandatory reading training courses over a three-year period. Students in grades 6 through 12 who demonstrated insufficient reading skills were provided remedial instruction.

Student Accountability. Beginning in 2003, Florida students were asked to pass a more demanding examination if they were to be given a high school diploma. In addition, Florida lawmakers, as discussed above, curtailed the social promotion of 3rd-grade students who performed at very low levels in reading. According to a careful evaluation by Jay Greene and Marcus Winters at the University of Arkansas (see “Getting Ahead by Staying Behind,” research, Spring 2006), the program had a positive impact on the performance of all 3rd graders, including those who were retained in that grade. Apparently, they benefited more from an additional year of instruction than they would have had they been pushed on to 4th grade when they were not well prepared for the more challenging material.

Teacher Recruitment.  Florida enacted new policies for broadening the pool from which teachers were being selected. Previously, teachers were required to earn a certificate by attending one of the state’s schools of education. Florida supplemented that channel of recruitment with a variety of alternative paths. The state opened “Educator Preparation Institutes” to facilitate the transition into teaching. Districts were allowed to offer alternative certification. Today, more than one-third of all new teachers in Florida are coming to the profession through alternative certification programs. The state’s teaching workforce has become the nation’s third most ethnically representative (see “What Happens When States Have Genuine Alternative Certification?” check the facts, Winter 2009).

The alternative certification program may have had a particularly significant impact on Hispanic students. Florida enjoys a large immigrant population that fled from Cuba in the years following the establishment of Castro’s communist regime. Many of the immigrants were middle-class professionals and entrepreneurs, and they have established a strong economic, political, and cultural presence in southern Florida. That population provides a pool of potential educators of high talent who speak both English and Spanish. Just how important alternative certification was to the recruitment of highly qualified bilingual instructors is unknown, but it cannot be ruled out as a potential explanation for the particularly large gains Florida’s Hispanic students have made. On the other hand, it cannot be the whole story, as African American and non-Hispanic white students also made strong gains during this period. Moreover, the percentage of Hispanics of Cuban origin has declined during the past decade (though this may not have affected the size of the pool of qualified bilingual teachers).

School Choice. Florida is well known for the range of school choice legislation it has enacted over the past decade. Charter schools, vouchers, tax credits, and online education all provide students and families with greater choice in 2008 than they had in 1998. For example, 105,329 students were enrolled in the state’s 358 public charter schools in 2007–08. That same year 19,852 students eligible for special education took advantage of the opportunity to use a voucher to attend private schools, and 21,493 students received scholarships averaging $3,750 from a tax credit program that opened private schooling to students from low-income families. The state-funded Florida Virtual School currently offers students more than 90 online courses (ranging from GED to Advanced Placement courses). Middle and high school students anywhere in Florida can access these classes free of charge. The state projects that 168,000 courses will be taken and completed during the 2008–09 school year (see “Florida’s Online Option,” features). Multiple evaluations, by organizations ranging from the Manhattan Institute to the Urban League, have found the choice programs to have had a positive impact on Florida public schools.

Despite the numbers, the school choice programs are not large enough to have had more than a limited statewide impact on the millions of students attending Florida’s public schools. Yet they helped create a climate in which public schools may have wanted to demonstrate their effectiveness for fear that choice opportunities would continue to expand.

Identifying what has caused the rise in Florida student performance cannot be done with perfect certainty. It might have been the accountability system, or the state’s reading program, or its decision to expect more from students, or its alternative certification program, or its plethora of school choice innovations, or some combination of all of them. But the results from Florida do suggest that concerted efforts to improve the quality of an education system can pay dividends for students. It is probably not a coincidence that the one state that has outdone the others in its efforts to reform its schools has made outsize gains in student performance. Exactly which of the many reforms Florida undertook was the key to success may never be known, but the reform package offers other states—and the nation as a whole—a clear path on which they, too, can move forward.

Matthew Ladner is vice president for policy at the Goldwater Institute. Dan Lips is senior policy analyst at the Heritage Foundation.

Cheating by educators comes in many forms, ranging from the subtle coaching of students to the overt manipulation of test results. For instance, a colleague of mine tells of a principal who would begin each morning’s announcements with a greeting to students, such as, “Good morning students, and salutations! Do you know what a salutation is? It means ‘greeting,’ like the greeting you see at the beginning of a letter.” Students learned the meaning of words like “salutation” from the principal’s daily announcements; they probably never learned that his choice of words like “salutation” was done with the vocabulary section of the state-mandated, norm-referenced test in hand. A recent U.S. News & World Report article described a case in Ohio, where one educator is accused of physically moving a student’s pencil-holding hand to the correct answer on a multiple-choice question. Another widely reported case involved a principal in Potomac, Maryland, who stepped down amidst charges that she had gone through students’ test booklets in the classroom and called them up to change or elaborate on their answers.

This isn’t just a recent phenomenon. More than a decade ago, at an evening reception following a conference for school district superintendents in one midwestern state, I happened upon a conversation among several superintendents who, with cocktails in hand, were chuckling and winking about how their quality-control procedures for student testing involved “pre-screening the kids’ answer sheets for stray marks.” What was so funny—I found out later from one of the superintendents—was that stray marks included things like wrong answers. Wink, wink. This practice apparently continues. In Texas, where the accountability system is particularly rigorous, 11 school districts were investigated for an unusually high number of erasures on students’ answer sheets in 1999.

Despite their headline-grabbing nature, such blatant cases of cheating are probably rare. There is usually more subtlety involved, such as when a teacher prods a student to review his or her answer: “Why don’t you take another look at what you wrote down for number 17?” Some examiners cheat by failing to monitor their students properly while proctoring an exam. Others cheat by omission, such as when a teacher reminds students who are likely to attain low scores that it would be okay for them to be absent on the day of the test. A more sophisticated version of cheating by omission occurred in the Austin, Texas, school district in 1999. School administrators entered incorrect student-identification numbers on the answer sheets of low-scoring students, which invalidated their scores and thus raised the schools’ average performance. In states that force schools to give absent students a score of zero for reporting purposes (which eliminates the incentive to encourage them to stay home), all students are encouraged to attend school on test day. But some students are afforded “testing disability accommodations” such as an individual aid, reader, extra time, or other assistance that isn’t usually a part of the student’s educational experience.

Perhaps the largest cheating scandal to date involved teachers and principals in the New York City school district. In December 1999, Edward Stancik, the district’s special commissioner of investigation, released an exhaustive study that found that cheating by 12 educators was “so egregious that their employment must be terminated and they should be barred from future work with the [Board of Education].” For instance, one teacher would have her students first write their answers on a piece of scrap paper. She would then correct their answers before they bubbled in their official answer sheets. The report named another 40 educators who were recommended for disciplinary action; 35 of them had engaged in actions that the investigators judged serious enough to warrant potential termination. The report concluded that the school district had known about the extensive cheating by educators “for years,” and that “educators were not held fully liable for their misconduct.” A follow-up report named another ten educators who had engaged in seriously inappropriate behaviors during testing in New York City, some of them so blatant—such as writing answers to test questions on the chalk board—that immediate termination of employment was recommended.

Some opponents of testing see cheating by educators as a reason to abandon high-stakes accountability systems. For instance, last year Alfie Kohn, a prominent critic of testing, told Congressional Quarterly, “The real cheating going on in education reform is by those who are cheating students out of an education by turning schools into giant test-prep centers.” These critics view cheating as the natural, and not so reprehensible, result of placing undue emphasis on a metric—standardized tests—that yields an incomplete picture of both student and teacher performance. Some even view cheating as a kind of civil disobedience. But, to collect useful information about educational progress, testing is an indispensable if imperfect tool. And if tests are to yield useful information, their validity must be ensured. The answer to cheating is not to abandon accountability. The answer is to limit if not eliminate the cheating. What follows is a survey of what we know about the extent of cheating and some proposals to guard against the subversion of accountability systems.

Testing Guidelines

Cheating can be defined as any action that violates the rules for administering a test. Commercial test publishers produce carefully scripted directions and clear guidelines for administering their tests. The guidelines lay out, in detail, all the actions that would compromise the test results. Similar instructions and rules accompany the customized tests that form the bedrock of many state accountability systems. Some states have strongly worded professional codes of ethics that explicitly define the responsibilities and boundaries associated with mandated testing. What constitutes cheating may even be codified in state law. For instance, the Ohio Revised Code proscribes “any practice that results solely in raising scores or performance levels on a specific assessment instrument without simultaneously increasing the student’s achievement level as measured by tasks and/or instruments designed to assess the same content domain.” The law provides for the termination of an offender’s employment, the suspension of an educator’s license for a violation, and the charging of an offender with a misdemeanor.

National organizations representing various professional associations have also developed standards for educators who administer standardized tests. For example, the American Federation of Teachers and the National Education Association helped to produce the Standards for Teacher Competence in Educational Assessment of Students, which require that teachers “should be skilled in recognizing unethical, illegal, or otherwise inappropriate assessment methods and uses of assessment information.” The most explicit statements regarding cheating can be found in the Standards for Educational and Psychological Testing, a product of collaboration among professional organizations in the fields of education and psychology. Among other guidelines, the standards explain that those involved in educational testing programs must “ensure that test preparation activities and materials provided to students will not adversely affect the validity of test score inferences” and “maintain the integrity of test results by eliminating practices designed to raise test scores without improving students’ real knowledge, skills, or abilities in the area tested.” In short, educators cannot plead ignorance; there has been no dissemination problem regarding what constitutes cheating. Professional codes of ethics that cover virtually every profession whose members work in school settings and state laws that govern those who have a license or credential in the field of education contain strict guidelines for administering tests. Anyone who is connected with testing in American education knows—or should know—how to conduct assessments that yield accurate and credible results.

The Consequences of Cheating

When cheating occurs, testing yields inaccurate information about individual students. The error is compounded when this information is then used for any educational purpose, and specific students wind up paying the price. One student may not receive the remedial instruction in reading that she needs. Another student may be incorrectly assigned to a special program for gifted and talented students that has a limited number of slots. Another may receive a scholarship that should have gone to one of his peers. And yet another may receive a diploma without having learned those minimum skills deemed necessary for success in college or on the job.

From the perspective of educational policy-making, the same invalidities that yield misleading test scores at the individual level also serve to muddle the interpretation of group test performance. Policy-makers and educational administrators have increasingly come to rely on group data to inform their decisions on staffing, curricula, professional development, and teacher-credentialing requirements and to measure the effectiveness of educational reforms. By distorting test results, cheating can lead to ill-advised initiatives, improperly focused resources, and inaccurate conclusions about the course of education reform. Given the confluence of achievement gains—in states ranging from Texas to North Carolina to New York—with the pervasive reports of cheating by educators, it is entirely reasonable to question how much of the former can be attributed to the latter.

Though not attempted in this article or elsewhere to my knowledge, the costs of cheating probably could be measured in dollars and cents. What cannot be measured are the effects of cheating at more fundamental levels. For example, when students learn that their teachers or principals cheat, what is the effect of this kind of role modeling? While fallen professional athletes might be able to say, “Don’t look at me as a role model, I am just an athlete doing a job,” educators cannot: a significant aspect of their job is the modeling of appropriate social and ethical behavior. Also, how might educator cheating affect students’ attitudes toward tests or their motivation to excel? How might it affect their attitudes toward education, their trust or cynicism with respect to other institutions, or their propensity to cheat in other contexts?

Research on Cheating

Shocking anecdotes don’t tell us much about how serious the cheating problem actually is. Just how prevalent is cheating by educators? Only a few studies have directly asked educators whether they have engaged in what have come to be referred to euphemistically as “inappropriate test administration practices.” The most common avenue of research is to poll educators regarding their general perceptions of cheating in their schools. One such study asked 3rd, 6th, 8th, and 10th grade teachers in North Carolina to report how frequently they had witnessed certain inappropriate practices. Of those polled, 35 percent said they had engaged personally in such practices or were aware of others’ unethical actions. The teachers reported that their colleagues engaged in a range of inappropriate practices two to ten times more frequently than they had. The practices included giving extra time on timed tests, changing students’ answers, suggesting answers to students, and directly teaching specific portions of a test. More flagrant examples included teachers’ giving their students dictionaries and thesauruses for use on a state-mandated writing test. One teacher revealed that she checked students’ answer sheets “to be sure that her students answered as they had been taught.” Other teachers reported using more subtle strategies, such as “a nod of approval, a smile, and calling attention to a given answer,” to enhance their students’ performance. In another study, of teachers who were drawn from two large school districts, 32 percent of the teachers surveyed reported allowing students to practice on old forms of standardized tests for two or more weeks.

A total of 40 schools were included in a study that was initiated in order to investigate suspected cheating in the Chicago public schools. Of the 40 schools, 17 served as “control” schools, which were compared with 23 “suspect” schools that had exhibited irregularities in the performances of their 7th and 8th grade students on the Iowa Tests of Basic Skills (ITBS). The irregularities consisted of unusual patterns of score increases in previous years, unnecessarily large orders of blank answer sheets for the test, and high percentages of erasures on students’ answer sheets. The researchers readministered the ITBS under more controlled conditions and found that, even after accounting for students’ reduced level of motivation on the retesting, the “suspect” schools clearly did worse than the “control” schools on the retest. The researchers concluded that they might have underestimated how much cheating was going on at some schools. A study of cheating in the Memphis school district revealed extensive cheating on the California Achievement Test, including a teacher who displayed correctly filled-in answer sheets on the walls of her classroom.

The Perceptions of Educators

The most troubling stream of research on cheating concerns the attitudes of educators toward cheating. They seem increasingly indifferent toward cheating, and there appears to be a growing sense that cheating is a justifiable response to externally mandated tests.

Several studies have attempted to investigate educators’ perceptions of cheating. In a 1992 study, 74 pre-service teachers were asked to judge the appropriateness of certain behaviors. Only 1 percent thought that either changing answers on a student’s answer sheet or giving hints or clues during testing was appropriate, and only 3 percent agreed with the idea that allowing more time than allotted for a test was acceptable. But 8 percent thought that practicing on actual test items was okay; 23 percent judged the rephrasing or rewording of questions acceptable; and 38 percent thought that practice on an alternative test form was appropriate.

The beliefs of pre-service teachers appear to translate into actual practices when they enter the classroom. In 1991, a large sample of 3rd, 5th, and 6th grade teachers in two school districts was asked to describe the extent to which they believed specific cheating behaviors were practiced by teachers in their schools. On the positive side, a majority of respondents (shown in Figure 1) said that, for all of the behaviors listed but one, they occurred rarely or never. Equally noticeable, however, is that in several cases, 15 percent or more of the respondents reported that a behavior occurred “frequently” or “often.” A full 23 percent of the teachers said that they thought teachers often gave hints to children who were having difficulty. Twenty percent said they thought teachers “frequently” or “often” gave students extra time to finish their tests. Likewise, 20 percent said they thought teachers “frequently” or “often” gave students practice on passages that were highly similar to those used on the test.

A 1991 survey examined perceptions about two specific kinds of “test preparation” practices: having students practice for a state-mandated, norm-referenced test using another form of the same test or having students practice on the actual test to be used. The survey polled six groups of educators, including teachers, principals, superintendents, and school board members in California. The results, shown in Figure 2, reveal fairly broad acceptance of these behaviors, even among board members. For instance, 36 percent of teachers in California thought it appropriate to practice with current test forms.

What Can Be Done?

What can be done to address the problem of cheating? At some point, we will need to reconceptualize testing entirely. We must find more effective ways to link, consistently and directly, successful test performance to student effort and effective instruction. If poor performance were accompanied by sufficient diagnostic information about a student’s weaknesses, then all concerned might view identification and remediation of those weaknesses as more beneficial then cheating. Such an initiative will require changing much of the status quo in curriculum, instruction, and assessment. But there are less far-reaching, more pragmatic actions that can be taken immediately. The following list provides a start.

Dissemination. It has been said that we more often stand in need of being reminded than we do of education. As mentioned earlier, every large-scale testing program provides a description of appropriate test-administration procedures; state regulations define the boundaries of legal conduct for test administrators; and education-related associations have produced guidelines for sound testing practice. Nonetheless, those caught cheating often protest that they did not know the behavior was wrong. If only as a reminder, every implementation of high-stakes tests should be accompanied by dissemination of clear guidelines regarding appropriate testing practices. Such reminders should be clearly worded, pilot-tested to refine the meaning that educators take from the guidelines, and distributed and signed by all who handle testing materials.

Procedures. Some minor procedural changes would hinder the ability of educators to cheat. For example, bar-coding or other methods of identifying testing materials plus a system of tracking testing materials would be easy to implement. Federal Express and United Parcel Service know the location of every package at any given time and can reconstruct precisely the hands that a package has passed through. The same tracking can be used for testing materials. Other simple steps would include the sealing of cartons and bundles of testing materials; delaying the delivery of testing materials to schools until just before test administration; and, once delivered, requiring that materials be maintained securely by a named person who is responsible for them.

“Truth in testing.” States with so-called truth-in-testing laws should reconsider their relative benefits. These laws often require that the content of state-mandated tests be disclosed following the administration of a test. They have the best of intentions, but the unforeseen consequence of such laws has been an increase in educators’ use of previous versions of tests for classroom practice, resulting in further narrowing of instruction. Moreover, the economic costs to “truth in testing” states have been staggering. Disclosing each year’s tests renders them useless, making it necessary to develop entirely new monitoring instruments one or more times each year.

Scaling Back. The expansion of testing and accountability systems has elicited two reactionary responses to the concurrent rise in cheating: 1) that large-scale testing for accountability be abandoned; or 2) that testing for accountability rely more heavily on constructed-response formats that, ostensibly, would be less prone to corruption. For instance, it is more difficult to forge or coach a student’s answer to an essay question or a science experiment than to alter a bubbled-in response or to provide the key to a multiple-choice item.

The difficulty with these reactions is that they fail to address the core issues. High-stakes pupil testing arose in the 1970s in reaction to the complaints of some business leaders—along the lines of, “We are getting high school graduates who have a diploma, but can’t read or write!” As UCLA Professor of Education James Popham observed at the time: “Minimum competency testing programs . . . have been installed in so many states as a way of halting what is perceived as a continuing devaluation of the high school diploma.” The public perception was that the gatekeepers were leaving the gates wide open. Perhaps a widespread misunderstanding of the relationship between self-esteem and achievement was to blame. Educators understandably wanted all students to have the personal esteem associated with high achievement. But awarding higher grades in order to boost self-esteem and stimulate further achievement too often had neither effect. The sense that grades weren’t accurate measures of achievement led to the imposition of externally developed and administered tests.

Thus the obvious error in calls to return to the past is that such a strategy only returns American education to the situation that caused accountability tests to be introduced in the first place. Moreover, though current tests are susceptible to cheating, the solution of returning to measures and procedures that are even more easily manipulated is unthinkable.

Nevertheless, we should consider limiting the amount of testing for accountability. Some tests serve solely an instructional function; they serve our need for solid diagnostic information on students in order to make informed decisions about their educational programs. When these tests are used also for accountability purposes, the expanded incentives to cheat can corrupt the information we have on student progress. Likewise, not all tests—especially those designed for purposes of decision-making—need have instructional value. If we clarify the purpose of each test, we can minimize the scope of mandated accountability tests, the time required for their administration, and the opportunities for cheating.

Consequences. In conjunction with limiting opportunities for cheating, we must revise the procedures that are used to unearth cheating and the penalties that are handed down. Many tests are currently administered behind closed classroom doors with little independent oversight; there are strong disincentives for educational personnel to report cheating; and, in most jurisdictions, the responsibility for investigating cheating rests with school personnel who have an inherent conflict of interest in ferreting out inappropriately high student achievement. Revised procedures should include: 1) random sampling and oversight of test sites; 2) increased protections for whistle-blowers; 3) stiffer penalties for cheaters, including permanent disqualification from teaching within a state and more coordinated sharing among the states of information regarding educators who have had their licenses revoked; and 4) assignment of investigative responsibilities to an independent authority.

A Qualified Boon

As we are learning, accountability is not an unqualified boon to American education. Nascent accountability systems have been difficult to implement and have had some undesirable consequences. For instance, in some situations the use of tests as the primary accountability mechanism has resulted in an extreme narrowing of what students are taught. When so much is at stake, educators tend to limit their instruction to the content that is covered on a mandated test. Moreover, some educators perceive the imposition of an externally mandated test as an inappropriate intrusion into an area of professional practice and discretion. They believe that their knowledge of a student’s true ability far exceeds whatever information can be gleaned from a single test; thus the idea that a single test should not be used against any student (for example, to deny grade-to-grade promotion or a high school diploma) is widespread among teachers. Here the same moral reasoning that many current teachers learned in their education courses during the 1970s may come into play: the ends do justify the means; cheating is a justified response to a system that punishes students and teachers on the basis of incomplete information.

But cheating itself leads to inaccurate information and misguided decisions about students. It signals that students have learned the skills we want them to, when in fact they haven’t. It threatens the values that we hope to impart to students via those we have charged with their education. It leads to mistaken conclusions about the efficacy and pace of needed educational reforms. Even if cheating is limited to a minority of educators, as it most likely is, its effects are devastating. It is no more justifiable than telling a sick patient that he is well and then sending him on his way.

–Gregory J. Cizek is an associate professor of educational measurement and evaluation at the University of North Carolina at Chapel Hill and the author of Cheating on Tests: How to Do It, Detect It, and Prevent It (1999).

More than seven years ago, President George W. Bush signed No Child Left Behind (NCLB) into law. Sweeping calls for testing, intervening in persistently low-performing schools, and policing teacher quality made it the most ambitious legislation on K–12 schooling in American history. The law, due for congressional reauthorization in 2007, still awaits legislative action. This spring, the Hoover Institution’s Koret Task Force issued 10 recommendations to guide reauthorization. In this forum, lead author of Learning from No Child Left Behind, EdisonLearning’s John Chubb, and education historian and task force member Diane Ravitch, who declined to sign the recommendations, weigh in on the future of the law.

EDUCATION NEXT: Is NCLB working? Should it be reauthorized?

Diane Ravitch: It is time to pull the plug on No Child Left Behind. It has had adequate time to prove itself. It has failed. After seven years of trying, there is no reason to believe that the results of NCLB will get dramatically better. Now is the time for fundamental rethinking of the federal role in education.

NCLB has produced meager gains in achievement. The National Assessment of Educational Progress (NAEP) assesses student achievement in reading and mathematics every other year. Despite the intense concentration on reading and mathematics required by the law, the gains registered on NAEP since the enactment of NCLB have been unimpressive.

In 4th-grade reading, the gains after implementation of NCLB, from 2003 to 2007, were small (three points) and exactly the same as the gains from1998 to 2003. Fourth graders in the bottom10th percentile of performance had a five-point gain after NCLB, but this did not compare to the 10-point jump in their scores from 2000 to 2002 pre-NCLB (see Figure 1).

In 8th-grade reading, there were essentially no gains from 1998 to 2007. Student performance was a flat line both before and after NCLB.

Mathematics was tested in 1996, 2000, 2003, 2005, and 2007. The gains preceding the adoption of NCLB were larger than those posted after NCLB. From 2000 to 2003, 4 th grade students recorded a nine-point gain in mathematics, compared to a gain of only five points from2003 to 2007. Among 4th-grade students in the lowest decile, there was an astonishing 13-point gain from 2000 to 2003 pre-NCLB; the same group saw a gain of only five points from 2003 to 2007. The same deceleration of student improvement was seen at all performance levels, from top to bottom.

In 8th-grade mathematics, gains also slowed after the passage of NCLB. Eighth graders saw a five-point gain from 2000 to 2003, but only a three-point gain from 2003 to 2007.

John Chubb: NCLB will and should be reauthorized. Absolutely, student achievement has grown much more rapidly in the last decade—the NCLB era—than during the 1990s, especially for the lowest achieving and most-disadvantaged students in the nation. Achievement is what NCLB is all about, so the law has met its most basic test. This is recognized by even the law’s critics which is why the only discussion in Washington is how to mend the law. The Obama administration recognizes that No Child Left Behind aims to help the federal government perform its most important education function: improving the education of students in greatest need. The new president is supported in this view by a bipartisan majority in Congress, which has worked for many years to ensure that poor kids get the help they require. The education needs that NCLB addresses are not going away, nor is the need for funding. Indeed, the economic stimulus bill passed in February increased funding for NCLB by 80 percent, and these provisions of the massive and controversial bill met no objections.

Over half of poor and minority students have reading and math skills far below grade level, whether measured by the tough performance standards of the NAEP or by the standards of the various states. Dropout rates, measured accurately only since NCLB made them part of Title I accountability, hover around 50 percent in many major cities.

NCLB is based on sound principles and should with time improve the achievement of all American children, especially economically disadvantaged and racial minorities. There is empirical evidence these principles are working. The Thomas B. Fordham Foundation recently completed an analysis of the top and bottom 10 percent of all students tested by NAEP. It found that the bottom 10 percent had gained far more than the national average since 2000 in math and reading, more than a full grade level in math. The top 10 percent had gained as well, providing no evidence that schools were ignoring the best students while focusing on the kids below proficient and subject to NCLB sanctions. Both groups of students had also gained more since 2000 than they had during the 1990s. The federal government’s own comprehensive analysis of Title I, mandated by Congress, conducted by RAND among others, and published in 2007 after several years of NCLB experience, found the largest academic gains since 2000 and 2003 among students in high-poverty schools. To be clear, the evidence in total is early, and the research is incomplete. But there is no question that American kids, especially the most disadvantaged, are making progress. It is absolutely mistaken to suggest, that NAEP changes pre- and post-2003 are evidence that NCLB has been counterproductive. Disadvantaged kids are achieving far more today than ever before, and those gains are attributable to higher standards and tougher accountability that began in the states in the 1990s and accelerated with NCLB.

EN: What are the strengths and weaknesses of the NCLB approach to assessment?

DR: Educators and the public are getting wise to the uselessness of the testing regime that has been foisted upon them. A year ago, North Carolina’s Blue Ribbon Commission on Testing and Accountability issued a report recommending a sharp reduction in the number of tests that the state required. The chairman of the commission, Sam Houston, said, “We’re testing more but we’re not seeing the results. We’re not seeing graduation rates increasing. We’re not seeing remediation rates decreasing. Somewhere along the way testing isn’t aligning with excellence.”

NCLB may in reality be dumbing down our children by focusing the attention of teachers and administrators solely on basic skills. Our students are not being prepared to compete with students from high-performing nations in the world. Many are not getting an education based on a coherent, content-rich curriculum in history, geography, the arts, science, foreign languages, and literature. They are not getting a good education. They are getting thin gruel. If we want a future workforce that is smart, creative, independent, and resourceful, we are not educating to get what we want.

JC: Perhaps the single greatest virtue of NCLB’s approach to assessment and accountability is that it shines a bright light on student performance, as measured against explicit standards of proficiency. The nation finally knows which schools are raising proficiency in reading and math and which are not. Before NCLB, such information was spotty at best. A weakness, however, is that the bright light does not shine on all subjects that matter for kids and their future.

The education the nation values is one that is rich in content. NCLB has unwittingly and unfortunately encouraged schools to focus instruction inordinately on reading and math, the subjects that NCLB requires be tested annually and to which it has attached the tough accountability regime. Students, however, need also to understand science, history, geography, civics, and more if they are to succeed in a 21st-century world of intense international competition and technological sophistication.

NCLB already requires science testing once each in grades 3–5, 6–9, and 10–12. This requirement should be extended to include three tests of social science, defined as U.S. history, world history, geography, and civics. The law should further specify that the science and social science assessments be cumulative and comprehensive, and not focused just on the content taught during the tested grade level. NCLB should require that scores be posted on state and district web sites and included in school report cards. State scores should be benchmarked against NAEP, to encourage high standards. But science and social science should not be part of Adequate Yearly Progress (AYP); the process of assessing and exposing performance should be ample to promote attention to these fields.

EN: How should proficiency be defined and measured?

DR: The federal demand that all students will be proficient by 2014 has led states to embrace a very loose definition of proficiency. Most states are now using NAEP’s “basic” achievement level as their definition of proficiency because NAEP’s “proficient” level is far beyond their reach. But many states go even lower than NAEP basic for their definition of proficiency. Tennessee, for example, says that 90 percent of its 4th-grade students are proficient in reading, while NAEP says that only 26 percent are. Only 61 percent of students in Tennessee are at basic or above, according to NAEP. Similarly, North Carolina tells the public that 86 percent of its 4th graders are reading proficiently, but NAEP says only 28 percent are (and 36 percent score “below basic”). These states and many others make inflated claims to satisfy NCLB’s ridiculous requirements.

JC: There is much room for improvement in how proficiency is defined and measured by NCLB—and we have practical suggestions for improving both. But the fundamental principles that NCLB advances represent a huge step forward for the nation. NCLB asks the nation to define what all students should know and be able to do in reading and math, and then measures progress toward these performance standards. This is a boldly democratic and egalitarian expectation and the very first time that the nation has asked its schools to perform at an explicit level. We should proudly defend these principles.

On a practical level, “proficiency” should describe the knowledge and skills necessary to be “college and career ready” in the 21 st century. Proficiency should capture the “common core” of competencies deemed necessary for all students to have a chance at success after high school.

NCLB should authorize the U.S. Department of Education to fund—after a competitive bidding process—up to three multistate consortia to develop standards, tests, and performance levels that support the overarching goal of college and career readiness. With federal funding, states will buy into one of the systems of national standards and tests, saving the huge expense of developing new standards alone. NCLB could, through these recommendations, give the nation standards both achievable and worth achieving, while preserving the rights of the states to determine what “national” standards should be.

EN: Are the law’s “remedy” provisions—including public school choice and supplemental educational services—working?

DR: The remedies the law prescribes—choice and tutoring—have proven to be ineffective. Less than 5 percent (and by some estimates, as low as 1 percent) of eligible students choose to leave their “failing” school to transfer to a school that made AYP. Some say it is because the students and families did not get adequate notice, but more likely students are not choosing to leave for other reasons. In many suburban and rural school districts, there may be no other school to transfer to. But perhaps more important, most students will not leave their school even if there is another school that is presumably better, by NCLB’s definition, and that is accessible. That is because most students are not in the group that is failing to make progress, and if they like their school, they don’t want to be separated from their friends.

The law assumes that the schools are bubbling over with discontented kids who are eager to escape, but that assumption is probably wrong. Or at least there is no evidence for it based on the lack of response to the choice provisions of NCLB. We have long known from polling data that the public is concerned about the quality of American education, but most parents are satisfied with their own children’s school. The failure of choice in NCLB reminds us of that consistent finding.

The other remedy in NCLB for failure to make AYP is tutoring, and that too has proved to be ineffective, though it has turned into a half-billion-dollar bonanza for tutoring companies. Evaluations in several states, including Maryland, Virginia, Tennessee, Alabama, Georgia, Michigan, and Kentucky, have reported that students who received tutoring did no better on state tests than their peers who did not receive tutoring. Only about 15 percent of eligible students have signed up for tutoring. Even when tutoring is free, conducted after school, and provided in a convenient location (sometimes in their own school building), most students don’t want it. Maybe it conflicts with their afterschool jobs or their sports or other commitments. Maybe they just don’t want to study for an additional hour or two when the school day is done. We need to know more about why 85 percent of eligible students avoid tutoring. We need to know why most eligible students are not showing up to be tutored, and why those who do show up are gaining so little from it.

JC: We know from ample research that choice can boost the achievement of students who avail themselves of it. We also know that tutoring is an effective means of remediating achievement deficits. RAND recently affirmed the effectiveness of SES tutoring in a well-controlled study. But choice and tutoring are not working nearly as well as they could in NCLB. This has nothing to do with the ideas of choice and tutoring but rather with the way NCLB provides for them.

Students in failing schools simply do not have enough choices. The law currently limits choice to schools not in improvement status, which often eliminates all nearby options. NCLB should increase the choices available by permitting families to judge school shortcomings for themselves. A school failing a single subgroup or barely missing AYP, for example, might be a better choice for a student in a school that is failing badly. Yet today those choices are not available.

NCLB should offer additional charter school start-up grants in any school district where failure is rampant, such as a district not making AYP. Students should be able to choose schools in neighboring school districts, subject to district approval. And private schools should be eligible to receive choice students, provided those schools charge no extra tuition and participate in the state testing program.

Students in failing schools should also have greater access to tutoring, sooner. There is no more effective way to help students who are struggling than to get them extra, focused, individualized attention. Yet only 20 percent of students eligible for tutoring under NCLB are receiving services, and the services often fall short of the quality offered in the private marketplace. This should be remedied.

First, make Supplemental Educational Services (SES) available as soon as schools are declared in need of improvement, the same time as school choice is offered. Second, ensure that students have access to the best possible tutors. Grant districts the right to provide SES, even if the district is failing to make AYP, but also require districts to provide a fair and competitive marketplace for all providers. Whatever access the district itself has to families, students, and facilities, it must also provide to private tutors—or the district loses the right to be a provider. To reinforce these measures, NCLB should require states to provide information on eligible students to approved providers. The states should be required to collect and post comparative information on the effectiveness of all tutors.

EN: Are NCLB’s sanctions for persistently failing schools effective? Are they fair?

DR: The law’s punitive sanctions are ineffective. By year six of failing, the schools may be turned into charter schools, taken over by the state or private management, closed, or restructured (e.g., replacing the entire staff). None of these sanctions had a research basis to justify its inclusion in the law. They were hopes or hunches, based on ideology, not evidence. Most states and districts choose the least onerous of the sanctions, which is restructuring. According to a 2008 report from the Center on Education Policy, restructuring itself needs to be restructured because there is no sure-fire way to turn around a chronically low-performing school. The federal Institute of Education Sciences recently published a research summary on how to achieve this admirable goal, but not one of its four recommended strategies was supported by evidence.

JC: Currently, NCLB’s escalating sanctions apply identically to schools that have failed massively and to schools that barely miss. This is a big mistake—but one that is easily fixed. NCLB should differentiate school improvement needs. Over time we expect more and more schools to succeed with the majority of their students, but to struggle with certain extra-needy subgroups. It is vital, as the nation expects increasing percentages of students to achieve proficiency, that we identify schools accurately for their performance. The Department of Education has approved nine states’ requests to implement “differentiated accountability” plans. NCLB should build on this good work and institutionalize a simpler system for all states.

Schools should be placed into one of two categories of “needs improvement.” “Limited” improvement is for schools whose shortcomings involve less than one-third of the student body. Limited improvement would offer students in year one of their school’s acquiring improvement status (two years of missing AYP) choice of another school and SES. If schools remain in limited improvement status, NCLB would require, in year four of improvement, that states develop with schools “limited corrective action plans.” Schools with limited improvement status should not face restructuring; states should have the flexibility to work with schools with limited problems as they see fit.

“Schoolwide” improvement is for schools that miss new AYP growth targets for all students or for subgroups that total more than one-third of school enrollment. Schoolwide improvement would require schools to proceed through restructuring, but NLCB should be revised to include only three means to restructure: First, a school may be reorganized as a charter school, giving it new governance. Second, the school’s management can be contracted out to an independent school management company, changing day-to-day control of the school. Finally, a school may be closed and reopened with 100 percent of the teaching staff and administration replaced. Each of these measures ensures a new day for the school and its students.

EN: Is NCLB’s goal of universal proficiency by 2014 one that should remain in a reauthorization of the law?

DR: The demand that all students be proficient by 2014 is absurd. This laudable goal has never been reached by any other nation or by any state. The only way it can be met is by defining “proficiency” to mean minimal literacy and numeracy. Meanwhile, the expectation that all schools will achieve this goal has created a trajectory of failure that guarantees a steady increase in the number of schools that are stigmatized for not making adequate yearly progress. In the 2007–08 school year, nearly 30,000 schools—or 35 percent of all public schools—joined that abysmal list; this was a 28 percent increase in the number of “failing” schools over the previous year. In Massachusetts, which has the highest-scoring students on NAEP in the United States, nearly half the public schools in the state were rated as being “in need of improvement.”

It does not take a statistician to figure out that NCLB is a recipe for disaster for American public education. An article in Science magazine last fall predicted that nearly 100 percent of all elementary schools in California would be failing schools by 2014.

JC: Universal proficiency is perhaps the most important principle of NCLB—certainly the most audaciously democratic one. It should and will be preserved. Who, after all, will be willing to say whose children should be proficient and whose should not? And, this is not just a matter of principle—the goal is doable.

But the states need to come together around standards that are worth accomplishing, that represent the common core of knowledge and skills that every child needs to be prepared in the 21st century for college or a productive career. Students with special needs or just beginning to learn English need to be provided alternative means to demonstrate proficiency. Universal proficiency in practice may mean 90 to 95 percent proficient, a high number but not an unattainable one. Finally, schools must be given time to realize goals worth achieving.

NCLB should extend the 2014 deadline for universal proficiency by six years—half the original NCLB timeline—to 2020, but only for states willing to adopt new high national standards.

EN: Has the federal leadership embodied in NCLB been a help or a hindrance to school improvement?

DR: Washington does not have the institutional knowledge or capacity to reform our nation’s schools. Congress is not the right institution to reform the nation’s schools. The U.S. Department of Education lacks the capacity to tell the nation’s schools what they should do to improve. Washington is too remote from schools to take responsibility for improving them. In their edited volume, No Remedy Left Behind, Chester E. Finn Jr. and Frederick Hess wrote that NCLB “amounts to a civil rights manifesto dressed up as an accountability system. This provides an untenable basis for serious reform, rather as if Congress declared that every last molecule of water or air pollution would vanish by 2014, or that all American cities would be crime-free by that date…. NCLB’s dogmatic aspirations and fractured design are producing a compliance-driven regimen that recreates the very pathologies it was intended to solve. It’s time to relearn the lessons of the Great Society, when ambitious programs designed to promote justice and opportunity were undone by utopian formulations, unworkable implementation structures, and a stubborn unwillingness to acknowledge the limits of federal action in the American system. In the end, Washington is not well-positioned to effect change to a program that depends on state and local action, or successfully to require states and districts to adopt measures whose efficacy hinges on gusto and creativity rather than compliance.”

A few tweaks here and a little tinkering there cannot fix this fundamentally flawed legislation. The time has come to discard it altogether and begin to think afresh about how the federal government can provide useful assistance to states, districts, and schools that are trying to improve. What we need is a clear recognition of the appropriate federal role in education and a deeper understanding of the meaning of a good education. Perhaps with a sense of the limits of federalism and of the limitless potential of education, we might be able to free ourselves from the sterility, rigidity, dogmatism, and narrow anti-intellectualism of NCLB.

JC: NCLB embodies a delicate balance between federal leadership and state execution. Despite the hue and cry from critics about federal over-reaching, NCLB provides ample discretion to the states. The role that NCLB sets out for the federal government—setting national goals while leaving states and districts to decide how to reach them—is sound, and surely superior to the hodge-podge of state accountability systems that preceded it. The challenge now is to improve how our federal-state partnership works. Experience can be a powerful guide.

Let’s face facts. The nation needs to boost its achievement even more now than when the law was passed. Our economic welfare depends more and more on education. We should learn from the law—as it is beginning to help our children learn—and not expect 50 uncoordinated states to get the nation where it needs to be in the demanding world of the 21st century.

What, in addition to what we have already suggested, would improve the federal-state partnership? A practical remedy on which all sides now agree: change how the law measures academic progress. NCLB currently recognizes achievement only when it lands a student above a state’s proficiency bar. The act does not recognize student progress by the lowest achievers, growing from, say, below basic to basic. The act also fails to recognize the growth of the nation’s top students: a school gets zero credit toward AYP for upper-end success. The Department of Education has approved 15 states’ requests to use “growth models” to measure achievement. NCLB should be revised to make growth the only measure of achievement. The act should require that each student’s achievement be judged, for purposes of determining AYP, against one simple standard: is the student on track to be proficient or better by the time of her last reading and math tests in high school? Those tests must be passed for high-school diplomas to be awarded.

Listen to the Podcast

Focus of School Reform Shifting to Teachers (12/17/09)
Are Middle Schools or Middle Schoolers the Problem? (12/10/09)
Biggest Spender in Politics: The NEA (12/4/09)
Saving Jobs or Stimulating Reform? (11/24/09)
Election Postmortem (11/19/09)
Will Congress Reroute the Preschool Juggernaut? (11/4/09)
Voters Choose Neighborhood Schools over Socioeconomic Diversity (10/29/09)
The Nobel Committee Isn’t the Only One Giving Speculative Prizes (10/22/09)
Will Michelle Rhee Triumph? (10/14/09)
Will the Federal Role in Education Double? (10/8/09)
Charter Schools Narrow Achievement Gaps in New York City (10/1/09)
What Congress Is Not Working On (9/24/09)
Charter Schools, Unions, and Linking Teachers with Student Achievement Data (9/17/09)

The session in the second-floor seminar room at the Nelson A. Rockefeller Institute of Government in Albany had already featured a detailed presentation of “charter facts” from the new executive director of New York’s Charter Schools Institute and a dozen friendly questions from the mostly pro-charter audience. Ward, the dignified and cerebral deputy director of the institute, seemed anxious to wrap things up. “So, thank you—”

He stopped in midsentence. A hand had shot up at the back of the crowded room. “Eva, did you want to ask a question?” he asked.

All eyes turned to the dark-haired woman sitting on a folding chair along the back wall of the room. Some eyes rolled, as most of the group knew Eva Joseph, the embattled superintendent of Albany Public Schools (APS). They had seen her at countless education forums, on the local nightly news, and in the daily paper at every turn of the school budget clock, determinedly defending her district and, increasingly, railing against charter schools. “I’ll make it quick,” said Dr. Joseph. “I do want to thank you for acknowledging the situation in Albany, but going to the heart of what’s real, we have 10 charter schools in Albany with a total public school population of 10,500 students. Compare that to 23 charter schools in the Big 5, with the exception of New York City. Buffalo, Syracuse, Rochester, Yonkers. Twenty-three total charter schools and you total up their enrollment. The proliferation here. The oversaturation, per pupil and per capita, is glaring. And it has serious implications for the district. It destabilizes it on many fronts….”

Standing a few feet away, as Joseph plunged on, a man leaned against the wall, smiling. It was not a smug or obvious smile, nor the smirk of a man who was mocking or scornful. Tom Carroll was smiling because he had heard the speech before and because he knew, as founder of the charter school foundation that had siphoned off nearly a quarter of Dr. Joseph’s 10,500 students, that he was at least an immediate cause of the vitriol. It was the smile of victory.

The Holy Grail of Charter Schooling
During much of the previous nine years, Carroll had overseen the launch of eight charter schools (see Table 1) in Albany, a small city (pop. 94,172), as Joseph suggested, for so many charters (to see additional images of the schools and their students please click here). Joseph, who began her Albany tenure as an assistant superintendent in 1997 and took the top job in September 2004, had been engaged in the charter battle for most of that time. “Fifth Albany Charter School Approved” was the headline just two months after she became superintendent. “SUNY adviser suggests city district cut staff, rent out extra space as students depart.”

What had been especially maddening for Dr. Joseph and her school board, which issued routine condemnations of the charters, was not just the presence of so many of the new schools—“the proliferation, the oversaturation”—but that they were so good. The destabilization was real and deep, creating not just viable, but quantifiably better, educational alternatives for children. And this is the singular accomplishment of Carroll’s charter organization: they had found the Holy Grail of charter schooling, quality and scale.

It is still an elusive goal for the charter school movement, which has grown to include more than 4,500 schools and 1.3 million students in 40 states and the District of Columbia. This remains a drop in the public school bucket (nationally there were more than 94,000 public K—12 schools and more than 49 million students in 2007), which is why “market share” is considered a crucial milestone, one of the few ways to pinch traditional schools in their pocketbooks. According to the National Alliance for Public Charter Schools, in 2007 New Orleans had reached 55 percent market share, Washington, D.C., 31 percent, and Southfield, Michigan, and Dayton, Ohio, 28 percent each.

Albany at that time came in at 17 percent, tied with Buffalo for 12th place nationally. But it had already distinguished itself as the only member of the market-share club with consistently high academic outcomes.

In fact, the failure of charters to offer a meaningful choice, i.e., a better education, has become a sore point among charter promoters. Education Next editor Chester E. Finn Jr., in a 2007 confessional in his Thomas B. Fordham Institute newsletter (the Education Gadfly), wrote, “Why are so many charter schools inadequate, even mediocre? What went wrong?” Finn noted that in his own “charter-saturated” Dayton, where Fordham was born, things had gone terribly awry. And though charters have taken to putting a good face on things by comparing themselves to their local district schools, which is fair, the truth about quality is uncomfortable. In the fall of 2008, for instance, the Dayton Daily News published a story headlined, “Most charter schools made gains; most Dayton district schools saw losses.” But an accompanying chart revealed that 12 of Dayton’s 19 K—8 charters did not make Adequate Yearly Progress (AYP); 8 of the 10 charter high schools fell short of their AYP benchmark.

By contrast, all of Tom Carroll’s Albany charter schools made AYP. Not only that, they had become the best schools in the city. “Last year our students were number one in math in every single grade,” says Carroll, who now runs the Foundation for Education Reform & Accountability (FERA), which furnishes research help to charters, and serves as chairman of the Brighter Choice Foundation (BCF), which provides start-up financing aid. “In English, we were number one in 4th and 7th grades. We’re expecting to do even better this year.” And they have. On the 2009 state test in English language arts, in four of the six grades tested, the top school in Albany was one of Carroll’s charters.

Albany public school parents, mostly black and mostly poor, not only have a choice; they have one that will make a significant difference in their children’s futures. The Brighter Choice network has turned largely forgotten students into serious achievers. These schools have not only closed the achievement gap; they have reversed it.

An Unlikely Road to School Reform
“You might say that our success is the revenge of the amateurs,” jokes Carroll, over a recent lunch at a downtown Albany bistro. “We didn’t really know anything about education when we started—and perhaps that’s why we have succeeded.”

A veteran of the sharp-elbowed politics of New York’s infamously dysfunctional state legislature (called the worst in the nation by the Brennan Center for Justice in 2004), Carroll assembled a team of equally determined and savvy colleagues, most of whom had honed their political skills in those same tough trenches. They sounded like a law firm: Carroll, Backstrom, Murphy, Bender, and Brooks. Eva Joseph called them, disparagingly, “the white guys.” In fact, they knew money and they knew politics, and when they stumbled on to the disastrous state of public education, they became determined to know schooling.

“We were all focused on budget and tax issues that would make New York a more job-friendly place,” recalls Peter Murphy, who had worked with Carroll in the state’s budget office. “But we did lots of work examining various parts of the government, including education.” Increasingly, more budget and tax roads led to education, which consumed more than a quarter of the state’s revenues.

While Murphy continued work with a think tank called the Empire Foundation for Policy Research, helping to produce a report on education choice in 1993, Carroll and Brian Backstrom, another veteran of the budget office, and Jason Brooks, a fresh-faced triple major (history, religion, and political science) from Syracuse University, started FERA in 1998. FERA would seal their fates as education reformers when Virginia Gilder, then the wife of one of their major political reform benefactors, Wall Street financier Richard Gilder, asked Carroll what she could do to help fix the schools. Since there was no charter law in New York at the time, the group launched a voucher program. And in an early test of their market-share strategy, Carroll and company decided to spend all of Gilder’s money at just one school, offering $2,000 to 153 students, a third of the student body, at Giffen Elementary, “one of the worst public schools in New York State,” according to Forbes magazine, which featured the program on its cover.

The focus on one school, the national attention, and Giffen being “within spitting distance of the State House,” as Forbes put it, ensured that FERA would be an education reform player and an immediate thorn in the local school district’s side. Even Fred LeBrun—an influential columnist for the Times Union who once called Carroll and his political friends “a blustery gathering of overstuffed three-piece suits with watch fobs”—sympathized, praising the group for “walking the walk, not just talking the talk.” (The message was lost on Albany Public Schools: to this day, Giffen remains hopelessly behind the academic eight ball, with just 46 percent of its 168 remaining students in grades 6—8 able to pass proficiency tests in English and 57 percent passing in math. Carroll still oversees the voucher program, which continues to provide options for 38 Albany public school students each year.)

The next opportunity to walk the walk came in 1998 when the group helped write the state’s charter school law, passed in December, making New York the 34th state to have one.

The Road to a Brighter Choice
Carroll and his colleagues, now fully engaged in their roles as school reformers, immediately established the Charter School Resource Center to offer technical assistance to anyone willing to set up a charter school.

They knew their market. Almost overnight 90 applications for charter schools (the law set a cap of 100) were submitted to the state education department’s Board of Regents and the newly formed Charter Schools Institute at the State University of New York (SUNY). “You know you’ve hit on something when there is that level of interest across the state for doing something different in public education,” says Backstrom.

Carroll, Backstrom, and Brooks traveled the country and spent months on the phone quizzing successful school leaders: what works in your school, what doesn’t work? “And time and time again we were struck with how similar the answers were,” says Backstrom. Longer school day. Longer school year. Content-rich curriculum. School uniforms. Even the single-sex school, they learned, which had been all but driven off the education landscape by Title IX, was being tried, if quietly, and was working.

In November 2000 the group submitted a 300-page application to the Board of Regents to open the Brighter Choice Charter School for Boys (BCCS-Boys) and Brighter Choice Charter School for Girls (BCCS-Girls), which, initially, would be housed in the same building.

“The City of Albany is an educational tale of two cities,” the applicants wrote. Some 30 percent of Albany students already attended private schools—almost double the state average—and only 2 of the city’s 15 public schools managed even “respectable test scores.” Failure rates in the rest of the schools ranged from 50 percent to 80 percent in 4th-grade reading tests; citywide, the failure rate was 64 percent.

This, said the charter applicants, was “not acceptable.” The Regents agreed and granted charters to the two schools. And the rest, as Eva Joseph and a city of doubters would soon learn, is history.

Political Savvy Meets Commitment to Excellence
Even though they would have fewer than 100 places when the two schools opened in September 2002 (with just two grades, K—1, and plans to expand through grade 5), the schools received a thousand applications. This for a place that promised longer days (an hour more than the regular public schools), an extra 25 days of school per academic year, tough discipline, uniforms, and rigorous academic standards.

Success, of course, was anything but guaranteed. “They fought us every step of the way,” recalls Carroll of the APS and the teachers union. The New York State United Teachers (NYSUT), which boasted 600,000 members and had a retirement fund of $70 billion, was considered the most powerful lobbying group in the state. “They bring the cash to the legislators in wheelbarrows,” says Peter Murphy, only half-joking, “and not a day goes by that they aren’t trying to kill charters.” On one occasion, NYSUT slipped an amendment on to an obscure law that would have limited the market share of charters in Albany to 5 percent. Thanks to their many legislative connections, Carroll and team heard about the move and sent a busload of parents and students to the legislative hearing room. “If we had not had a legislative political background,” says Carroll, “they would have taken us to the cleaners.”

And then came Chris Bender, who was 10 minutes into pitching his school insurance product to Carroll when Carroll said, “You should be on my board.”

It was not an impetuous offer. Carroll knew that Bender was a fifth-generation Albany native and heir to a local publishing fortune. “We had people who knew a lot about charter schools,” explained Carroll. “But Chris Bender’s family had been here for 400 years, since the Dutch arrived. Between the two of us there are very few people of any significance in Albany that one of us doesn’t know. If we needed advice on construction, for instance, we could get to the best people. Who knows historic preservation? Who knows environmental regulation? We could figure all that out. If you were just starting out, if you helicoptered in from another country and tried to do it, you would trip over yourself a million times. With a master’s degree in education from Teachers College at Columbia University, Bender was also the only member of the team with real education credentials.

Three months after joining the board Bender became executive director of the group’s new nonprofit, the Brighter Choice Foundation, a technical and financial resource organization for Albany charter schools that would become the key to scale.

Through BCF, Carroll expanded the “what works” operating principle to include not just re-creating specific proven policies and practices, but replicating whole schools. The Achievement Academy middle school, which opened in the fall of 2005, was modeled after the Amistad Academy in New Haven, Connecticut, a pioneer of charter success. Albany Prep, which also opened in 2005, was modeled on the International Baccalaureate program and offered extended instructional periods for core academic subjects. The BCF would not only scout out potential charter school operators; it would build them a building; arrange for financing; and provide operational start-up money and free technical assistance, including community relations, politics, media relations, vendor advice, and legal advice if needed. The BCF lobbied KIPP (Knowledge Is Power Program) founder David Levin to open a school in Albany, offering KIPP both financial and logistical support and the freedom to focus on academic excellence. KIPP Tech Valley, a middle school, opened in 2005.

The financing scheme is both ingenious and, Carroll argues, replicable (see sidebar, page 36). For instance, BCF turned a $15 million loan from the Walton Foundation into a “revolving loan fund,” explains Carroll, that “allows us to build a facility and then take out a mortgage on it.” Once the building is built and the kids are in the chairs, BCF secures a short-term mortgage, which it pays while the new school is starting. When the school reaches full enrollment, usually within four or five years, it can then issue tax-exempt bonds and buy the building from BCF.

Getting It Right
The financing strategy, which includes start-up grants to new schools of up to $500,000, “helps schools start smaller, so they can start good,” says Chris Bender, who also believes that the smart leveraging of small amounts of money is what makes the BCF model replicable.

With such a model in hand, “a paradigm shift for us,” says Carroll, Brighter Choice was able to go to scale with quality and begin to make deep changes in the city’s educational system. The strategy is working so far.

At BCF’s flagship schools, Brighter Choice for Girls and Brighter Choice for Boys, 3rd and 4th graders have been outperforming their district counterparts almost from the beginning on the statewide English language arts and mathematics exams (see Figure 1 for the 2009 test results).

“The Brighter Choice and KIPP schools are even outscoring the white suburban districts surrounding Albany,” says Carroll. “KIPP beat Bethlehem Middle School and Shaker Junior High (both in Albany’s affluent white suburbs), which is North Colonie’s middle school, in 7th grade English and math last year. The point is that we are not just beating crappy mediocre Albany schools; we’re beating the top public school districts in the area.”

Things have not always gone perfectly for Carroll and company. There was early backsliding in performance indicators at the Brighter Choice charter schools, for instance. The Brighter Choice board, which Carroll chairs, immediately asked the state department of education for permission to postpone adding a 5th grade as had been planned. They then did what traditional public schools seem so reluctant to do: they immediately changed leadership personnel. Brighter Choice went through three principals in three years, “until we got it right,” says Carroll.

Carroll’s missteps highlight the secrets to Brighter Choice’s success: constant vigilance, constant adaptability. When the Achievement Academy scores tanked, coming in below the district average in 2006, its second year, the school’s board immediately fired the principal and several teachers (who work with one-year contracts), changed textbooks as well as some systems and routines, and saw, the following year, a marked increase in test scores.

Scale Counts
The second part of the Brighter Choice story has been taking that quality to scale, to provide the kind of pressure to change that will improve the educational landscape.

“In a country this size, creating 50 or 60 really good schools barely creates a ripple,” explains Carroll. “It has a profound impact, of course, on the children educated in them, but it doesn’t challenge any of the institutional structures.”

Scale also provides a certain element of political protection, as Carroll has learned. “If you’re a single school and they close it down, most political people are willing to take that hit. But they’re not willing to close down schools serving thousands of kids.”

Though no one on the Brighter Choice team ever imagined shepherding eight schools into existence (the ninth, a girls’ high school, Albany Leadership, will open in 2010, and applications for two new middle schools are in the pipeline), they were even less sanguine about their chances of moving Albany’s public school system to change.

“We learned early on,” says Brian Backstrom, “that you couldn’t get the schools you needed by changing the schools you had. So that’s why we decided to build new schools, from the bottom up.”

But there have been signs of change at Albany Public Schools, which suggests that scale can count. “They have done some things,” says Backstrom. “Uniforms in one school. They’ve renovated all their facilities, gotten small class sizes—of course, they’ve done that in part because we’ve taken 2,200 kids away from them, but they’ve done it.”

The district has also lengthened its school day, albeit by only 30 minutes, but it is the first change in the union-contracted school day since the district was created in the mid-1970s. And they’ve even closed a perennially underperforming school (the Livingston Middle School). “Admittedly,” says Carroll, “our decision to locate [KIPP Tech Valley] charter school across the street from their weakest school was not subtle.”

Jason Brooks went through the minutes of Albany’s board of education meetings and discovered that, in fact, they were indeed listening, watching—and talking about Brighter Choice. At a March 17, 2005, meeting, for instance, board member Bill Barnett exclaimed, “I think that it’s high time for this district and the Board to have an in-depth discussion around the implications and the associated cost of increasing the number of instruction days.”

Added board member Scott Wexler, “We cannot compete with charter schools with 200 days while our…calendar committee [has] not [had an] instructional discussion [but] an employee benefit discussion. Our employees apparently need to understand our desire to have more time on task so we can be more successful.”
It’s a hopeful start and Brooks created a memo called “The Positive ‘Ripple Effect’ of Charter Schools” to note it.

And, though the district won’t admit the connection, Carroll believes that the recent overall rise in district elementary- and middle-school test scores is the result of “competition from charter schools [which] has forced an increased district focus on measurable outcomes.”

Finally, on March 25, 2009, just two weeks after her Rockefeller Institute charter critique, Eva Joseph announced her resignation. “She said the job has consumed so much of her time, sometimes seemingly 24 hours a day, that she looks forward to relaxing mentally,” wrote Scott Waldman in the Albany Times Union. “Under Joseph, Albany also became ground zero for the charter school movement, with the city’s 11th school expected to open soon. Joseph and the city’s charter school leaders often were at odds.”

“The question remains,” says Carroll, “as charter schools continue to grow in the city—within a year of this September roughly a third of public school children in Albany will be in charter schools—will the district put its head in the sand or finally be forced to reform its schools in order to compete?”

Carroll and company are not waiting around to see what happens. “Our opponents would love to freeze us in our tracks through a moratorium or a market-share cap,” notes Carroll. “But, as long as Albany has a shortage of good schools, the demand from parents and students for more charter schools will not diminish. Why would we want to stop creating charter schools to meet this demand when we know the alternative is for these children to attend bad schools?” A good question.

Peter Meyer, former news editor of Life magazine, is a freelance writer and contributing editor of Education Next.

Researchers have recently begun to learn more about the effects of time spent on learning from natural experiments around the country. This new body of evidence, to which we have separately contributed, suggests that extending time in school would in fact likely raise student achievement. Below we review past research on this issue and then describe the new evidence and the additional insights it provides into the wisdom of increasing instructional time for American students.

We also discuss the importance of recognizing the role of instructional time, explicitly, in accountability systems. Whether or not policymakers change the length of the school year for the average American student, differences in instructional time can and do affect school performance as measured by No Child Left Behind. Ignoring this fact results in less-informative accountability systems and lost opportunities for improving learning outcomes.

Emerging Evidence

More than a century ago, William T. Harris in his 1894 Report of the Commissioner [of the U.S. Bureau of Education] lamented,

The boy of today must attend school 11.1 years in order to receive as much instruction, quantitatively, as the boy of fifty years ago received in 8 years…. It is scarcely necessary to look further than this for the explanation for the greater amount of work accomplished…in the German and French than in the American schools.

The National Education Commission on Time and Learning would echo his complaint one hundred years later. But the research summary issued by that same commission in 1994 included not one study on the impact of additional instruction on learning. Researchers at that time simply had little direct evidence to offer.

The general problem researchers confront here is that length of the school year is a choice variable. Because longer school years require greater resources, comparing a district with a long school year to one with a shorter year historically often amounted to comparing a rich school district to a poor one, thereby introducing many confounding factors. A further problem in the American context is that there is little recent variation in the length of school year. Nationwide, districts generally adhere to (and seldom exceed) a school calendar of 180 instructional days. And while there was some variation in the first half of the 20th century, other policies and practices changed simultaneously, making it difficult to uncover the separate effect of changes in instructional time.

Among the first researchers to try to identify the impact of variation in instructional time were economists studying the effect of schooling on labor market outcomes such as earnings. Robert Margo in 1994 found evidence suggesting that historical differences in school-year length accounted for a large fraction of differences in earnings between black workers and white workers.

Using differences in the length of the school year across countries, researchers Jong-Wha Lee and Robert Barro reported in 2001 that more time in school improves math and science test scores. Oddly, though, their results also suggested that it lowers reading scores. In 2007, Ozkan Eren and Daniel Millimet examined the limited variation that does exist across American states and found weak evidence that longer school years improve math and reading test scores.

Work we conducted separately in 2007 and 2008 provides much stronger evidence of effects on test scores from year-to-year changes in the length of the school year due to bad weather. In a nutshell, we compared how specific Maryland and Colorado schools fared on state assessments in years when there were frequent cancellations due to snowfall to the performance of the very same schools in relatively mild winters. Because the severity of winter weather is inarguably outside the control of schools, this research design addresses the concern that schools with longer school years differ from those with shorter years (see research design sidebar).

While our studies use data from different states and years, and employ somewhat different statistical methods, they yield very similar results on the value of additional instructional days for student performance. We estimate that an additional 10 days of instruction results in an increase in student performance on state math assessments of just under 0.2 standard deviations. To put that in perspective, the percentage of students passing math assessments falls by about one-third to one-half a percentage point for each day school is closed.

Other researchers have examined impacts of instructional time on learning outcomes in other states, with similar results. For example, University of Virginia researcher Sarah Hastedt has shown that closures that eliminated 10 school days reduced math and reading performance on the Virginia Standards of Learning exams by 0.2 standard deviations, the same magnitude we estimate for the neighboring state of Maryland. Economist David Sims of Brigham Young University in 2008 took advantage of a 2001 law change in Wisconsin that required all school districts in that state to start after September 1. Because some districts were affected while others were not, he was also able to provide unusually convincing evidence on the effect of changes in the number of instructional days. He found additional instruction days to be associated with increased scores in math for 4th-grade students, though not in reading.

Collectively, this emerging body of research suggests that expanding instructional time is as effective as other commonly discussed educational interventions intended to boost learning. Figure 1 compares the magnitude of the effect of instructional days on standardized math scores to estimates drawn from other high-quality studies of the impact of changing class size, teacher quality, and retaining students in grade. The effect of additional instructional days is quite similar to that of increasing teacher quality and reducing class size. The impact of grade retention is comparable, too, though that intervention is pertinent only for low-achieving students.

Although the evidence is mounting that expanding instructional time will result in real learning gains, evidence on the costs of extending the school year is much scarcer and involves a good deal of conjecture. Perhaps the best evidence comes from a recent study in Minnesota, which estimated that increasing the number of instructional days from 175 to 200 would cost close to $1,000 per student, in a state where the median per-pupil expenditure is about $9,000. The total annual cost was estimated at $750 million, an expense that proved politically and financially infeasible when the proposal was recently considered in that state. Comparing costs of expanding instructional days with the costs of other policy interventions will be an analytic and policy exercise of real importance if the call for expanded instructional time is to result in real change.

Complicating this analytic task are differences in costs that exist across schools and states. Utilities, transportation, and teacher summer-labor markets vary widely across geographic areas, and all affect the cost of extending the school year. So, while the benefits of extending the school year may exceed the costs in some states or school districts, they may not in others. A further complication is the possibility of diminishing returns to additional instructional time. Our research has studied the effect of additional instructional days prior to testing, typically after approximately 120 school days. The effect of extending instructional time into the summer is unknown. Also, our research has focused on the variation in instructional days prior to exams, or accountable days. The effect of adding days after exams could be quite different.

Costs of extending school years are as much political as economic. Teachers have come to expect time off in the summer and have been among the most vocal opponents of extending school years in several locations. Additional compensation could likely overcome this obstacle, but how much is an unresolved and difficult question.

Teachers are not the only ones who have grown accustomed to a summer lasting from June through August. Students and families have camps, vacations, and work schedules set up around summer vacation. “Save Our Summers” movements have for years decried the benefits of additional instructional days and proclaimed the benefits of summer vacation, and the movements have grown as states have considered extending the school year and individual school districts have moved up their start dates. Longer school years might reduce tourism and its accompanying tax revenue. These additional costs likely vary by state and district, but are clearly part of the analytic and political calculus.

Time and Accountability

As education policymakers consider lengthening the school year and face trade-offs and uncertainties, it is important to recognize that expanding instructional time offers both opportunities and hazards for another reform that is well established, the accountability movement. Educators, policymakers, parents, and economists are sure to agree that if students in one school learn content in half the time it takes comparable students at another school to learn the same content, the first school is doing a better job. How students would rank these schools is equally obvious. Yet state and federal accountability systems do not account for the time students actually spent in school when measuring gains, and so far have no way of determining how efficiently schools educate their students.

One implication of this oversight is that accountability systems are ignoring information relevant to understanding schools’ performance. Year-to-year improvements in the share of students performing well on state assessments can be accomplished by changes in school practices, or by increases in students’ exposure to school. Depending on the financial or political costs of extending school years, those with a stake in education might think differently about gains attributable to the quality of instruction provided and gains attributable to the quantity.

To see how the contributions of these inputs might be separated, consider data from Minnesota. Between 2002 and 2005, 3rd graders in that state exhibited substantial improvements in performance on math assessments, a fact clearly reflected by Minnesota’s accountability system. But during that period, there was substantial year-to-year variation in the number of instructional days students had prior to the test date. In Figure 2, we plot both the reported test scores for Minnesota 3rd graders (the solid line) and the number of days of instruction those students received (the bars). Useful, and readily calculated, is the time series of test scores, adjusting for differences in the number of instructional days (the dotted line).

Comparing the reported and adjusted scores is useful for at least two reasons. First, it illustrates the role of time as a component of test gains. Overall, scale scores increased by 0.4 standard deviations from 2001–02 to 2004–05. Of this increase, a large portion was attributable to expansion in instructional time prior to the test date. Adjusting for the effect of instructional days, we estimate that scores increased by roughly 0.25 standard deviations, nearly 40 percent less than the reported gains.

Second, the comparatively steady gain in adjusted scores over the period provides evidence of improvements in instructional quality, independent of changes in the amount of time students were in class. The fast year-to-year increases in the first and last periods result in large part from increases in the amount of time in school, while the negligible change in overall scores between 2003 and 2004 does not pick up real gains made despite a shortened school year. Adjusted scores pick up increases in learning gains attributable to how schools used instructional time, such as through changing personnel, curricula, or leadership. The point here is that time-adjusted scores provide information that is just as important as the overall reported scores for understanding school improvements. A robust accountability system would recognize that more instructional time can be used to meet goals, but that more time is neither a perfect substitute for, nor the same thing as, better use of time.

The Hazards of Ignoring Time

Failing to account for the role of time in student learning not only means missed opportunity, it also creates potential problems. First, it can allow districts to game accountability systems by rearranging school calendars so that students have more time in school prior to the exam, even as the overall length of the school year remains constant. Beginning in the 1990s, districts in a number of states began moving start dates earlier, with many starting just after the first of August. The question arose whether these changes might be linked to pressures on districts to improve performance on state assessments. David Sims showed that Wisconsin schools with low test scores in one year acted strategically by starting the next school year a bit earlier to raise scores. Evidence of gaming soon emerged in other states as well. Wisconsin passed its 2001 law requiring schools to begin after September 1 to prevent such gaming; similar laws were recently passed in Texas and Florida.

The motives driving earlier start dates could spill over into other instructional policies. Minnesota moved its testing regimen from February to April in the wake of accountability standards, while Colorado legislators have proposed moving their testing window from March into April, with advocates suggesting that the increased time for instruction would make meeting performance requirements under No Child Left Behind more feasible for struggling schools. While administering the test later in the year has potential benefits in measured performance, grading the tests over a shorter time frame costs more, estimated at some $3.9 million annually in Colorado. Schools thus sacrifice educational inputs (such as smaller classes or higher teacher salaries) to pay for the later test date.

A second hazard involves fairness to schools at risk of being sanctioned for poor performance: these schools can face longer odds if weather or other schedule disruptions limit school days. The impact of instructional time on learning means that one factor determining the ability of schools to meet performance goals is not under the control of administrators and teachers. We illustrate the effects of time on making adequate yearly progress (AYP) as defined by No Child Left Behind by comparing the performance of Maryland schools the law identified as underperforming to estimates of what the performance would have been had the schools been given a few more days for instruction.

We begin with data from all elementary schools in Maryland that did not make AYP in math and reading during the 2002–03 to 2004–05 school years. We adjust actual performance by the number of days lost in a given year multiplied by the marginal effect of an additional day on test performance as reported in Marcotte and Hemelt’s study of Maryland schools. This allows us to estimate what the proficiency rates in each subject would have been had those schools been open for all scheduled instructional days prior to the assessment. We then compare the predicted proficiency rate to the AYP threshold.

We summarize the results of this exercise in Figure 3. The light bars represent the number of schools failing to make AYP in math and reading in various years. The dark bars are the number of those schools that we predict would have failed to make AYP if the schools had been able to meet on all scheduled days. We make these estimates assuming that low-performing schools would have made average gains with each additional day of instruction.

The average number of days lost to unscheduled school closings varied substantially over the period, from more than 10 to fewer than four and a half. Many schools that did not make AYP likely would have had they not lost so many school days. For example, we estimate that 35 of the 56 elementary schools that did not make AYP in math in 2002–03 would have met the AYP criterion if they had been open during all scheduled school days. Even if these schools were only half as productive as the typical school, 24 of the 56 flagged schools would likely have made AYP if they had been open for all scheduled days.

There is, however, a way to reduce risks like these for schools and to limit incentives for administrators to move start or test dates at the same time: that is to recognize and report time as an input in education. A simple and transparent way to do this is for state report cards, which inform parents about school outcomes and summarize the information on AYP status, to include information about the number of instructional days at test date as well as the total number of instructional days for the year. This information is readily available and already monitored by schools, districts, and states. Local and state education authorities could use it when assessing performance, for example, in hearing an appeal from a school that failed to meet its AYP goals. Further, this information could be used to estimate test scores adjusted for instructional days, to be used alongside unadjusted changes in performance. Distinguishing between gains due to expanded instruction time and better use of that time can enrich accountability systems and provide more and better information to analysts and the public alike.

Looking Ahead

There can be no doubt that expanding the amount of time American students spend in school is an idea popular with many education policymakers and has long been so. What makes the present different is that we now have solid evidence that anticipated improvements in learning will materialize.

Practical obstacles to the extension of the school year include substantial expense and stakeholder attachment to the current school year and summer schedule. The benefits of additional instructional days could diminish as school years are lengthened. Further, it is unknown how teachers would use additional instructional days if they are provided after annual testing is already finished. Simply extending the year well after assessments are given might mean that students and teachers spend more days filling (or killing) time before the end of the year. This would make improvements in learning unlikely, and presumably make students unhappy for no good reason.

Though the issue has seen little movement in the past and faces real opposition going forward, the policy climate appears likely to be favorable once the fiscal challenges now facing public school systems recede. It is our hope that policymakers and administrators who try to take advantage of this window of opportunity don’t harm reforms that have succeeded in improving learning outcomes and don’t implement reforms in a manner that would fail to do the same. Advocates for extended school years have so far said virtually nothing about whether or how accountability systems should accommodate longer school years.

Across the country, a small number of schools and districts are modifying or extending the academic year. The Massachusetts 2020 initiative has provided resources for several dozen schools to increase the number of instructional days they offer from 180 to about 200. Other examples include low-performing schools that have lengthened their school day in an effort to improve, and the longer school days, weeks, and years in some charter schools. However, such initiatives remain rare, with no systemic change in the instructional time provided to American students. Our work confirms that increasing instructional time could have large positive effects on learning gains. Encouraging schools and districts to view the school calendar as a tool in the effort to improve learning outcomes should be encouraged in both word and policy.

Dave E. Marcotte is professor of public policy at the University of Maryland, Baltimore County. Benjamin Hansen is a research associate at IMPAQ International, LLC.

Please join me on a short, visionary tour circa 2025, and let us glimpse the central role that data have come to play in American K–12 education.

Perhaps the most profound advance since 2010 is that individual achievement and attainment records for every subject are saved (with elaborate safeguards) in cyberspace and secure state databases, where “unique student identifier” numbers make it possible for data to be readily aggregated without revealing individual identity and for analysts to investigate things like learning gains by pupils in various schools and circumstances.

Student assessments (formative, summative, informal) are completed electronically, many through adaptive online programs. Software automatically analyzes the resulting information to create a data dashboard for each pupil, showing what has been mastered and what still needs work. Most assessments are graded by computer, although teachers read essays and occasionally offer separate “hand-graded” scores on other assignments. Instant preliminary feedback is the norm, and the official results, checked over by a data team, are available soon thereafter.

An artificial intelligence program periodically “sifts” each student’s cumulating education record to answer—especially for parents, teachers, and counselors—such key directional questions as whether the student is on track for college when she completes high school. Are there any warning signs of academic (or other) problems that warrant a change of course, maybe even a swift intervention?

Parents can log on and view their child’s cumulative report card, which is continually updated, not just with test results but also with sample work, attendance data, and teacher comments.

Multiple teacher web sites offer resources for planning lessons and obtaining supplementary materials. These include most everything an instructor might need, from student readings, workbooks, assignment ideas, web links and mini-tests to audio and video snippets for classroom use. The online curriculum vault includes thousands of videos of master teachers delivering lessons, and interactive web sites host discussion groups (most enable participants to view as well as hear and read each other). Increasing portions of students’ days are given over to virtual education: watching lectures, participating in online discussions, making productive use of software programs, e-mailing or conversing with distant experts, and teaming up with peers as much as half a world away.

Principals keep electronic files of data (as well as eyewitness impressions, pupil and parent and peer ratings) on individual teachers’ pedagogical strengths and weaknesses. Linked teacher and student databases are used to formulate professional development activities for each teacher. Classroom sessions are periodically recorded and viewed by online mentors who offer quick feedback to new or struggling teachers. Pupil achievement consultants review students’ data files and advise teachers on working with challenging students.

Schools regularly calculate gain scores for each pupil and every state has a Tennessee-style value-added scoring system that spits out data on the effectiveness of its teachers, schools, and districts. Analysts can now control for outside factors affecting achievement. Districts and schools can also use them to evaluate the effects of particular textbooks, teaching units, and professional development activities.

Information about individual performance is aggregated across pupil populations at the classroom (and teacher), school, district, state, and national levels and cumulated over time. Such data enable principals, superintendents, and state officials to determine which institutions, programs, and individuals are on track to attain their targets. The public gets data, too, and can gauge the return on its education investments. Media outlets faithfully publish England-style “league tables” showing raw scores, value-added results, and change over time for every school.

The progress in education data over the past two decades surpasses that made during the entire previous century. Considering the size and decentralized nature of U.S. education, the sluggishness with which it has reacted to many demands for reform, and the modest political oomph behind such mundane activities as crafting data systems, the gains are remarkable. The best explanation seems to be that the millions of people in public education have finally come to realize that the more you know the better off you are.

Chester Finn is president of the Thomas B. Fordham Institute and senior editor of Education Next.

The Education Equality Project champions accountability, pay reform, and school choice, while the Broader, Bolder coalition insists we must attend to health care, preschool, and parenting skills if students are to succeed in school. The Obama administration must negotiate this split in pursuing education reform; indeed, Secretary of Education Arne Duncan was the only individual to serve as a founding member of both groups.

In this forum, president of Democrats for Education Reform Joe Williams speaks for the Education Equality Project and Pedro Noguera offers the Broader, Bolder perspective on improving K–12 schooling, the early record of the Obama administration, and the challenges that lie ahead.

Education Next: What principles unify the signers of the coalition [Education Equality Project or A Broader, Bolder Approach to Education]? Can you explain the key reforms the coalition is calling for?

Pedro Noguera: The basic principle underlying the Broader, Bolder Approach to school reform is that efforts to raise student achievement cannot ignore the unmet social needs of children, particularly those related to concentrated poverty—inadequate health, housing, and nutrition. These conditions have a tremendous impact upon child development and learning.

Poverty does not cause academic failure, but it is a factor that profoundly influences the character of schools and student performance, in at least three broad and interrelated ways: 1) in most cases, considerably less money is spent on the education of poor children. Per-pupil spending has bearing on the quality of facilities, the availability of learning materials, and the ability of schools to attract and retain highly qualified personnel. While high levels of funding do not guarantee that children will receive a quality education, money matters, and many of the most acclaimed charter schools spend more per pupil than public schools, even though they generally serve fewer high-need students (i.e., special education or English language learners); 2) the unmet, nonacademic needs of children (social, emotional, and psychological) often have an impact on learning; 3) schools serving large numbers of poor children typically lack the resources and expertise to respond to their academic and social needs.

This does not mean that poor children cannot learn or that until we eliminate poverty and related social issues we will not be able to educate all children in this country. There are schools across the country—some are charter, some are private, and many are traditional public—that have shown us that it is possible for poor children to achieve at high levels when we respond to their needs and create conditions that are conducive to learning. However, the fact that a small number of schools have experienced a degree of success does not mean that we can simply blame other schools for their failures or ignore what is happening to children outside of school. Many, though not all, schools that succeed with poor children devise strategies to mitigate the effects of poverty with site-based social services and extended learning opportunities.

BBA advocates providing universal access to health care for children, quality early-childhood education, and expanded access to extended learning opportunities, after school and during the summer. While these measures alone will not guarantee higher student achievement or large-scale school improvement, they are essential for creating a context in which other education reforms can be effective.

Joe Williams: The Education Equality Project is a coalition of leaders (from education, civil rights, government, public policy, and business) who believe that what happens inside schools (and in the politics surrounding schooling) plays a tremendous role in shaping the achievement gap that exists in this country between the haves and the have-nots. The focus for reform, therefore, should be on what happens between teachers and students. That isn’t meant to be glib; we keep finding ourselves debating that key distinction with people who argue that the external forces in a child’s life represent obstacles too large for even great schools to overcome. While we are very sympathetic to the obstacles that impoverished children face to their physical, emotional, and educational development, and support policies to address these deficiencies, we believe that when conditions outside of the classroom are less than stellar, it is even more important that we get the schooling piece right.

One of the beliefs that has tied together the signatories of EEP thus far is a commitment to eliminating the racial and ethnic achievement gap in this country. This is not just an education issue, but a civil rights issue. If we neglect the education needs of our children, we are depriving them of the kinds of opportunities that the American dream can offer.

The EEP has called for an effective teacher for every child (paying teachers as professionals, giving them the tools and training to do their work effectively, and making tough decisions about ineffective teachers); empowering parents by allowing them to choose the best schools for their children; holding grown-ups at all levels accountable for the education of our children; and, very important, having enough strength in our convictions to stand up to anyone who seeks to preserve a failed system.

EN: Is it fair to expect all students to meet a uniform performance baseline? Is it reasonable to hold schools and educators responsible for ensuring that students meet that bar?

JW: Yes, these expectations are fair and reasonable. The key is making sure that schools and educators have the tools to provide students with the kind of education they need to clear the bar, including resources, the ability to build teams of excellent educators, and enough flexibility at the school level to adjust the length of the school day and year (among other things). This will likely require both additional resources and smarter use of education budgets around the country. Newark mayor Cory Booker often talks about the fact that we allow time spent on education to be the constant, while achievement is the variable. We need the flexibility to flip that notion so that time is the variable and achievement is the constant.

PN: Setting high academic standards for schools and students to meet is important but relatively easy to do. The harder and more important task is to adopt and implement standards that create optimal conditions for learning. This means ensuring that all children, regardless of where they live, have access to high-quality schools. This is what government policy must strive to achieve. We have quality standards for airports, highways, food, drugs, and water, but no state has adopted standards for learning environments, and many poor children attend under-resourced, inferior schools.

In fact, the most troubled schools typically serve students with the greatest needs. These schools cannot solve problems related to inequality and poverty without additional support. Yet this is essentially what No Child Left Behind (NCLB) and most education reforms that preceded it have expected. Almost eight years after the enactment of NCLB, high dropout rates and low achievement are still pervasive throughout this country, particularly in schools where poor children are concentrated.

EN: Do you think the administration’s actions thus far on school choice and charter schooling have been too aggressive or not aggressive enough?

PN: School choice is an idea that should be supported in principle. It is good for parents to have a variety of schools from which to choose because not all children have the same needs or interests. The greater challenge is ensuring that there are many high-quality schools to choose from and ensuring that choice does not contribute to further segregation in schools. Unfortunately, in many communities that have enacted choice plans, well-organized and informed parents do their best to gain access to the better schools, and invariably, others are left out. Racial segregation in schools has increased in the last 20 years, and poor children have become concentrated in the worst schools. Furthermore, in most choice systems it’s not parents but schools that really do the choosing. The better schools are often able to screen out needy students and limit enrollment. Because of high demand, they can be selective about whom they choose. This often occurs even in charter schools that use lotteries to determine admission but set criteria that are difficult for low-income parents to meet. Those who are not chosen by the superior schools invariably end up in lower-quality public schools with fewer resources.

Many, but not all, charter schools have demonstrated considerable success in educating poor children. Secretary of Education Arne Duncan has expressed his support for charter schools, even though in several states, such as Texas and Arizona, the charter schools are often no better, and in some cases are worse, than the public schools. As a trustee of the State University of New York, I am proud to say that the charter schools we authorize consistently outperform similar schools in the communities where they are located. If such quality-control measures can be adopted in other communities, charter schools should be supported as a means to increase the supply of good schools available to poor children.

JW: Choice, in and of itself, won’t bring about the kind of systemic change that we need. But it is difficult to imagine how we can drive that systemic change without choice playing a role. The administration’s actions to limit the D.C. Opportunity Scholarship (K–12 vouchers), for example, were perplexing, if only because the actions were accompanied by empty rhetoric about doing what is best for children. How do we look at low-income families with a straight face and tell them they can’t send their children to better schools because it isn’t the right policy to pursue for the broader system? We need to be doing everything we can to reform the larger system, but by all means, let’s help those families who need good schools now. All of that said, President Obama and education secretary Arne Duncan have provided tremendous cover for the public charter-school movement and have helped shift the focus toward identifying those schools that are doing an outstanding job of educating students and giving them the green light to bring their models to scale.

I have never believed that a voucher or a charter can teach a child to read or do math at exceptionally high levels. That stuff happens in great schools, and vouchers and charter school lotteries offer access to those schools for families who can’t afford to live in affluent neighborhoods or send their children to effective private schools. The key is ensuring that they have an abundance of great schools from which to choose. The public charter-school movement, in addressing both the supply and demand sides of this equation, has emerged as the most promising development in the broader attempt to save public education. The question is whether the charter movement will provide the political spark needed to fundamentally transform our public schools.

EN: Is basing pay on teacher performance essential to school improvement? Is it possible to craft a merit-pay plan that the National Education Association (NEA) and the American Federation of Teachers (AFT) will endorse? Are teachers unions and existing collective-bargaining agreements an impediment to school quality?

JW: I think we have gotten way too far ahead in this discussion. We are talking about merit pay and performance pay in school systems that recognize neither merit nor performance. Teachers unions are understandably squeamish about this topic because today’s testing regimens were not created to serve this purpose. Until people feel confident in the tests that we are using, it will be difficult to build compensation systems on them.

This is an issue we can’t afford to ignore, however. The unions set out to create a standard of fairness for all teachers. The end result, in many cases, is a system that doesn’t allow itself to view great teachers any differently than it does mediocre teachers. Evaluations rate teachers as merely “satisfactory” or “unsatisfactory.” As long as excellence is irrelevant in our schools, we will continue to be stuck in this holding pattern. Wouldn’t it be something if we could strive for systems filled with “excellent” teachers, where excellence actually means something? We’re going to need a lot of help from the NEA and AFT in getting there, since they are holding the keys right now.

PN: Addressing the effectiveness of teachers must be an essential part of education reform in this country. However, judging teachers and awarding bonuses simply on the basis of test scores is problematic. We have already witnessed a large number of schools that have adopted scripted curricula and a narrow focus on test preparation as one way to raise test scores. This tendency will undoubtedly increase if teachers are evaluated exclusively on that basis. Such an approach is likely to discourage good teachers from working in high-need schools and to widen the gap between poor and affluent students. A narrow focus on raising test scores is also likely to deny poor students access to an enriched curriculum that encourages the development of higher-order thinking skills.

It makes more sense to devise incentives, including increased pay, to attract teachers with a track record of effectiveness, to high-need schools and classrooms. Such teachers can be identified through systematic evaluations carried out by principals and peers. If we could combine such a strategy with lower class sizes and extended learning opportunities after school, we could see major gains for struggling students.

In many cities, unions have resisted giving districts greater flexibility in how teachers are assigned, and in too many cases they have made it difficult to remove teachers who are ineffective and inept. Since it seems likely that teachers unions will be around for many years to come, it would be wise to find ways to collaborate with them to devise peer review programs like those that have shown promise in districts such as Toledo, Ohio, and Rochester, New York. In these districts, ineffective teachers are removed in greater numbers than in districts that rely on principal evaluation. Districts should also be encouraged to use the negotiation process to push for greater flexibility in how teachers are assigned to schools.

EN: The president has touted the $5 billion for preschool in the stimulus bill. How can we be confident that the money will fund difference-making programs?

PN: Most of the nations that outperform the United States in educational outcomes provide universal access to quality preschool. Research in child development has shown that the learning that occurs during infancy establishes a foundation for learning throughout life. It is cost effective and in our national interest to expand access to quality early-childhood education for all children.

We know two important things about early childhood education: 1) children who have access to quality programs generally outperform children who do not, and 2) the benefits of quality preschool can be further enhanced if quality of education is maintained in the K–12 system. The situation is similar for elementary schools. Throughout the country we have seen a growing number of successful primary schools and increases in test scores. However, these gains often are not sustained in middle school. This should not be used as a justification to question the value of elementary school nor should similar logic be used to limit expansion of early childhood education.

JW: If high-quality pre-K isn’t such a good idea, why are rich people in my neighborhood running around thinking that the Earth will implode (and their kid won’t get into Harvard someday) if they don’t get a slot in the most sought-after preschool programs? Providing access to high-quality preschool opportunities to the have-nots is an important part of the overall reform effort, as long as those programs successfully help students prepare for the world that awaits them in kindergarten and beyond.

Critics note that finding “high-quality” early-childhood programs, just like finding high-quality K–12 schools, is where the proposition gets iffy. My organization, Democrats for Education Reform, has been pushing to extend state charter-school laws so that charter schools can offer pre-K while being held accountable for their results. Connecting pre-K to early childhood programs that run through 3rd grade would close the gap that exists between what is taught in pre-K and what students need to be able to do in the later grades.

This is about making sure that all students are starting off on as close to a level playing field as possible, whether or not they can afford to make a $100,000 contribution to get a leg up on preschool enrollment.

EN: The Broader, Bolder Approach has made the case that school reform must attend to the “physical health, character, social development, and non-academic skills” of students. Should schools and educators be tasked with this? At what point can or should we start to hold educators responsible for student outcomes?

JW: Students clearly have needs that extend beyond merely learning to read and do math. In the most successful schools serving low-income students, we see a wide range of child development activities, including sports, dance, art, chess, and citizenship enrichment activities. The notion that these activities are distractions from academic instruction assumes this is an either/or proposition. The best schools out there today seem to nail both.

This is where issues like better use of time come into play. Many educators decided long ago (seemingly correctly) that it is not possible to meet the complex needs of their students with a school day that ends at 3 p.m. This is particularly true for students who are two and three years behind where they are supposed to be academically.

PN: It is impossible and undesirable to separate academic performance from physical health, character development, and a variety of nonacademic skills. Sick and unhealthy children generally don’t do as well in school as healthy ones, and children who have trouble getting along with others typically don’t do very well either. From their very beginning, public schools have been charged with preparing children for work and citizenship, and such preparation has never focused solely upon academic skills.

To educate the “whole child,” schools must provide students with an enriched education that includes art, music, physical education, and character development in addition to the core subjects. The fact that skills in these areas cannot be easily assessed should not trouble us since most middle-class and affluent children receive such an education already and typically no one asks for evidence that such an approach has an impact on their test scores.

The highest-performing schools never focus exclusively on student achievement. In fact, what typically distinguishes the best schools from the others is the culture—shared expectations, values, norms, and beliefs—that permeate the school environment.

EN: The president has suggested that the American Recovery and Reinvestment Act, especially the $5 billion in “innovation” education funds, provides an opportunity to “transform” schooling. What are a couple of developments that give you cause for optimism or pessimism? How will we know in a few years if these education funds were spent wisely?

PN: While many public schools, especially in urban areas, are in dire need of reform, I am concerned that there is a lack of clarity about why past reforms have failed and insufficient understanding about the direction change must take if we are to obtain better results. Why do we still have dropout rates of 50 percent and higher in several cities eight years after the enactment of No Child Left Behind, and why are so many schools still foundering after substantial investments of public and private funds on reform? Several studies have shown that reforms have failed because we have ignored the nonacademic needs of children, because we have ignored school culture, because we have not evaluated reforms and insisted upon accountability, and because we have been too quick to pursue fads and gimmicks (small schools, technology, testing) while ignoring more substantive issues that support teaching and learning.

More funding is needed in many districts to address the lack of resources, but given the recession, we will need to rely upon better coordination between schools, nonprofits, and local government to respond to student needs. And money alone will not solve the problems facing America’s schools. We need a new vision and a new approach. A Broader, Bolder Approach offers part of the way forward. This must be combined with strategies that improve the quality of teaching and increase the accountability and responsiveness of schools to the communities they serve.

JW: The president and Secretary Duncan seem to have figured out that the leverage that comes from insisting that $5 billion be attached to innovation is tremendous. Even before a single dime was disbursed from the “Race to the Top” fund, we saw state legislatures take actions to support things like charter school expansion: Massachusetts, Illinois, Indiana, Tennessee, and Rhode Island were not exactly lining up to help charter schools until Duncan made clear that it would impact these states’ applications for federal funding. For a state like Tennessee, which risked losing $100 million in Duncan’s discretionary spending, the conversation quickly changed. A charter-school expansion bill that had been declared dead and tagged by the political coroners came back to life before our very eyes.

The challenge will come when it is time to convert the leverage Duncan has discovered into ongoing federal appropriations. This will launch a dramatic transformation of the role of the federal government in education. This is where we should be optimistic.

Politically, Duncan and Obama are going to need to tell good stories about what has been unleashed here through the stimulus package. If successful school operators like KIPP (Knowledge Is Power Program), Uncommon Schools, and Achievement First can get help (financially and legislatively) in bringing their models to scale, and if successful education programs can be brought to more and more students, there will be a compelling story to tell. Public education will be on its way to saving itself.

EN: What does BBA’s proposed accountability system look like? How does it differ from NCLB?

PN: The BBA proposal for accountability emphasizes qualitative and quantitative evaluations of schools. That is, rather than relying exclusively on test scores to judge schools, BBA calls for the creation of an inspectorate, similar to that used in other countries with high-performing education systems, that is comprised of experienced educators, policymakers and scholars, to evaluate schools and make recommendations about how they might be improved. Such an approach could be used to provide schools with detailed feedback on how to make better use of resources and employ strategies that will enable them to become more successful in raising achievement and overcoming obstacles to learning.

Under NCLB, schools are judged largely on the basis of test scores, and many schools have figured out that the system can be gamed simply by targeting groups of students with intensive test preparation. Schools that are faced with greater challenges are simply labeled “failing” and targeted with threats and humiliation. The underlying assumption is that the educators are lazy and that pressure can be used to force them to improve. Accountability is essential if we are going to bring about school improvement on a larger scale, but it must be accompanied by real assistance and support.

In some cases, shutting down failing schools, as Secretary Duncan has suggested, may be necessary, but we must acknowledge ahead of time that the number of failing schools is simply too great for this to be the only strategy that we use. It is more constructive and effective to find out why a school has failed and to work with educators and local stakeholders to address the causes.

EN: In the context of EEP’s proposed reforms, how will an expanded federal role make a significant difference? How should new federal funds be distributed?

JW: An expanded federal role will allow our entire nation to cut through some of the political fog that has prevented good, sound ideas about how to change our schools from getting the go-ahead to proceed as part of a major systemic reform strategy. This is about using the tremendous leverage of the federal government to force some really blunt conversations at the state and district level, the kinds of conversations that make people uncomfortable and often lead to political paralysis. We have this tendency, if policy conversations make people feel uncomfortable, to sweep important issues under the rug. This is one of the reasons so little has actually changed despite waves and waves of reforms. We have an opportunity to change that dynamic, but only if President Obama holds firm on his commitment to bring change to public education.

Questions of educational adequacy and school spending have long been a point of contention in school reform. Amid the recent economic turmoil and gaping state budget shortfalls, questions of whether court-ordered funding remedies have delivered—and why they have or have not—have taken on particular import. This forum offers two sharply different takes on our experiences to date, and what lessons they offer going forward. Eric Hanushek and Alfred Lindseth are the authors of Schoolhouses, Courthouses, and Statehouses: Solving the Funding-Achievement Puzzle in America’s Public Schools (Princeton University Press, 2009), in which they propose a system of performance-based funding focused on improving student achievement. Michael Rebell is executive director of the Campaign for Educational Equity at Teachers College, Columbia University, and is the author of Courts and Kids: Pursuing Educational Equity through the State Courts (University of Chicago Press, forthcoming), in which he proposes a new functional separation of powers among the executive, legislative, and judicial branches to promote education reform and student achievement.

Education Next: Over the past four decades, many states have revised their funding of schools, through either judicial or legislative initiatives, in an effort to improve schools serving disadvantaged children. Too often, however, these actions have not yielded improved student achievement. Looking to the future, what kinds of judicial or legislative remedies are most likely to fulfill the promise of improved student outcomes?

Eric Hanushek and Al Lindseth: This question is particularly timely, as national policies on education embodied in the federal No Child Left Behind (NCLB) law are in a state of flux and likely to change under President Obama. At the same time, an economic crisis has engulfed not only our country, but most of the world, suggesting that significant increases in funding for education budgets are unlikely in the foreseeable future. The challenge is to find ways to develop a well-educated workforce that are not only more effective than those relied on in the past, but also do not depend on significant annual increases in education appropriations.

Since about 1970, the achievement levels of U.S. students on the reading and math tests of the National Assessment of Educational Progress (NAEP) have remained largely flat despite massive financial and other efforts to improve them. The problem is particularly acute for poor and minority students, with average black and Hispanic students lagging three or four grade levels behind the average white student. While lack of sufficient funding is often cited as the principal reason for low student performance, the United States already spends more on K—12 education than all but a few countries. Moreover, spending has increased dramatically over the past several decades, with today’s per-pupil expenditures almost four times, in inflation-adjusted dollars, what they were in 1960.

The underlying system, which governs how money is spent, has remained largely unchanged over that period. It is characterized by, among other things, a compensation scheme that pays teachers and administrators without regard to the results they get in the classroom; rules that make it extremely difficult to terminate unqualified teachers or assign the good ones where they are most needed; an assessment and rating system that discriminates against good teachers who are assigned to schools with significant numbers of at-risk students; a monopolistic structure that insulates public schools from competition; and numerous union and other work rules that prevent principals from effectively running their schools. It is a system more concerned with the adults and their rights than it is with ensuring the success of its students. Although some reforms have taken place in the last decade or so—the adoption of statewide standards, limited choice options, and increased accountability—they have not been sufficient to overcome the obstacles posed by the underlying system.

Given this sobering assessment, what can be done in the future to improve student achievement? The solution, we believe, lies in performance-based funding: a system of integrated education policies and funding mechanisms designed to drive and reward better performance by teachers, administrators, students, and others involved in the education process. Such a system will ensure more effective use of education dollars through better decisionmaking, eliminate perverse incentives that reward mediocrity or failure, and most important, energize and motivate those involved in the education of our young people. The essential components of a performance-based funding system cannot be ordered à la carte. These components interlock and depend on each other for their success. While various states have adopted some of these components—state-level academic standards, for example—none have implemented the integrated system we recommend, and the results have been clearly unsatisfactory.

A performance-based system of funding would contain the following nine features:

1) A focus on improving outcomes rather than on increasing inputs. States must set high and uniform achievement goals for every child to strive to meet. While every child may not reach the highest goals, high expectations will encourage children to do their best.

2) Local school administrators and teachers with the flexibility to determine how their schools can best meet high standards. Often even the most dedicated teachers and principals are hampered by severe limitations on spending and programmatic decisions by ineffective state regulations, constraints such as those that come with categorical funding, and a variety of state and local laws and contractual arrangements. The idea is to let those who are most familiar with the problems faced in the schools take the lead in deciding how to solve them.

3) Rewards for both teachers and administrators based on their success in improving student achievement. In almost every school district in the country, teachers are currently paid based solely on their years of experience and degree level, despite a consensus in the scientific community that these two factors bear little relationship to their success in improving student performance. The single-salary pay schedule—which makes it virtually impossible to pay good teachers more, to offer bonuses for teaching in hard-to-staff schools, and to pay higher salaries to teachers in shortage areas, such as math, science and special education—must go, and a pay system implemented based upon the just named considerations.

4) Greater accountability commensurate with increased authority and discretion. Teachers, schools, and principals must be held accountable for results. Just as they should be rewarded if they are successful, they must experience the consequences if they are not. Each state should adopt an accountability plan that sets clear goals as well as significant and enforceable consequences if goals are not achieved within a reasonable period.

5) Rewards and accountability based on factors within the control of the local district. Schools, teachers, and administrators should be judged and, if appropriate, rewarded based on the “value” they add during the school year, not on absolute test scores. The latter may be influenced by students’ homes and neighborhoods and may give teachers in middle-class suburban communities an advantage over those teaching in less advantaged communities. Under current practice, schools with disadvantaged students are almost always labeled “failing,” no matter how good the teachers are. Once value-added assessments are put in place, it will be possible to isolate the contributions made by the schools, teachers, and programs in raising achievement from external factors also affecting achievement and to act accordingly by following a model of continuous improvement.

6) Schooling options for parents and children who judge their school less than satisfactory. Schools must know that, if they are not successful, parents have alternatives for their children. Therefore, the finance system should also support charter and other choice schools.

7) Reasonable funding levels based on the needs of particular student enrollments and other factors outside of district control, but also discretion by local district taxpayers to augment the funding of their schools. Base funding would adjust for district poverty and external labor-market factors. Supplementation should incorporate “equalization” funds by the state to recognize differences in the ability of districts to raise funds locally when levying the same tax rate, but would permit parents and taxpayers to express directly their satisfaction with educational plans and policies.

8) Transparency incorporating value-added measures. Parents, taxpayers, and other stakeholders can then readily gauge how good a job the schools are doing.

9) A commitment to evaluating school and programmatic effectiveness. Expensive new strategies, such as large-scale class-size reduction programs, should be implemented only if they also provide for regular, independent evaluations to determine their effectiveness. Unsuccessful programs should not be allowed to continue and proliferate year after year just because they have strong sponsors.

The path to such reform will not be an easy one. While elements such as state standards, accountability measures, and value-added measures either are not controversial or are gaining acceptance, other important components, especially performance-based pay and increased choice options, are opposed by powerful forces with vested interests in the current system. Most powerful are the politically connected teachers unions. They, for example, clashed with Washington, D.C., schools chancellor Michelle Rhee over a proposal to couple higher pay with greater risk of termination because of ineffectiveness. The unions vigorously opposed those efforts, leading Rhee to move instead to improve the teaching force by terminating unqualified teachers. Unless this system is changed, it seems unlikely that outcomes will measurably improve in the district, already one of the highest funded and worst performing in the country.

The responsibility to enact and implement performance-based funding systems will fall primarily on the political branches of government, the state legislatures and governors. Although judicial remedies have played a significant role in school finance in the past, that era is drawing to a close. Beginning in the early 1970s, advocacy groups, frustrated with legislative efforts, began turning to the courts, initially to seek more equity in the allocation of education funds and later to seek vastly increased appropriations from state legislatures through “educational adequacy” lawsuits based on vaguely worded state constitutional provisions. A significant number of state courts responded positively to plaintiffs’ pleas and ordered unprecedented increases in K—12 funding in their states. Unfortunately, basic problems in the underlying systems of delivering education services were often ignored. In this sense, the courts mirrored what had been going on in the state legislatures, and the results were, not surprisingly, much the same: large amounts of money expended, but little or no improvement in student outcomes. An analysis in our recently published book examines the NAEP test-score trends in the four states that have implemented court remedies the longest, and demonstrates that, despite spending increases amounting to billions of dollars, the achievement patterns in three of them—Wyoming, New Jersey, and Kentucky—are largely unchanged from what they were in the early 1990s, before the court-ordered remedies commenced. Only in Massachusetts, where much deeper and broader reforms were instituted, has there been some improvement, although even there the state’s black students have not benefited from the remedy.

Perhaps due in part to this track record, the courts have begun to step back, opting instead to leave decisions regarding education policy and appropriations in the hands of the political branches of government, where they have traditionally resided. In the last five years, court decisions in approximately 15 states have disposed of educational adequacy cases, and, with one or two minor exceptions, the courts have either dismissed the cases or granted minimal relief. While this could change in a number of cases still pending, we believe the likelihood of significant court-ordered remedies in the foreseeable future is small.

Performance-based funding is not by itself a panacea that will solve all problems of substandard achievement or eliminate the achievement gap. Many of the problems that plague American education are beyond the control of the schools and will have to be addressed by other means. Performance-based funding will, however, put the nation’s schools back on the right track, help to raise the achievement of all students significantly, and once again make our students competitive on the world stage.

President Obama has called for increased funding to support NCLB, and Congress has provided substantial stimulus money for schools. A wise use of that money would be to underwrite transition costs in states moving to implement a performance-based funding system. For example, support for the improvement of student testing, for the development of improved databases and value-added measures, and for initial payments of expanded salaries under performance-based pay could provide important incentives for the states to move toward more logical and more effective funding systems.

Michael Rebell: The basic premise of the book and essay by Eric Hanushek and Al Lindseth—and of the question posed by Education Next—is that although “massive” amounts of money have been spent on education over the past 40 years, the results have been meager. Hanushek and Lindseth claim that states in which courts have ordered “extraordinary spending increases,” or at least the select few they have studied, have shown no improvement in student test scores. They then argue that certain “performance-based” accountability mechanisms that they recommend, rather than increased funding, should be the focus of future efforts.

I strongly dispute these premises, and I doubt that the reforms that Hanushek and Lindseth recommend are feasible, or that if enacted, they would constitute the panacea for the nation’s education ills that they imply. Extensive inequities in education funding, by which students with the greatest needs receive the fewest funds, still prevail in many parts of the United States; for that reason, state courts continue to have a critical role in ensuring meaningful educational opportunities for all children. The evidence strongly indicates that money well spent does make a significant difference in student achievement, and as Education Sector’s Kevin Carey has noted in reviewing one of Mr. Hanushek’s books: “There is little evidence that starving schools of needed funds is a catalyst for innovation, or that well-funded schools are more likely than others to be inefficient.” Moreover, although I agree that additional accountability measures are needed, continued involvement of the state courts, working in concert with the executive and legislative branches in a new, functional separation-of-powers mode, is essential for holding all parties accountable and for attaining the nation’s education goals.

Let me first put the spending issues into perspective. Hanushek and Lindseth claim that per-pupil spending in the U.S. has quadrupled since 1960. This is a gross exaggeration. According to recent analyses by Economic Policy Institute research associate Richard Rothstein, the cost of school services, when adjusted by the consumer price index, increased by 157 percent from 1967 to 2005, but when adjusted by the more relevant net services index (which omits shelter rent and medical care. the increase was only 92 percent. Moreover, these general statistics mask the fact that much of this increase has gone to special education, a sector that has dramatically expanded and substantially improved the lives of millions of students with disabilities over this time period. According to Rothstein, from 1967 to 2005 the share of educational expenditures going to regular education dropped from 80 to 55 percent and the share going to special education increased from 4 to 21 percent.

Second, for the past two decades, the United States has been committed to the historically unprecedented mission of simultaneously promoting excellence and equity in education. The standards-based reform movement seeks both to equip all of our high-school graduates to compete in the global marketplace and to narrow the achievement gap between our advantaged and disadvantaged student populations. Obviously, attaining these critical goals will require substantial resource infusions, especially for the high-need schools that historically have been treated inequitably by state education finance systems. Thus far, neither Congress, which has not even come close to fully funding the No Child Left Behind Act, nor most states, which have raised their academic standards but not their funding levels to a commensurate degree, have stepped up to the plate.

Third, Hanushek and Lindseth assert that “the United States already spends more on K—12 education than all but a few countries.” Although the U.S. is fourth among the 30 industrialized democracies that comprise the OECD (Organisation for Economic Co-operation and Development) in per-pupil spending on K—12, it is in the middle of the pack (13) in education spending as a percentage of GDP. Moreover, since, comparatively speaking, the U.S. starves health care, economic security, housing, and other areas of social welfare provision, the schools must bear an enormous burden in overcoming the impact of concentrated poverty for the poor and minority children they are committed to educating to high levels. In 2005, the childhood poverty rate in the U.S. was 21.9 percent, the highest, with the exception of Mexico, of the 24 OECD countries listed, and far higher than the 3 percent childhood poverty rate of countries like Denmark and Finland.

Given the extent of poverty in our society and the heavy burden that has been placed on the schools to alleviate its impact, it is astounding how much educational progress has been made. For example, from 1990 to 2007, black students’ scale scores increased 34 points on the NAEP 4th-grade mathematics tests (compared with a 28-point increase for whites), and the black-white achievement gap declined from 32 to 26 points during this period. Nevertheless, even greater progress can and should be made. I doubt, however, that “performance-based funding,” the solution Hanushek and Lindseth offer, will prove to be the silver bullet that will “help to raise the achievement of all students significantly, and once again make the nation’s students competitive on the world stage.”

Hanushek and Lindseth announce their performance-based funding prescriptions as if they will instantly solve the nation’s educational ills. But test-based outcomes, merit pay for teachers, rewards and sanctions, and voucher and charter alternatives have been part of the reform agenda of most states for years. Studies of each of these approaches have generally shown mixed results, and there is no strong empirical basis for dramatically expanding their use. Hanushek and Lindseth have an answer to this criticism: This is not a menu of options that can be ordered à la carte, they say. These components interlock, and they must be implemented as an “integrated system.”

Leaving aside the objections I have to many aspects of their program, full implementation of their “integrated systems approach” is clearly a pipe dream. In a democratic polity, no single reform approach can ever be fully put into effect, much less maintained, in its pure form. Policymaking for public education in a democracy inevitably is shaped by politics, and any reform proposal will inexorably be subject to compromise and modification. Although there was an unprecedented degree of bipartisan support for passage of the No Child Left Behind law in 2001, for example, that support came at a high price. As Education Next editors Rick Hess and Chester Finn recently observed, NCLB is a “Christmas tree of programs, incentives, and interventions that are more an assemblage of reform ideas than a coherent scheme. NCLB’s remedy provisions bear all the marks of concessions to various ideologies, advocates, and interest groups, with scant attention paid to how they fit together, the resources or authority they require, or whether they could be sensibly deployed through the available machinery.”

How to forge a better package of education reforms out of the positive aspects of NCLB is the main education policy challenge for the Obama administration, and how to make standards-based reform really work is the parallel problem that state education policymakers need to face. Hanushek and Lindseth’s performance-based funding proposal adds little of real value to this equation. However, the state courts’ wide experience in recent decades with fiscal equity and education adequacy litigations, which these authors roundly criticize, does provide significant possibilities for developing productive policy compromises and significantly advancing prospects for meaningful education reform.

Since 1973, when the U.S. Supreme Court held that education was not a “fundamental interest” under the federal constitution, education advocates, frustrated by continuing inequities in the funding of public education, have turned to the state courts. As Hanushek and Lindseth acknowledge, a “significant number of state courts responded positively to plaintiffs’ pleas.” In fact, during this era, cases have been filed in 45 of the 50 states, and plaintiffs have won more than 60 percent of them; since 1989, when the legal emphasis shifted from “equity” cases that seek equal funding levels for all students to “adequacy” cases that look to provide all students a basic quality education consistent with state standards, plaintiffs have prevailed in two-thirds of the final high-state-court decisions.

Hanushek and Lindseth claim that the courts have begun to step back from their support of constitutional rights in this area. But, in fact, there has been no diminution in the willingness of state supreme courts to issue strong rulings on students’ basic constitutional right to an adequate education. What has changed in recent years is that more cases have reached the remedy stage and more courts are experiencing difficulty in seeing constitutional compliance through to a successful conclusion. Some courts have cut short their remedial oversight out of frustration with the political complications and complexity of effecting meaningful change.

In other words, the adequacy movement has matured, and the courts are now grappling with many of the same implementation and compliance issues that have stymied governors and legislatures for years. The problems raised by judges in these remedial proceedings call for thoughtful responses and nuanced solutions, rather than the cavalier rejection of “judicial activism” that Hanushek and Lindseth and other opponents of adequacy articulate. (The title of a recent book that Hanushek edited and to which Lindseth contributed accuses judges of “harming our children.” This kind of hyperbole is clearly unwarranted.) As University of Wisconsin law professor Neil Komesar has insightfully pointed out, “All societal decision makers are highly imperfect.” Governors, state education departments, legislatures, and the federal Congress have been unable to solve the nation’s educational problems over the past half century, so why should anyone expect judicial interventions to achieve immediate, decisive results?

Where courts have persevered in their efforts, there have often been substantial improvements in student achievement. Hanushek and Lindseth allude to NAEP test-score trends in a few states with long-standing court orders that they claim have resulted in no improvement in student achievement in three out of four cases. The NAEP scores they focus on do not correspond in most of the cases to the relevant years in which the court orders were actually implemented; they ignore the fact that, as in Kentucky, initial increases in funding are sometimes followed by substantial decreases in later years; and their use of NAEP scores makes no sense in a state like New Jersey, where the court orders covered only a subset of the state’s students ( i.e., students in 31 poor urban school districts) and not the full statewide populations represented by NAEP scores. Recent, more finely tuned data for New Jersey, provided by Peg Goertz, a University of Pennsylvania researcher who has closely followed developments in the Garden State, indicate that from 1999 to 2007 substantial gains were made in the Abbott districts, which were the focus of the judicial remedies. For example, in 4th-grade mathematics, the achievement gaps between the Abbott districts and the rest of the state were cut by more than one-third. Similarly, Kentucky, which was near the bottom of the national rankings in virtually all performance indexes before its 1989 court decision, now ranks above the national averages in reading and science and almost at the national average in math.

Despite these gains, to fully meet our nation’s challenging goals for excellence and equity in our public school systems, clearly more needs to be done. What is required is a concerted effort by all three branches of government to bring their relative functional strengths to bear on ensuring constitutional compliance and solving the nation’s educational ills. In a forthcoming book, I propose a “successful remedies model” that is based on the extensive empirical experience that dozens of courts have had in dealing with legislatures, governors, and state education departments in crafting remedies. It is a process approach that is compatible with Hanushek and Lindseth’s performance-funding focus or any other policy perspective, or as is more likely, whatever mix of policies a state’s elected representatives choose to endorse. This process seeks to ensure that, whatever reform path state policymakers pursue, the compromise package they assemble is cohesive, adequately funded, and consistently implemented; moreover, the state should be committed to seeing the reforms through over time so that lasting results can be achieved. “Success” in implementing standards-based reforms under this model is defined not in terms of test scores in a limited number of subject areas, but broadly, in terms of providing all students a sound basic education on a sustained basis.

To achieve such success requires effective, programs and ongoing “colloquy” among the three branches of government. The courts’ role in this process is to outline in general, principled terms the expectation that the legislative and executive branches will develop challenging standards, fair and adequate funding systems, and effective programs and accountability measures, but to leave to the programs and the political branches the full responsibility for actually formulating these policies. Legislatures should make basic educational policy decisions; state education departments and local school districts should determine how best to implement educational reforms. Once the state has decided on its policy position, however, a judicial presence should be maintained to ensure that the chosen policy is fully funded, is implemented in a coherent manner, and results in substantially improved student performance, as measured by validated assessments of academic achievement and of students’ ability to function as capable citizens and workers.

Since significant compliance cannot be achieved overnight, in most cases courts will need to maintain nominal jurisdiction for a multiyear period, probably 10 to 15 years. The mere fact that judicial oversight remains in place can ensure continued adherence to implementation of stated policy goals, and actual interventions should be rare, especially if it is clearly understood that all the courts would be enforcing are the state’s own policy goals. A judicial presence is especially important to ensure that the reform process—and reasonable funding levels—are maintained in times of economic stress or recession like the present, where children’s needs and constitutional values are often given short shrift.

In short, then, my answer to the question posed by the editors of Education Next is that what is most likely to fulfill the promise of improved student outcomes in the future is not any silver bullet remedy, but rather a pragmatic process that allows courts, legislatures, state education departments, and school districts to work collaboratively to implement meaningful reforms on a sustained basis.

Hanushek and Lindseth: Notwithstanding his obfuscation, Michael Rebell’s solution is essentially more of the same. Beginning by misstating spending increases (based on incorrect data and flawed adjustments) and ignoring pertinent performance data, he rewrites the constitution of every state to give judges the major policy-setting role in a “new, functional separation of powers mode.” He further recommends that judges and legislators be guided in their efforts by a “successful remedies model” to be drawn from previous adequacy litigation—perhaps tempting if such “successful” models actually existed. Quite surprisingly, he cites New Jersey’s tortured 35-year-old Abbott litigation as an example of “success,” but neglects to mention that the state’s black students, the principal beneficiaries of the remedy, are still scoring at about the same relative levels on the NAEP tests as in 1992. In Kentucky, he relies on data for all students, which mask the fact that black students, the state’s principal minority group, have regressed compared to their peers nationally during the remedial period.

Our solution may not be a “silver bullet” for everything that ails American education, but it surely presents a better chance for our children than continuing the demonstrably failed practices of the past. In the end, Rebell basically concludes that political forces are too strong to bring about the fundamental changes we recommend, so we should just continue plowing more money into the current system. If we do, no one should be surprised in 2040 when our students are still performing, as they are now, at 1970 levels.

Rebell: If I didn’t know that Rick Hanushek was an outstanding economist and that Al Lindseth was a master litigator, I would think from some of the provocative phrases they use in their writings that they were sensationalist journalists, looking to attract readers with shocking but misleading headlines and catchphrases. They claim that I am proposing to “rewrit[e] the constitution of every state to give judges the major policy-setting role.” A detailed examination of the positions they actually take in their writings, and especially in their recent book upon which this Forum is based, indicates, however, that we agree that money—if well spent—does matter, that education finance cases have had a significant equalizing effect on state education funding formulas, and that court orders can “support legislators who want to address serious problems in education.”

The fact is that the unproven, business-model, and privatization practices they propose as education reforms have no chance of being adopted as an “integrated system,” especially in the present political climate. I would, therefore, ask Hanushek and Lindseth to stop tilting at windmills and to join with me in instituting a dialogue in major areas in which we do agree, like the fact that courts can and should hold states and school districts accountable for better performance, and that “school funding policies must recognize the underlying heterogeneity of students and their educational challenges and ensure that all schools have the means to succeed” (Hanushek and Lindseth, Schoolhouses, Courthouses, and Statehouses, page 218). That kind of conversation might help to promote real changes that might provide truly meaningful educational opportunities to all of our children.

For as long as there have been struggling schools in America’s cities, there have been efforts to turn them around. The lure of dramatic improvement runs through Morgan Freeman’s big-screen portrayal of bat-wielding principal Joe Clark, philanthropic initiatives like the Gates Foundation’s “small schools” project, and No Child Left Behind (NCLB)’s restructuring mandate. The Obama administration hopes to extend this thread even further, making school turnarounds a top priority.

But overall, school turnaround efforts have consistently fallen far short of hopes and expectations. Quite simply, turnarounds are not a scalable strategy for fixing America’s troubled urban school systems.

Fortunately, findings from two generations of school improvement efforts, lessons from similar work in other industries, and a budding practice among reform-minded superintendents are pointing to a promising alternative. When conscientiously applied strategies fail to drastically improve America’s lowest-performing schools, we need to close them.

Done right, not only will this strategy help the students assigned to these failing schools, it will also have a cascading effect on other policies and practices, ultimately helping to bring about healthy systems of urban public schools.

A Body at Rest Stays at Rest

Looking back on the history of school turnaround efforts, the first and most important lesson is the “Law of Incessant Inertia.” Once persistently low performing, the majority of schools will remain low performing despite being acted upon in innumerable ways.

Examples abound: In the first year of California’s Academic Performance Index, the state targeted its lowest-performing 20 percent of schools for intervention. After three years, only 11 percent of the elementary schools in this category (109 of 968) were able to make “exemplary progress.” Only 1 of the 394 middle and high schools in this category reached this mark. Just one-quarter of the schools were even able to accomplish a lesser goal: meeting schoolwide and subgroup growth targets each year.

In 2008, 52 Ohio schools were forced to restructure because of persistent failure. Even after several years of significant attention, fewer than one in three had been able to reach established academic goals, and less than half showed any student performance gains. The Columbus Dispatch concluded, “Few of them have improved significantly even after years of effort and millions in tax dollars.”

These state anecdotes align with national data on schools undergoing NCLB-mandated restructuring, the law’s most serious intervention, which follows five or more years of failing to meet minimum achievement targets. Of the schools required to restructure in 2004–05, only 19 percent were able to exit improvement status two years later.

A 2008 Center on Education Policy (CEP) study investigated the results of restructuring in five states. In California, Maryland, and Ohio, only 14, 12, and 9 percent of schools in restructuring, respectively, made adequate yearly progress (AYP) as defined by NCLB the following year. And we must consider carefully whether merely making AYP should constitute success at all: in California, for example, a school can meet its performance target if slightly more than one-third of its students reach proficiency in English language arts and math. Though the CEP study found that improvement rates in Michigan and Georgia were considerably higher, Michigan changed its accountability system during this period, and both states set their AYP bars especially low.

Though alarming, the poor record for school turnarounds in recent years should come as no surprise. A study published in 2005 by the Education Commission of the States (ECS) on state takeovers of schools and districts noted that the takeovers “have yet to produce dramatic consistent increases in student performance,” and that the impact on learning “falls short of expectations.”

Reflecting on the wide array of efforts to improve failing schools, one set of analysts concluded, “Turnaround efforts have for the most part resulted in only marginal improvements…. Promising practices have failed to work at scale when imported to troubled schools.”

Like Finding the Cure for Cancer

The second important lesson is the “Law of Ongoing Ignorance.” Despite years of experience and great expenditures of time, money, and energy, we still lack basic information about which tactics will make a struggling school excellent. A review published in January 2003 by the Thomas B. Fordham Foundation of more than 100 books, articles, and briefs on turnaround efforts concluded, “There is, at present, no strong evidence that any particular intervention type works most of the time or in most places.”

An EdSource study that sought to compare California’s low-performing schools that failed to make progress to its low-performing schools that did improve came to a confounding conclusion: clear differences avoided detection. Comparing the two groups, the authors noted, “These were schools in the same cities and districts, often serving children from the same backgrounds. Some of them also adopted the same curriculum programs, had teachers with similar backgrounds, and had similar opportunities for professional development.”

Maryland’s veteran state superintendent of schools, Nancy Grasmick, agrees: “Very little research exists on how to bring about real sea change in schools…. Clearly, there’s no infallible strategy or even sequence of them.” Responding to the growing number of failing Baltimore schools requiring state-approved improvement plans, she said, “No one has the answer. It’s like finding the cure for cancer.”

Researchers have openly lamented the lack of reliable information pointing to or explaining successful improvement efforts, describing the literature as “sparse” and “scarce.” Those attempting to help others fix broken schools have typically resorted to identifying activities in improved schools, such as bolstering leadership and collecting data.

However, this case-study style of analysis is deeply flawed. As the U.S. Department of Education’s Institute of Education Sciences (IES) has noted, studies “that look back at factors that may have contributed to [a] school’s success” are “particularly weak in determining causal validity for several reasons, including the fact that there is no way to be confident that the features common to successful turnaround schools are not also common to schools that fail.”

Researchers have noted that the Department of Education has signaled its own ignorance about what to do about the nation’s very worst schools. One study reported, “The NCLB law does not specify any additional actions for schools that remain in the implementation phase of restructuring for more than one year, and [the Department] has offered little guidance on what to do about persistently struggling schools.” Indeed, the IES publication, “Turning Around Chronically Low-Performing Schools” practice guide, purportedly a resource for states and districts, concedes, “All recommendations had to rely on low levels of evidence,” because it could not identify any rigorous studies finding that “specific turnaround practices produce significantly better academic outcomes.”

Still in Its Infancy?

The prevailing view is that we must keep looking for turnaround solutions. Observers have written, “Turnaround at scale is still in its infancy,” and “In education, turnarounds have been tried rarely” (see “The Big U-Turn,” features, Winter 2009). But, in fact, the number and scope of fix-it efforts have been extensive to say the least.

Long before NCLB required interventions in the lowest-performing schools, states had undertaken significant activity. In 1989 New Jersey took over Jersey City Public Schools; in 1995 it took over Newark Public Schools. In 1993 California took control of the Compton Unified School District. In 1995 Ohio took over the Cleveland Metropolitan School District. Between 1993 and 1997 states required the reconstitution of failing schools in Denver, Chicago, New York City, and Houston. In 2000 Alabama took over a number of schools across the state, and Maryland seized control of three schools in Baltimore.

Since NCLB, interventions in struggling schools have only grown in number and intensity. In the 2006–07 school year, more than 750 schools in “corrective action,” the NCLB phase preceding restructuring, implemented a new research-based curriculum, more than 700 used an outside expert to advise the school, nearly 400 restructured the internal organization of the school, and more than 200 extended the school day or year. Importantly, more than 300 replaced staff members or the principal, among the toughest traditional interventions possible.

Occasionally a program will report encouraging success rates. The University of Virginia School Turnaround Specialist Program asserts that about half of its targeted schools have either made AYP or reduced math and reading failure rates by at least 5 percent. Though this might be better than would otherwise be expected, the threshold for success is remarkably low. It is also unknown whether such progress can be sustained. This matter is particularly important, given that some point to charter management organizations Green Dot and Mastery as turnaround success stories even though each has a very short turnaround résumé, in both numbers of schools and years of experience.

Many schools that reach NCLB’s restructuring phase, rather than implementing one of the law’s stated interventions (close and reopen as a charter school, replace staff, turn the school over to the state, or contract with an outside entity), choose the “other” option, under which they have considerable flexibility to design an improvement strategy of their own (see “Easy Way Out,” forum, Winter 2007). Some call this a “loophole” for avoiding tough action.

Yet even under the maligned “other” option, states and districts have tried an astonishing array of improvement strategies, including different types of school-level needs assessments, surveys of school staff, conferences, professional development, turnaround specialists, school improvement committees, training sessions, principal mentors, teacher coaches, leadership facilitators, instructional trainers, subject-matter experts, audits, summer residential academies, student tutoring, research-based reform models, reconfigured grade spans, alternative governance models, new curricula, improved use of data, and turning over operation of some schools to outside organizations.

It’s simply impossible to make the case that turnaround efforts haven’t been tried or given a chance to work.

A Better Mousetrap?

Despite this evidence, some continue to advocate for improved turnaround efforts. Nancy Grasmick supports recognizing turnarounds as a unique discipline. Frederick Hess and Thomas Gift have argued for developing school restructuring leaders; Bryan Hassel and Emily Ayscue Hassel have recommended that states and districts “fuel the pipeline” of untraditional turnaround specialists. NewSchools Venture Fund, the Education Commission of the States, and the research firm Mass Insight have offered related turnaround strategies.

And the Obama administration too has bought into the notion that turnarounds are the key to improving urban districts. Education secretary Arne Duncan has said that if the nation could turn around 1,000 schools annually for five years, “We could really move the needle, lift the bottom and change the lives of tens of millions of underserved children.” In the administration’s 2009 stimulus legislation, $3 billion in new funds were appropriated for School Improvement Grants, which aid schools in NCLB improvement status. The administration requested an additional $1.5 billion for this program in the 2010 budget. This is all on top of the numerous streams of existing federal funds that can be—and have been—used to turn around failing schools.

The dissonance is deafening. The history of urban education tells us emphatically that turnarounds are not a reliable strategy for improving our very worst schools. So why does there remain a stubborn insistence on preserving fix-it efforts?

The most common, but also the most deeply flawed, justification is that there are high-performing schools in American cities. That is, some fix-it proponents point to unarguably successful urban schools and then infer that scalable turnaround strategies are within reach. In fact, it has become fashionable among turnaround advocates to repeat philosopher Immanuel Kant’s adage that “the actual proves the possible.”

But as a Thomas B. Fordham Foundation study noted, “Much is known about how effective schools work, but it is far less clear how to move an ineffective school from failure to success…. Being a high-performing school and becoming a high-performing school are very different challenges.”

In fact, America’s most-famous superior urban schools are virtually always new starts rather than schools that were previously underperforming. Probably the most convincing argument for the fundamental difference between start-ups and turnarounds comes from those actually running high-performing high-poverty urban schools (see sidebar). Groups like KIPP (Knowledge Is Power Program) and Achievement First open new schools; as a rule they don’t reform failing schools. KIPP’s lone foray into turnarounds closed after only two years, and the organization abandoned further turnaround initiatives. Said KIPP’s spokesman, “Our core competency is starting and running new schools.”

A 2006 NewSchools Venture Fund study confirmed a widespread aversion to takeover-and-turnaround strategies among successful school operators. Only 4 of 36 organizations interviewed expressed interest in restructuring existing schools. Remarkably, rather than trusting successful school operators’ track records and informed opinion that start-ups are the way to go, Secretary Duncan urged them to get into the turnaround business during a speech at the 2009 National Charter Schools Conference.

The findings above deserve repeating: Fix-it efforts at the worst schools have consistently failed to generate significant improvement. Our knowledge base about improving failing schools is still staggeringly small. And exceptional urban schools are nearly always start-ups or consistently excellent schools, not drastically improved once-failing schools.

So when considering turnaround efforts we should stop repeating, “The actual proves the possible” and bear in mind a different Kant adage: “Ought implies can.”

If we are going to tell states and districts that they must fix all of their failing schools, or if we are to consider it a moral obligation to radically improve such schools, we should be certain that this endeavor is possible. But there is no reason to believe it is.

Turnarounds Elsewhere

Education leaders seem to believe that, outside of the world of schools, persistent failures are easily fixed. Far from it. The limited success of turnarounds is a common theme in other fields. Writing in Public Money & Management, researchers familiar with the true private-sector track record offered a word of caution: “There is a risk that politicians, government officials, and others, newly enamored of the language of failure and turnaround and inadequately informed of the empirical evidence and practical experience in the for-profit sector…will have unrealistic expectations of the transformative power of the turnaround process.”

Hess and Gift reviewed the success rates of Total Quality Management (TQM) and Business Process Reengineering (BPR), the two most common approaches to organizational reform in the private sector. The literature suggests that both have failed to generate the desired results two-thirds of the time or more. They concluded, “The hope that we can systematically turn around all troubled schools—or even a majority of them—is at odds with much of what we know from similar efforts in the private sector.”

Many have noted that flexibility and dynamism are part of the genetic code of private business, so we should expect these organizations to be more receptive to the massive changes required by a turnaround process than institutions set in what Hess calls the “political, regulatory, and contractual morass of K–12 schooling.” Accordingly, school turnarounds should be more difficult to achieve. Indeed, a consultant with the Bridgespan Group reported, “Turnarounds in the public education space are far harder than any turnaround I’ve ever seen in the for-profit space.”

Building a Healthy Education Industry

We shouldn’t be surprised then that turnarounds in urban education have largely failed. The surprise and shame is that urban public education, unlike nearly every other industry, profession, and field, has never developed a sensible solution to its continuous failures. After undergoing improvement efforts, a struggling private firm that continues to lose money will close, get taken over, or go bankrupt. Unfit elected officials are voted out of office. The worst lawyers can be disbarred, and the most negligent doctors can lose their licenses. Urban school districts, at long last, need an equivalent.

The beginning of the solution is establishing a clear process for closing schools. The simplest and best way to put this into operation is the charter model. Each school, in conjunction with the state or district, would develop a five-year contract with performance measures. Consistent failure to meet goals in key areas would result in closure. Alternatively, the state could decide that districts only have one option—not five—for schools reaching NCLB-mandated restructuring: closure.

This would have three benefits. First, children would no longer be subjected to schools with long track records of failure and high probabilities of continued failure.

Second, the fear of closure might generate improvement in some low-performing schools. Failure in public education has had fewer consequences (for adults) than in other fields, a fact that might contribute to the persistent struggles of some schools. We should have limited expectations in this regard, however. Even in the private sector, where the consequences for poor performance are significant, some low-performing entities never become successful.

Third, and by far the most important and least appreciated factor, closures make room for replacements, which have a transformative positive impact on the health of a field. When a firm folds due to poor performance, the slack is taken up by the expansion of successful existing firms—meaning that those excelling have the opportunity to do more—or by new firms. New entrants not only fill gaps, they have a tendency to better reflect current market conditions. They are also far likelier to introduce innovations: Google, Facebook, and Twitter were not products of long-standing firms. Certainly not all new starts will excel, not in education, not in any field. But when provided the right characteristics and environment, their potential is vast.

The churn caused by closures isn’t something to be feared; on the contrary, it’s a familiar prerequisite for industry health. Richard Foster and Sarah Kaplan’s brilliant 2001 book Creative Destruction catalogued the ubiquity of turnover in thriving industries, including the eventual loss of once-dominant players. Churn generates new ideas, ensures responsiveness, facilitates needed change, and empowers the best to do more.

These principles can be translated easily into urban public education via tools already at our fingertips thanks to chartering: start-ups, replications, and expansions. Chartering has enabled new school starts for nearly 20 years and school replications and expansions for a decade. Chartering has demonstrated clearly that the ingredients of healthy, orderly churn can be brought to bear on public education.

A small number of progressive leaders of major urban school systems are using school closure and replacement to transform their long-broken districts: Under Chancellor Joel Klein, New York City has closed nearly 100 traditional public schools and opened more than 300 new schools. In 2004, Chicago announced the Renaissance 2010 project, which is built around closing chronically failing schools and opening 100 new public schools by the end of the decade.

Numerous other big-city districts are in the process of closing troubled schools, including Detroit, Philadelphia, and Washington, D.C. In Baltimore, under schools CEO Andrés Alonso, reform’s guiding principles include “Closing schools that don’t work for our kids,” “Creating new options that have strong chances of success,” and “Expanding some programs that are already proving effective.”

Equally encouraging, there are indications that these ideas, which once would have been considered heretical, are being embraced by education’s cognoscenti. A group of leading reformers, the Coalition for Student Achievement, published a document in April 2009 that offered ideas for the best use of the federal government’s $100 billion in stimulus funding. They recommended that each state develop a mechanism to “close its lowest performing five percent of schools and replace them with higher-performing, new schools including public charter schools.”

A generation ago, few would have believed that such a fundamental overhaul of urban districts was on the horizon, much less that perennial underperformers New York City, Chicago, and Baltimore would be at the front of the pack with much of the education establishment and reform community in tow. But, consciously or not, these cities have begun internalizing the lessons of healthy industries and the chartering mechanism, which, if vigorously applied to urban schooling, have extraordinary potential. Best of all, these districts and outstanding charter leaders like KIPP Houston (with 15 schools already and dozens more planned) and Green Dot (which opened 5 new schools surrounding one of Los Angeles’s worst high schools) are showing that the formula boils down to four simple but eminently sensible steps: close failing schools, open new schools, replicate great schools, repeat.

Today’s fixation with fix-it efforts is misguided. Turnarounds have consistently shown themselves to be ineffective—truly an unscalable strategy for improving urban districts—and our relentless preoccupation with improving the worst schools actually inhibits the development of a healthy urban public-education industry.

Those hesitant about replacing turnarounds with closures should simply remember that a failed business doesn’t indict capitalism and an unseated incumbent doesn’t indict democracy. Though temporarily painful, both are essential mechanisms for maintaining long-term systemwide quality, responsiveness, and innovation. Closing America’s worst urban schools doesn’t indict public education nor does it suggest a lack of commitment to disadvantaged students. On the contrary, it reflects our insistence on finally taking the steps necessary to build city school systems that work for the boys and girls most in need.

Andy Smarick is a distinguished visiting fellow at the Thomas B. Fordham Institute and adjunct fellow at the American Enterprise Institute.

Michelle Rhee’s senior staff meeting has all the ceremony of lunchtime in the teachers’ lounge. News is exchanged. Ideas tumble around. Rhee sits at the head of the table but doesn’t run the meeting or even take the conversational lead. Staffers talk over her as often as she talks over them. If consensus is the goal, the ball is far upfield.

But then, Rhee wades in with, “Here’s what I think,” or “What I don’t want,” or “This is crap,” or “I want someone to figure this out,” or “I’m gonna tell you what we’re gonna do; we can talk about how we’re gonna do it.” And that is that. Next order of business, please.

Rhee’s style—as steely as the sound of her peekaboo high heels on a linoleum-tile hallway—has angered much of Washington, D.C., and baffled the rest since she arrived as schools chancellor in June 2007. But it is also helping her gain control of a school system that has defied management for decades: that hasn’t kept records, patched windows, met budgets, delivered books, returned phone calls, followed court orders, checked teachers’ credentials, or, for years on end, opened school on schedule in the fall.

When I asked Rhee to name her most significant achievement in her two years in Washington, her answer suggested that any progress is, so far, only incremental. “We have begun—begun—begun—to establish a culture of accountability,” she said, with a long pause between each “begun.” A teacher had recently e-mailed her about a personnel matter, she went on, and was thrilled that Rhee had replied. “It’s sorta sad because the expectations are so low. The fact that you just get a response is celebrated,” she said.

Rhee tells parents and taxpayers that they should judge her on “student performance.” Are test scores rising? Are students graduating? So far, there’s some evidence that they are, although some teachers and parents say that even that evidence is suspect.

But not much learning gets done without institutional support, and for decades in Washington, not much has. When I asked Kenneth Wong, director of Brown University’s urban-education policy program, on what measures Rhee should be judged, he answered with a long list. It included how well the schools work with other city agencies (to get sidewalks plowed in the winter, for example), how many and which colleges new teachers come from (the wider the net, the better), how quickly managers return phone calls, and whether teacher absenteeism is down. Only at the end of the list did he get to student performance. “The other stuff are the necessary conditions to get to student achievement,” he said.

That’s not particularly glamorous for a national media darling who has been celebrated on magazine covers, on Capitol Hill, and by the president, but it is a start.

Rhee tells parents and taxpayers to judge her on “student performance.”

Rock Bottom?

It’s not news that Washington’s schools are among the most woeful in the country, but even a cynic has to gasp. The mismanagement is legendary: consider the 5 million personnel records Rhee says she found piled on a storeroom floor when she took office. Marc Borbely, a former teacher, filed a Freedom of Information Act request in 2004 to find out how many work orders were outstanding at the central maintenance office. The answer: 25,000.

Teachers complained of out-of-control students: The city’s Ballou High School was closed for a 35-day cleanup after students stole chemistry-lab thermometers and scattered the mercury around hallways. In most school districts, mercury thermometers had been replaced years earlier.

The system churned through six superintendents in 10 years, usually after brutal head butting with the city council and community activists. That made Washington the La Brea Tar Pits of strategic plans: Each one sank into oblivion as its drafters moved on. The school funding formula changed four times under as many superintendents.

Academic measures were miserable. The 2007 National Assessment of Educational Progress (NAEP), administered before Rhee’s arrival but announced five months after her term began, found that 61 percent of the city’s 4th graders had below-basic reading skills, which means they could barely read. Just 8 percent of its 8th graders were proficient—that is, at grade level—or above in math.

Scores on the district’s own tests for the 2006–07 school year, the last before Rhee’s arrival, were higher but still dismal. Just 38 percent of elementary-school children were at grade level or above in reading, and 27 percent of high schoolers were at grade level or above in math. Districtwide, fewer than 30 percent of African American students were reading at grade level, compared to 87 percent of whites, a 57-percentage-point gap.

Rhee arrived to find that all 10 of Washington’s comprehensive high schools had failed to meet federal No Child Left Behind (NCLB) adequate yearly progress goals and that 48 of its 67 elementary schools were in some level of NCLB-mandated corrective action. The high-school dropout rate hovered at about 50 percent, and just 9 percent of entering 9th graders ever graduated from college.

On the SAT—a test presumably only the most ambitious students take—43 percent of district students who took the exam in 2009 scored 390 or below on the 800-point math test, which awards 200 points just for showing up. African Americans citywide averaged 773 on the 1600-point reading and math tests combined, or about 400 points less than they’d need for admission to the nearby University of Maryland.

Community pressure to “do something” about the schools’ performance had never materialized, though. Political leaders had seen no upside to taking on a school system that employs thousands of African Americans in a city where African Americans account for a majority of the population, the voter rolls, the city council, local-government posts, and union leadership. And in the weary way that people get used to dysfunction, no one else complained. Rhee says she marvels that her decision to shut down 23 failing schools in her first year drew howls of protest, while keeping failing schools open doesn’t excite anyone.

The Money Question

Washington’s business community has fussed for years about the schools because they turn out so few employable graduates and at a huge cost. The Chamber of Commerce says that only one in four jobs in the city is held by a D.C. resident now, and that 44 percent of Washingtonians don’t have even a high-school diploma.

Education expenditures can swing wildly depending on how students are counted and what spending is included in the calculation. But the U.S. Census Bureau, in a survey of education finances released in July 2009, says Washington spent $14,324 per public-school student in the 2006–07 school year, or about $6,300 more than the national average. The only states to spend more were New Jersey and New York, which have vastly larger corporate tax bases and far more upper-income taxpayers. The U.S. Department of Education reports that the federal government pays 12 percent of Washington’s education budget, a percentage largely determined by the city’s high poverty rate. That puts it well below Louisiana and Mississippi, but well above the 9 percent national average for federal support.

A simpler way of looking at it: Washington has budgeted $760 million for its traditional public schools in the fiscal year beginning October 2010. Using Rhee’s enrollment estimate of 45,000, that works out to $16,800 per student. Using the city council’s estimate of 41,500 students, it’s $18,300.

As costs have risen, enrollment has plummeted (see Figure 1). Affluent or activist parents enroll their youngsters in three or four largely autonomous elementary schools in white neighborhoods, or move to private schools, charter schools, or the suburbs. Between 2004 and 2008, Washington’s traditional public schools lost 13,500 students, while its charters gained 10,200.

What may be Washington’s last hope of stopping the slide from dismal to disastrous rests on the reform course chosen by its mayor, Adrian Fenty, an African American Democrat who has staked his political career and considerable ego on his pledge to improve the schools. After his January 2007 inauguration, Fenty courted and then summoned Rhee to Washington through her mentor, New York schools chancellor Joel Klein, even though Rhee says she initially “was not blown away” by the mayor or the job. Fenty quickly pushed through legislation that abolished the disputatious school board, won Rhee the authority to fire hundreds of central-office workers, and “has not flinched once through any of this, never,” she says.

Rhee’s Roots

Rhee speaks often about her Teach For America (TFA) tour in a Baltimore classroom between 1992 and 1995: how she struggled the first year until pairing with another teacher to team-teach a class of 2nd and 3rd graders. But Rhee’s experience a few years later with The New Teacher Project (TNTP) is a better window on how she’s doing her job in Washington.

Political leaders have seen no upside to taking on a school system that employs thousands of African Americans in a city where they are a majority.

As Ariela Rozman, TNTP’s current CEO, tells it, superintendents had begun asking TFA founder Wendy Kopp for help attracting and training teachers like those Kopp was sending them. Rhee was finishing a graduate program at Harvard and had never had a management role at TFA, but Kopp tapped Rhee to head the teacher project as a spin-off in 1997. “The idea came from TFA clients, but Michelle brought the vision,” Rozman told me.

Rhee was a no-nonsense manager. She was so determined to fund The New Teacher Project out of the revenues it was generating through its training contracts with schools that she sorely underpaid her staff. For years, she resisted pressure even from Kopp to take foundation funding, said Kati Haycock, who is chair of the project’s board and president of the Education Trust. Even so, the project attracted a talented staff with high morale, little turnover, and fierce loyalty to Rhee. Richard Nyankori, who moved with Rhee to Washington from TNTP and now heads special education for the district, says Rhee teases him that he would throw himself under a bus for her, “and she’s right. I probably would.”

Rhee’s greatest success at The New Teacher Project may be how she left it. Start-ups frequently struggle when a strong-willed manager leaves: Staffers move on, backers temporize, and contracts slow as the new leader finds her footing. But Ariela Rozman says The New Teacher Project has grown since Rhee left, from 140 people and a $20 million budget to this year’s staff of 210 and budget of $32 million.

Kaya Henderson, who also moved to Washington with Rhee as her deputy chancellor, says The New Teacher Project’s management style moved with them. Policy differences are hashed out at the weekly senior staff meetings and at biweekly meetings of a strategy committee, which considers major initiatives. “We’re not going to leave the meeting until one group has convinced the other group. We all have to be good with the decision,” Henderson told me. Still, “part of being a good leader is knowing when to say ‘this is a good thing to do,’” a prerogative Rhee doesn’t shy from, Henderson added.

Rhee has pledged to stay to the end of a second Fenty term—January 2015, if he is reelected—and Henderson says “the rest of us are probably in it for the same.”

Bumpy Ride

Six weeks into the job, Rhee called her staff together with the message that “We are not here to do the bureaucracy better,” Nyankori says. Rhee told them that “that’s what all of our friends are doing in reform all around the country: They’re trying to make the trains stay on the track and go faster. We are here to derail those trains.”

If upheaval was the goal, Rhee has succeeded. Teachers say she has set black teachers against whites and young teachers against veterans with her controversial 2008 contract offer. Congressional Democrats worry that she has put them between a policy goal, school improvement, and their teachers-union allies. Education reformers are nervous that her outta-my-way approach will wound their movement if it backfires.

Almost everyone has a Rhee story. As when the chancellor closed those 23 schools and scheduled a community meeting at each one but on the same evening, so she couldn’t attend most of them. Or suggested the elected city council was irrelevant and resisted its invitations to testify. Or arrived for a meeting with the Chamber of Commerce board with—surprise!—a television news crew in tow. Chamber president Barbara Lang says Rhee never thanked the chamber for testifying in favor of Mayor Fenty’s takeover of the schools, legislation that will be pivotal to Rhee’s success.

Businesses, foundations, and civic groups that funded and ran after-school and enrichment programs were similarly dismissed. A Chamber of Commerce project that taught jobs skills to high schoolers was dropped. The World Bank had outfitted and staffed college-prep resource centers at some of the city’s toughest high schools. When Rhee put the outside groups on hold, the bank diverted its $1 million a year in youth programming to local nonprofits.

Parent groups that used to be solicited—even begged—to help make decisions about dress codes, building budgets and staffing, renovations and construction, and principal selection now find themselves shut out. “Parents feel pushed aside,” says Cathy Reilly, who started a parents’ group to exchange news about their kids’ high schools.

Rhee urges parents to e-mail her with questions, and she answers late into the night (she says she answered 99,000 e-mails her first year). But at the public meetings I attended last spring, Rhee sat alone at the front of the room, talked over parents, moved about with an ever-present photographer, and left immediately afterward in a chauffeured Chevy Tahoe.

Rhee and her loyalists say with jaw-dropping insouciance that none of that matters because, as she told me, she’s “doing what’s right for kids.”

“The conventional rules and the people who play by them don’t get much change,” says the Education Trust’s Haycock. “Hordes” of people come to their table when she and Rhee dine out together, Haycock adds, and “I have never heard anyone say anything except ‘keep on keeping on.’”

Rhee and her senior staff believe that the ed-reform stars are aligned as they never have been in Washington, and that they have the brains, focus, and work ethic to leap at the opportunity. In all of that, they’re probably right.

Rhee visits with first grader Sasha Simpson.

The Front Line

Rhee and her top aides don’t talk much about curriculum change; their focus is people. “Strong principals, strong teachers—that’s what turns schools around,” says Nyankori. “That’s why we feel so strongly about this union contract.”

The Washington Teachers Union and its parent American Federation of Teachers (AFT) feel just as strongly, of course, about a contract that undercuts such union cornerstones as tenure, seniority, and worker solidarity, and that would set a national precedent. Rhee’s proposal to pay six-figure salaries to teachers who agreed to link their paychecks to classroom outcomes: that’s the “green” option. Teachers who choose the “red” option (green, go; red, stop—get it?) would collect far-smaller pay increases, but would retain job security.

Rhee didn’t say how she would pay for the salary boosts, although she implied that foundations would pick up much of the tab. Meanwhile, foundation endowments have plunged and local tax revenues have shrunk since Rhee offered the plan in summer 2008.

AFT president Randi Weingarten, who has largely taken over the negotiations from the local union, insists that the teachers and Rhee “share the same goals, the issue in contract negotiations is how to get there.” She proposes rewarding teachers equally with school-based bonuses, a nonstarter with Rhee, who is zealous about getting rid of those she calls “bad teachers.” Stakes are so high for both sides that they appear to be working on a compromise that gives Rhee some, but by no means all of the staffing and firing flexibility she is after.

Still, Rhee has some tools that other school heads don’t have. Congress gave her the power to impose a teacher-evaluation system without negotiating its terms with the union. The new evaluations, set to begin in the 2009–10 school year, will include student test scores and five classroom observations of each teacher each year. Henderson, the deputy chancellor, has let the union know that the district will likely begin observing teachers by video, too.

And then there are some test-score gains, which Rhee is counting on to build public support for her plans and ease the doubts about her style. Two years after Rhee’s arrival, scores on district-administered tests are up: 49 percent of elementary school students were reading at grade level, a 21-percentage-point jump in two years, according to test results released in July 2009. Among secondary-school students, 40 percent were at grade level in math, up 13 points. Rising proficiency levels should win Rhee new clout in the city’s political circles, new respect among parents and civil groups, and more leverage to turn the troubled system around.

Rising proficiency levels should win Rhee new clout in the city’s political circles.

Taking Stock

Rhee’s other successes aren’t exactly the stuff of headlines. Erich Martel, who has taught social studies in the D.C. schools for 40 years, says teachers are doing more lesson prep and trying to make their classes more interesting. “There are teachers who need someone looking over their shoulder and they’re getting it,” he says.

Long-neglected school buildings are being renovated or rebuilt, which could make them more competitive with some better-housed charters. Spending on professional development has quadrupled. There are art and music classes in every school, the district says.

Rhee’s most important achievement might be in the management fixes most people can’t see. High-school transcripts, which the schools used to hold on to and sometimes alter to boost graduation rates, are being centralized and scrubbed (the audit found that one-third of students weren’t taking the classes they need to graduate). Nyankori says he has lured back 155 of the district’s 2,400 special-ed youngsters who are in private schools, at a yearly cost of $141 million, with more programs and better case management, and has set a target return date for each of the others. Quarterly diagnostic tests have been aligned with year-end assessments: Unbelievably, the two were designed by different consultants, and didn’t predict or reflect the outcome of the other.

That isn’t to say that Rhee is anywhere near achieving her often-stated goal of making Washington the best urban district in the country. Even she attributes much of the test-score gains in her two years to the district’s ability to pick what she calls “low-hanging fruit.” Saturday test-prep classes have helped borderline kids pass their year-end tests, even while thousands of other children remain far behind because of weak basic skills. Accounting changes helped boost results, too: Children who were absent on test day now are counted as no-shows; before, they were counted among those with failing scores.

The graduation rate—as opposed to the drop-out rate, which is calculated differently—was up a few percentage points in 2009 to 70 percent, the district says. But some teachers and parents attribute that to a new “credit recovery” program that lets failing students retake courses after school. Martel, the long-time social studies teacher, says credit-recovery classes ran 82 hours per quarter at his school compared to 125 hours for classes held during the school day, and that teachers were told not to give homework.

Despite the celebrity surrounding Rhee and Fenty, the traditional public schools are still bleeding students, which is perhaps the ultimate, market-driven judgment. Washington’s State Office of Education—yes, this nonstate has a state office—says enrollment in the traditional schools dropped to 45,200 in the 2008 school year from 49,500 just the year before. Charters grew to 25,700 from 22,000. Charter enrollment is even more impressive if you look at the fine print: In 2008, charters enrolled 48 percent of public-school 6th graders, up from 36 percent a year earlier.

Michael Herreld, who is president of PNC Bank’s Washington region and sits on several local school-reform committees, worries about what he calls the “disintegration” of the city’s traditional public schools if Rhee can’t stop the enrollment decline. Any urgency to fix things would wane, and so would the schools’ claim on public revenue. That would have practical consequences: Washington doesn’t have school buses, for example. If more schools are closed, youngsters could be miles from the nearest kindergarten and its free breakfast and lunch programs.

The only way to stop the attrition is to “grow good neighborhood schools,” says Nyankori. Rhee illustrated the obstacles to that when a woman asked her about her plans for math and science education during a meeting in the spring of 2009 in the city’s northwest quadrant, where most adults have at least one degree and, often, two or three. Rhee said she had ordered more computers to support math and science programs, but learned when they arrived that most schools didn’t have three-pronged electrical outlets for the computers’ three-pronged plugs. “This is the level where we are…subzero,” she said, as the audience stifled a collective eye roll.

High Stakes

Rhee seems irked that policymakers see Washington as the laboratory of the education-reform agenda. “That is the dumbest thing I’ve ever heard,” she said, at the same spring meeting at which she bemoaned the lack of proper sockets. What matters is Washington’s kids, not a national agenda, she insisted.

In fact, both are at stake. Washington is a natural petri dish, whether Rhee disdains the idea or not. It’s small and deeply troubled, is a foundation darling, has creative new leadership, and is pursuing the popular academic ideas of the day. Its big charter sector almost begs researchers to compare the two systems, and it sits in the spotlight of the U.S. Capitol.

I asked Rhee to name her biggest mistake in two years and she offered this: She could have done a “better job of communicating with teachers” when she presented her contract proposal and averted some of the antagonism that dogs her relationship with them. Since then, she has met with teachers a few times a week, she said, and finds the exchanges “incredibly heartening.” There are other tiny signs that Rhee may be trying to calm the waters she has roiled. With contract talks going nowhere in the spring of 2009, she wrote a Washington Post op-ed in which she insisted that “[t]hose who categorically blame teachers for the failures of our system are simply wrong.”

Around the same time, at a banquet at the Federal City Council, a premier business and civic group, Rhee thanked a consulting group for undertaking, pro bono, the school-records audit. “It was the first time I’ve heard her thank anyone for anything,” said the head of a major nonprofit. Her staff now concedes that a Time magazine cover of Rhee—standing grim-faced in an empty classroom, holding a broom—was a mistake.

That may be about it. I asked The New Teacher Project’s Ariela Rozman if Rhee ever called to cry on her shoulder. “Michelle doesn’t cry,” Rozman said. That’s probably a good thing.

June Kronholz is a former foreign correspondent, bureau chief, and Washington-based education reporter for the Wall Street Journal.

To receive a standard state certification in most states, prospective teachers not only must be college graduates but also must have taken a specific set of education-related courses that comprise approximately 30 credit hours of coursework. Prospective teachers are well advised to pursue studies at a college or university within the state where they expect to teach, because it is often only within that state that students can get the courses required for state certification in the subject area and for the grade levels that they will be teaching.

Such certification requirements limit the supply of certified teachers, and as a result, serious teaching shortages are regularly observed. For example, in California, one-third of the entire teacher work force, about 100,000 teachers, will retire over the next decade and need to be replaced, compounding what the governor’s office calls a “severe” current teacher shortage. Other states are facing a similar situation. The National Council of Teachers of Mathematics projects a shortfall of 280,000 qualified math and science teachers by 2015. As former National Education Association president Reg Weaver put it, “At the start of every school year, we read in the newspaper…stories about schools scrambling to hire teachers.”

Teachers of minority background are in especially short supply. In 2004, only 14.1 percent of the nation’s teachers were African American or Hispanic, even though these ethnic groups comprised 26.5 percent of the adult population. That shortage has led to calls for remedial action. In the words of Weaver, “An impressive body of research confirms that recruiting and retaining more minority teachers can be crucial to” raising the achievement of minority students. “States and school districts need to develop programs…[that] reach out to minorities still in school, offering encouragement and incentives to enter the teaching profession. We need more minority teachers. School districts need to aggressively recruit them.”

The Certification Debate

Both colleges of education and teachers unions oppose any relaxation of certification requirements. In Weaver’s view, “The solution is not to develop alternative routes of entry into the profession or to increase the supply of recruits by allowing prospective teachers to skip ‘burdensome’ education courses or student teaching. The solution is to show a little R-E-S-P-E-C-T, and show us the money.”

According to this point of view, certification is necessary to ensure teacher quality, because teaching, like other professions (law, medicine, the sciences, and so forth), requires mastery of an esoteric body of substantive and pedagogical knowledge that cannot be obtained without undergoing a rigorous training program. Arthur Wise, former head of the National Council for Accreditation of Teacher Education, told the House Committee on Education and the Workforce, “Rigorous teacher preparation is key to ensuring that no child is left behind…. Content knowledge is only one indicator of readiness to teach…. [Schools of education] must prepare new teachers to teach the great diversity of students who are in America’s classrooms today.”

But Kate Walsh, president of the National Council on Teacher Quality, says that the “certification process” has only a “crude capacity for ensuring” quality teachers, because pedagogical “knowledge can be acquired by means other than coursework.” Teachers learn to teach by practicing the craft, not by taking coursework in its history or psychology. If that is so, then both the teaching shortage and the paucity of minority teachers can be alleviated by opening the classroom door to all college graduates, not just to those who have taken the required courses associated with state certification.

Although no state has abandoned its traditional certification programs in response to calls for broader recruitment paths into education, all but three states have set up some kind of alternative certification pathway, and the number of alternatively certified teachers has steadily grown. In 2001, just 20,000 alternative teaching certificates were issued. By 2006, nearly 60,000 alternatively certified teachers were entering the teaching force each year, roughly one-fifth of new entrants (see Figure 1).

In many states, however, the alternative certification requirements closely resemble traditional ones. As Chester Finn, president of the Thomas B. Fordham Institute (and Education Next senior editor), explains, “Typical alternative certification programs have come to mimic standard-issue pre-service college of education programs…. Alternative certification has been co-opted, compromised, and diluted. Education schools—brilliantly turning a threat into an opportunity—have themselves come to dominate this enterprise, blurring the distinctions that once made it ‘alternative.’”

Symbolic vs. Genuine Alternatives

To see which states have established genuine alternative certification programs and which have only symbolic ones of the kind described by Finn, we compared alternative certification rules to requirements for traditional certification in each state. We obtained information from state web sites as well as from publications available for prospective teachers at the Harvard Graduate School of Education. When necessary, we contacted relevant officials in state departments of education (see sidebar).

Our research revealed that the design of alternative certification programs varies considerably from one state to the next. In 26 of the 47 states that allow alternative certification, the situation is as Finn describes, similar to standard-issue education school programs. But in the other 21 states, alternative teacher certification can be obtained without completion of most of the coursework associated with traditional certification (though the states may require additional coursework after a teacher secures a classroom position).

The variation in state practice allows us to provide some preliminary answers to questions frequently raised about alternative certification: Do states that provide a genuine alternative, not simply a symbolic one, recruit more teachers who take the alternative certification route? Do they open the classroom door to more minority teachers? What is the impact on student learning?

The answer to the first question is very clear: it makes a good deal of difference whether alternative certification is meaningful or symbolic. In the 30 states that do not have genuine alternative certification, only 5 percent of the newly certified teachers chose the alternative route in 2004–05 (the most recent year for which information is available). Hardly anyone bothers with an alternative certificate if the requirements are essentially the same as for the traditional one. In the 21 states that offer genuine alternative certification, however, 28 percent of newly certified teachers utilize the option (see Figure 2). Altogether, 92 percent of those with an alternative teaching certificate received it in one of the 21 states that have such certification in reality as well as in name.

Minority Representation

To the second question, there is also a fairly clear answer: minorities are represented in the teaching force to a greater extent in states with genuine alternative certification than in other states. Information on the number of minority teachers in each state is available from the Office for Civil Rights and the U.S. Department of Education. Together with census data, that information allows for the calculation of the ratio of minority teachers to minority adults for each state, which serves as an index of minority representation. If the index for a given state has a value of 1.0, then the percentage of minority teachers equals the percentage of minority adults in that state. If the index exceeds 1.0, then minorities are overrepresented in the teaching population; if the index is below 1.0, minorities areunderrepresented.

In 2004 (the most recent year for which data are available) minority teachers were seriously underrepresented in the nation’s public school classrooms. The nationwide ratio in 2004 was 0.53, which means that only a little more than half as many minority adults were teachers as one would expect, given the minority composition of the adult population.

The index of minority representation was nonetheless considerably higher in the 21 states with genuine alternative certification than in the 30 states with a symbolic substitute or no alternative certification (see Table 1). In the states with genuine alternative certification, the weighted average index of representation was 0.6, while in the states with a symbolic substitute or no alternative certification, the index was 0.2. In the three genuine alternative certification states with the largest total populations (California, Texas, and Florida), the index of representation was 0.56, 0.68, and 0.72, respectively, while in the three largest states with a merely symbolic or no alternative certification option (New York, Illinois, and Ohio), the index of representation was 0.38, 0.33, and 0.51, respectively. In other words, genuine alternative certification seems to give minority adults interested in a career in education greater opportunity to become a teacher.

Of course, minority teachers may be recruited by avenues other than alternative certification. Sixteen states report the ethnic background of alternatively certified teachers to the U.S. Department of Education. In 14 of them, the percentage minority for those alternatively certified exceeds by a wide margin the percentage minority of the state’s teaching force as a whole. In Mississippi, for example, the disparities are massive: 60 percent of the more than 800 teachers who were alternatively certified in 2004–05 were of minority background, while the overall Mississippi teaching force is just 26 percent minority. Other states where percentage differences between the two groups exceed at least 10 percentage points include California, Delaware, and Texas. In other words, there is every reason to believe that alternative certification is key to recruiting more minorities into the teaching profession.

Impact on Student Learning

But is alternative certification, however desirable a device for recruiting minorities into the teaching profession, impairing student learning? That is ultimately the justification for traditional teacher certification, regardless of its consequences for teacher shortages or for the recruitment of minority teachers. And that apparently is the rationale for those who interpret the federal No Child Left Behind (NCLB) act as requiring a certified teacher in every classroom, the basis for the complaint in the California case Renee v. Spellings. That suit objects to the presence in classrooms of teachers who have not received traditional certification.

Most studies show very little, if any, connection with a teacher’s classroom effectiveness and certification status. Harvard economist Thomas Kane and his colleagues (see “Photo Finish,” research, Winter 2007) found no difference between the amount of learning taking place in New York City classrooms under the direction of non-certified teachers and in those under the direction of regularly certified ones.

Nor is there convincing evidence that minority teachers are less effective at teaching minority students. On the contrary, the results from an experiment conducted in Tennessee (see “The Race Connection,” research, Spring 2004) indicate that minority students learn more from teachers of their own ethnicity than from other teachers.

Our results are consistent with that research and other studies that have found little reason to equate certification with “highly qualified.” Students attending schools in states with genuine alternative certification gained more on the National Assessment of Educational Progress (NAEP) between 2003 and 2007 than did students in the other states. The finding holds, even when one adjusts for changes in the ethnic composition, free-lunch eligibility,class size, and education expenditures for each state (see Figure 3).

In states that had genuine alternative certification, test-score gains on the NAEP exceeded those in the other states by 4.8 points and 7.6 points in 4th- and 8th-grade math, respectively. In reading, the additional gains in the states with genuine alternative certification were 10.6 points and 3.9 points for the two grade levels, respectively. Among African Americans, test-score gains were also larger in the states with genuine alternative certification.

It is possible that the disproportionately large gains in test-score performance in the states with genuine alternative certification were due to some other factor, possibly other education reforms those states were introducing at the same time they were widening the door to the teaching profession. We cannot dismiss this possibility, inasmuch as we were able to control only for changes in certain demographic and policy variables, not for changes in other state policies that might explain the difference in test-score gains among states.

But the burden of proof would now seem to shift to the plaintiffs in the Renee v. Spellings case, who argue that traditional state certification is necessary to ensure teacher quality. Genuine alternative certification opens the door to more minority teachers, and student learning is more rapid in states where the reform has been introduced. Meanwhile, scientific evidence that alternative certification harms students remains somewhere between scant and nonexistent.

-Paul E. Peterson, director of the Harvard Program on Education Policy and Governance (PEPG), is editor-in-chief of Education Next. Daniel Nadler is a PEPG research associate.

But we can only hope their similarities don’t extend to education policy. Patrick calls education his “singular pursuit.” Yet after winning election in a 2006 landslide fueled by strong support from the Bay State’s powerful teachers unions—including $3 million in contributions—he has pursued the systematic dismantling of reforms that have made Massachusetts the national leader in public education.

The Massachusetts Education Reform Act of 1993 dramatically increased school funding in return for high academic standards, accountability, and enhanced school choice. In the years following, the Commonwealth’s independent board of education, founded in 1837 with Horace Mann at the helm, implemented a set of reforms that have unquestionably been the nation’s most successful.

In 2005, Massachusetts became the first state ever to finish first in four categories of the National Assessment of Educational Progress (NAEP): 4th-grade reading and math and 8th-grade reading and math. The next time the test was administered, Bay State students did it again. Late last year, results from the Trends in International Mathematics and Science Study (TIMSS) demonstrated that Massachusetts students are not only the best in the country, they are globally competitive as well. The Commonwealth’s 8th graders tied for first in the world in science and were sixth in math; 4th graders scored second in science and third in math.

Despite the clear success of more than a decade of education reform in Massachusetts, Governor Patrick’s administration has turned its back on the very forces behind that success: it is wavering on standards, choice is under continual fire, and the board of education has been stripped of the independence that for 170 years was Horace Mann’s legacy and had allowed the board to implement reform with a singular focus on improving student achievement.

In June 2008, Governor Patrick released the recommendations of his “Readiness Project,” an unwieldy 168-member, 13-subcommittee behemoth charged with developing a long-term “action agenda” for education. The plan calls for full-day kindergarten, universal pre-K, consolidation of school districts, and differentiated pay for teachers—all worthy goals. But the report maintains Patrick’s steadfast resistance to raising caps on charter schools. (Charter schools have the same effect on some of his supporters in the education establishment as Nancy Pelosi has on Rush Limbaugh.) Although the governor claimed during his campaign that he would open more charter schools once he “fixed” the formula by which they are funded, the Readiness Project is virtually silent on charters and their funding.

The Boston Globe, which enthusiastically endorsed the governor in both the Democratic primary and the general election, was not impressed. An editorial titled “Adrift in the edu-sphere” noted, “It’s nice to explore the educational cosmos. But taxpayers can’t be expected to pay for such a trip…when the likely cost of implementing Patrick’s full-blown plans could exceed $2 billion per year.”

Yet another commission, this one tasked with determining how to pay for Patrick’s action agenda, was appointed in June 2008. By the time its report was released, in the midst of a snowstorm on New Year’s Eve, the bottom had fallen out of the economy. Instead of identifying revenues to support new programs, the report focused mostly on cost-saving measures designed to preserve the current level of quality, although a majority of the commission’s 23 members did endorse raising the Commonwealth’s sales tax from 5 to 6 percent.

Success Story

All of this is particularly bizarre in light of the dramatic strides the state has made in improving its schools. The U.S. Chamber of Commerce published a state-by-state report card on educational effectiveness in 2007 that rated the Commonwealth’s public schools number one in the nation. The combination of funding, standards, accountability, and choice has brought real, measurable gains in student achievement (see Figure 1). A look at the condition of public education prior to reform shows just how far Massachusetts has come. During the 1980s, the Commonwealth’s verbal SAT scores were below the national average; math scores were below average as late as 1992. A funding system that was overly reliant on local property tax revenue created vast discrepancies from district to district in student achievement, class size, and the availability of resources like textbooks, libraries, and technology.

Since 1993 the Commonwealth has pumped more than $40 billion in new state money into public education, matched by $40 billion-plus in new local funding. Each district’s foundation budget, the minimum expenditure needed to provide an adequate education, is determined by formula, along with the amount each city and town can afford to contribute. The Commonwealth fills in the gap between the local contribution and the foundation budget. The result is a funding formula in which the vast majority of state education aid goes to the poorer school districts, making Massachusetts one of the national leaders in this respect as well (see Figure 2).

To ensure high academic standards and school-level accountability, state curriculum frameworks provide a subject-by-subject outline of the material that should form the basis of local curricula. To ensure implementation of the frameworks, students are tested each spring. Since 2003, passing the Massachusetts Comprehensive Assessment System (MCAS) tests (based on the liberal arts-rich content of the frameworks) has been a high school graduation requirement. High-stakes testing also extends to new teachers, who must pass tests that measure communication and literacy skills as well as subject-area knowledge.

The state’s NAEP scores shot up after the curriculum frameworks were completed and the MCAS test was first administered in 1998. By 2007, the average Massachusetts 4th grader was performing at a higher level in math than the average 6th grader had been in 1996. Achieve, Inc., a national education organization established by governors and business leaders, found in 2001 that Massachusetts was the only state among the 10 it examined that had both strong standards and strong assessments. A 2007 study by the Massachusetts Board of Higher Education confirmed the tests’ validity, finding a strong correlation between MCAS results and college performance.

Noted educator and developer of the Core Knowledge curriculum E. D. Hirsch lauded the Massachusetts approach in a February 2008 op-ed in the Washington Post. “Consider the eighth grade NAEP results from Massachusetts, which are a stunning exception to the nationwide pattern of stagnation and decline,” he wrote. “That is because Massachusetts decided…students (and teachers) should learn explicit, substantive things about history, science and literature, and that students should be tested on such knowledge.”

Choice and Charters

In Massachusetts, public charter schools are the principal vehicle for offering educational choice, and the Thomas B. Fordham Foundation has described the Commonwealth’s charter-school approval process as the nation’s most rigorous. Today, roughly 25,000 students (about 2.6 percent of the total public school population) attend Massachusetts charter schools, and another 21,000 are on wait-lists. Admission to an oversubscribed school is by lottery. When a student chooses to transfer to a charter school, funding follows from the district to the charter school. Despite the fact that districts are reimbursed for three years after a student leaves (100 percent the first year, 60 percent the second, and 40 percent in the third) and despite the 2004 adoption of district-friendly changes to the charter-funding formula, the flow of money has made charter schools controversial.

That controversy has fueled a one-step-forward, two-steps-back treatment of charters over the years. Caps on the number of schools have been raised just twice and now stand at 72 for the original type (known as Commonwealth charter schools) and at 48 for Horace Mann charters (a unionized, in-district model sanctioned after Commonwealth charters were established). Other limitations have been placed on both types of charter schools. The statewide share of public school students who can attend charters is capped at 4 percent. In any year in which a new charter school is approved, at least three of the newly approved charters must be located in low-performing districts. The law limits to 9 percent the portion of district spending that can be transferred to charter schools. More than 150 communities, mostly in poorer areas with low-performing schools, are bumping up against that cap, which places a de facto moratorium on charters.

Charter school results have been strong. A 2006 Massachusetts Department of Education study found that 90 percent of charter schools performed as well as or better than the districts from which their students came and 30 percent outperformed sending districts by a substantial margin. Their success has been particularly striking in urban areas, where most charters are located. Several urban charter schools, like Community Day in Lawrence, and MATCH, Boston Prep, and Excel Academy in Boston, serve overwhelmingly low-income and minority populations, yet outscore even the best suburban schools on MCAS tests.

SABIS International Charter School in Springfield is among the schools that have had remarkable success in narrowing achievement gaps based on race and economic status, a clear priority for the next phase of education reform. By 10th grade, Hispanic and African American students, who together make up 60 percent of the school’s student body, outperform white students statewide on the MCAS English exam and are virtually even with statewide averages for white students in math. More than 2,500 students sit on the school’s waiting list. Every member of all seven graduating classes has been accepted to college.

A study conducted by a team of Harvard and MIT researchers and published in January by the Boston Foundation showed that Boston charter schools dramatically outperform both district and pilot schools (semi-autonomous district schools created in response to charters). It found that the academic impact from a year spent in a Boston charter is comparable to that of a year spent in one of the city’s elite exam schools and, in middle school math, equivalent to one-half of the achievement gap between black and white students (see Figure 3).

One might expect the governor would support schools that the state’s own analysis and others have found to be successful. Indeed, prior to release of the Readiness Project recommendations, Governor Patrick said, “Everything is on the table because our future is at stake.” Everything, it seems, except expanding the kind of educational choice that transformed the governor’s own life.

Patrick earned a scholarship from A Better Chance, an organization that provides educational opportunities to young people of color. The scholarship transported him from the South Side of Chicago to Milton Academy, an elite Massachusetts preparatory school, and put him on a trajectory that led to Harvard, a top position in the U.S. Department of Justice, the corporate world, and ultimately the governor’s office.

But the Readiness Project includes precious little that would give others the opportunity to choose their school. The Readiness Project proposes “readiness schools,” which would have some of the autonomy of charter schools and some of the features of pilot schools. Teachers in a district school could come together and vote to convert to a readiness school, or districts could initiate the conversion. As an inducement to adopt the newly proposed schools, S. Paul Reville, whom Patrick appointed first to chair the state board of education and then as secretary of education, floated the possibility of a freeze on charter schools in districts that embrace readiness schools. When the trial balloon became public, he quickly backpedaled in the face of a torrent of opposition.

The administration’s current position on the charter freeze is unclear. When questioned, Patrick said the charter school debate had reached “stalemate” and his plan includes new ways to achieve the same goals.

Dismantling Success

In 2000, the Commonwealth created the Office of Educational Quality and Accountability (EQA) as an independent state agency to measure the effectiveness of school-district managers at implementing reform. Beginning in 2002, EQA conducted comprehensive audits of more than 175 school districts. The audits scrutinized MCAS performance, district leadership, curriculum and instruction, teacher and student assessment and evaluation, and business and financial operations. All findings were made public.

Soon after taking office, Patrick moved to eliminate the EQA. Opponents particularly disliked the agency because it did its job so well—auditing school districts and reporting when they came up short. Two studies by Boston-based think tank Pioneer Institute analyzed agency data and found that low-performing urban districts in particular were not aligning curricula with state frameworks and not using MCAS data effectively to improve achievement by tailoring lessons to student weaknesses.

More than a year after the EQA was scuttled, the co-chairs of the state legislature’s Joint Committee on Education filed a bill, later enacted, creating a new Advisory Council on District Accountability and Assistance. The new agency amounts to the fox guarding the accountability henhouse, replacing the EQA’s independent 5-person board with a 13-member panel that includes representatives of the very people it’s supposed to audit: the Massachusetts Association of School Superintendents; American Federation of Teachers Massachusetts; Massachusetts Teachers Association; Massachusetts Association of School Committees; Massachusetts Secondary School Administrators Association; and the Massachusetts Elementary School Principals Association.

The administration’s proposal to overhaul the Commonwealth’s education governance structure gained legislative approval in February 2008. An education commissioner who reported to the board of education, not the governor, had long directed primary and secondary public education in Massachusetts. The Patrick proposal resurrected the state secretary of education post, which had been created and abolished twice since the 1970s.

Reville, who then chaired the board of education, claimed Governor Patrick’s plan kept appropriate distance between politics and education policy. But when a far weaker education secretariat had been proposed in 2003, Reville testified before the legislature in opposition to the plan, saying, “No matter how well constituted, an education secretariat creates a competing center of power that vies with and against the state’s chief school officer, the Commissioner of Education and the state education agency.”

Governor Patrick himself contradicted Reville’s claim that the new proposal was more respectful of independent education policymaking. At its unveiling, the governor said his plan “will be different in that (the secretary) will have real authority.”

But the administration’s main target was the state board of education. In a move reminiscent of FDR’s court-packing plan, the overhaul added two seats to the board, opened up two more slots by removing the commissioner of early childhood education and the chancellor of higher education, made the new secretary a voting member, and truncated the terms of members least likely to agree with the administration.

Even more importantly, it stripped the renamed Board of Elementary and Secondary Education of its independence, placing it firmly under the governor’s control by giving the new secretary final say over budget requests and veto power over its selection of future commissioners of education. The board had just selected Mitchell Chester, an Ohio education official, to be the next commissioner. Chester beat out Karla Baehr, who was superintendent of schools in the city of Lowell, had gained some prominence among urban superintendents (see sidebar), and was widely seen as the choice of the education establishment and the governor. Baehr was later hired as a deputy commissioner.

The usually affable Patrick also used the unveiling of his governance proposal to send a message to those concerned about charter schools, saying they should “grow up.” Later, after release of the Boston Foundation study, Patrick called the debate about raising charter caps “a red herring because we’re not at the cap,” despite the fact that Boston is among the urban communities bumping up against the 9 percent of school district spending limitation, with only 111 charter seats remaining and about 7,000 students languishing on wait lists.

In February 2008, the board, still chaired by Reville (he assumed the new secretary of education post on July 1), became the first to reject a charter school recommended for approval by the commissioner of education. The focus of the board’s discussion about the proposed SABIS regional charter school in the city of Brockton was a 2005 state department of education (DOE) report that identified problems at the Springfield SABIS charter school. Days after the new school’s application was rejected, a 2006 DOE letter surfaced that said the Springfield school had successfully addressed all the major issues raised in the earlier report. Company officials who attended the board meeting were not allowed to respond to Reville’s criticisms.

A Boston Globe editorial noted that the “rejection raises thorny questions about just how hard the Patrick administration is willing to push to achieve equity in education.” Like SABIS’s successful Springfield charter school, the proposed school would have served troubled communities. The most current data available prior to the proposed school’s rejection showed that 20 of Brockton’s 23 schools failed to make adequate yearly progress (AYP) under federal law, and all 6 failed to make AYP in nearby Randolph, where the schools are in such bad shape the district was required to submit a plan to stave off state receivership. During the board’s debate over the proposed charter school, Patrick appointee and board PTA representative Ruth Kaplan commented that charter schools are too focused on sending students to college, saying “families…don’t always know what’s best for their children.”

During the spring of 2008, Reville charged a “21st Century Skills Task Force” with rewriting curricula and ensuring that Massachusetts students are prepared to succeed in a fast-changing economy. The task force’s report, published in November, proposes revamping MCAS and using the U.S. History test to try out project-based assessments that require students to demonstrate skills like “global awareness,” a change likely to crowd out topics like the Constitution or causes of the Civil War. It calls on the teachers unions, school committees, and superintendents that have fought education reform for 15 years to determine how to integrate 21st-century skills in our schools.

In a sad irony, the task force report claims that “Massachusetts can learn from the experience of West Virginia” on ways to incorporate the needed skills. West Virginia students score below the national average on the NAEP tests, and the state was among the seven that saw the largest declines in reading scores between 1998 and 2005.

A month after release of the task force report, former state senate president Thomas Birmingham, one of the architects of education reform, delivered an address in which said he was “discomforted” by the direction of the Readiness Project and that the 21st Century Skills Task Force “may threaten to…drive us back in the direction of vague expectations and fuzzy standards.”

Teacher testing has also come under fire. In April 2008, the state senate voted to allow some teachers to be licensed even if they failed the required exam three times. The administration announced that it was looking at alternative criteria for aspiring teachers, even though most of the tests are at a high-school level of difficulty. Reville told the Globe the test “isn’t necessarily the best venue for everyone to demonstrate their competency.”

The move to back away from teacher testing sparked another firestorm of opposition. In a Boston Globe op-ed, Charles Glenn, then dean ad interim of Boston University’s School of Education, wrote, “It would be a gross disservice for our public school children to be taught by teachers who do not meet the standards set by our current teacher tests.” Reville later said the administration didn’t support the senate vote after all.

With the pillars of reform under attack, Joe Williams, executive director of Democrats for Education Reform, wrote in the Globe, “You have to wonder why Massachusetts seems intent on retreating from its own nationally recognized success. The backward slide is already evident.”

The Wrong Path

The Commonwealth’s 15-year track record of successful education reform gave Governor Patrick a clear path ahead on education policy. Instead of undoing the reforms of his predecessors, the governor could have built on the state’s success by carrying on the commitment to high standards, fine-tuning a successful accountability system, and maintaining the governance structure that had successfully insulated critical education policy decisions from special-interest pressure. He could extend to others the educational opportunity that transformed his own life by raising from 9 to 20 percent the cap on the amount of money that can be transferred from school districts to charter schools in districts whose MCAS scores are in the bottom 10 percent statewide.

So far, he has chosen instead to dismantle reform and replace the singular focus on student achievement that was the key to education reform’s success with a wish list that would likely cost taxpayers an additional $2 billion per year. With the new Board of Elementary and Secondary Education stripped of independence, there is no entity left that can operate outside the political arena with the sole mission of improving academic performance.

Results released in September 2008 showed a sharp drop in MCAS pass rates and flat or declining scores in the elementary and middle school grades and in many urban districts. While 15 years of progress will not be undone overnight, as the Patrick administration’s efforts to dismantle reform continue, such drops are likely to become the rule. It is the price we will pay for Massachusetts policymakers snatching defeat from the jaws of the Commonwealth’s historic education-reform victory.

As for President Obama, during the primaries he played to the teachers unions that are a critical Democratic Party constituency by assailing the evils of forcing teachers to “teach to the test.” But once the nomination was secured, he moved to the center, unveiling proposals that included merit pay for teachers and doubling federal charter-school funding. His selection of Arne Duncan, Chicago’s charter-friendly school superintendent, as education secretary also bodes well. Let’s hope that as president he continues down that path rather than the one Governor Patrick has chosen, and that he applies the lessons from the successful reforms in Massachusetts to federal education policy.

Charles Chieppo is a senior fellow and James Gass is director of the Center for School Reform at Pioneer Institute, a Massachusetts public-policy think tank.

CORRECTION: The printed version of this article contains two errors, which have been corrected here. The number of students on waiting lists for charter schools in Boston is about 7,000. The Readiness Finance Commission reportedly favored raising the state sales tax rather than the income tax from 5 to 6 percent.

In this report, we use 2007 test-score information to evaluate the rigor of each state’s proficiency standards against the National Assessment of Educational Progress (NAEP), an achievement measure that is recognized nationally and has international credibility as well. The analysis extends previous work (see “Johnny Can Read…in Some States,” features, Summer 2005, and “Keeping an Eye on State Standards,” features, Summer 2006) that used 2003 and 2005 test-score data and finds in the new data a noticeable decline, especially at the 8th-grade level. In Figure 1, we rank the rigor of state proficiency standards using the same A to F scale teachers use to grade students. Those that receive an A have the toughest definitions of student proficiency, while those with an F have the least rigorous.

Measuring Standards

That states vary widely in their definitions of student proficiency seems little short of bizarre. Agreement on what constitutes “proficiency” would seem the essential starting point: if students are to know what is expected of them, teachers are to know what to teach, and parents are to have a measuring stick for their schools. In the absence of such agreement, it is impossible to determine how student achievement stacks up across states and countries.

One national metric for performance does exist, the National Assessment of Educational Progress. The NAEP is a series of tests administered under the auspices of the U.S. Department of Education’s National Center  for Education Statistics. Known as the Nation’s Report Card, the NAEP tests measure proficiency in reading and math among 4th and 8th graders nationwide as well as in every state. The NAEP sets its proficiency standard through a well-established, if complex, technical process. Basically, it asks informed experts to judge the difficulty of each of the items in its test bank. The experts’ handiwork received a pat on the back recently when the American Institutes for Research (AIR) showed that NAEP’s definition of “proficiency” was very similar to the standard used by designers of international tests of student achievement. Proficiency has acquired roughly the same meaning in Europe and Asia, and in the United States—as long as the NAEP standard is employed.

This is not to say students are proficient either in this country or elsewhere. According to NAEP standards, only 31 percent of 8th graders in the United States are proficient in mathematics. Using that same standard, just 73 percent of 8th graders are proficient in math in the highest-achieving country, Singapore, according to the AIR study. In other words, bringing virtually all 8th graders in the United States up to a NAEP-like level of proficiency in mathematics constitutes a challenge no country has ever mastered.

Comparing the States

Three states—Massachusetts, South Carolina, and Missouri—have established world-class standards in math and reading as the goal for all students. Every other state has established a lower proficiency standard, and some states (for example, Georgia and Tennessee) declare most students proficient even when their performance is miles short of the NAEP standard. By setting widely varying standards, states render the very notion of proficiency meaningless. If Billy and Sally cannot read in South Carolina, they should not be able to pass muster simply by crossing the state’s western border.

We gauge the differences among states by comparing how students do on state assessments with how they perform on NAEP tests. By comparing the percentage of students deemed proficient on each, it is possible to determine whether states are setting expectations higher, lower, or equal to the NAEP standard. If the percentages are identical (or roughly so), then state proficiency standards can be fairly labeled as “world-class.” If state assessments identify many more students as proficient than the NAEP, then state proficiency figures should be regarded as inflated. In short, comparing state assessment results to NAEP scores can help reveal whether states are giving parents and voters the real scoop about where the state’s children stack up when measured against world-class benchmarks.

In Figure 1, we give Massachusetts, Missouri, and South Carolina an A for establishing rigorous expectations regarding what proficient students must know and be able to do. Note that a grade of A does not indicate students are performing at the highest level. Rather, the high grade indicates that the three states have set a high bar for students to reach if they are to be deemed proficient. So, for example, only 25 percent of 8th graders in South Carolina were deemed proficient on both the state reading test and on the NAEP reading test—an honest, if embarrassing, reckoning of the education situation in the state.

The remaining 47 states (information is not yet available for the District of Columbia) had distinctly lower standards. Three states—Georgia, Oklahoma, and Tennessee—expected so little of students that they received the grade of F. The state of Georgia, for instance, declared 88 percent of 8th graders proficient in reading, even though just 26 percent scored at or above the proficiency level on the NAEP. According to our calculations, Georgia 8th-grade reading standards are 4.0 standard deviations below those in South Carolina, an extraordinarily large difference. Thus, while students in Georgia and South Carolina perform at similar levels on the NAEP, the casual observer would be misled by Georgia’s reporting that its students achieve proficiency at three times the rate that South Carolina’s students do.

Twelve states—Alabama, Alaska, Idaho, Illinois, Michigan, Mississippi, Nebraska, North Carolina, Texas, Utah, Virginia, and West Virginia—received Ds because they had pitched their expectations far below other states. Illinois set its proficiency bar for 8th-grade reading at a level that is 1.01 standard deviations below the national average. If you believe those who set the Illinois standards, 82 percent of its 8th graders are proficient in reading, even though the NAEP says only 30 percent are.

In general, the states of the Northeast have the highest standards, while the states of the South and Midwest have the lowest. Western states fall in between.

A Downward Trend

There is some evidence of slippage in standards since our original report card was published in 2005 (see Figure 2). In 8th-grade reading, for example, standards overall are down by 0.2 standard deviations. This means that, in 8th-grade reading, states are reporting a substantial improvement that is not evident on the NAEP. The smallest amount of slippage was in 4th-grade math, where standards fell by 0.06 standard deviations. Most of the slippage at the 4th-grade level is due to the lower standards adopted by those states that were initially slow in complying with the NCLB accountability system; those that have had standards since 2003 have not altered them significantly. But at the 8th-grade level, standards are falling across the board—in both reading and math, and among both the states that had standards in 2003 and the states that have only adopted them more recently.

We also see slight convergence among the states. For example, the variation in 4th-grade math standards narrowed 0.11 standard deviations between 2003 and 2007. The good news is that differences among state standards are shrinking; the bad news is that states are converging downward, not upward.

By and large, the changes that are taking place in individual states are fairly small, perhaps so they do not stir controversy. A few states, though, have made big adjustments since 2003. Colorado and Texas have raised their proficiency bars enough to warrant a grade one letter better than the one given initially. Five states—Arizona, Illinois, Maine, Michigan, and Wyoming—have lowered the bar enough that their grades have dropped by a full letter.

Two years ago, we could see small evidence for a decline in standards but detected no race to the bottom. That is still true for 4th graders. But 8th-grade standards, if not exactly racing downward, are moving steadily away from world-class standards. Those responsible for NCLB reauthorization, as they struggle forward, should first and foremost establish a clear and consistent definition of grade-level proficiency in reading and math, even if it means giving up the cherished but decidedly unrealistic goal of proficiency for all students by 2014.

Paul E. Peterson and Frederick M. Hess are editors of Education Next.

If states choose to spend a fortune on such tests, it’s because their leaders believe, in the words of the National Governors Association (NGA), that doing so will lead to “policy solutions to U.S. education system shortcomings.” This is the first of two justifications given for these efforts, which are widely referred to as “international benchmarking.” The second purpose is the straightforward comparison of U.S. students and their peers in other countries. Here, the resulting “league tables” are the center of attention, and these results are often described, and almost as often decried, as a “horse race” or an “Olympics contest” with an attendant fixation on who’s winning and who’s losing. Absent is an acknowledgment that these assessments embody particular assumptions and beliefs about what should be measured.

As states consider funding and administering international assessments for themselves, such facts must be kept in mind and key questions addressed. Among them are, what will states get as a result of participation? How much will it cost? Are there easier and cheaper ways of obtaining the desired information? We begin with a comparison of PISA and TIMSS, the two most prominent international assessments (see sidebar for additional information).

PISA v. TIMSS
PISA is a self-proclaimed “yield study” assessing the “literacy” of 15-year-olds and is not tied to any specific curricula. PISA claims to be assessing the skills young adults will need in the emerging global economy. Never modest, the report releasing the 2003 PISA results was titled “Learning for Tomorrow’s World” and the 2006 PISA results “Science Competencies for Tomorrow’s World.”

TIMSS is more grade- and curriculum-centered and far more modest in its claims. As described in the release of the 2007 data, “TIMSS is designed to align broadly with mathematics and science curricula in the participating countries. The results, therefore, suggest the degree to which students have learned mathematics and science concepts and skills likely to have been taught in school.”

PISA has not hesitated in making the most of these differences, and the high visibility of the Organisation for Economic Co-operation and Development (OECD) has ensured that PISA is the most prominent of the international assessments. Indeed, while in public the NGA is careful to say that it is in favor of international benchmarking in general and not in favor of any particular test, in practice the organization has tended to favor PISA. For example, a background paper the NGA released in February 2008 concludes by promising that “the National Governors Association will work with states interested in state-level PISA administration to identify available resources and cost-savings options to support state participation.” (In this three-page memo, TIMSS rated just one sentence.)

More recently, the NGA, the Council of Chief State School Officers, and Achieve released a report titled “Benchmarking for Success: Ensuring U.S. Students Receive a World-Class Education,” calling for state scores on international assessments. PISA was mentioned far more frequently than TIMSS, and the head of the PISA project at OECD, Andreas Schleicher, was repeatedly quoted.

The Alliance for Excellent Education has made an even stronger appeal for state participation in PISA, calling our failure to participate at that level “myopic.”

Most state interest is in the science and math assessments of PISA and TIMSS. In 1995, TIMSS participants included five states and one consortium of school districts. In 1999, just before PISA was launched, 13 states plus 14 consortia of school districts participated in TIMSS. Only two states, Massachusetts and Minnesota, funded their own participation in the 2007 TIMSS. No U.S. states have independently participated in PISA or PIRLS thus far.

The results of the three assessments are often compared to those of the National Assessment of Educational Progress (NAEP), which in the United States currently dwarfs the international assessments in scope. In 2007, the NAEP 8th-grade math exams involved more than 150,000 students in approximately 7,000 schools. The 8th-grade 2007 TIMSS assessed some 7,400 U.S. students in fewer than 250 schools. PISA in 2006 involved only 5,600 15-year-olds in around 170 schools.

The difference in size of these assessments is important: in reading and math in 4th and 8th grade, NAEP can report state-by-state performance every two years and, through the Trial Urban District Assessment program, NAEP can present data on a growing number of large school districts. In contrast, the small size of the PISA and TIMSS samples in most states makes state-level reporting difficult, though not impossible.

Statistical Benchmarking
One relatively inexpensive way in which state performance can be gauged against international measures is through statistical linking. Figure 1 shows TIMSS scores for 8th-grade math for seven states plus the District of Columbia and an equal number of countries. This figure is based on work by Gary Phillips of the American Institutes for Research, who has estimated 2007 TIMSS scores for all states by placing TIMSS and NAEP on the same scale. Such statistical linking can be more easily done with TIMSS than with PISA because TIMSS is administered at the same grade levels as NAEP, and their purposes and frameworks are similar.
Figure 1 presents patterns more precisely than we might infer from NAEP or TIMSS alone. For example, we know from the 2007 NAEP math report that Massachusetts and Minnesota are high-performing states, that South Carolina and Missouri perform around the national average, that Mississippi and Alabama are low-performing states, and that the District of Columbia, is the lowest-performing “state” in the nation. The 2007 TIMSS ranks countries, ranging from the highest-performing, including Chinese Taipei (Taiwan) and the Republic of Korea, through the middle ranks, including the United States, and going down to the lowest-performing countries, including Turkey (the lowest shown in Figure 1), Syria, Egypt, Algeria, and Colombia.

By combining these two sets of results, we know that even the best-performing American states do not score nearly as high as Chinese Taipei or Korea, that the average-performing American states are about on par with England, the Russian Federation, and Lithuania, and that the District of Columbia’s performance is more comparable to those of Thailand and Turkey.

While the Phillips method creates this interwoven list at minimal cost (because he is estimating state scores from already extant data), obtaining actual state-level scores is very expensive. How much a more precise list would be worth is an issue that states must consider.

Generating Policy Advice
Figure 1 reflects the “how are we doing” aspect of benchmarking. But what policy advice can be garnered from these international studies? The quick answer is not as much as many would have you believe. This is particularly true of PISA, which is very aggressive in drawing policy implications out of the data. Reflecting the mission of the OECD, which combines both statistical and policy functions, the very organizational structure of PISA combines the collecting and release of statistical data with policy advice in a single unit. In contrast, in the United States, strict Office of Management and Budget guidelines separate the release of federal statistical reports by time and place from any policy statements.

The pressure from policymakers for advice based on PISA interacts with this unhealthy mix of policy and technical people. The technical experts make sure that the appropriate caveats are noted, but the warnings are all too often ignored by the needs of the policy arm of PISA. As a result, PISA reports often list the known problems with the data, but then the policy advice flows as though those problems didn’t exist. Consequently, some have argued that PISA has become a vehicle for policy advocacy in which advice is built on flimsy data and flawed analysis. Much of the critical work has come from Germany, where PISA has had a profound effect. There has also been debate in Finland, in part because students do well in PISA but did relatively poorly on TIMSS on 1999.

Among the limitations are the fact that both PISA and TIMSS produce cross-sectional (single point in time) data and therefore do not allow longitudinal analysis (comparisons over time) at the student level, which is how researchers prefer to measure growth in student achievement and to identify the factors associated with such changes. But too often, countries and education ministers demand information and don’t want to know how cross-sectional data using flawed measures can lead to bad advice or advice that seems “right” but has little or no basis in fact.

To be sure, international data can in some cases be useful for addressing some policy questions, and often these data are the only way to examine the consequences of differences in policy that vary across countries. But the obstacles to drawing strong causal inferences based on such data are substantial. While both the International Association for the Evaluation of Educational Achievement (IEA) and the OECD work hard to make sure that measures are comparable, there are large variations in how people in different countries understand similar questions, and how statistical agencies measure and report indicators. This is especially true for the United States, where, as a federal system of continental size in which education is still predominantly a state and local function, variance is often more important than the mean national value calculated by many international studies. Unfortunately, both TIMSS and PISA data are used, too often with little regard for their limitations, to formulate policy recommendations.

It should be noted that NAEP has taken a different approach to releasing its results, one that PISA has rejected. While early NAEP reports resembled the current voluminous PISA reports, over time, the Nation’s Report Cards have become shorter and more tightly focused on data that show cross-sectional results by state and by reporting categories called for in U.S. legislation (for example, race or ethnicity and income). NAEP reports also compare the performance of students at the same grade level over time; these comparisons have become a critical tool for measuring the nation’s progress. In short, these are highly focused and valuable benchmarking reports. Further, a typical NAEP release is far more accessible to policymakers, reporters, and the general public than PISA reports are. And NAEP reports do not contain policy implications or recommendations: the dividing line between statistical reporting and policy prescription is honored.

Within guidelines designed to protect confidentiality, data from all three assessments are made widely available both online and through other avenues of dissemination, allowing researchers to explore statistical patterns and the policy implications of the data. However, with PISA, the OECD megaphone attracts the attention of policymakers and drowns out alternate interpretations.

Accountability Problem
OECD is a high-level intergovernmental organization with wide sweep across many domains, including finance, trade, environment, agriculture, technology, and taxation as well as education. Its pronouncements have a gravitas that cannot be matched by the IEA. Just as OECD does not have a modest presence in describing the scope of PISA, its voice in pushing the policy lessons that can be “learned” from PISA has been equally loud. But that loud voice has a few cracks in it.

For example, consider Chapter 5 of the 2006 PISA report, “Science Competencies for Tomorrow’s World,” which is devoted to analyzing the effects of school resources on student performance. Clearly, the stakes are high in identifying the best ways to allocate money, teachers, school leadership, and the like. Box 5.1 on page 215 notes all the appropriate caveats: PISA is cross-sectional; some things are not measured well; and other important factors are unmeasured. Moreover, it notes that the school characteristics measured are from the student’s current school, and in many countries a 15-year-old might have been in that school for only a year or two. The report warns that “these restrictions limit the ability of PISA to provide direct statistical estimates of the effects of school resources on educational outcomes.”

Nonetheless, the chapter proceeds with dozens of charts and tables relating different school resources to student outcomes. Finally, the chapter ends with “policy implications,” even though the foundations for these implications are weak. For example, the report reads, “what is noticeable about the strongest effects measured in this chapter is that they are not the ones most closely associated with finite material resources, such as the distribution of good teachers. Rather, such effects are related to how schools and the school system are run—for example, the amount of time that students spend in class and the extent to which schools are accountable for results.”

This is powerful stuff: hold schools accountable, and other issues (such as the allocation of good teachers) don’t matter. What is the evidence for PISA’s confident endorsement of accountability?

On close examination, “accountability” turns out to mean merely the public posting of school-level results. No other measures of accountability were tied to PISA scores at a level of statistical significance once students’ socioeconomic status was introduced. But the report uses the highly evocative word “accountable” rather than “posting.” The reader can easily have the mistaken impression that a broad set of conditions has been found to be important.

Standards of evidence and the research practices that are found in this analysis would not pass muster in the equivalent statistical agencies and among most researchers in the United States. Indeed, the National Center for Education Statistics (NCES) frequently objects to many of the findings in PISA reports, but is almost as often politely (and sometimes not politely) ignored.

Questions Remain
Despite these limits, the chorus for state-level assessments will continue to grow louder. But how would state-level international assessments fit into the already complex world of large-scale assessments? In the 2006 PISA assessment, the U.S. barely made the minimal school participation rate to be included in the analysis. If governors and chief state school officers are behind a state administration of PISA or TIMSS, getting school participation may be easier, but student engagement in low-stakes tests declines as students get older—likely a situation that even gubernatorial exhortations cannot resolve.

If it is true that what gets tested gets taught, are governors and chief state school officers really willing to allow OECD or IEA to drive their curricula? In the rush to employ rigorous international standards, we need to keep in mind that TIMSS and PISA each embody particular views about what those standards should be.

Can PISA really inform a state’s policymakers about how to improve their school system? PISA assesses mathematical and science “literacy,” a broad domain encompassing skills and knowledge learned both inside and outside of school. PISA 2006 cautions, “If a country’s scale scores in reading, scientific or mathematical literacy are significantly higher than those in another country, it cannot automatically be inferred that the schools or particular parts of the education system in the first country are more effective than those in the second. However, one can legitimately conclude that the cumulative impact of learning experiences in the first country, starting in early childhood and up to the age of 15 and embracing experiences both in school and at home, have resulted in higher outcomes in the literacy domains that PISA measures.” In other words, PISA itself admits that it cannot reliably identify which parts of the education pipeline are working well and which need improvement.

How expensive are these tests, and are there less costly ways to get these or similar data? To get a state-level score, about 1,500 students in the state need to be tested. The cost to test that number of students is around $700,000 for PISA and, based on the experience of Massachusetts, approximately $500,000 per grade for TIMSS. Nationwide, that’s around $25 million per PISA assessment and $15 million per grade in TIMSS if all states participated. There are less expensive alternatives, such as the TIMSS estimates for the states generated by Gary Phillips and noted earlier. While the alternatives would not produce all the details that might come from the full assessment (and mercifully avoid the temptation to use these data for unwarranted policy analysis), they could produce reliable estimates of state performance relative to international performance. Phillips has recently worked out the technical details for obtaining international benchmarks with PISA, by embedding PISA items within the state assessment. The PISA items on the state assessment are then used to link the state assessment to the PISA scale. Once this is done, the state can benchmark its performance against PISA without having students take the entire PISA assessment.

An alternative to statistical linking for PISA would be “small area estimation.” These estimates are model-based and “borrow” information from other data available for the state together with any state-level PISA data collected. The results are known as “indirect” projections to distinguish them from standard or “direct” estimates. NCES produced such state and country estimates of adults with low literacy based on the National Assessment of Adult Literacy (NAAL), for example. Further research would be needed to determine the feasibility of conducting small area estimates to generate state-level PISA scores. And it is likely that the PISA sample would have to be increased somewhat. This would add costs, but far less than full state-level assessments.

Double-Edged Sword
Compared to students in most other OECD countries, American students do not perform well on PISA. In the aftermath of the dismal results from the 2006 PISA, the clamor for producing state-level PISA scores was thunderous. State interest in administering PISA has since receded, with the current financial crisis and the fading of the fanfare surrounding PISA 2006, but the economy will eventually turn around, PISA 2009 is in the offing, and the Obama administration and many governors are sympathetic to the idea of international benchmarking.

While TIMSS has its partisans and states have chosen to participate in past years, momentum is behind PISA. If more states do choose to participate in PISA, what should they expect? First, they would get a PISA score that would allow them to compare themselves to other PISA participants. In some cases, this would provide bragging rights (“Our students are better than those in Turkey”). In most states, disappointing results would provide reform-minded governors with ammunition to push for their own legislative agenda. But if states choose a full state assessment, along with the PISA scale score would come all of the OECD’s policy advice and its approach to standards, which might make it harder for reform-minded governors to choose the options they prefer. Caveat emptor.

Mark Schneider is a vice president at American Institutes for Research and former commissioner of the National Center for Education Statistics.

“Experimenters separated seventh- and eighth-grade students into two groups—strong and weak readers as measured by standard reading tests,” Hirsch wrote. “The students in each group were subdivided according to their baseball knowledge. Then they were all given a reading test with passages about baseball.  Low-level readers with high baseball knowledge significantly outperformed strong readers with little background knowledge. Hirsch continued

The experiment confirmed what language researchers have long maintained: the key to comprehension is familiarity with the relevant subject. For a student with a basic ability to decode print, a reading-comprehension test is not chiefly a test of formal techniques but a test of background knowledge.

Dan Willingham has a nice summary of the implications in a recent blog post at the Washington Post:

We tend to teach comprehension as a series of ‘reading strategies’ that can be practiced and mastered. Unfortunately it really doesn’t work that way. The mainspring of comprehension is prior knowledge—the stuff readers already know that enables them to create understanding as they read.

Prior knowledge is vital to comprehension because writers omit information. For example, suppose you read ‘He just got a new puppy. His landlord is angry.’ You easily understand the logical connection between those sentences because you know things about puppies (they aren’t housebroken), carpets (urine stains them) and landlords (they are protective of their property.)

The writer could have included all that information. The writer gambled that the reader would know about puppies, carpets and landlords. A writer who doesn’t assume some prior knowledge on the part of her readers will write very boring prose.

What happens if the reader doesn’t have the prior knowledge the writer assumed she had? The reader will be confused and comprehension breaks down.

What this means is that even if a student has mastered “reading strategies” such as identifying the thesis and how it is argued, without any background knowledge in the subject matter, those strategies don’t carry the student very far. Readers without it can’t fill in blanks and pick up subtexts and discern points of view.

This is why, when the American Federation of Teachers issued its Setting Strong Standards report in 2003, it emphasized repeatedly the importance of knowledge in all the disciplines, including English. Some statements are worth repeating here:

—–”a set of standards must embody the knowledge essential to each of the core subjects, and this cannot be accomplished by trying to fit disciplinary knowledge into broad over-arching, non disciplinary categories such as ‘critical thinking’ and ‘problem solving.’”

—–”Strong standards must provide clear guidance to teachers, curriculum and assessment developers, textbook publishers, and others so that one person’s interpretation of the core knowledge and skills students should learn in a particular grade level or education level—elementary, middle, or high school—will be fairly similar to someone else’s.”

—–”English: The basic skills and knowledge that are the foundations of learning how to read (e.g., letter/sound recognition, decoding skills, vocabulary), reading comprehension (e.g., exposure to a variety of literary genres), writing conventions (e.g., spelling, writing mechanics), and writing forms (e.g., narrative, persuasive, expository).”

—–”It is not enough for standards to emphasize the skills students should learn but leave the content to local discretion.”

Hirsch, Willingham, and the AFT are powerful voices arguing against one of the sorriest trends in English Language Arts over the years, namely, the attempt to convert it into a skills discipline that emphasizes cross-disciplinary capacities (critical thinking, “media literacy,” reading comprehension strategies, etc.) and downplays English knowledge.

And what is “English knowledge”? Three things: philology (history and development of the English language); literary history (major periods and movements, major authors and works); criticism (theories of, approaches to, and great examples of interpretation).

In the committee room, however, those knowledges are fraught with tension. English literary history bears exclusions that people don’t like, philology asks for competences that people just don’t have, and criticism seems beyond the level of high school students.

But, counter Hirsch, Willingham, and AFT, without them, we can expect reading scores for high school students to continue as they have since NAEP started tracking them 40 years ago: a more or less flat line.

What is worse than sluggish NAEP scores is their combination with steady, substantial increases in per-student spending in public schools. Teachers unions have done well in improving the income and perks of their members, but not the achievement of our students, whose interests have been poorly represented by federal and state legislators and school board members. Only the East Asian countries have sustained high levels of learning at surprisingly low costs, which is substantially attributable to a large private school sector. And Sweden, the one economically advanced Western country with sharp achievement gains, has a national voucher system, which forces schools, including a growing number of educationally and financially successful for-profit schools, to compete for students. If socialist Sweden can substantially improve its school system via choice, why not the United States?

The trend is flat; it’s a plateau. Scores are not going anywhere, at least nowhere important. That means that eight years after enactment of No Child Left Behind, the problems it set out to solve are not being solved, and now we’re five years from the deadline and we’re still far, far from the goal.

But there is a silver lining for DC schools chancellor Michelle Rhee (portrayed as “DC’s Braveheart” in a new Education Next profile): her schools, and those in just four states, were the only ones to post gains in both fourth and eighth grades over the past two years. That should give her reform efforts a much-needed boost.

Update:  The NAEP results can be found here.

EDUCATION NEXT: Should the United States have a national curriculum?

Chester Finn: Absolutely, positively yes, provided that we properly define “curriculum,” and ensure that the states’ participation remains voluntary. In the core subjects of English, math, science, and history (including geography and civics, never say “social studies”), there is absolutely no reason why we ought not ask all young Americans to learn most of the same things while in the elementary and secondary grades. That doesn’t mean all teachers should follow identical lesson plans, that everybody needs to read the same poems and plays, or that a rigid “scope and sequence” should be clamped onto all schools and school systems. But the basic content of, say, 4th-grade English or 6th-grade math or 8th-grade science should be the same from Portland, Maine, to Portland, Oregon. And that content should be married to national standards of “proficiency” in these subjects at these grade levels, and joined to national exams by which we determine how well and by whom this is being accomplished.

The curriculum should cover grades K–12 and leave plenty of room for state, local, and building- and classroom-level variation and augmentation. Particularly in grades 11 and 12, it would make sense to offer (as high schools do today) some choice among courses in science, history, and English; one English class might focus on drama, another on creative writing. A charter or magnet school might specialize in art and music, while another concentrates on science and math, in addition to the academic core.

One way to picture the core is the “1,000 question” approach, which blends standards, curriculum, and assessment. Here’s a simple version: The testing body (perhaps a consortium of states, possibly a spin-off from the National Assessment of Educational Progress [NAEP]) would publish—this is all totally transparent—maybe 1,000 possible exam questions dealing with, say, 7th-grade science. A generous portion would be open-response and deep-thought queries that probe a student’s ability to make sense of what he or she is learning, not just parrot it back or fill in bubbles. The national end-of-course exam in 7th-grade science would consist of a subset of those questions. Any student able to answer all 1,000 would likely get a perfect score on the exam.

But 1,000 is obviously too many to drill students on, so effective teaching of 7th-grade science would cover most if not all of the subject matter spanned by those questions. The teacher would be free to cover it however she likes—any sequence, any course structure, any instructional materials. If the state or school system or charter school wants to systematize this (and assist its teachers) by setting forth a scope and sequence, textbooks, units, midcourse assessments, and such, that’s fine, too.

Obviously, the testing body needs to ensure that there’s a logical, sequential relationship between the 7th-grade science questions and the 8th-grade questions and so forth. Indeed, the questions would surely overlap in part—and would cumulate, over the 13 grades, to a solid science education.

That’s pretty much the way the best extant national curriculum works, at least through grade 8. Of course I’m thinking of the Core Knowledge Curriculum developed by University of Virginia professor and Cultural Literacy author E. D. Hirsch. (Alas, it doesn’t yet include the high school grades.) Hirsch says it’s supposed to occupy roughly half of the school day. That feels about right to me. Maybe even two-thirds.

The national curriculum would cover only content, not pedagogy or instruction. For that we depend on professionals, and we assume and expect that they will differ from one another in their skills, enthusiasms, preferences, and values. One school might rely heavily on “virtual” instruction, for example, making extensive use of Internet offerings and opportunities. Others may team teach several subjects via two or three teachers who like working together and whose subjects lend themselves to blending. Still others will resemble the traditional self-contained-classroom schools of yesteryear. This is as it should be. The United States has some 54 million schoolkids, 3 million teachers, and 100,000 schools. They differ in many ways and ought to. But today the absence of a common core is a critical handicap, particularly for the neediest kids, weakest teachers, and least advantaged schools. Equity demands that we rectify this.

Most successful modern nations have something akin to a national curriculum, whether explicitly or through their exam systems. Japan has “national curriculum standards” and insists that individual schools use them as the basis for planning what they will actually teach and how they teach it. Singapore publishes syllabi for each major course or subject, usually divided between primary and secondary. England spells it all out in considerable detail, and France famously standardizes even its lesson plans. To my knowledge, no two nations do this in quite the same way—and some of the other “federal” countries, such as Australia and Canada, are still working on how to do it at all. In Canada, for example, several provincial education ministries have voluntarily joined together to develop “pan-Canadian” curricula, starting with science.

Rather than starting with a federal mandate, a consortium of states and private organizations (such as some combination of Achieve, the state school “chiefs,” and the governors) could develop the curricula and tests, ideally with initial support from major national foundations. States would then be free to join if they like. Possibly we’ll wind up with more than one consortium, and states would have choices among them. Picture the Southern Regional Education Board spearheading the second of these. Maybe the four small New England states that have already joined forces on testing will become the starting point for a third. (The Brits do something like this with their multiple testing bodies.) Uncle Sam’s role is to encourage movement in this direction, probably by giving states that join such consortia some breaks on No Child Left Behind (NCLB) and its successors, perhaps a bit more money, perhaps automatic approval of their standards and tests without further inspection or negotiation.

I don’t expect every state will join, at least not soon, so the federal government’s additional responsibility is to maintain NAEP as the external auditor of all states. We’ll find out over time whether kids in schools and states that join in the common curricula and exams do better (or worse) than in those that maintain their curricular independence.

Deborah Meier: I have five concerns with Chester Finn’s proposal for a national curriculum.

First, what’s (positively) special about the U.S.A. is that it doesn’t have an official line, above all, on ideological and intellectual matters. This is part of our unusual history and reflects tolerance for diverse origins and beliefs. It has always been a struggle, never quite won, but it is a strength that is always tempting for us to abandon. Doing so would be at a cost we would someday rue.

Second, there is no way in which a federally approved curriculum can avoid the trap of selection bias—no matter who might design it. Even if I were to design my ideal history curriculum, whatever I decided to spend more time on or (God forbid) omit altogether will be influenced by my biases. The sources I require my students to be familiar with; the differences of opinion I tolerate versus those I feel compelled to correct; how I “simplify” without losing the most important truths—all these are fraught with inevitable biases. What I believe everyone must know may be different from what you believe. Do we vote on “the truth” and then put a camera in each classroom to ensure it’s carried out?

I truly cannot imagine how supporters of a federally approved curriculum solve these issues. It’s not merely political bias, mind you, but academic and intellectual views that may or may not have political implications. Historians at my graduate school differed on whether history was a science or a field of humanities, for example. So they offered both! Not an insignificant difference of opinion. Biologists and physicists may have different views about which science is more critical, and within each field there are controversies about the nature of science and which scientific ideas are more important.

A panel of righteous and well-educated people is not an answer. So, you might ask, are multiple bodies of righteous and well-educated people any better? Yes, because it leaves the door open for more controversy; offers escape hatches for unexpected views; and leaves contenders, alternatives, and authority in many hands.

Third, attempting to avoid bias by including everyone’s biases only generates more problems. Precisely in order to avoid charges of bias, the tendency of textbooks (and curricula and tests) is already to include snippets of all viewpoints, thus becoming long-winded and boring. The effort of the national science community a decade or so ago to outline what every 18-year-old should know about science was so extensive that it invited either rote memorization (in defiance of the heart of science instruction being recommended) or studying nothing but science in order to cover it all. The science teachers at my old high school admitted that they were only secure in their knowledge of one or at most two of the fields covered. What part of the fascinating study of mathematics is a “must” for 18-year-olds? What knowledge of music or art?

The focus on testing also has an interesting side effect: it makes it hard for wise educators to take advantage of the teachable moment in their concern to stick with the stuff that will appear on the test. For example, teachers should be able to use the recent election as a moment for understanding our political system or the financial crisis to examine how money and finance works.

My fourth problem is that any curriculum leads—as Finn acknowledges—to assessment issues. My colleague Diane Ravitch suggests we decouple the two ideas. I think, as Finn does, that the one inevitably follows the other. At that point the best intentions of a good curriculum come screeching to a halt. In reality, the test becomes the curriculum, and the scoring guide for the test becomes the bible.

Of course, we can do our best to develop tests that are more nuanced, that require strong written and oral exposition, opportunities to defend one’s ideas, to think critically and persuasively, etc. But it’s highly unlikely, almost utopian, to imagine we could do it on a national scale, and far more likely are precisely the kinds of assessment tools that undermine a strong education.

But my greatest concern is none of the above!

I’m concerned that all of this is a way to avoid a real conversation about the purposes of public education and then to acknowledge our ignorance about “ensuring” success. Our own children are worth more than money can buy, but no parent can offer a guarantee.

Whether the first discussion might ever lead to a substantial consensus I don’t know. What math must we “all” know, and why? Like music, mathematics is a subject of beauty, as well as a practical study of import. But which aspects of math must we all—as citizens—have at our fingertips regardless of our vocational goals? In this debate not only experts in math must have a voice.

So, too, with debating what literature is indispensable. How tempting it is to add a little bit of everything to please all camps rather than engaging with a few works in great depth. The development of a “taste” for literature—fiction and nonfiction alike—is hardly something we’re good at teaching, Not to mention the dilemma about how to teach literacy of the new media that will constitute the bulk of the next generation’s “reading.”

Perhaps we can reach agreement that one purpose stands apart from the rest—that the indispensable core purpose of a public education system is that it prepares people for public life in a democracy, with all that this implies. But even that would be far from settling matters. How we define democracy, and what constitutes the intellectual underpinnings of a democracy are open to endless discourse. But it’s the “litmus” test.

I also know that the second question—how to make it work—is equally knotty and that no one has a monopoly on the right answers.

EN: What should be taught, in your view, and how will educators figure out effective ways to do this?

CF: If I were king, I’d probably install Core Knowledge in the primary and middle grades and the International Baccalaureate (IB) in high schools. I don’t think it’s any coincidence that the most highly respected high school courses in America today are Advanced Placement (AP) and IB courses, which have quite a lot of nationwide prescription as to their content. (It’s true that AP shuns a prescribed “syllabus,” but veteran AP teachers are clear as to what they must cover in order to prepare their pupils for those exams—and the College Board isn’t shy about clueing in new teachers.)

With these standards and assessments in place, the question reasonably arises, where do educators find the curricular materials that best help them tackle the standards? Some will develop their own or pull them off the Internet. I think we can be confident that major publishing companies will develop and market commercial versions. I’d favor staging a competition among prospective curriculum suppliers, maybe have a jury evaluate and grade their products. Perhaps then we could make all their products available to states, districts, and schools, and let the market select among them. Wikipedia-style (or Zagat-style) open-source rating systems will enable product users to rate and comment on what works best in what circumstances. Having a national curriculum doesn’t mean we need confine ourselves to just one option.

Textbook publishers (and their modern-day successors, such as virtual-curriculum developers) will align their products with the national standards rather than with the whims of California and Texas. (That assumes California and Texas join the multistate ventures, of course.) The total amount of testing should diminish and, if it doesn’t, it will have to be better aligned to the end-of-course expectations and exams that states will administer. Commercial tests such as the Stanford and Iowa may evolve into something more like formative assessments meant to assist teachers rather than be used for external accountability.

Teacher preparers and professional developers, and those who try to set standards for them (e.g., National Council for Accreditation of Teacher Education, National Board for Professional Teaching Standards), will need to take seriously the obligation to align their expectations for instructors with the common expectations for students. All this is mostly good—and, yes, a little bit risky, if the national standards go squishy or the national curriculum falls into the hands of zealots. That’s why it needs to stay voluntary, so any jurisdiction that can’t abide it need not stick with it.

A big grown-up country in the 21st century needs common (and ambitious!) curricular standards for all its children, at least in core subjects, and it needs common assessments, too. If we’ve learned anything from the NCLB experience (and its antecedent “Goals 2000” and “Improving America’s Schools” legislation), it’s that having these things vary from state to state produces mediocrity, cacophony, waste, duplication, and confusion (see Figure 1). Survey after survey makes clear that (if the question is asked correctly) parents favor national standards and tests. Instead of letting “That’s the first step toward a national curriculum” serve as a conversation stopper, let’s deploy it as a conversation starter. Let’s acknowledge that “curriculum,” loosely defined, is supposed to be aligned with standards and appraised by assessments.

Let me note, finally, that I’m unimpressed by Meier’s “habits of mind” alternative to content (see below). It’s wonderfully seductive, but the serious psychologists with whose work I’m acquainted (see, for example, “Reframing the Mind,” check the facts, Summer 2004) don’t put much stock in this Howard Gardner–originated proposition that youngsters can learn skills devoid of content. It’s the absence of essential core content from her view of schooling that lies at the heart of our curricular disagreement.

EN: What other options are there for bringing our nation’s public education system to a higher level?

DM: At the schools I led for nearly 40 years, as part of the work of the Coalition of Essential Schools, we spent a lot of time exploring the “why” questions and developing an approach that was aimed at answering them. This discussion was at the heart of the school’s existence and included all parties to it. Like Coalition founder Ted Sizer, we figured if we could grab hold of that, we’d see how much else we could teach and, more importantly, how everything we taught and did helped to reinforce “the essentials,” influencing not only our students’ hours in school (or doing homework), but every waking hour of their lives. We even saw misbehavior as an opening, an opportunity to teach such habits and not an obstacle to it.

We boiled it down to five “habits of mind” that we claimed (somewhat pompously) underlay all the academic disciplines as well as the mental and social disciplines needed for living in a complex modern society: (1) How do you know what you know? What’s the nature of your evidence? How credible is it? Compared to what? (2) Are there other perspectives? What affects our points of view? How otherwise might this be seen? (3) Are there patterns there? A sequence? A theory of cause and effect? (4) Could it be otherwise? What would happen if? Supposing that x had not happened? and (5) Who cares? Why does it matter? As you can see, they blend into each other and, in a way, just define a mind state of skepticism and informed empathy. It suggests having to take seriously the idea that one might be wrong, and so could others. We added “habits of work” like meeting deadlines and being on time and “habits of the heart” like caring about one’s impact on others.

We developed rubrics that spelled out specific formats in which students could demonstrate their proficiency in each discipline. The diploma from our high school rested on convincing an internal and external evaluation committee that the student met the standards set by the faculty. Students’ oral presentations and defenses were based on written essays and other performances in each of the major disciplines as well as subjects of the student’s and faculty’s choice.

Could the five habits of mind become a national curriculum? Democratic habits of the sort we laid out at Central Park East can be taught in the process of learning math with its powerful logical habits, its attentiveness to patterns, as well as its multiple approaches to getting “right answers.” They can be taught in science, with its scrupulous attention to detail, specificity, and evidence, not to mention its humility in the face of the unknown. They can be taught in literature through our capacity to empathize with otherwise unacceptable protagonists, connecting us to people and worlds we otherwise would or could never choose. They can be taught in the way one handles discipline!

Isn’t democratic culture best served if all citizens are accustomed to such habitual ways of thinking, not just knowing how to do various things? I know how to do a lot of things—like putting my keys in the right compartment in my purse—that I don’t practice, especially in times of stress. What would it mean to teach so well that we’d hang on to such “habits of mind” in times of stress? Are our five a fair representation of what democratic intellectual habits amount to? Fair questions.

A school community that holds itself to high standards must risk such everlasting debate among, at the very least, the adults in charge and ideally all members of the community. But nothing I’ve said works if it’s simply adopted to try to “cover” the likely contents of a test with which ordinary teachers, families, and students cannot argue or differ. The habit of mind of “supposing that” is best learned from adults who are in a position to choose, revise, and rethink their own viewpoints in the presence of the young.

Andreas Schleicher recently blogged in response to my Ed Next article, “The International PISA Test,” but his response fails to deal with important issues I raised in my article: the quality of PISA’s analysis and the degree to which PISA’s reports ignore the limits of the data in support of conclusions that seem to have been determined in advance of the analysis. Another issue I raised is how PISA should fit into the U.S. system of testing and data collection. The United States already spends well over $130 million each year on NAEP. Moreover, after this year’s State Longitudinal Data System grants are announced, the federal government will have invested approximately $400 million in student-based longitudinal data—an investment that does not include the large amounts of money invested by the states in developing these systems.

How PISA fits into our national testing, data, and research landscape is a serious policy issue, and Mr. Schleicher’s superficial comparison of the United States to countries that have not made similar investments is singularly unhelpful. In addition, while Mr. Schleicher implies that I am a luddite for refusing to accept all the technological advances that he advocates, I should note that the last part of my article explored new ways that states could obtain state-level PISA results without spending additional millions of dollars.

I am quite surprised by the ad hominem nature of Mr. Schleicher’s remarks. It was my honor to serve the American people, and I am proud of what I accomplished as the Commissioner of Education Statistics. But I was only a part of the policy process in the U.S. Department of Education. The positions I took regarding PISA represented the decisions of the United States Government and reflected budget allocations and other policies decided by the U.S. Congress. During the years I served, Mr. Schleicher often tried to change U. S. policies, and he engaged in such egregious lobbying behavior that leaders in the Department of Education complained about him to the State Department, to the U.S. Ambassador to OECD, and to Mr. Schleicher’s supervisors in the OECD.

Mr. Schleicher perhaps too closely identifies with PISA to realize that he works for an international membership organization and that nations have their own interests in how they approach OECD services such as PISA. I believe that Mr. Schleicher’s emotional response to my criticism has led him astray on several other points. Of course, I know that nations can request individual reports that produce specific policy recommendations and that these reports are not produced by PISA, but by another unit of OECD. But this distinction may have more meaning to Mr. Schleicher than to readers of the PISA reports. These reports are filled with policy implications, and by billing them as “implications” rather than “recommendations” PISA is free to offer advice based on weak evidence, avoiding the more intense scrutiny that a country report would get. If the “policy implications” about accountability in the PISA report had been presented to the United States in a country report, we would have rejected them out of hand as not evidence-based. Indeed, the U.S. has frequently tried to limit some of the more fanciful conclusions of PISA reports, but often without success.

I should also note that Mr. Schleicher is so sure that he knows best about PISA that not only do the conclusions of PISA reports often seem predetermined, he also seems to know the policy decisions of the PISA Governing Board even before they do: At one PISA Governing Board meeting, Mr. Schleicher made the mistake of showing his summary, conclusions, and recommendations of the Board’s meeting before it had even reached the halfway point.

Mr. Schleicher also notes that the U.S. had “to withdraw key outcomes from PISA 2006 because of technical flaws by Schneider’s subcontractors.” First, I should note that these were not my subcontractors—contracts are awarded by the United States government based on a rigorous review process and governed by detailed contract law. Further (and fortunately) the results that were withdrawn were for a minor domain and the U.S. did report its results for science, the major subject tested that year, and for mathematics. Mr. Schleicher states that “much of the rest of the world was simply bewildered.” I was on the PISA Governing Board when this error occurred, and the only feedback I got was sympathy from the other members, not bewilderment. Many other countries have run into problems in administering this complex assessment (in fact, in 2003, the United Kingdom could not report any results). Moreover, I forced the contractor, RTI International, to reimburse the U.S. government for the error. I should note that the other alternative would have been to intentionally release flawed data, which would have been contrary to everything that NCES stands for.

Finally, despite what Mr. Schleicher infers, I am not opposed to PISA. Indeed, I was the highest-ranking U.S. government official ever to serve on the PISA Governing Board—and I did not serve on that board to sabotage PISA, but to make sure that U.S. interests were better represented.

There will be continued evolution in how PISA will be used in the United States. And, as in the past, the United States might choose to participate in some parts of PISA and not in others. But those decisions will (and rightfully should) be governed by the needs of the United States, as determined by American policymakers, and made in light of scarce resources.

Mr. Schleicher’s clear pique at what he sees as unenlightened policy decisions and the ad hominem nature of his response show that he cannot clearly distinguish between serious policy discussions and anything less than the full adoration of PISA.

In an article in the Fall 2009 issue of Education Next, “The International PISA Test,” Mark Schneider argues that American states ought to think twice before participating in the PISA exam and that the policy advice offered in connection with PISA is not based on solid research.

If Mark Schneider has doubts about the usefulness of PISA, he should wonder whether the United States has, under his leadership, used PISA effectively. Among the G8 economies the U.S. assessed the second smallest number of students for PISA and collected the least contextual information, limiting the inferences that can be drawn for states and narrowing the usefulness of PISA for policy. While much of the industrialized world has opted to extend PISA towards interactive electronic tests, the U.S. stuck to paper-and-pencil versions. While Schneider rightly notes that longitudinal studies are needed to establish causality, countries like Australia, Canada or Denmark are already implementing them, keeping track of the students assessed in PISA to find out how their knowledge and skills shape their subsequent life opportunities.

In virtually every other federal nation, whether Canada or Mexico in North America; Belgium, Germany, Italy, Spain, Switzerland or the United Kingdom in Europe; or Australia in the Pacific, individual states, provinces, and regions have implemented PISA successfully, are using it effectively, and have found it useful for policy formation. They recognize that the yardstick for educational success is no longer improvement by state and national standards alone, but the best-performing education systems internationally. States, like the nations that now make up almost 90 percent of the world economy, are getting a number of benefits from PISA:

*            an internationally benchmarked assessment of the overall performance of their student populations in key school subjects

*            indicators of the proportion of their students who reach global standards of excellence as well as the proportion of their students failing to reach baseline performance standards

*            indicators of equity-related performance, in terms of the extent to which student and school performance is influenced by the socioeconomic context of students and schools

*            indicators of coherence in educational standards, in terms of the extent to which high standards are consistently achieved throughout the system

Schneider worries that international assessments, like any assessment, embody judgments about what should be measured and that such judgments may differ from those judgments being made at the school, state, or federal levels. While those countries that have studied this empirically have generally found PISA well aligned with their national goals and standards, much of the value of international assessments lies precisely in allowing states to see their own standards through the prism of the judgments that the principal industrialized countries make collectively as to what knowledge and skills matter for the success of individuals in a global economy. The decision to have PISA go beyond the reproduction of subject-matter content and examine to what extent students can extrapolate from what they have learned and apply their knowledge in novel situations was made deliberately by the governments of the principal industrialized nations, because that is what is at a premium in modern economies. No nation uses that perspective to replace its national curriculum, but many states and nations now use PISA to benchmark their own standards. When a nation discovers that its students are unable to do things that students in other countries can do, knowing whether this better success was because students in other countries learned those things in school or out of school is secondary only. The crucial question is, do our students need these things too, to be able to succeed? If the answer is yes, then it is worthwhile to have a serious look at state standards and assessments, to improve them in case these things are covered but not learned, or to include them if they are not covered.

Do international assessments provide causal evidence on what makes school systems succeed? No, they don’t and they do not claim to. But they can shed light on important features in which education systems show similarities and differences and, by making those features visible, they can help to ask the right questions. They can show what is possible in education, in terms of the quality and equity in outcomes achieved by the best-performing education systems, and they can foster better understanding of how different education systems address similar problems. They can also assist with gauging the pace of progress and help review the reality of education delivery at the frontline. Are the contextual data currently used for this perfect? Certainly not, and OECD nations therefore constantly review and refine them. The methods used for this are, however, far more robust than the ways in which Schneider’s organization is patching together data from U.S. states and international assessments to suggest to states that they can pass over a process of thorough international benchmarking.

And, yes, on the request of individual states and nations the OECD does carry out peer reviews for these nations that can result in policy recommendations. Several nations have found this a useful way to translate experiences from other nations into their own policy context. However, different from what Schneider claims, that is done through an entirely different organizational structure and governance arrangement within the OECD than the one that manages PISA and collects the data, and the results are collectively reviewed by OECD member nations before they are finalized and published.

The United States was the driving force behind the ideas that shaped PISA and, together with Japan, chaired the governing board of member countries that manages the program. The fact that the United States has lost much of its intellectual leadership in the field of comparative assessment may be because other countries have shown growing interest and are doing better, but may also reflect the fact that the U.S. has reduced comparative assessments to a stale horse race. When the U.S. had to withdraw key outcomes from PISA 2006 because of technical flaws by Schneider’s subcontractors, much of the rest of the world was simply bewildered.

Update: Mark Schneider has written a response to this blog entry, which can be found here.

“Crossing the Finish Line …demonstrates the high value of information locked away in administrative databases, and suggests new and potentially powerful approaches to increasing the nation’s population of college-educated citizens,” Whitehurst writes.

Four areas that will have to be part of our approach to increasing college graduation rates, according to Whitehurst: improving elementary and middle school preparation, making college more affordable, reducing leaks in the pipeline to graduation, and freeing up information about institutional performance.

Whitehurst is not persuaded that reducing “undermatch”—that is, reducing the number of students who attend less selective colleges, when they have high school GPAs and test scores that would qualify them to attend more selective colleges—will boost college graduation rates. He calls for more research on this matter, suspecting that students who choose to attend less selective colleges might be different from students who choose to attend more selective colleges.

[Earlier, Education Next interviewed Matt Chingos, one of the authors of Crossing the Finish Line. The video is available here.]