We Know the Answer, But What Is the Question?

By 01/28/2013

6 Comments | Print | NO PDF |

For almost 50 years, the United States and a number of other countries have periodically participated in international math and science assessments.  Until quite recently, little attention has been given to the fact that U.S. students have never performed very well on these tests.  With, however, attention from Secretary Duncan and others, awareness has been elevated, leading to broader discussions not only of how to interpret the apparently mediocre scores of U.S. students but also of what to do about them.

Martin Carnoy and Richard Rothstein, in a recent report for the Economic Policy Institute, now tell us that performance is not as bad as you think and that Secretary Duncan should stop making “exaggerated and misleading statements” about the performance of U.S. students.

To arrive at this conclusion and the accompanying one-liner for media consumption, Carnoy and Rothstein begin with the fact that U.S. students are disproportionately disadvantaged when compared to students in a sample of high performing countries (Canada, Finland, and Korea) and in a sample of post-industrial countries (France, Germany, and the United Kingdom).  They then re-weight average PISA test scores in the U.S. and these six countries by the distribution of socio-economic status (measured in their analysis by numbers of books in the home) in an attempt to equalize statistically the distribution of family backgrounds across countries.

Adjusted for books in the home, U.S. students in math still lag behind Canadian, Finnish, Korean, and German students but pull even with those in the U.K. and come close to those in France.  They then repeatedly work to convince us that everything is not as bad as thought and may even be pretty good.

But what is the question that these calculations answer?  The reason that Secretary Duncan and others, including me, are concerned about the performance of U.S. students is that the international achievement scores in math say a great deal about the skills that our students will take to the labor force.  These human capital differences, according to historical data, bear a direct relationship to growth of the national economy.  And the economic implications of mediocre performance are enormous.

The Carnoy and Rothstein calculations simply do not deal with the relevant differences in skills across countries.  We have the population that we have – not the population of Finland.  So their adjustments cannot address the question of how well prepared our future labor force might be.  How well prepared we are depends on the skills of all of our population, not just those that statistically look like Finns.   In fact, as prior analysis by Paul Peterson, Ludger Woessmann, and me shows, looking at just our most advantaged socio-economic subgroup (children of college educated parents) does little to erase the deficits with other countries in advanced math and science performance, which is the preparation for science and engineering careers.

There is a hint – although largely unstated in the report’s 100 pages – that these calculations can be used to answer the question ‘how well are our schools doing?’  Specifically, they might absolve our schools of guilt, because it is the parents of these disadvantaged children that lead to much of the difference in scores, perhaps even all of the difference between scores with the U.K. students.

Even if their adjustment removed all of the measured international skill gap, why should we chide Secretary Duncan for pointing out that our students are less prepared than those of the other six sampled countries?

It might mean that our schools face more difficult educational challenges than found in the other countries, but surely this does not imply that we should “rest on our laurels” (such as they are).  First, most of the international gaps remain after standardization.  Even after adjustment, the U.S. remains at best in a three-way tie for fifth out of seven.  Second, we are past saying that there is nothing we can do to educate our racial and ethnic minorities and our economically disadvantaged; hundreds of schools have proven this wrong.  Third, this is the population that we have.

The U.S. does look somewhat better in comparison if we use international reading assessments.  I personally am skeptical about our ability to obtain valid and reliable reading comparisons across different languages.  It also appears that math and science assessments say more about skills valued in the labor market than reading assessments.  Yet, again, even after adjustment, we know from the international reading assessments that we have a long way to go to compete with the top countries.

Historically, when it comes to economic performance, the U.S. has remained the strongest economy in the world.  A variety of advantages have allowed us to overcome the shortcomings of skills that are exposed by these tests.  This historic economic strength reflects our earlier historical commitment to universal secondary school attainment; our strong and well-developed economic system; our secure property rights and free movement of labor and capital; our world’s best universities that can overcome some of the low entering skills;  and our use of skilled immigrants to bolster our innovative capacity.

Unfortunately, all of these advantages over our economic competitors are going away as many have made great strides in emulating and even surpassing these strengths of the U.S.  In the future, we will simply have to rely just on our skills if we are to sustain our current economic standing – and skills are what are measured by the international assessments.

In sum, we cannot paper over the fact that a large number of other countries have shown that it is possible to develop considerably higher skills in their youth than we are doing.  Our future depends on the skills of our population, and it is time to recognize that we are lagging.  Secretary Duncan is neither exaggerating nor misleading about this.

Eric Hanushek is a senior fellow at the Hoover Institution of Stanford University and a member of its Koret Task Force on K-12 Education.

Comment on this article
  • Karl Wheatley says:

    “But what is the question that these calculations answer?”

    Presumably, Dr. Hanushek is well aware that public school teachers have been bashed in the press and by think tanks for decades now, generally based on the claim that average test scores supposedly show that teachers are failing.

    To fairly compare the performance of different teachers (or nations) serving very different populations, Dr. Hanushek is also well aware that one must compare for powerful background characteristics that strongly influence student outcomes. That is why they are making an adjustment.

    Discussing how well are students are doing versus how well our teachers schools are performing (given the particular conditions of their work) are very different conversations. However, the media, pundits, and politicians generally conflate the two and blindly leap into assuming that average student achievement data answers what we want to know about student learning and what we want to know about school effectiveness. In general, such test score data is insufficient for intelligently judging either.

    You state …

    #1) “I personally am skeptical about our ability to obtain valid and reliable reading comparisons across different languages.”

    and then …

    #2) ” …we know from the international reading assessments that we have a long way to go to compete with the top countries.”

    If you believe #1, there is no logical way to claim #2.

    You say …

    “Second, we are past saying that there is nothing we can do to educate … our economically disadvantaged; hundreds of schools have proven this wrong.

    No, they haven’t. In fact, all those schools have proven is that it is possible to cherry pick the more able and motivated from among a low-SES district, pressure out those who can’t then keep up with an accelerated curriculum, and then claim you’ve solved the problem of educating the poor. No one anywhere, not KIPP nor anyone, has proven they know how to educate all kids in a poor urban district to a level commensurate with their wealthier peers. Even Geoffrey Canada booted out a whole bunch of kids. The task of educating everyone well is fundamentally more challenging than that of educating the easier to educate.

    We need to continue to do a better job of helping many students learn high-end STEM skills, along with languages, knowledge of cultures, design, etc. However, the vast majority of new jobs simply don’t require these skills, a recent report finds that roughly half of the American workforce is overqualified for their jobs, and there is no broad shortage of scientists or engineers in America.

    We can if we wish, dedicate American education to chasing the #1 ranking on international tests, but if you understand development and learning, you’ll understand why Howard Gardner called this “a fool’s errand.” In fact, if I were sent from a foreign nation to undermine America’s educational system, I could scarcely devise a better scheme than our current obsession with standardized test scores in a few subjects. It is no exaggeration to claim that our pursuit of higher test scores is one of the central obstacles to quality education.

  • phillip cantor (@phillipcantor) says:

    I completely agree with Mr. Wheatley.

    I’m a public school teacher in Chicago, and I have three children in CPS schools. We’re up to about 15 days of standardized testing in most schools. Even kindergarten teachers spend huge chunks of the school year testing students. Teachers and parents are starting to realize that we are testing our schools into oblivion. In my experience many of the tests are poorly designed and are neither valid nor reliable. They also cause a narrowing of curriculum and put a mind-numbing damper on creativity in the classroom.

    The point that Carnoy and Rothstein make is that the “failure” in our test scores is not one that our schools alone can solve. The point of controlling for poverty and level of parental education when comparing the US with other nations is to demonstrate that, overall, we’re doing a pretty good job of closing the gap between rich and poor (compared to other nations.) If we want to compete with other nations we should do something about the growing income inequality in the US. This does not absolve the Dept of Ed, state legislatures and local school boards from their responsibility to adequately fund and support schools and students – especially those who must overcome obstacles that our society has put in their way.

    Using testing as the main measure of educational quality will damage schooling and further separate rich from poor. The emphasis on schools competing for students through so called “choice” markets causing greater stratification and segregation in our schools… is this the way to become more like Finland or Canada in terms of education? Let’s stop pretending that we can use average scores on international tests to measure our success or failure without looking at the deeper issues that are causing the real problems with this nation.

  • stats geek says:

    This is a straw man and a disappointing straw man at that. The important point of Carnoy and Rothstein is that the US is comparing their lowest achieving students to higher achieving cohorts in other countries. That fact that are lowest achieving students are achieving so well is simply shocking.

    However, if our lowest achieving students are not competing well against other nations, we can be assured that these students will not be overqualified.

    Kudos for a better understanding of the consequences of high stakes test. Hopefully Hanushek’s will you use his cachet among ultra conservative politicians to redirect their misguided ideology.

  • Toady says:

    Perhaps the problem isn’t with the school (or even, eh, the wealthynot paying their fair share) but with the home environment?

    “In their study of family-based expectations for Caucasian, Asian-American, African-American and Hispanic adolescents, Steinberg, Dornbusch, and Brown (1992) found strong ethnic and cultural differences related to students’ beliefs about the consequences of negative school failure. Asian-American students overwhelmingly believed a bad education would have negative effects on finding a good job, and African-American and Hispanic students predicted few negative consequences of a bad education. Although students across all ethnic groups reported that their parents valued education, African-American and Hispanic students “devoted less time to homework, perceived their parents as having lower standards, and were less likely to believe academic success comes from working hard”

    International tests
    Evidence that US schools are doing a decent job with the non-Black & non-Hispanic subset of the population.

    Quote: “However, the picture changes when low achievers in the U.S. are broken out by race. White and Asian students in the United States are generally among the best-performing pupils in the world; black and Hispanic students in the U.S. have very high rates of low achievement.”

  • John Merrifield says:

    Subjective accountability to schooling customers, partially informed by test scores, would produce better accountability than the current test score driven, supposedly high stakes, top-down accountability of educators to public officials. Choice and genuine choices are needed to empower schooling customers.

    That said, a critical issue, is that absolute skills matter more than how the US outcomes compare to other countries. Dr. Hanushek could have mentioned another of his critical findings to help make this point. He found that replacing the 5-8% worst teachers with average teachers would vault the US outcomes to among the best. If we can leave a myriad of inefficiencies in place – huge hindrances – and still become world class, then world class is still pretty bad. That’s important!!

    We can learn from other school systems (probably many do some things better than we do), even from systems with overall results worse than ours, but trying to find a current system to copy, or to get excited about being better, but still pretty bad, is a fool’s errand.

  • […] city kids like those other countries that outperform us (China, she’s looking at you). Actually, many countries that have large minority populations and educate everyone do it better than the U.S., including […]

  • Comment on this Article

    Name ()


    Sponsored Results

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform