No Matter How Hard You Try, You Cannot Deny U. S. Math Performance is Terrible

By 01/12/2011

9 Comments | Print | NO PDF |

A few weeks ago, I, together with Eric Hanushek and Ludger Woessmann, issued a report showing that the United States ranked 31st in the world at bringing 15 year olds up to an advanced level of math achievement.  Since the study caught the attention of the mainstream media (including the Atlantic, NBC’s Nightly News, and the Boston Globe), it could hardly expect to escape scrutiny.

Still, I had not expected a critic to characterize our work as “deceptive,” the adjective chosen by University of Georgia’s Jeremy Kilpatrick.  We are further charged with “inaccurately exaggerat[ing] small differences” and engaging in other “misleading practice[s].” Even though Kilpatrick writes under the auspices of a union-financed policy center, we were surprised by the tone of Kilpatrick’s claims, as we had taken pains to make our assumptions and methodology fully transparent, as is evident from the fact that our critic spends much of his time and energy iterating points we had addressed in the report.

He also faults us for failing to make policy recommendations, a strange critique of a scientific paper. None of the authors shrink from the responsibility of recommending needed policies when the context is appropriate. But to have added them to this report would have been a distraction from its basic purpose: Providing accurate, comparative information on the percentages of U. S. students performing at the highest level.

An odd criticism is highlighted in Kilpatrick’s summary. He objects to the stringency of the definition of “advanced” upon which our report relies on the grounds that it shows that only 6 percent of U. S. eighth graders reach the advanced level.  To set the standard, we relied upon the judgments of the National Assessment Governing Board, which administers the National Assessment of Educational Progress (NAEP),  as well as an international testing agency, Trends in Mathematics and Science Study (TIMSS),  whose work Kilpatrick applauds.

Our standard is not set so high that it keeps 15 percent of all Canadians, 17 percent of all Dutch students, and 23 percent of Korean students from reaching it. But in the end, the key fact is not the standard’s stringency as that a smaller proportion of American students meet that standard than do the students in 30 other countries.  Nor would results change much if you shifted the standard downward, as the average performance of U. S. students also lags behind almost all of these same countries.

Later in the report, Kilpatrick argues at great length that the PISA and NAEP math tests measure different things, making results utterly incomparable. But the performance of countries on the PISA math test correlates at the 0.93 level with TIMSS, the very international math test Kilpatrick prefers and one that he and others say is similar to NAEP.  To say the different tests are incomparable strikes us as somewhere between implausible and daft.

But for the sake of argument, let’s concede Kilpatrick’s point.  Let’s assume you cannot place each U. S. state on the international scale by linking the NAEP to the PISA, as we have done.  Let’s assume the implausible, that Kilpatrick, on this point, is dead right, and we are dead wrong.  Where are we then?  We still can compare every state with every other state within the United States and come up with the same ranking that shows Massachusetts and Minnesota first and second and Mississippi in last place.  No other state’s ranking vis a vis one another changes by so much as a rabbit’s hair.

And the rank of the United States as a whole relative to other countries remains stuck at 31, as that ranking depends entirely on U. S. performance on the same PISA test taken at roughly the same time as by students in all the other participating countries.

To deny that the U. S. ranks number 31 in math performance requires the claim that PISA is a “deceptive”, “misleading” test that “exaggerates small differences” something Kilpatrick wants to imply but never dares to argue, as it would run in the face of PISA’s adoption by educational ministries throughout the industrial world, including the U. S. Department of Education.

Kilpatrick’s has a hodge-podge of trivial technical concerns:

1. He points out that the international students who took the PISA were not always from the same graduating class as the U. S. students taking the National Assessment of Educational Progress (NAEP).  As we stated in our report, that is significant only if math performance fluctuates noticeably from one student cohort to the next, which Kilpatrick fails to show. On the contrary, the percentage of U. S 8th graders performing at the advanced level changes only very modestly over time.

2. Kilpatrick objects that scores of 8th graders do not prove anything about performance of students in high school. A point well taken, it cuts in exactly the opposite direction than Kilpatrick would like to take the reader.  The performance of older U. S. students on the NAEP is even more discouraging than the performance of 8th graders.  Our report very likely under-estimates—it does not over-estimate–the math performance of U. S. high school graduates.

3. Kilpatrick argues—without offering evidence–that PISA 2006 math results may be imprecise for some countries. We provide tests of statistical significance that indicate the extent to which one can have confidence in any specific comparison.

-Paul E. Peterson

Comment on this article
  • Anne Clark says:

    The most important recent development in math education in the US, arguably, is the adoption of the Common Core standards by most states. Even though I do not wholeheartedly agree with either the placement of some topics within particular grades, nor the overall pedagogy embedded in many of them, the Common Core adoption presents an enormous opportunity for improvement in mathematics achievement, if it is seized.

    I would love to get more detailed information than I have found to-date on specific issues related to implementation. For instance, many districts in New Jersey, and across the US, use Everyday Math. If someone has done an analysis of the match between Everyday Math textbooks and the Common Core, it would be great for it to be posted on the web. Same thing for other textbook series – enVision, HSP, Investigations,…

    Of course, there is Common Core opponent Eric Milou’s analysis here: But it is hard to take such an analysis seriously when he is such an opponent of CCSS. It is amazing that NCTM used Dr. Milou to create it’s powerpoint presentation about CCSS given the derogatory comments he makes. Seems to me the NCTM is trying to maintain the relevance of its Focal Points to sell more PD and materials.

    Is there a place on the web that NGA or CCSSO or Hunt or anyone else who does not have a commercial conflict is gathering best practices on CCSS adoption?

  • Mike in Texas says:

    No wonder they criticized you; the NAEP scores were deemed to be invalid and unreliable more than a decade ago by both the General Accounting Office AND the National Academy of Sciences.

  • Barry Garelick says:

    Dr. Kilpatrick in the past has weighed his words carefully so that it was difficult to tell whether he was for or against fuzzy math. His criticism here makes it a bit more obvious what side he’s on. He was also involved somewhat with the Connected Math Program (CMP) a notably constructivist y math middle school math program which was funded by NSF-EHR. Georgia itself is imbued with constructivist math. The link below is to a video that shows a “discovery learning” approach to math. (The problem being discussed is from CMP).

    Given the direction that ed schools and others have pushed the teaching of math, is it any wonder why the US performs poorly in that subjcct?

  • Ze'ev Wurman says:

    To Anne Clark,

    Here is the critique of Everyday Math authors of the March 2010 version of the Common Core Standards: . Clearly, as it is, Everyday Math is not aligned with Common Core by their own admission.

    The Standards became slightly more rigorous between March and their final issue in June, which makes claims of certain schools and teachers found on the web, that Everyday Math is “aligned” with Common Core, even more ridiculous.

    On a separate note, I find Dr. Milou’s critique generally sound and well done. Further, he is not alone criticizing it — check this report and particularly its appendix for a scathing critique from a Stanford mathematician and a member of Common Core’s own validation panel.

  • Barry Garelick says:


    Everyday Math criticized the Common Core standards for emphasizing paper and pencil “traditional” algorithms, among other things. The June version, interestingly, seemed to bend to EM’s wishes and put off teaching standards multiplication algorithms, from 2nd grade to 4th grade, and long division in 6th grade. So they got a little of what the moaned about; whether it’s aligned is another matter, but EM’s marketing experts are very good at what they do. Perception is reality, after all.

  • Alan Cook says:

    National math test scores continue to be disappointing. This poor trend persists in spite of new texts, standardized tests with attached implied threats, or laptops in the class. At some point, maybe we should admit that math, as it is taught currently and in the recent past, seems irrelevant to a large percentage of grade school kids.

    Why blame a sixth grade student or teacher trapped by meaningless lessons? Teachers are frustrated. Students check out.

    The missing element is reality. Instead of insisting that students learn another sixteen formulae, we need to involve them in tangible life projects. And the task must be interesting.

    A Trip To The Number Yard is a math book focusing on the building of a bungalow. Odd numbered chapters cover the phases of the project: lot layout, foundation, framing, all the way through until the trim out. The even numbered chapters introduce the math needed for the next stage of building and/or reviews the previous lessons.

    This type of project-oriented math engages kids. It is fun. They have a reason to learn the math they may have ignored in the standard lecture format of a class room.

    If we really want kids to learn math and to have the lessons be valuable, we need to change the mode of teaching. Our kids can master the math that most adults need. We can’t continue to have class rooms full of math drudges. Instead, we need to change our teaching tactics with real life projects.

    Alan Cook

  • Jacob says:

    As a long time Bungalow builder, I can attest to this. Every year it gets harder and harder to find qualified bungalowers to build my bungalows. Why, just last week I had to help a new bungalower expand the Maclaurin series for the natural log x over sin function! How do we expect our nation to continue building the quality bungalows that we’re known for if our young people can’t derive Taylor’s theorem or an infinite sum?

  • Concerned Mom says:

    You can find a complete comparison of Everyday Math vs. Massachusetts Frameworks here:
    Click on the following links in the column on the left for a complete comparison grade by grade.
    Gr 1 Math_Lit.pdf
    Gr 2 Math_Lit.pdf
    Gr 3 Math_Lit.pdf
    Gr 4 Math_Lit.pdf
    Gr 5 Math_Lit.pdf

  • Nik says:

    Concerned mom and Ze’ev Wurman, your link is broken. Can you please re-post the links. I would like to read them.

  • Comment on this Article

    Name ()


    Sponsored Results

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform