A few weeks ago, I, together with Eric Hanushek and Ludger Woessmann, issued a report showing that the United States ranked 31st in the world at bringing 15 year olds up to an advanced level of math achievement. Since the study caught the attention of the mainstream media (including the Atlantic, NBC’s Nightly News, and the Boston Globe), it could hardly expect to escape scrutiny.
Still, I had not expected a critic to characterize our work as “deceptive,” the adjective chosen by University of Georgia’s Jeremy Kilpatrick. We are further charged with “inaccurately exaggerat[ing] small differences” and engaging in other “misleading practice[s].” Even though Kilpatrick writes under the auspices of a union-financed policy center, we were surprised by the tone of Kilpatrick’s claims, as we had taken pains to make our assumptions and methodology fully transparent, as is evident from the fact that our critic spends much of his time and energy iterating points we had addressed in the report.
He also faults us for failing to make policy recommendations, a strange critique of a scientific paper. None of the authors shrink from the responsibility of recommending needed policies when the context is appropriate. But to have added them to this report would have been a distraction from its basic purpose: Providing accurate, comparative information on the percentages of U. S. students performing at the highest level.
An odd criticism is highlighted in Kilpatrick’s summary. He objects to the stringency of the definition of “advanced” upon which our report relies on the grounds that it shows that only 6 percent of U. S. eighth graders reach the advanced level. To set the standard, we relied upon the judgments of the National Assessment Governing Board, which administers the National Assessment of Educational Progress (NAEP), as well as an international testing agency, Trends in Mathematics and Science Study (TIMSS), whose work Kilpatrick applauds.
Our standard is not set so high that it keeps 15 percent of all Canadians, 17 percent of all Dutch students, and 23 percent of Korean students from reaching it. But in the end, the key fact is not the standard’s stringency as that a smaller proportion of American students meet that standard than do the students in 30 other countries. Nor would results change much if you shifted the standard downward, as the average performance of U. S. students also lags behind almost all of these same countries.
Later in the report, Kilpatrick argues at great length that the PISA and NAEP math tests measure different things, making results utterly incomparable. But the performance of countries on the PISA math test correlates at the 0.93 level with TIMSS, the very international math test Kilpatrick prefers and one that he and others say is similar to NAEP. To say the different tests are incomparable strikes us as somewhere between implausible and daft.
But for the sake of argument, let’s concede Kilpatrick’s point. Let’s assume you cannot place each U. S. state on the international scale by linking the NAEP to the PISA, as we have done. Let’s assume the implausible, that Kilpatrick, on this point, is dead right, and we are dead wrong. Where are we then? We still can compare every state with every other state within the United States and come up with the same ranking that shows Massachusetts and Minnesota first and second and Mississippi in last place. No other state’s ranking vis a vis one another changes by so much as a rabbit’s hair.
And the rank of the United States as a whole relative to other countries remains stuck at 31, as that ranking depends entirely on U. S. performance on the same PISA test taken at roughly the same time as by students in all the other participating countries.
To deny that the U. S. ranks number 31 in math performance requires the claim that PISA is a “deceptive”, “misleading” test that “exaggerates small differences” something Kilpatrick wants to imply but never dares to argue, as it would run in the face of PISA’s adoption by educational ministries throughout the industrial world, including the U. S. Department of Education.
Kilpatrick’s has a hodge-podge of trivial technical concerns:
1. He points out that the international students who took the PISA were not always from the same graduating class as the U. S. students taking the National Assessment of Educational Progress (NAEP). As we stated in our report, that is significant only if math performance fluctuates noticeably from one student cohort to the next, which Kilpatrick fails to show. On the contrary, the percentage of U. S 8th graders performing at the advanced level changes only very modestly over time.
2. Kilpatrick objects that scores of 8th graders do not prove anything about performance of students in high school. A point well taken, it cuts in exactly the opposite direction than Kilpatrick would like to take the reader. The performance of older U. S. students on the NAEP is even more discouraging than the performance of 8th graders. Our report very likely under-estimates—it does not over-estimate–the math performance of U. S. high school graduates.
3. Kilpatrick argues—without offering evidence–that PISA 2006 math results may be imprecise for some countries. We provide tests of statistical significance that indicate the extent to which one can have confidence in any specific comparison.
-Paul E. Peterson
