Mark Schneider is a vice president at American Institutes for Research and former commissioner of the National Center for Education Statistics.
Andreas Schleicher recently blogged in response to my EdNext article, “The International PISA Test,” but his response fails to deal with important issues I raised in my article: the quality of PISA’s analysis and the degree to which PISA’s reports ignore the limits of the data in support of conclusions that seem to have been determined in advance of the analysis. Another issue I raised is how PISA should fit into the U.S. system of testing and data collection. The United States already spends well over $130 million each year on NAEP. Moreover, after this year’s State Longitudinal Data System grants are announced, the federal government will have invested approximately $400 million in student-based longitudinal data—an investment that does not include the large amounts of money invested by the states in developing these systems.
How PISA fits into our national testing, data, and research landscape is a serious policy issue, and Mr. Schleicher’s superficial comparison of the United States to countries that have not made similar investments is singularly unhelpful. In addition, while Mr. Schleicher implies that I am a luddite for refusing to accept all the technological advances that he advocates, I should note that the last part of my article explored new ways that states could obtain state-level PISA results without spending additional millions of dollars.
I am quite surprised by the ad hominem nature of Mr. Schleicher’s remarks. It was my honor to serve the American people, and I am proud of what I accomplished as the Commissioner of Education Statistics. But I was only a part of the policy process in the U.S. Department of Education. The positions I took regarding PISA represented the decisions of the United States Government and reflected budget allocations and other policies decided by the U.S. Congress. During the years I served, Mr. Schleicher often tried to change U. S. policies, and he engaged in such egregious lobbying behavior that leaders in the Department of Education complained about him to the State Department, to the U.S. Ambassador to OECD, and to Mr. Schleicher’s supervisors in the OECD.
Mr. Schleicher perhaps too closely identifies with PISA to realize that he works for an international membership organization and that nations have their own interests in how they approach OECD services such as PISA. I believe that Mr. Schleicher’s emotional response to my criticism has led him astray on several other points. Of course, I know that nations can request individual reports that produce specific policy recommendations and that these reports are not produced by PISA, but by another unit of OECD. But this distinction may have more meaning to Mr. Schleicher than to readers of the PISA reports. These reports are filled with policy implications, and by billing them as “implications” rather than “recommendations” PISA is free to offer advice based on weak evidence, avoiding the more intense scrutiny that a country report would get. If the “policy implications” about accountability in the PISA report had been presented to the United States in a country report, we would have rejected them out of hand as not evidence-based. Indeed, the U.S. has frequently tried to limit some of the more fanciful conclusions of PISA reports, but often without success.
I should also note that Mr. Schleicher is so sure that he knows best about PISA that not only do the conclusions of PISA reports often seem predetermined, he also seems to know the policy decisions of the PISA Governing Board even before they do: At one PISA Governing Board meeting, Mr. Schleicher made the mistake of showing his summary, conclusions, and recommendations of the Board’s meeting before it had even reached the halfway point.
Mr. Schleicher also notes that the U.S. had “to withdraw key outcomes from PISA 2006 because of technical flaws by Schneider’s subcontractors.” First, I should note that these were not my subcontractors—contracts are awarded by the United States government based on a rigorous review process and governed by detailed contract law. Further (and fortunately) the results that were withdrawn were for a minor domain and the U.S. did report its results for science, the major subject tested that year, and for mathematics. Mr. Schleicher states that “much of the rest of the world was simply bewildered.” I was on the PISA Governing Board when this error occurred, and the only feedback I got was sympathy from the other members, not bewilderment. Many other countries have run into problems in administering this complex assessment (in fact, in 2003, the United Kingdom could not report any results). Moreover, I forced the contractor, RTI International, to reimburse the U.S. government for the error. I should note that the other alternative would have been to intentionally release flawed data, which would have been contrary to everything that NCES stands for.
Finally, despite what Mr. Schleicher infers, I am not opposed to PISA. Indeed, I was the highest-ranking U.S. government official ever to serve on the PISA Governing Board—and I did not serve on that board to sabotage PISA, but to make sure that U.S. interests were better represented.
There will be continued evolution in how PISA will be used in the United States. And, as in the past, the United States might choose to participate in some parts of PISA and not in others. But those decisions will (and rightfully should) be governed by the needs of the United States, as determined by American policymakers, and made in light of scarce resources.
Mr. Schleicher’s clear pique at what he sees as unenlightened policy decisions and the ad hominem nature of his response show that he cannot clearly distinguish between serious policy discussions and anything less than the full adoration of PISA.
