My Response to Marc Tucker’s Defense of Surpassing Shanghai



By 04/18/2012

8 Comments | Print | NO PDF |

In a reply posted on the Ed Next blog that is longer than my original review of his book, Surpassing Shanghai, Marc Tucker throws quite a bit of dust in the air – more than I can address in this brief response – but one thing remains perfectly clear: Marc Tucker does not understand basic principles of research design.  The “best practices” method that is gaining popularity among more-impressionable education policy wonks and that Tucker used in Surpassing Shanghai simply cannot support causal claims about “what works.”

The fundamental problem is that “best practices” analyses lack variation in the dependent variable – they only examine in detail successful organizations or countries – so they can’t link particular practices or policies to success.  To make such a link they would need to observe that the presence or absence of those practices or policies is related to the presence or absence of success.  If they only look at successful organizations, then they can’t know whether they would have been less (or more) successful had they not adopted a particular policy or practice.  They also do not rule out the possibility that others who have adopted the “best practices” do so without success.

But Tucker claims that he didn’t only look at successful countries because “the strategy we used was to compare the top performing countries to the United States.”  Making (mostly implicit) comparisons to the United States does not solve the problem.  Again, without considering a broad spectrum of successful and unsuccessful countries it is impossible to attribute the superior performance of another country to any particular policy or practice.

There are many things that are different between the U.S. and Shanghai, Finland, Japan, Singapore, and Canada.  How can Tucker or anyone know which differences caused the superior performance?  Tucker just picks and chooses the policies and practices he favors, ignoring that his recommendations are not even universally present in the handful of successful places he examines.  And by limiting variation in the dependent variable to exclude places that perform worse than the United States, Tucker is unable to discover whether lower-achieving countries are also employing the practices and policies he recommends, which would debunk his claim of having found the formula for success.

I’m far from being the only one who is aware of the problems with Tucker’s method of “selection on the dependent variable.”  Virtually every introductory text on research design warns readers not to do as Tucker and other best practices enthusiasts do when they focus only on successful organizations or countries.  For example, Gary King, Robert Keohane, and Sidney Verba, in their classic Designing Social Inquiry, make the point emphatically:

That brings us to a basic and obvious rule: selection should allow for the possibility of at least some variation on the dependent variable. This point seems so obvious that we would think it hardly needs to be mentioned. How can we explain variations on a dependent variable if it does not vary? Unfortunately, the literature is full of work that makes just this mistake of failing to let the dependent variable vary…. The cases of extreme selection bias—where there is by design no variation on the dependent variable—are easy to deal with: avoid them! We will not learn about causal effects from them.

In my review I recommend analyses of international policies and practices done by Eric Hanushek, Ludger Woessmann, Martin West, Michael Kremer, Karthik Muralidharan and Charles Glenn because, unlike Tucker and other “best practices” gurus,  they avoid the error of selection on the dependent variable by considering the full range of outcomes, not just focusing on successful places.

Tucker is apparently unable to understand the difference between what he and these reputable researchers do when he mistakenly declares:

Greene appears to realize that his war on “best practices” has led him to inadvertently attack the kinds of studies done by people whose policy prescriptions he prefers, like Ludger Woessmann and Eric Hanushek, who have done well-regarded statistical analyses of survey data from OECD-PISA and other sources…. So, in the end, all the methods we used meet with Jay Greene’s approval.  It is only our conclusions that are odious.

Tucker’s inability to understand the difference and his dismissal of the selection on dependent variable criticism as “highfalutin language” is just plain embarrassing.  It’s not so much embarrassing for him, since he appears to be proud in his ignorance, as it is embarrassing for the Gates Foundation that pays for his work and the supporters of Common Core who rely on Tucker as one of their principal architects and advocates.

There is a cynical habit in the education policy world to fund and promote analyses that people know or should know to be faulty as long as those analyses advance their cause.  Shaming those who engage in this cynical practice by revealing the obvious flaws in Tucker’s work was the purpose of my review.  I fear that it will not end the use of “best practices” in education, but I hope it will exact a price for those who engage in such hucksterism.

-Jay Greene




Comment on this article
  • Niki Hayes says:

    Great response, Jay. Thank you, thank you for putting clarity on Marc Tucker’s very, very, very long huffing and puffing about your review.

  • Will Fitzhugh says:

    I can say that for all his great work in bringing Sandra Stotsky to Arkansas, and for whatever expertise he may have in research design, Mr. Greene, in contrast to Mr. Tucker, has shown, in my experience, zero interest in the academic reading and writing of secondary students in the United States, at least insofar as I have tried over several years to draw his attention to the issues.

  • Jay P. Greene says:

    Hi Will — I admire the work you’ve done with the Concord Review and I agree with your call for a focus on writing in high school. But I don’t go around talking about it a lot because it is not something I’ve ever studied and I don’t claim to be an expert on it. I leave that to people whose expertise I trust, like Sandra Stotsky.

    Just because others feel expert on all issues and whisper pretty words in your ears doesn’t mean that they actually support the reforms you favor or that you should support them.

  • Zeev Wurman says:

    Will,

    Let’ assume for a moment that you are completely correct than Jay has no interest in the academic reading and writing of secondary students in th U.S. (Not that I believe so.)

    What does it have to do with understanding — or misunderstanding — of sociel science research methods and causality?

  • jeffrey miller says:

    Jay, your doctorate is in political science, not education. Which as it happens, suits your disposition towards all comers who would quibble with your view of reform. So far, experiments with choice in education haven’t worked out too well, have they? Have you ever worked as a classroom teacher? An administrator in an urban school? Because, you know what, it doesn’t seem you understand anything other than getting into debates about how many angels can dance on the head of a pin.

    In 2000, you had this article published http://www.city-journal.org/html/10_3_the_texas_school.html in which you make a very unscientific claim about the apparent success of the Texas educational system, including the reign of George W. However, the state of Texas makes the astonishing claim that more money (finance reform) to poor areas accounted for the TAAS improvements. http://www.window.state.tx.us/border/ch03/ch03.html Not surprisingly, the reform W brought to the nation has not worked, either. It’s not about the business model or testing or choice, Jay. It should be clear, even to you, that the countries that have been successful in education are not doing what you and your like-minded friends are trying to do here. Oh, and btw, it’s possible to have strong national expectations, a strong union, and curricular guidelines and give teachers a great deal of autonomy as in Finland. You might also read some of your own work sometime. Here you seem to accept that more money will yield results in public schools, which seems odd given your politics and ideology but hey, maybe there’s hope for you http://jaypgreene.com/2012/01/24/are-charter-schools-models-of-reform-for-traditional-public-schools/

  • Michael says:

    Jeffrey Miller:

    You said, “So far, experiments with choice in education haven’t worked out too well, have they?”

    That’s actually not a very accurate reading of the random-assignment research literature on school choice. The overwhelming majority of random-assignment school choice (voucher and charter) studies find a positive relationship (particularly for low-income and minority students) between choice and achievement. Some find no effect. But almost no random-assignment studies on school choice find a negative impact.

    Were you not aware of this?

  • Scott Widmeyer says:

    Jay–I would not question your intellectual capacity and shame you the way you have made every attempt possible to shame the work of Marc Tucker and others who fall in the “best practices” sector. I will make one point that all one needs to do is look north to Ontario as well as Asia, some Scandinavian countries and districts in Minnesota, Maryland, Massachusetts who are structuring their schools around benchmarked best practices. We need the will in America to make these needed changes and not wait for decades on the numbing research that you and your disciples are spinning.

  • Trace Pickering says:

    I believe Jay is trying to point out something that social systems scientists – like Russ Ackoff and Jamshid Gharajedaghi – have been trying to teach us for years. Put simply, the whole is not simply a sum of the parts. Can we be informed and guided in our own designs by seeing what appears to be working in other countries? Sure. But there’s where it stops. Why? 2 simple reasons: 1) there is no such thing as a universal best practice when it comes to human learning, and; 2) the moment a “best practice” is extracted from its containing system it loses all its essential properties. Placing it in a new system is a completely new proposition because it isn’t the action of that part, whatever it is, it is how that part interacts with all the other parts. This is why all-star teams are often not the best teams – assembling the best parts together without concern for their interactions leads to failure. How did the US bounce back from its embarrassing Olympic basketball performances a decade or so ago? They no longer just take the 10 best players, they select more carefully for the chemistry – how the parts interact. If we are serious about our US education system we will work to redesign it to achieve the ends we want to achieve, not try to emulate other countries so we can catch up and especially not just cherry-pick the best-practices. I don’t see anything in our history that suggests the American ethos is: “boy, I hope we can keep pace with X!” We lead, we don’t mindlessly copy.

  • Comment on this Article

    Name ()


    *

         8 Comments
    Sponsored Results
    Sponsors

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform

    Sponsors