The Dead End of Scientific Progressivism



By 01/20/2011

12 Comments | Print | NO PDF |

In Education Myths I argued that we needed to rely on science rather than our direct experience to identify effective policies.  Our eyes can mislead us, while scientific evidence has the systematic rigor to guide us more accurately.

That’s true, but I am now more aware of the opposite failing — believing that we can resolve all policy disputes and identify the “right way” to educate all children solely by relying on science.  Science has its limits.  Science cannot adjudicate among the competing values that might attract us to one educational approach over another.  Science usually tells us about outcomes for the typical or average student and cannot easily tell us about what is most effective for individual students with diverse needs.  Science is slow and uncertain, while policy and practice decisions have to be made right now whether a consensus of scientific evidence exists or not.  We should rely on science when we can but we also need to be humble about what science can and can’t address.

I was thinking about this while reflecting on the Gates Foundation’s Measuring Effective Teachers Project.  The project is an ambitious $45 million enterprise to improve the stability of value-added measures while identifying effective practices that contribute to higher value-added performance.  These are worthy goals.  The project intends to advance those goals by administering two standardized tests to students in 8 different school systems, surveying the students, and videotaping classroom lessons.

The idea is to see if combining information from the tests, survey, and classroom observations could produce more stable measures of teacher contributions to learning than is possible by just using the state test.  And since they are observing classrooms and surveying students, they can also identify certain teacher practices and techniques that might be associated with greater improvement.  The Gates folks are using science to improve the measures of student progress and to identify what makes a more effective teacher.

This is a great use of science, but there are limits to what we can expect.  When identifying practices that are more effective, we have to remember that this is just more effective for the typical student.  Different practices may be more effective for different students.  In principle science could help address this also, but even this study, with 3,000 teachers, is not nearly large enough to produce a fine-grained analysis of what kind of approach is most effective for many different kinds of kids.

My fear is that the researchers, their foundation-backers, and most-importantly, the policymaker and educator consumers of the research are insensitive to these limitations of science.  I fear that the project will identify the “right” way to teach and then it will be used to enforce that right way on everyone, even though it is highly likely that there are different “right” ways for different kids.

We already have a taste of this from the preliminary report that Gates issued last month.  Following its release Vicki Phillips, the head of education at the Gates Foundation, told the New York Times: “Teaching to the test makes your students do worse on the tests.”  Science had produced its answer — teachers should stop teaching to the test, stop drill and kill, and stop test prep (which the Gates officials and reporters used as interchangeable terms).

Unfortunately, Vicki Phillips mis-read her own Foundation’s report.  On p. 34 the correlation between test prep and value-added is positive, not negative.  If the study shows any relationship between test prep and student progress, it is that test prep contributes to higher value-added.  Let’s leave aside the fact that these were simply a series of pairwise correlations and not the sort of multivariate analysis that you would expect if you were really trying to identify effective teaching practices.  Vicki Phillips was just plain wrong in what she said.  Even worse, despite having the error pointed out, neither the Gates Foundation nor the New York Times has considered it worthwhile to post a public  correction.  Science says what I say it says.

And this is the greatest danger of a lack of humility in the application of science to public policy.  Science can be corrupted so that it simply becomes a shield disguising the policy preferences of those in authority.  How many times have you heard a school official justify a particular policy by saying that it is supported by research when in fact no such research exists?  This (mis)use of science is a way for authority figures to tell their critics, “shut up!”

But even if the Gates report had conducted multivariate analyses on effective teaching practices and even if Vicki Phillips could accurately describe the results of those analyses, the Gates project of using science to identify the “best” practices is doomed to failure.  The very nature of education is that difference techniques are more effective in different kinds of situations for different kinds of kids.  Science can identify the best approach for the average student but it cannot identify the best approach for each individual student.  And if students are highly varied in their needs, which I believe they are, this is a major limitation.

But as the Gates Foundation pushes national standards with new national tests, they seem inclined to impose the “best” practices that science identified on all students.  The combination of Gates building a national infrastructure for driving educator behavior while launching a gigantic scientific effort to identify the best practices is worrisome.

There is nothing wrong with using science to inform local practice.  But science needs markets to keep it honest.  If competing educators can be informed by science, then they can pick among competing claims about what science tells us.  And they can learn from their experience whether the practices that are recommended for the typical student by science work in the particular circumstances in which they are operating.

But if the science of best educator practice is combined with a national infrastructure of standards and testing, then local actors cannot adjudicate among competing claims about what science says.  What the central authorities decide science says will be infused in the national standards and tests and all must adhere to that vision if they wish to excel along these centralized criteria.  Even if the central authority completely misunderstands what science has to say, we will all have to accept that interpretation.

I don’t mean to be overly alarmist.  Gates has a lot of sensible people working for them and there are many barriers remaining before we fully implement national standards and testing.  My concern is that the Gates Foundation is being informed by an incorrect theory of reform.  Reform does not come from science identifying the right thing to do and then a centralized authority imposing that right thing on everyone.  Progress comes from decentralized decision-makers having the freedom and motivation to choose among competing claims about what is right according to science.

- Jay P. Greene

Addendum

I just wanted to add a few thoughts to my post yesterday.  Readers may be wondering what is wrong with using science to identify the best educational practices and then implementing those best practices.  If they are best, why wouldn’t we want to do them?

Let me answer by analogy.  We could use science to identify where we could get the highest return on capital.  If science can tell us where the highest returns can be found, why would we want to let markets allocate capital and potentially make a lot of mistakes?  Government could just use science and avoid all of those errors by making sure capital went to where it could best be used.

Of course, we tried this approach in the Soviet Union and it failed miserably.  The primary problem is that science is always uncertain and susceptible to corruption.  We can run models to measure returns on capital, but we have uncertainty about the models and we have uncertainty about the future.  Markets provide a reality test to scientific models by allowing us to choose among competing models and experience the consequences of choosing wisely or not.  Science can advise us, but only choice, freedom, and experience permit us to benefit from what science has to offer.

And even more dangerous is that in the absence of choice and competition among scientific models, authorities will allow their own interests or preferences to distort what they claim science has to say.  For an excellent example of this, check out the story of Lysenko and Soviet research on genetics.  For decades Soviet science was compelled to believe that environmental influences could be inherited.

Science facilitates progress through the crucible of market tests.  Science without markets facilitates stronger authoritarianism.




Comment on this article
  • Matt says:

    I was with you all the way up to “But science needs markets to keep it honest.” If you’re talking about competition between ideas based on their ability to predict outcomes – that is science. The fact that “science is always uncertain and susceptible to corruption” is because it is a human enterprise, not because it lacks the [incorruptible?] “crucible of market tests”

  • Jay P. Greene says:

    Hi Matt,

    Markets are not incorruptible (since they are also human institutions), but they are less susceptible to corruption than government because they make people more likely to experience the consequences of their decisions. If I choose the wrong science in a market context then I suffer some adverse consequence. If I choose the wrong science through government, I can force others to experience that adverse consequence and perhaps escape it myself.

    Besides, if the government enforces the one true way, there will be little opportunity for choice and competition between ideas. Free markets facilitate free ideas.

  • Peter says:

    Great article Jay. It’s certainly one that I’ve tweeted and e-mailed out and will refer to in the future.

    http://educrati.com/

  • john thompson says:

    Yes, “Different practices may be more effective for different students.”

    So if Gates MET was a serious piece of science, they’ve have evaluators create a control by evaluating instruction according to their normal rubric.

    Then they’d fully brief evaluators on the IEPs, the mental illness stats and conduct disorders, and SEDs, the parole jackets of students in classrooms, the minutes of Facuty Meetings where policies are laid down, and school’s disciplinary stats to understand the context of the class being videotaped.

    Then they compare the evaluation resulting from watching videotaped instruction being unaware of peer and school effects, with results of evaluators who could put instruction in context.

  • Jo-Anne Gross says:

    I ,like most of you,have watched a deluge of educational fads come and go.
    I really think science plays a major role in proof of efficacy for math and Reading,Spelling and Writing,the K-3 space.
    Otherwise,we are just winging it.
    I have seen clever marketing take a program out that had absolutely not grounding in what the research has proved to be scientifically sound instruction.That`s just morally wrong.
    If the kids are doing relatively well academically by grade 3 we then look at behaviour , motivation and engagement during the content acquisition phase of education.

  • Dick Schutz says:

    You haven’t defined “science,” Jay. My definition of choice is “Doing your damnedest with your mind, no holds barred.” Under this definition, the MET project is puny science, for reasons that you and other comments give. Many other reasons could be presented.

    But the larger fallacy is that scientific information can be “applied” to the schooling endeavor. Research can inform the D in R&D, but the Development to create usable tools that can obtain reliable consequences is virtually absent in the education sector. This “how to” is termed “technology.” But in in education “technology” references equipment, usually electronic, rather than “how to.”

  • Parry says:

    I’m really struggling with this one, for a couple reasons. First is this quote:

    “The very nature of education is that different techniques are more effective in different kinds of situations for different kinds of kids. Science can identify the best approach for the average student but it cannot identify the best approach for each individual student.”

    I agree that, in education, there is no one-size-fits-all instructional approach. But there are clearly some approaches that are less effective (e.g., lecturing to a group of kindergarteners for 90 minutes) and some approaches that are more effective (e.g., providing students with individualized feedback, no matter their level of ability of achievement). I think what a scientific approach can do is help us tease out the differences between the more effective and less effective approaches. (I do, however, agree with your point that many educational approaches can end up being presented as absolutes when the reality is that somebody somewhere did a study that kinda says that maybe a certain approach might have been effective in a certain situation.)

    My second struggle is that, in prior work of yours that I have read, much of your argument for choice in public education seems to rely on the results of scientifically-conducted research. I like the research you do and am impressed by your results. But couldn’t this current article be used as an argument to say “Sure, maybe some school choice initiatives seem to have helped students raise test scores in some specific situations, but that’s no evidence to suggest that school choice initiatives with different types of students in different contexts would work”.

    What am I missing?

    Parry

  • Robert Sweet Jr says:

    This is an interesting discussion. Applying the findings of research in medical science is a given, is it not? Who would take any of the myriads of pills for curing our “ills” unless we were certain they had undergone extensive clinical trials? And yet, when “science” and “clinical trials” are suggested for, lets say, reading methodology it is dismissed. Illiteracy continues in America unabated, year, after year, after year…seems to me that giving serious consideration to applying settled research findings to a skill like reading instruction would be a logical way to address this blight on America.

  • [...] using scientific evidence in policy — Here is an interesting somersault by Jay Greene, the provocative and prolific think tanker/researcher/commentator, who for many years [...]

  • Jay P. Greene says:

    Thanks for all of the thought-provoking comments. Let me briefly address some of the issues raised.

    I am in no way abandoning my commitment to using science to inform educational policy and practice. My concern is that many policies and practices in education have highly varied effects — they are good for certain students in certain circumstances and bad for other students in other circumstances. To the extent that educational interventions have relatively uniform effects, science is very helpful because it can identify mean effects with rigor. But to the extent that is not the case, science is much less helpful and we have to rely more on judgement and experience.

    The same is true for medicine. The difference is that there are many more interventions in medicine that have uniform effects than in education. Education is more akin to parenting and there is no well developed science of how to parent. And this is for very good reason — the right parenting technique is highly dependent on the individual child and particular circumstance.

    Of course, there are a good number of important policies and practices in education (as well as parenting) that have uniform effects and where science is extremely useful. Locking any child in the dark like Kaspar Hauser is unlikely to be an effective parenting strategy just like lecturing to a class of 90 kindergarten students probably doesn’t work for anyone. But there are a host of practices and policies where what works for some students in some circumstances doesn’t work for others. In those situations we want to be guided by what science says for the typical child but also be sensitive to differentiated effects.

    Because so much of education has differentiated effects, having a system of choice and competition will be better than attempting to impose the “one true way” on all students. Perhaps it is because of this that scientific evaluations of choice programs tend to show better results. But I recognize that even the effects of choice and competition are likely to be varied. Having more options may be better for most student but there will be some who would be better off with less choice. We can’t realistically customize the amount of choice for each student, so in that circumstance we have to be guided by the mean effects and do what is best for most students. Where we can customize, however, we can use more judgement and experience.

  • Kyle Peck says:

    Important insights in this article! Thanks for helping people see that there is value in high-quality research, but that the value is in providing information with which to make decisions about what might work for individual students.

    Try to think ahead a decade, when it might be possible to provide better information on what works FOR WHAT STUDENTS, rather than for aggregated groups. Think about how possible it is and will become to gather data on student success along with other data (some demographic, and some about successes and failures with other interventions). I can envision a database that helps teachers willing to use individualized approaches figure out what is most likely to work for a twelve year old student who has had trouble learning fractions, but was successful when he worked at his own pace with computer-based tutorials, or for the nine year old girl who seems not to benefit from computer-based experiences, but thrives when working with hands-on materials in group settings.

    I realize that that is a big jump from here, the pieces are all in place.

    My favorite line was Jay’s last… “Progress comes from decentralized decision-makers having the freedom and motivation to choose among competing claims about what is right according to science.” In the future we’ll be able to support that decision making with data-based evidence.

  • [...] Jay Greene, discussing the use of evidence to inform policy, manages to range from the Gates Foundation’s Measures of Effective Teaching project to the evils of communism in a single blog post. Here’s the crux of his argument before it drifts off course: I am now more aware of the opposite failing — believing that we can resolve all policy disputes and identify the “right way” to educate all children solely by relying on science.  Science has its limits.  Science cannot adjudicate among the competing values that might attract us to one educational approach over another.  Science usually tells us about outcomes for the typical or average student and cannot easily tell us about what is most effective for individual students with diverse needs.  Science is slow and uncertain, while policy and practice decisions have to be made right now whether a consensus of scientific evidence exists or not.  We should rely on science when we can but we also need to be humble about what science can and can’t address. [...]

  • Comment on this Article

    Name ()


    *

         12 Comments
    Sponsored Results
    Sponsors

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform

    Sponsors