Choosing Blindly

By Guest Bloggers 05/17/2012

3 Comments | Print | NO PDF |

Students learn principally through interactions with people (teachers and peers) and instructional materials (textbooks, workbooks, instructional software, web-based content, homework, projects, quizzes, and tests).  But education policymakers focus primarily on factors removed from those interactions, such as academic standards, teacher evaluation systems, and school accountability policies.  It’s as if the medical profession worried about the administration of hospitals and patient insurance but paid no attention to the treatments that doctors give their patients.  With over half of fourth graders doing math problems from their textbooks daily, we surely ought to care about what’s in those books.

There is strong evidence that the choice of instructional materials has large effects on student learning—effects that rival in size those that are associated with differences in teacher effectiveness.  For example, in a large-scale methodologically rigorous evaluation of the differential impact of four leading mathematics curricula, second-grade students taught using Saxon Math scored on average 0.17 standard deviations higher in mathematics than students taught using Scott Foresman-Addison Wesley Mathematics.  By way of comparison, the difference in the impact on student achievement of a teacher at the 75th percentile of effectiveness compared to an average teacher is only 0.11 to 0.15 standard deviations.  But whereas improving teacher quality through changes in the preparation and professional development of teachers and the human resources policies surrounding their employment is challenging, expensive, and time-consuming; making better choices among available instructional materials should be relatively easy, inexpensive, and quick.

Administrators are prevented from making better choices of instructional materials by the lack of evidence on the effectiveness of the materials currently in use.  The vast majority of materials either have no studies of their effectiveness or have no studies that meet reasonable standards of evidence.  Not only is little information available on the effectiveness of most instructional materials, there is also very little systematic information on which materials are being used in which schools.  In every state except one, it is impossible to find out what materials districts are currently using without contacting the districts one at a time to ask.

This scandalous lack of information will only become more troubling as two major policy initiatives—the Common Core standards and efforts to improve teacher effectiveness—are implemented.  Publishers of instructional materials are lining up to declare the alignment of their materials with the Common Core standards using the most superficial of definitions.  The Common Core standards will only have a chance of raising student achievement if they are implemented with high-quality materials, but there is currently no basis to measure the quality of materials.  Efforts to improve teacher effectiveness will also fall short if they focus solely on the selection and retention of teachers and ignore the instructional tools that teachers are given to practice their craft.

In our Brookings Institution report, we show how this problem can be fixed by states with support from the federal government, non-profit organizations, and private philanthropy.  First, state education agencies should collect data from districts on the instructional materials in use in their schools.  The collection of comprehensive and accurate data will require states to survey districts, and in some cases districts may need to survey their schools.  In the near term, many states can quickly glean useful information by requesting purchasing reports from their districts’ finance offices.  Building on these initial efforts, states should look to initiate future efforts to survey teachers, albeit on a more limited basis.

The federal government’s National Center for Education Statistics should aid states in this effort by developing data collection templates for them to use through its Common Education Data Standards (CEDS), and providing guidance on how states can use and share data on instructional materials.  The most recent version of CEDS contains 679 data elements for K–12 education, none of which relate to instructional materials in use.

Organizations with an interest in education reform should support this effort.  For example, the National Governors Association (NGA) and Council of Chief State School Officers (CCSSO) have put their reputations on the line by sponsoring the Common Core State Standards Initiative.  Research based on current and past state standards indicates that this initiative is unlikely to have much of an effect on student achievement in and of itself.  The NGA and CCSSO should put their considerable weight behind the effort to improve the collection of information on instructional materials in order to create an environment in which states, districts, and schools will be able to choose the materials most likely to help students master the content laid out in the Common Core standards.

States facing severe budgetary pressures may be reluctant to undertake new data collection efforts.  Philanthropic organizations such as the Bill and Melinda Gates Foundation and the Lumina Foundation for Education could have a major impact by providing the start-up funding needed to collect data on instructional materials and support the research that would put those data to use.

In 1955, educational psychologist Lee J. Cronbach wrote that “The sheer absence of trustworthy fact regarding the text-in-use is amazing.”  It is more than a half-century later and we still don’t know.  How can we tolerate ignorance on something that is as critical to student learning as instructional materials?

Matthew M. Chingos and Grover J. (Russ) Whitehurst, who are research director and director of the Brown Center on Education Policy at the Brookings Institution, are the authors of Choosing Blindly: Instructional Materials, Teacher Effectiveness, and the Common Core.

Comment on this article
  • jeffrey miller says:

    Very welcome examination of educational materials. Given the shoddy quality of what I am provided by the district, I routinely make my own curricular materials. Plus, I know my students better than a standardized text and I can vary assignments to avoid boredom or overkill and meet differing needs. A survey of teachers might just reveal that my practice is not unusual. And, let’s not always go to the well of support from foundations and think tanks for research, why not ask universities with teacher/education programs to sic their grad students on local district materials and have them do master’s or doctoral research. Fwiw, it worked for me and that was some of the most valuable grad work I did.

  • Barry Garelick says:

    Excellent idea whose time has long since come. In addition to reporting on the instructional material in use, there should also be a breakdown of how many students (per each such school identified using particular materials) are receiving outside help via tutors, learning centers such as Sylvan, or parents. The reason for such additional information is that in districts that are relatively affluent, such services are used, and the test scores may reflect the effectiveness of the tutors/learning centers and their instructional materials, rather than the schools’.

  • Lisa Jones says:

    Wow… Edutopia 101… States do not need the involvement of the federal government, NGA, CCSSO, or anyone else to accomplish what you are proposing. In fact, every taxpayer should demand to have this information – consider it an inexpensive quarterly report to stock holders!! Colleges and universities should compile this information when students take their placement tests. When students from k-12 districts using inferior teaching materials place into remedical courses upon entering college, certainly all taxpayers have a right to know!! Especially their parents-who end up paying for their education twice, sometimes three times if/when they’ve had to hire tutors along the way!


  • Comment on this Article

    Name ()


    Sponsored Results

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform