Why Not Have Open Tests?



By 09/14/2011

7 Comments | Print | NO PDF |

The news is full of stories about incidents of cheating on various accountability tests.  The Secretary of Education has urged all state commissioners to focus on testing integrity. In response, states are asking task forces to develop new security protocols, are hiring consultants to evaluate erasure patterns on test booklets, and are contemplating how they can change the pressures for cheating.

A  more complete integration of testing, accountability, and teaching would be superior to dealing with the integrity of testing in isolation.   Let’s put the tests out in the sun instead of trying to lock them up in more and more secure rooms.

The use of student outcome measures for accountability is now firmly entrenched and is not about to go away.  But a variety of complaints about the current testing system exist.   First, the tests tend to narrow the teaching to just what is expected on the tests – with excessive teaching to the test and drilling on  practice tests.  Second, the tests are too easy for students in some schools and too hard for others, either wasting the time of some or frustrating others.  Third, using tests for accountability purposes encourages cheating and other ways to evade scrutiny.

To address these different issues, we need to think differently about aims and means.  Here is a brief proposal to deal with all of the problems.  It starts with developing a large item bank of test questions of varying difficulty.  Imagine 1,500 questions for fourth grade math that cover the entire scope of appropriate material from basic to advanced topics.  Next, make all of the test items – not just sample items – publicly available and encourage teachers to teach to the test, because the items cover the full range of the desired curriculum.  Making the items public will also ensure the quality of the test items.  One could invite feedback ratings or open sourcing to provide a path to improving the questions over time.  Then, move to computerized adaptive testing, where answers to an initial set of questions move the student to easier or more difficult items based on responses.  This testing permits accurate assessments at varying levels while lessening test burden from excessive questions that provide little information on individual student performance.  Such assessments would not be limited to minimally proficient levels that are the focus of today’s tests, and thus they could provide useful information to districts that find current testing too easy.  Students would be given a random selection of questions, and the answers would go directly into the computer – bypassing the erasure checks, the comparison of responses with other students, and the like.

This proposal actually follows the current testing by the FAA of knowledge needed to obtain a private pilot license.  While there are commercial books on these tests, replete with questions and answers, the efficient way to prepare for the tests is simply to learn the underlying concepts.  It is not to attempt to memorize the answers, because it is easy to confuse such an attempt.

What are the potential problems?  Some say developing test items would make this too costly, but remember that it is only necessary to have one item bank, not the continually changed banks of today.  Some worry about ensuring that sufficient computers are available in all schools, but with all of the digital devices currently in use, surely there are a range of possibilities to deliver the tests effectively and efficiently.  There is the problem that the testing companies would not particularly like the proposal.   They find they are happy with mindlessly developing slightly different variants of existing tests for different states, years, and administrations.  But, maybe there are more productive ways for them to enter into the process.

The proposed system would yield quick and reliable feedback on student achievement, would deal with the various cheating and gaming issues, and would more effectively define what students should know than the currently available standards.

-Eric A. Hanushek

Eric Hanushek is a senior fellow at the Hoover Institution of Stanford University and member of the Koret Task Force on K-12 Education.




Comment on this article
  • Kathy McKean says:

    Georgia tried this…I recall being in meetings in which their state department of education officials talked about how it was going to work. I don’t know how it worked out or if they ever got it off the ground, but I don’t believe they do it now.

  • Morgan says:

    This is a terrific idea, and it could potentially solve a lot of the problems with our assessment system. Foremost among these in my mind is the poor alignment of our tests with our standards – if the item bank really covered the full domain of the standards, teachers would have a much clearer picture of what to teach. I wrote about this with Andy Porter in an article in AERJ this summer: http://aer.sagepub.com/content/early/2011/06/15/0002831211410684.abstract

  • Wayne Bishop says:

    Ancient history… I was on the California Math Task Force in 1995 and this proposal was almost unanimously agreed upon as a recommendation including MARS Balanced Assessment guru Alan Schoenfeld, Dean of Education at UC Berkeley. I was an enthusiastic supporter of the concept and still am – a national “voluntary” exam would be great. Unfortunately, my cynical side said then that there wasn’t a chance that this would ever come to pass (it is too reasonable) and my cynical side still says the same.

    My worst fear is that these exams will come out far too similar to those meaningless MARS Balanced Assessments. No national exams at all would be preferable.

  • [...] a recent post in EducationNext, Eric Hanushek of the Hoover Institution asked why not have open tests? An open test would mean [...]

  • [...] Jacobs summarized an article I saw earlier this week with a pretty good suggestion by Erik Hanushek for how state tests could be changed to be more relevant/useful. She asked if the idea makes [...]

  • Delbert Dyar says:

    I have often thought that a state or federally run assessment system would lessen the potential of cheating on “high stakes” testing. Advances in technology have allowed many school districts and states to develop online tests that address state standards that make administration on a national level a viable option.

    A national assessment system would need to be concerned about the verification of the student’s learning in some form of face-to-face observed performance. Online students are also very capable of misrepresenting their true abilities on web based tests, sometimes in a cabal that includes their parents.

    A further concern is that current federal and state requirements for the assessment of students with disabilities would likely make such a system impossible to implement, extremely costly, or highly flawed from its inception without major reform in federal special education laws and regulations.

  • miriam kurtzig freedman says:

    It’s always great to have out-of-the-box proposals, so I’m grateful for this one! But, I’m not sure I understand it….

    I’m thinking that this proposal would solve the issue of invalid test results due to the way tests are modified (allowing too many invalidating ‘accommodations’) for students with disabilitites…. because each test would be individualized from the get go. Is that right? If so, this is another aspect of this proposal worth considering–though it was not mentioned.
    If this is not right, please clarify.

  • Comment on this Article

    Name ()


    *

         7 Comments
    Sponsored Results
    Sponsors

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform

    Sponsors