When we examine the results of standardized test scores we typically think we are seeing evidence of what students know. As it turns out, that is only partially right. Test scores capture both what students know as well as their willingness to exert effort to show us what they know.
A new paper by my colleagues, Gema Zamarro, Collin Hitt, and Ildefonso Mendez uses multiple, novel techniques to demonstrate that between 32% and 38% of the variation in PISA test performance across countries can be explained by how much effort students are willing to exert rather than what they know. The implications of this finding for ed reform are huge. When we see low test score performance we are often misdiagnosing the problem as poor content instruction when it may in fact be insufficient development of student character skills. If we focus all of our energy on the former without addressing the later, we’ll fail to make as much progress.
So, how do Gema, Collin and Ildefonso know that between 32% and 38% of variation in PISA test performance across countries is explained by effort? They used three different methods to measure the influence of effort. First, they took advantage of the fact that the order of questions without the PISA was randomly ordered. They then compared how well students performed on the first set of items relative to the last. Because the order of items was randomized the first and last questions were, on average, of equal difficulty. The decline in getting items correct from the start to the end of the exam is therefore a function of the decline in effort students are willing to exert, not the difficulty of the items.
If you compare performance in the US and Greece (as can be seen in the figure above), students in the two countries do about as well at the beginning of the test. That means that students in Greece and the US know about the same amount of stuff. But students in Greece decline much more rapidly across the test, which means that those students are less willing to exert consistent effort. When we compare PISA results from the US and Greece we wrongly conclude that content instruction in Greece must be much worse. In reality, Greek students know as much as students in the US but simply exert a lot less effort.
A second way the paper measures effort is by examining responses students gave to a survey that was administered at the same time as the PISA. Using novel techniques that have been validated in previous research, they measure the extent to which students skip answers (or say “don’t know”) as well as the extent to which students give careless answers as proxies for their effort. Both skipped answers and careless answers yield very similar results to what they find from the decline across the test.
Some people have expressed skepticism about the focus on “non-cog” or character in education research because they believe that these capture personality traits that are largely inherited and immutable. This research contradicts that claim. Unless we think there are big and important genetic differences across countries, the variation in effort across countries has to be explained by factors that are social constructed and, at least in theory, could be changed. In addition, great work by Gema and Albert Cheng has found that student effort can actually be changed when students are randomly assigned to different teachers who themselves possess different character skills.
The evidence is becoming clear that character matters and is subject to influence by the education system.
—Jay P. Greene
Jay P. Greene is endowed chair and head of the Department of Education Reform at the University of Arkansas.
