Brilliant New Measure of Non-Cognitive Skills



By 08/12/2013

3 Comments | Print | NO PDF |

My student, Collin Hitt, and colleague, Julie Trivitt, have an amazing paper on how we can efficiently measure an important non-cognitive skill that is strongly predictive of later life outcomes. A growing number of researchers have come to realize that lifetime success is partially a function of traditional academic achievement (cognitive skills) and partially a function of what are called non-cognitive skills, such as hard work, self-discipline, determination, etc… Schools may play a central role in conveying both types of skills, but for the most part we have only been collecting information on cognitive skills in the form of standardized test results. The main difficulty in expanding the types of measures we collect to include non-cognitive skills is that we have not developed efficient mechanisms for doing so.

Hitt and Trivitt have taken an enormous step forward to solve this problem. They have discovered that student non-response on surveys (not answering questions or saying they don’t know) is an excellent measure of non-cognitive skills that are strongly predictive of later life outcomes. In particular they examined survey response rates from the National Longitudinal Study of Youth (NLSY) given to students ages 13 to 17 in 1997. The number of items that students answered was predictive of the highest level of education students attained by 2010, controlling for a host of factors including measures of their cognitive ability. If students care enough to answer questions on a survey they are more likely to care enough to pursue their education further.

They then examined another data set to see if they found the same relationship. They did. The number of items that students in Milwaukee answered in a survey when they were in 9th grade was predictive of whether they graduated high school and went to college later, controlling for their academic achievement and other factors.

If this holds up when examined with multiple data sets, it will be an amazing breakthrough for researchers. We will finally have a fairly easy to obtain measure of an important non-cognitive skill that is predictive of later life success.

When studying voucher or other school choice programs, for example, we have observed modest test score benefits for participants, but fairly large attainment benefits. This suggests that school choice has larger effects on non-cognitive skills, but up until now we haven’t been able to observe these non-cognitive benefits without waiting nearly a decade to see if students graduate high school and go on to college. With the Hitt and Trivitt measure, we will have an early warning indicator of whether students are acquiring non-cognitive skills and are more likely to have higher attainment later.

I am not suggesting that the Hitt and Trivitt measure can be used in an accountability system, since it is certain not to work once high stakes are attached. But for research purposes it could be incredibly useful.

Developing an accurate and efficient measure of non-cognitive skills is especially important because one commonly considered measure, the self-reported “grit scale” developed by Angela Duckworth, may not be holding up very well. In the recent Dobbie and Fryer evaluation of the Harlem Promise Academy, it actually appears that the Duckworth scale was a contrary indicator of later life success. That is, students who rated themselves higher on the grit scale were less likely to succeed. We have also tried the Duckworth scale in an experiment and found that it was uncorrelated with other, behavioral measures of non-cognitive skills, such as time devoted to a challenging task and delayed gratification. But the self-reported grit scale was related to a student self-assessment of honesty, suggesting that the Duckworth scale may really measure how highly students will rate themselves more than actual grit or other non-cognitive skills.

Of course, the Hitt and Trivitt measure requires a lot more testing and field research, but it is one of the more exciting recent developments in education research.

—Jay P. Greene




Comment on this article
  • Collin Hitt says:

    Jay’s comments on our paper on our paper are quite generous. Similarly, I should pay a debt of gratitude. Those that read the paper will see that we owe an intellectual debt to David Hedengren and Thomas Stratman at George Mason University. Their working paper, “The Dog That Didn’t Bark,” provided the first compelling evidence that survey item response rates might be a measure of effort and conscientiousness. Our paper was the first to test whether, over time, item response rates were predictive of educational outcomes, as we would expect. Indeed they are – a result that we’re excited about and which I hope brings greater attention to Hedengren and Stratman’s work.

  • Michael Norton says:

    Clever insight; I’m glad the research supported your expectation.

    A question: “The number of items that students answered was predictive of the highest level of education students attained by 2010, controlling for a host of factors including measures of their cognitive ability.”

    Instead of controlling for cognitive ability, what about comparing the two, number of items answered vs. cognitive ability? Is more predictive of attainment than the other? Are they equally in their ability to predict attainment?

  • Ada says:

    Seriously? Not only do you people want to “measure’ intelligence, now you want to measure our “grit.” What are human beings now a days, your lab rats? Good luck with that. While your creating your measurements, I’ll be leaving a free and responsible life without psychological manipulation or an education system that acts more and more like a moral busybody.

  • Comment on this Article

    Name ()


    *

         3 Comments
    Sponsored Results
    Sponsors

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform

    Sponsors