I’ve written several times before about the disconnect between changing test scores and changing life outcomes. In general, even when we can rigorously identify changes in math and reading test results caused by schools or programs, we have weak and inconsistent evidence that this produces commensurate changes in later life outcomes. Keeping in mind that test-based accountability mostly focuses on the level of test scores, not changes, and virtually never relies upon a rigorous identification of how test scores are caused by schools and programs, we have no way of knowing that that the kinds of schools, programs, and practices that we are pushing in education will actually help kids later in life. We might actually be hurting them and are certainly foreclosing potential opportunities based on our false confidence that we (policymakers, researchers, and pundits) are better at judging quality than are families.
Rather than contemplate the implications of our ignorance, people in our field are inclined to press ahead. Yes, our measures are imperfect, they’ll admit, but we have to do the best we can with what we have. Or they will say we just have to try harder to develop better measures of knowledge acquisition or expand our measures to include non-cognitive skills that provide a more complete picture of the recipe for success.
But what if there is simply no recipe for success? Or more precisely, what if the recipe for success is highly context-dependent so that distant policymakers, researchers, and pundits are unable to prescribe what programs and schools should be cooking? Maybe only parents, communities, and local educators are well-enough positioned to make reasonable (if imperfect) judgments about what each child needs.
If you were holding out hope that the expansion of educational measures to include non-cognitive skills would give policymakers, researchers, and pundits a stronger ability to prescribe how and where students should be educated, I have some bad news for you. The disconnect between educational measures and later life outcomes is at least as severe in non-cognitive measures as it is in test scores. Let’s leave aside the fact that most non-cognitive measures are too easily gameable to be used for any accountability purposes. Even if only for research purposes, there does not appear to be a straightforward and consistent connection between non-cognitive measures and later life outcomes.
A new study led by Nicholas W. Papageorge at Johns Hopkins University and IZA examines the connection in Great Britain between teacher reports about behavior when students are 11 and later life outcomes for those students. Because non-cognitive measures are in their infancy, we aren’t entirely sure how to slice and dice the measures and do not have a clear system for labeling the related concepts we are measuring. In this study, if we simply lumped all of the teacher reports of misbehavior together we would find that students who misbehave more tend to do worse later in life.
But if we split misbehavior into two categories — one that captures misbehavior directed toward others (externalizing) and another that captures whether students are misbehaving because they are withdrawn (internalizing) — the picture gets more complicated. Students who score poorly on measures of internalizing misbehavior still seem to fare poorly later in life. But for students who score lower on the externalizing misbehavior, how they fare later depends on their social class. If students are from more advantaged backgrounds, externalizing is actually associated with higher earnings, while for more disadvantaged students externalizing seems to have no effect on earnings.
This null to positive effect on earnings for a certain type of misbehavior occurs despite that fact that externalizing is associated with lower levels of educational attainment. That is, students who misbehave toward others don’t go as far in school, but they earn more in the workplace if they come from more advantaged backgrounds despite that negative effect on educational attainment.
Yet again, we see that what we think is “good” performance on a a near-term educational measure is highly dependent on context and is not connected to later outcomes in a straightforward way. Policymakers, researchers, and pundits are inclined to say that scoring higher on a measure of behavior is better, but that is not necessarily the case. It depends on the type of behavior and who the student is.
In addition, things we do to increase educational attainment may come at the expense of later earnings. Increasing compliance with school authorities may help students go further in school, but may stifle the initiative and ambition necessary to make larger contributions to the economy later. Policymakers, researchers, and pundits cannot simply identify a set of educational measures (test scores, non-cog measures, or educational attainment) and judge from afar which programs and schools are going to help students succeed later in life. You can’t just maximize these measures and expect uniformly good results.
None of this should be surprising to parents engaged in the complicated task of raising their children. We often want our children to possess certain qualities — but not too much of those qualities. We want our children to be obedient, but not too obedient. We want them to be ambitious, but not too ambitious. We want them to value abstract knowledge, but not too much abstract knowledge. We want them to value being in school, but not stay in school forever. And we know that the approaches we take to produce these balanced outcomes vary for each child, even within our own family let alone across all families.
As it turns out, educating children is simply an extension of raising children to be the kinds of adults we hope they can be, requiring all of the same nuance, balance, and judgment. Solutions imposed by distant policymakers, researchers, and pundits are no more likely to be effective in educating our children than they would be in raising them. Yes, parents, communities, and local educators will make mistakes in raising children as well as educating them. But they are better positioned to understand the context and achieve the appropriate balance for each child than are distant policymakers and experts, even if they are well-intentioned and highly knowledgeable.
– – Jay P. Greene
