Call me a crotchety old guy and you have a point, but I need to vent my angst over a new report from the NewSchools Venture Fund, authored by the very capable Stacey Childress and her colleagues. They’ve landed with many feet on the social and emotional learning (SEL) bandwagon. Indeed, their new “insight brief” proudly declares that “enthusiasm for social emotional learning has reached a fever pitch among policymakers and funders.” That may well be so, but readers should bear in mind that while some fevers do good—the bodily kind can boost one’s immune system, not to mention getting overcome by election fever, a feverish desire to end poverty, etc.—other fevers can kill (typhoid, dengue, etc.) and still others simply drive you nuts (cabin fever).
SEL fever may be all of the above—and it may already be time to apply an ice pack and antibiotics. Months ago, Rick Hess and I wrote a cautionary piece showing how SEL could do much good if deployed in pursuit of academic learning, but also runs multiple risks of going off the rails, much as happened to earlier “whole child” education movements such as “self-esteem” and “values clarification.” We offered seven cautions lest overheated SEL boosters nudge it in the latter direction.
The sixth of our cautions was about metrics. Here’s what we had to say:
Make It a Priority to Develop Valid, Reliable, Intuitive Metrics for SEL—and Be Honest About Their Limits. It will be important to identify and insofar as possible develop bona fide outcome measures that feel credible to a broad swath of parents and educators. This will lend legitimacy to the insistence that SEL be evidence based, while enabling its measures to be viewed alongside measures of academic performance…. School climate surveys are a start, but let’s not kid ourselves. They share the vulnerabilities of all subjective “how do you think things are going” polls and questionnaires, including selective answers by adults who want their school to look good (or bad!) and plain old game-playing by students. SEL needs more reliable instruments. Just how practical it will be to develop them remains an open question, which should be addressed with a transparency and modesty too often absent in recent years in high-profile efforts to promote other sorts of novel measures such as student assessment and teacher evaluation.
It’s also vital to resist overselling the instruments that we do have. Some of what we most value in SEL may ultimately prove difficult to measure systematically or credibly…. Transparency and a willingness to continuously solicit feedback from skeptical students, parents, and teachers—not just supportive ones—will prove invaluable in addressing these tensions. A relentless commitment to evidence will help guard against goofiness while also helping reassure parents and educators. When the evidence is shaky or uncertain, SEL advocates need to forthrightly acknowledge the fact—not duck it or downplay it….
The NewSchools team, like many others in education today, is all about “expanded measures of student success” and how these can be used for school improvement. Others are especially keen about how such expansion can be used for improved school accountability that gives a fuller picture of school performance that’s less dependent on reading and math test scores. There’s considerable wisdom and common sense in the new brief’s advice about “non-academic indicators,” which they’ve been working on for two years in conjunction with TransformEd and some forty district and charter schools. Along the way, they whittled some sixty potential indicators (“skills, traits, mindsets, and values”) down to fourteen, half of which they term “social emotional competencies,” the other seven “culture/climate factors.” As they gather data, they look for (and don’t always find) correlations between these indicators and standard academic measures, and this quest is yielding a few useful initial insights. For example, the brief reports that “growth mindset” and “perceptions of school safety” are the two SEL indicators “most strongly associated with academic performance” while “sense of belonging” isn’t. Good to know, and even better when turned into formative guidance for school people.
I wanted to know a bit more about how these fourteen indicators work, in particular how they generate the data that enable correlations even to be sought. It’s well known that measuring this sort of stuff is difficult and lots of smart people are working in various places to get better at it. When I lifted the veil on the GreatSchools metrics, it was not too surprising to find that they’re based entirely on surveys administered to schoolkids, i.e., on self-reporting by students. (Also popular in SEL—and some ESSA accountability—circles are gauges of “school climate” resulting from surveys that include teachers.).
You can learn a lot from asking kids questions, and lots of survey questions and scoring scales have been tested and validated in various ways. TransformEd has done a lot of that and can cite much research—as did the big Aspen Commission on SEL that prompted Rick and me to write. One can learn a lot about the kids’ preferences, their opinions about themselves and their peers and their schools, and much more. But how reliable are they when used to gauge changes over time in a child or a school, particularly when seeking correlations with other things that are also changing over time? How many of these indicators are actually malleable and to what extent? And—especially—to what extent can they be altered by actions undertaken by schools and educators?
Consider, for example, what kids are asked that is meant to reveal the extent of their “growth mindset,” which we know from Carol Dweck, Angela Duckworth, et al.—and are again being told—correlates with, if it does not actually enhance, academic achievement over time.
They’re given statements such as “There are some things I am not capable of learning” and “Challenging myself won’t make me any smarter.” They’re given a scale of responses from “not at all true” to “completely true.” On questions like those, of course, the less true the statement in kids’ eyes, the stronger their growth mindset. (You can see all fourteen indicators if you turn to Appendix A of a TransformED working paper on the topic.)
Makes sense, sure. But now ask yourself to what extent are these things susceptible to change as a result of things that teachers and schools do or don’t do, or do well or poorly? And to what extent does this sense of self among kids get far more powerfully shaped by home and neighborhood, by the cultures they grow up within (including different ethnic and gender attitudes and stereotypes), by prior experience with anything from riding a bike to baking a cake? And how much are they contingent on context and subject matter?
On the first of those two illustrative statements, for instance, I would have displayed zero growth mindset because I probably would have imagined learning astrophysics, Amharic, or skydiving. This would have happened on the second statement, too, as I would have thought of how challenging myself in one realm—high jumping, say—wouldn’t have made me any “smarter.”
Now think about placing statements like these before both eight-year-olds and fifteen-year-olds. Before A students and D students. Homeless kids and upper-middle -class kids. During math class or P.E. On Monday morning or Friday afternoon. Think about it. Of course you’ll learn stuff from the results, and of course that stuff may help school people understand what they’re dealing with, what challenges they need to try to help kids with, and which kids have which challenges. But just how reliable and valid are these data? How accurately do they trace change over time? What inferences can you legitimately make about what causes changes in them? To what extent are they malleable by schools?
Some things in the SEL realm should certainly be malleable, especially on the “school culture/climate” side of the indicators. School safety and “rigorous expectations” are obvious. It gets more problematic on the “social-emotional competency” side, but over time—assuming kids remain there for multiple years—a first-rate school can probably deepen one’s curiosity, encourage perseverance, and foster self-regulation. Yet it’s really hard to imagine schools and teachers having great influence over those things—and especially over kids’ self-perception of those things—when compared with the other forces that shape and influence children. Yes, I’m harkening back to a tiny bit of “original research”—it took about five minutes—that I did several decades ago when I calculated that the typical American child spends only 9 percent of their time on earth between birth and age 18 under the school roof—and 91 percent elsewhere. (And that assumes full-day kindergarten and perfect attendance for thirteen years.)
Two risks loom. Most troubling—and I’m not saying NewSchools is pushing this—is what happens if indicators like these make their way into accountability systems and are used for judgments that get made about schools and teachers. It’s simply wrong to hold people and institutions to account for things over which they have such limited control—and which rely on indicators that more than a few parents and politicians will find unacceptably nosey and personal. Perhaps the risk is small—Angela Duckworth herself has warned against doing it—but in the quest to widen the accountability net, it shouldn’t be ignored.
Also troubling, and far more likely: What happens when school leaders, teachers, and ed reformers get so preoccupied with the SEL indicators themselves that they neglect English, math, science, and history or forget how much those matter in the real world? Recall the research showing that kids in Asian countries that were trouncing us on achievement measures actually didn’t feel all that positive about themselves and their work. Remember, too, the research showing that many young Americans who were bursting with self-esteem could neither multiply two numbers nor write a cogent paragraph.
To repeat: Resist overselling. Don’t get run over by bandwagons. Avoid catching most fevers.
Chester E. Finn, Jr., is a Distinguished Senior Fellow and President Emeritus at the Thomas B. Fordham Institute. He is also a Senior Fellow at Stanford’s Hoover Institution.
This post originally appeared in Flypaper.
