Evidence Speaks focuses weekly on what we know about a relevant topic that can help policy makers make better decisions. For example, last week Mark Dynarski provided evidence on professional development for teachers. A search of Amazon turns up several books, in different policy areas, called “Data-Driven Decision Making.”
Although evidence clearly contributes to thoughtful policy-making, evidence cannot and should not drive policy decisions. When we make decisions, or policies, we are driven by a desire to achieve a set of goals. The role of data is to provide evidence on how our choices are likely to affect the realization of our goals. Evidence informs decisions so that, if the evidence is good and we interpret it well, the results of our decisions align better with what we value.
A challenge for many decision makers is to think clearly about the values they are seeking to realize. In education, decision makers are often motivated by the desire to improve student outcomes and increase educational equity. Yet both “student outcomes” and “equity” are vague terms. Which student outcomes, or combination of outcomes, are most valuable? Do we care about students’ understanding of trigonometry or their ability to run fast? Do we want to work towards all students having more equal cognitive skills or to increasing the skills of the least well off? Without more precise understandings of which outcomes we care about and which distributions of those outcomes are fair, decision makers lack orientation. Their decisions may end up relying on data about outcomes that happen to be available rather than about outcomes that align with their goals.
In a new book, Educational Goods: Values, Evidence, and Decision-Making (University of Chicago Press, 2018), we seek to spell out a set of educational values and distributive principles and to illustrate how they, along with a small number of non-education values, can be combined with the relevant evidence to improve education decision making. Two of us (Ladd and Loeb) are social scientists who bring a familiarity with the use of evidence, and two (Brighouse and Swift) are philosophers who operate in the realm of values.
The book provides new language to talk about the valued outcomes of the education process. We call the student outcomes that matter educational goods. The term “goods” signifies that they are valued. The adjective “educational” signifies that they arise from a broad educational process that includes families and communities as well as schools. Educational goods are the knowledge, skills, dispositions and attitudes needed for people to flourish and contribute to the flourishing of others.
At first, the approach may seem unhelpfully abstract. But it forces decision-makers to think carefully about the capacities needed to flourish in and contribute to society. For example, today in the US, adults need whatever knowledge, skills, dispositions and attitudes enable them to participate productively in the economy; to make their own, independent but informed decisions about how to go about their lives; to participate competently in democratic decision making; to make and sustain healthy interpersonal relationships; to treat others as equals; and to be fulfilled in their personal life. These capacities are specific enough for decision makers to collect relevant data and, then, to make trade-offs among them.
The overall level of educational goods is not the only concern. We also aim to distribute them fairly across students. Thirty years ago Christopher Jencks commented that “the enduring popularity of equal educational opportunity probably derives from the fact that we can all define it in different ways without realizing how profound our differences really are.” The same could be said today about the ideas of “equity” or “social justice”, vague terms that can mask a great deal of disagreement.
In Educational Goods, we shift the focus from those vague concepts to three specific distributive goals: adequacy, equality, and benefits to the disadvantaged. Adequacy values every child having at least enough educational goods to flourish above some threshold. The appeal of this value is obvious: a society that does not provide everyone with the developmental resources needed to live a quality life seems unjust. Adequacy, however, provides no insights into what would be a fair distribution for those above the threshold, and, in practice, choosing such a threshold is difficult and somewhat random. A second distributive goal is equality of educational goods. We may value all students having equal capacities to contribute to the workforce. No child is responsible for their social background or even for their level of natural talent, so allowing either to affect educational outcomes is problematic. However, achieving full equality of educational goods likely would require measures that seem unacceptable. For differences in natural ability not to affect educational outcomes would require at best neglect of, and at worse disabling, those with high levels of ability, to the ultimate detriment of all. Given what we know about how families work, preventing social class from influencing outcomes similarly would require considerable interference in family life. A third distributive value avoids the leveling down problem without requiring identifying an adequacy threshold, the idea of prioritizing benefit to the less advantaged. The idea is simple: inequalities are permissible, even good, if they work to benefit the less advantaged. The value allows for investment in those with natural talents if there’s a reasonable expectation that they will be productive in ways that benefit everyone. Some educational choices will likely improve all three distributive values but, at other times, achieving one distributive aim might require tradeoffs with the others.
While educational goods and their distribution are central to education policy decisions, other values come into play as well. While it may at first seem like these additional values are too numerous and ill-defined to specify, in fact, only a small set of values – we identify five of them – typically come into play in education decision making. Think again of the possibility that equalizing educational goods would require extensive intervention in the family. Respect for parents’ interests limits the pursuit of distributive goals. Think of another independent value – what we term childhood goods. Childhood goods are the experiences that students have in childhood that contribute to their flourishing even if they do not build their capacities. These goods may include purposeless play, as well as the joys of learning or laughing. We may be unwilling to undertake an educational approach that develops students’ educational goods if it, in turn, makes the students miserable in the process. The other independent values – respect for the democratic process, freedom of residence and occupation, and other goods (e.g. heath care or housing) – may also put a brake on what should be done to pursue educational goods and their valuable distribution.
Some educational decisions are quite simple with available evidence. For example, we may be faced with choosing between two early literacy programs, one of which has shown positive results for students and one of which has shown no benefit in similar studies. Here we use the evidence to simply choose the more effective program. However, one program may be better for some students than for others and, in this case, our decision would benefit from consideration both of the different distributive values that we hold and of the evidence about which students will benefit. By thinking carefully about how the programs might affect the distribution of outcomes, and then by collecting evidence on these effects, we will be in a better position to make the choice.
Many education decisions are substantially more complicated than choosing between two well-researched reading programs. Consider a district with two schools, one with a large concentration of students from the poorest area in the district and one with a large concentration of students from the richest area, with the first school showing far lower achievement than the second school. The district superintendent needs to decide whether to use her political capital to change the school zones so that the student composition is more equal or to reallocate available funds from the higher performing school to the lower performing school.
She wants to base her decision on evidence, but what evidence would be most useful? Certainly, findings about the effects of desegregation and funding on measures of achievement would be useful. But the choices are also likely to affect other educational goods – perhaps students’ capacities to treat others as equals or to develop close personal relationships. The choice may also affect childhood goods, as the experiences of the students may change, and it may affect whether parents choose to move from the district or send their children to non-district schools. To make these decisions, the superintendent needs evidence relevant to her goals. By identifying these goals more specifically than “student outcomes” and “equity,” she will be better positioned to align her decisions to her values. Researchers, similarly, will be in a better position develop the evidence that is most relevant to decisions aiming at valued goals.
We often hear the mantra that we need data-driven decision making or evidence-based decision making. The call to use evidence for decisions is the right one. But we need to know how our educational decisions will affect our valued goals. Evidence is useful when it is aligned to goals. One policy choice may improve test scores but if it reduces student motivation or childhood goods than it may be a poor choice, all things considered. Both decision makers and researchers could benefit from a more systematic consideration of the values at play so that the evidence is both relevant and comprehensive. Our goal in Educational Goods: Values, Evidence, and Decision-Making is to provide a framework to ease this process.
— Harry Brighouse, Helen F. Ladd, Susanna Loeb and Adam Swift
Harry Brighouse is a Professor of Political Philosophy, Philosophy of Education at the University of Wisconsin. Helen F. Ladd is the Susan B. King Professor Emeritus of Public Policy at Duke University, Samford School of Public Policy. Susanna Loeb is the Barnett Family Professor of Education at Stanford University, faculty director of CEPA Labs, and a co-director of Policy Analysis for California Education. Adam Swift is a Professor of Political Theory at the University of Warwick
This post originally appeared as part of Evidence Speaks, a weekly series of reports and notes by a standing panel of researchers under the editorship of Russ Whitehurst.
The author(s) were not paid by any entity outside of Brookings to write this particular article and did not receive financial support from or serve in a leadership position with any entity whose political or financial interests could be affected by this article.
