One of the greatest and longest-lasting education accomplishments of the George W. Bush Administration, in which I was honored to serve, was the creation of the Institute of Education Sciences. Thanks to the vision, courage, and persistence of IES’s first director, Grover “Russ” Whitehurst, education research is no longer the laughingstock of the social sciences. Every week we find multiple studies published on important topics, employing rigorous methodologies, and yielding findings that can inform classroom practice. This is an enormous accomplishment. (Getting educators to follow the evidence is another matter.)
Still and all, we’re not nearly as far along when it comes to publishing rigorous research that is relevant to policymakers, especially state leaders and lawmakers, who make many of the big decisions when it comes to K–12 schooling.
It may be that IES, and the scholars that it funds, are doing the best they possibly can. As Rick Hess (among others) has long argued, many questions in policy and governance simply cannot be answered by evidence. We often turn to the healthcare system for inspiration when it comes to research, looking at randomized field trials of pharmaceuticals as a model for what we’d like to do with research on instructional practice. Yet we don’t see randomized controlled trials doing much to inform the Obamacare debate, or helping to understand the relative wisdom of, say, insurance mandates versus high-risk pools.
Perhaps it’s the same in education. It’s far easier to figure out how to study a new math curriculum or blended learning app than it is to determine whether one particular approach to accountability is more effective than another. Partly it’s because some questions are fundamentally normative, involving trade-offs between competing priorities (like liberty versus equality). And partly it’s because some questions can’t be answered with an “n” of 50, or in a political system in which it’s impossible to randomly assign policies to states.
Consider charter school policy. A few weeks ago, I asked some very smart people who help states draft charter laws for a list of the key policy design questions that their policymakers tend to ask. When Kentucky legislators sat down to draft their new charter school law, for example, what questions did they need to answer? Here is what we came up with:
1. Should states limit charter schools’ numbers, locations, or types?
a. Should states place caps on the number of charter schools allowed to open, and/or the number of students allowed to be served?
b. Should states limit charter schools to certain geographic areas, such as urban communities or those with a high concentration of low-performing traditional public schools?
c. Should states allow for-profit companies to manage the operation of charter schools?2. Should states set any limits on charter schools’ open-enrollment policies?
a. Should states mandate that charter schools serve a minimum percentage of special education students, English language learners, or other subgroups?
b. Should states require charters to “backfill” when students leave mid-year?3. What policies should states adopt around charter school authorizing?
a. Which entities should be allowed to authorize charters?
b. Should states regulate the minimum or maximum length of a charter contract?
c. Should charter authorizers be held accountable? How so?
d. Should states adopt automatic closure policies for chronically low-performing charter schools?4. How should states fund charter schools?
a. How much funding should charter schools receive?
b. Should states provide charter facilities funding?
c. Should districts losing students to charters receive a share of the associated funding for a phase-out period?5. Should states set any regulations on charter school teachers?
a. Should charter teachers be required to be licensed?
b. Should charter teachers be required to participate in the state pension system, prohibited from participating, or given a choice?6. Should states allow online charter schools to open?
Note what isn’t even mentioned here: the basic question about whether charter schools “work.” For lawmakers who want to write a charter school law, that question is already answered. What they want to know is how to make their state’s charter sector work as well as possible—how to write a law in such a way that many high-quality schools will result.
I wondered if there was extant research that could answer these questions, so I turned to an excellent resource, a chapter on charter schools in the Handbook of the Economics of Education by Dennis Epple, Richard Romano, and Ron Zimmer. This is a remarkable document, covering in extensive detail the design and results of dozens of charter school studies from recent years. That topic has been examined every which way from Sunday. Yet I struggled to find answers to the policy design questions that policymakers actually face. I learned plenty about whether charter schools outperform district schools, and in which conditions, and whether competitive effects from charter schools can improve the traditional public school system. There is also a wealth of information available on the demographics of charter schools and whether they are representative of their neighborhoods.
All of those studies are helpful when we debate the fundamental question of whether we should embrace charter schools, and whether they are having a positive impact on students—both their own and those who remain in district-operated schools. But very few of these studies actually provide helpful information about how to increase the odds of spurring a large, high-quality charter school sector, at least without a great deal of translation and interpretation.
Why is that? My sense is that the policy design questions are exceptionally hard to answer with studies that are rigorous enough methodologically to make it into a chapter like this one. On the other hand, they are the exactly the kinds of questions that think tanks like Fordham, the Center on Reinventing Public Education, and many others have been struggling to answer as best we can. See, for example, our report from 2015, Pre-K and Charter Schools: Where State Policies Create Barriers to Collaboration. It provides concrete advice to lawmakers around two important policy areas, yet no one would mistake it for IES-style, gold-standard research or evaluation.
Many policy questions simply cannot be answered with hard evidence, at least entirely, regardless of rigor and methodology, because they’re the kinds of questions that have to be answered by common sense, ideology, and plain old experience. Groups like the National Alliance for Public Charter Schools and the Center for Education Reform have developed views over the years on what makes a strong charter law. These views are informed by evidence where possible, but they are also based on other considerations, beliefs, and direct experience.
On Thursday morning, the Fordham Institute and the Knowledge Alliance are bringing together policy wonks and academics to discuss whether and how we can build better bridges across the research-to-policy divide. We will talk about whether that divide is inevitable and perhaps unbridgeable, which would mean we must accept that much policymaking will never be evidence-based. I truly hope that’s not our conclusion. Join us if you can, or watch the recording when you’re able, and see for yourself. But be warned that your own conclusion, while surely informed by evidence, may not be scientific.
— Mike Petrilli
Mike Petrilli is president of the Thomas B. Fordham Institute, research fellow at Stanford University’s Hoover Institution, and executive editor of Education Next.
This first appeared on Flypaper.
