The purpose of my new book, Charter School City, is to describe the unprecedented New Orleans school reforms, piece together more than three dozen studies about them from the center I direct, and explain what all of this might tell us about the future of schooling nationally. As a first-of-its-kind reform, New Orleans is a fascinating case study, which I describe by intermingling evidence with stories and interviews from those most intimately involved in the reforms.
Jay Greene, who penned a review of the book for Education Next, and I both seem to agree on many of the book’s main conclusions. We agree that the reforms yielded large and positive effects on a wide variety of metrics: test scores, high school graduation rates, college-going, achievement/opportunity gaps, and parent satisfaction. We agree that these effects were at least partially due to the unusual conditions of New Orleans and therefore may not generalize to other cities and states. We agree that it is important to understand why those effects emerged in New Orleans.
That might be where the agreement ends, but it’s difficult to tell. Greene focuses his unsuccessful critique on one specific conclusion that “closing and taking over low-performing schools was the factor above all others, that explains the improved student outcomes,” and that choice and competition were not key factors. The first problem with his critique is its focus on such a narrow part of the book.
The second problem is that even this narrow line of attack, as he has written it, doesn’t make much sense. Greene would like you to believe that my conclusion on school takeover is based on a single flimsy analysis. In reality, it is based on five different and compelling types of evidence:
- We used a difference-in-differences strategy, comparing New Orleans students who experienced takeover, before and after the takeovers occurred, to a comparison group. This is a widely accepted method for identifying causal effects. Our version goes a step further and combines the difference-in-differences strategy with another that selects the comparison groups via matching methods. Greene only argues that the difference-in-differences approach “sometimes produces spurious findings.” That’s true. In fact, all research methods can yield spurious findings if they’re carried out poorly or with limited data. However, he offers no specific critique at all of our analysis. He also doesn’t mention that we carried out many different versions of the analysis, all of which yielded the same conclusion.
- The same study he mentions also provides evidence about why the takeover process was so effective in this case. In particular, my co-authors and I find that it was effective for a very simple reason: New Orleans students ended up in better schools (measured by school value-added) when their schools were taken over, which further reinforces the conclusion that takeover was a key contributor to improved student outcomes. It’s almost common sense that students do better in better schools, but this simple point is something that has been missed in much of the national discussion of school takeovers. The book also shows that the same pattern holds when we look at studies of takeover in other states. He doesn’t mention or question this part of the analysis either.
- We also compared the experiences of New Orleans with our neighbor, Baton Rouge. In contrast to New Orleans, the Baton Rouge results do not yield positive benefits for students. This turns out to be easy to explain using point 2 above. Baton Rouge students ended up in lower performing schools after takeovers, while New Orleans students ended up in higher performing ones. Greene labels this an “ad hoc explanation.” Does he believe that students don’t do better when they move to better schools? I don’t know because he, again, provides no specific critique.
- Rather than rely only on rigorous analysis of student outcomes, I also talked to New Orleans education leaders. This is a helpful general strategy for understanding why effects emerge. If the quantitative analysis discussed above had conflicted with what educators told me, then I would have been circumspect about the quantitative analysis. But, without exception, New Orleans education leaders said the takeover process was key. Though Greene recommends that “education reform might benefit from…greater acceptance of qualitative analysis,” he makes no mention of the qualitative evidence in the book.
- The performance-based takeover process seemed so important in that first study that we developed a novel way to decompose school improvement into multiple parts. That study, “From Evolution to Revolution: Market Dynamics in School Value-Added and Marketed Program Offerings Under the Post-Katrina School Reforms in New Orleans,” suggests that we probably under-estimated the role of the takeover process in the study he cites. To be fair to Greene, that additional study was not published when I was writing the book. Also, I didn’t see a need to add it in the final book edits, given how compelling the rest of the evidence was (see points 1-4).
Put another way, to argue that school takeovers were not the driving force behind improved student outcomes, as Greene seems to suggest, you have to believe most of the following: (a) we made some kind of unspecified error in the New Orleans analysis—an error that neither he nor any other peer reviewer has identified; (b) that those errors had no effect, or the opposite effect, in Baton Rouge; (c) that students actually do better in low-performing schools or that our measures of school performance (value-added) are wrong, despite considerable evidence to the contrary; (d) that all the education leaders in New Orleans I spoke with are wrong about what drove the measurable improvement; and (e) that we made an another large, unspecified error in the follow-up study decomposing the effects into different parts. I will let readers judge for themselves whether this scenario is likely. But the key is that Greene does not attempt to make a case that any of this is true.
Another way Greene might have approached his critique would have been to argue that factors other than school takeover mattered even more. He mentions choice and competition as an alternative possibility, but he does not attempt to rebut the evidence I present that these were not key factors. I present evidence that families, in choosing schools, focus on practical factors like distance and after-school care, as well as extracurricular activities, not the outcome measures whose improvement I was trying to explain. I present evidence that essentially all schools were guaranteed sufficient enrollment, which greatly reduced competitive pressures. I summarize the large national research base showing that the effects of competition, while positive, are widely considered to be small. He does not question any of this evidence or even propose a theory about why these findings might be misleading.
Greene does point out, correctly, that I argue competition had some unintended consequences, but, again, he offers no rebuttal. Instead, he suggests that I add in discussion about Arizona, which cannot possibly inform a debate about whether takeover worked in New Orleans. He also gets sidetracked on the question of whether the National Association of Charter School Authorizers’ process for reviewing applications to operate charter schools was successful in identifying the best applications. This point is relevant to why school takeovers mattered, but not to the larger point about whether they mattered. If this were a movie review, his approach here would be equivalent to trashing the movie because he doesn’t like one of the extra actors hanging out in the background of the scenes.
His review gets more puzzling as it moves along. Greene implies that school takeover was the explanation I “prefer” for the New Orleans experience. In coming to your own judgment, you have to ask, why would I prefer this? School takeover is one of the least popular ideas in the history of education and not one I had ever written positively about in the past. The reality is that I prefer this explanation because this is what the data tell me.
Another problem with the review comes when he writes that the importance of school takeovers in New Orleans is the “heart of [my] argument.” But he never actually states what my argument is. This makes his review hard to follow, especially as the main focus of the book isn’t on making arguments at all.
My best guess is that he is referring to the conclusion that I draw late in the book that the government has some important roles to play in schooling vis-a-vis markets. (This is one of the most interesting and surprising things we learned, especially given the measurable success of the New Orleans reforms and their market orientation.) Specifically, I propose a framework that I call Democratic Choice, which suggests five general roles for government: providing accountability for schools, ensuring access to quality schools for all students, providing information and acting with transparency, ensuring engagement of families and the community in system-level decisions, and providing (some forms of) choice. Notably, this framework doesn’t call for active school takeovers, which belies his assertion that school closure is the “heart of [my] argument.” It also explicitly includes an idea—choice—that he suggests I am biased against. Puzzling indeed.
In any event, you don’t need to support any element of the Democratic Choice framework to get something useful from the book. In fact, my greatest concern with Greene’s review is that it gets the essence of the book wrong. The value of Charter School City, in my view, is that it presents a comprehensive picture of the New Orleans school reforms, not a simplistic story, as Greene would have you believe. If you are interested in learning about the complexity of the school reforms, if you want to hear from supporters and opponents of the reforms in their own words, if you want to hear the vast trove of evidence (in New Orleans and beyond) woven together, if you want to hear what went well and what did not, if you want to be surprised and engaged, and if you want to use all of this to think deeply about the roles of government and markets, then I think you will find the book worthwhile.
I can understand why Greene would oppose a book that affirms some roles for government. He generally opposes roles for government (beyond funding), and sometimes he makes interesting and important points to support that view. Especially in this time of political polarization, we need to listen to people who think differently than ourselves. But having a different perspective is not enough. We have to really listen to one another’s arguments, think about the elements of those arguments and supporting evidence, and analyze each part. This is where Greene’s review falls short. At the end of his critique, he writes, “well-done case studies … make their accounts compelling … [using] standards of evidence and logical argumentation.” I agree, and I wish he had applied that same standard to his review.
The New Orleans school reforms certainly do draw strong reactions. This is because the stakes are high. After a quarter-century of mostly small experiments with school choice, charter schools, and reduced job protections for teachers, New Orleans was the first to try them all at once, at scale. For that reason, we’ll be talking about the city’s experiences for years to come. I believe Charter School City is a valuable resource in that discussion.
Douglas N. Harris is professor and chair of the economics department at Tulane University and is author of Charter School City. He is also Tulane’s Schlieder Foundation Chair in Public Education and director of the Education Research Alliance for New Orleans and the National Center for Research on Education Access and Choice (REACH).
