Driven in part by pressure from the Obama administration’s “Dear Colleague” guidance on school discipline more than 50 of America’s major school districts have implemented district-wide discipline reforms, and 27 states have revised their laws with an eye toward limiting suspensions. Discipline reformers, convinced that racial differences in student suspensions are largely attributable to teacher bias rather than student behavior, declare victory wherever suspension rates fall. But whatever the potential benefits to the relatively small share of students who avoid suspension, we know next to nothing about discipline reform’s potential costs.

As Matthew Steinberg and Johanna Lacoe noted in these pages, we have remarkably little conclusive evidence as to whether the difference in suspension rates is a product of racial bias, whether suspensions harm students, and how alternative disciplinary approaches stack up. They conclude that there is a “great need” for evidence, “on the impact of school discipline reforms and on their potential unintended consequences.”

There’s little evidence on the effects of discipline reform because the reforms are so recent and reliable data is scarce. Fortunately, the largest school district in America, New York City Public Schools, has administered an annual school-climate survey and publicly reported school-level results for the past decade. In the last five years, NYCPS has cut suspensions nearly in half under two reforms, both associated with approximately a 16,000 suspension decrease. Former Mayor Bloomberg’s reform, implemented in fall semester 2012, told teachers that they could no longer suspend a student for a first-time, low-level infraction. Mayor de Blasio’s reform, implemented in spring semester 2015, required principals to seek central office approval to suspend a student who committed repeated infractions.

To try to assess how school climate has shifted during these reforms, my recently released Manhattan Institute report, School Discipline Reform and Disorder, examines student and teacher responses to school-order related questions on the NYC School Survey. This is a descriptive examination not a causal analysis. It can tell us how school climate changed during discipline reforms but not whether school climate changed because of them. Still, the report ought to give policymakers pause: The data show that while school climate remained largely stable under Bloomberg, hundreds of schools serving hundreds of thousands of students became less respectful, less orderly, and more violent under Mayor Bill de Blasio.

The NYC School Survey

The NYC School Survey is administered every spring to parents, teachers, and students in grades 6-12. Response rates are generally high, hovering between 81-83% for the past five years. To assess climate shifts around each reform, I examined school-level survey responses from 2011-12 to 2015-16, and divided my observations into two periods: Period 1 from 2011-12 to 2013-14, and Period 2 from 2013-14 to 2015-16. In each three-year period analyzed, Year 1 is before a discipline reform, Year 2 is when a discipline reform is implemented, and Year 3 brings us well into full implementation.

Out of the 27 school order-related questions on the 2013-14 survey, the de Blasio administration deleted or substantively modified 22, leaving only 5 fit for comparison with the 2015-16 survey:

Student Questions:
1. At my school, students get into physical fights;
2. Most students at this school treat each other with respect;
3. At my school, students drink alcohol, use illegal drugs, or abuse prescription drugs while at school;
4. At my school, there is gang activity;

Teacher Question:
5. At my school, order and discipline are maintained.

The answers to these questions give us a window into school order. The survey appears to be a reliable instrument; student answers to school-order questions are strongly correlated with each other as well as with teacher answers.

To calculate shifts in school climate, I examined and displayed school-level changes in each period to the percentage of negative responses (e.g., teachers who “disagree” or “strongly disagree” when asked if order and discipline are maintained). I display those changes using a distribution of differences, stipulating that a change of less than five percentage points represents no shift, a change of five to fifteen represents a shift, and a change of 15+ represents a substantial shift. (Elementary schools are excluded from the analysis, as NYC DOE does not survey elementary school students.)

Results

Figure 5 shows whereas school climate remained stable under Bloomberg, under Mayor de Blasio’s reform: More than half of schools saw a deterioration in mutual respect compared to a fifth saw an improvement; on physical fighting, gang activity, and drug use, three times as many schools saw a deterioration as saw an improvement; to teachers, the shift was negative though more muted.

Figure 7 shows the distribution of differences by whether a school suspension rate dropped by more or less than three percent. Three things stand out from this figure: (1) In Period 1, the distribution of differences was more favorable among schools that saw substantial suspension reductions; (2) In Period 2, the distribution of differences in student responses is even across schools that saw substantial suspension reductions and those that did not; and (3) in Period 2, there were many schools where suspensions improved in the eyes of teachers, but not students.

Figures 10 and 11 show the distribution of differences by school composition, based respectively on percentage of minority students and students in poverty. In Period 1 under Bloomberg, the distribution was balanced and even regardless of school composition. In Period 2, however, there appears to be a disparate impact, by race and socio-economic status, to the negative shifts in school climate.

Discussion

The data suggest that school climate remained steady during Bloomberg’s discipline reform then deteriorated substantially during de Blasio’s. This is a purely descriptive study. Concurrence is certainly not causation and any number of other variables could be at play.

Still, given the fact that policymakers are almost completely in the dark regarding the effects of school discipline reform and must act based on the best evidence available, they should take this negative shift in school climate seriously. If we grant the (intuitive) assumption that this shift in school climate was related to a shift in discipline policy, what are the implications?

The answer partly depends on whether the negative shift in Period 2 was a cumulative and delayed response to both discipline reforms, or a rapid response to de Blasio’s alone. If the former, it suggests that lowering suspension rates is an entirely bad idea. If the latter, it suggests that the nature and character of a particular discipline reform matters much more than the number of suspensions it reduces.

Bloomberg’s reform prevented teachers from using suspensions as a tool-of-first-resort. Given the favorable distribution of differences in schools that saw a sizable suspension decrease in Period 1, this may have been a win-win that cut out unnecessary suspensions while fostering a more positive school environment.

De Blasio’s however, likely undermined teacher authority significantly. Students are always pushing the envelope. Knowing that teachers have an incentive to not suspend (even after repeated disciplinary infractions) is a recipe for schoolhouse mischief.

These are, of course, conjectures based on a plausible reading of descriptive evidence. But hopefully this evidence will help take the policy conversation beyond the facile assumption that lower suspension rates are necessarily a sign of progress. Hopefully it will also encourage further study into how various suspension reforms are playing out in practice.

We know next to nothing about the effects of these reforms. And given that most major districts lack a robust school climate survey, we simply can’t know much of anything in most places. But if we assume that what’s happened in NYC in the last two years is indicative of what’s been happening across the country, then we have to contend with the possibility that these reforms have harmed millions of America’s most vulnerable students.

Given that mere plausibility, the Trump administration should rescind Obama’s misguided “guidance” on discipline, which coerced school district administrators into second-guessing their teachers on how best to keep schools orderly, respectful, and safe. Once it’s no longer our official national policy to mistrust and second-guess teachers, we can have more serious, nuanced, local conversations on how best to approach school discipline… hopefully based on new evidence rather than old ideological assumptions.

—Max Eden

Max Eden is a senior fellow at the Manhattan Institute.