Money Does Matter After All

 This is a response to “Money Matters After All?” by Eric Hanushek, published July 17, 2015 on the EdNext blog, which was a response to “Boosting Educational Attainment and Adult Earnings,” by C. Kirabo Jackson, Rucker C. Johnson and Claudia Persico, published in the Fall 2015 issue of Education Next.  That article was based on “The Effects of School Spending on Educational and Economic Outcomes: Evidence from School Finance Reforms,” The Quarterly Journal of Economics (forthcoming). Eric Hanushek has responded to this piece in a blog entry published on July 20, 2015 on the EdNext blog.

We would like to thank Eric Hanushek for his comments and interest in our work.  We appreciate the opportunity to offer a brief response. Hanushek provides an accurate description of our study and is correct that the methodological details matter. His critique, however, is not an objection to any of our methodological choices; he instead disputes our results. He states “while these [questions about measurement and how spending reactions to court decision is measured…] are important methodological issues, it is more useful to focus on the substance of their findings.” We take this as clear evidence that Hanushek finds our methodology sound. When the methods are sound, the results must be taken seriously.  We appreciate that Hanushek has done so in this case. His single, important critique of our key results is the “time trend” argument. Following the summary of our findings below, we present the “time trend” argument and highlight its flaws. We then discuss how we overcome the problems of the previous studies on which Hanushek bases his opinions. Finally, we discuss how our results differ from previous literature because (a) existing studies suffered from biases, and (b) the spending increases analyzed in our analysis were spent on more productive inputs than the spending increases examined in other studies.

Overview of our findings:

In most states, prior to the 1970s, most resources spent on K–12 schooling were raised at the local level, through local property taxes (Howell and Miller 1997; Hoxby 1996). Because the local property tax base is typically higher in areas with higher home values, and there are persistently high levels of residential segregation by socioeconomic status, heavy reliance on local financing contributed to affluent districts’ ability to spend more per student. In response to large within-state differences in per-pupil spending across wealthy/high-income and poor districts, state supreme courts overturned school finance systems in 28 states between 1971 and 2010, and many states implemented legislative reforms that spawned important changes in public education funding. The goal of these school finance reforms (SFRs) was to increase spending levels in low-spending districts, and in many cases to reduce the differences in per-pupil school-spending levels across districts. By design, some districts experienced increases in per-pupil spending while others may have experienced decreases (Murray, Evans, and Schwab 1998; Card and Payne 2002; Hoxby 2001). Our key finding is that increased per-pupil spending, induced by court-ordered SFRs, increased high school graduation rates, educational attainment, earnings, and family incomes for children who attended school after these reforms were implemented in affected districts. We find larger effects for low-income children, such that these reforms narrowed adult socioeconomic attainment differences between those raised in low- vs. high-income families.

What we do not find:

There are two misunderstandings about our findings that critics appear to make. As such, we feel it is helpful to outline what we do not conclude from our study.

1. We do not find that merely increasing spending will improve student outcomes irrespective of how it is spent. Though Hanushek’s critique may lead readers to think otherwise, at no point in our paper do we make claims suggesting that “policy makers…only have to concern themselves with how much money was provided to schools and not with how money was used.” We are very careful to highlight that how money is spent matters. We find that increased spending that leads to reductions in class sizes, increased teacher salaries and more instructional school days in a year improved outcomes. As such, one of our key conclusions is that, while how much money one spends does clearly matter, how it is spent is very important. The final lines of our full paper read, “Money alone may not be sufficient, but our findings indicate that provision of adequate funding may be a necessary condition. Importantly, we find that how the money is spent may be important. As such, to be most effective it is likely that spending increases should be coupled with systems that help ensure spending is allocated toward the most productive uses.”

2. We do not find that increasing spending by 22.7 percent will eliminate all differences in outcomes by socioeconomic status. This is a common misunderstanding of our findings that is also made by Hanushek. We find that a 22.7 percent spending increase is large enough to eliminate the average outcome differences between the poor (those with family incomes below twice the poverty line) and the non-poor (those with family incomes above twice the poverty line). Because there are large differences by socioeconomic status among those in each income group (e.g., the wealthy tend to have better outcomes than the average non-poor person, and the very poor tend to have worse outcomes than those just above the poverty line) eliminating the average difference in outcomes across the two broad groups does not eliminate all differences by socioeconomic status within each group. Simply put, just because a 22.7 percent spending increase is large enough to eliminate the average outcome differences between the poor and non-poor it does not mean that a 22.7 percent spending increase is large enough to eliminate the difference in outcome between the very poor and the very wealthy or differences across other measures of socioeconomic status. Also, we do not speculate that this spending increase will eliminate differences in outcomes by other categories such as race and gender. To illustrate this logic, consider the following simple mathematical example.

Illustrative example: There are 4 people in a society of different income levels. Persons are ranked by income level so that Person 1 is the richest and person 4 is the poorest. Richer individuals tend to have better outcomes such that Person 1 has 20 years of education, Person 2 has 18 years, Person 3 has 18 years and person 4 has 16 years of education. The average educational attainment for the two richest persons is 19 years and the average educational attainment for the two poorest persons is 17. The average gap between the high income group and the low income groups is 2 years. However, the gap between the richest and poorest person is 4 years. If one could increase the level of education for both lower income persons (persons 3 and 4) by 2 years, the average gap across the two groups would be eliminated. However, the richest person would still have 2 more years of education that the poorest person. This simple example illustrates that eliminating the average difference across the two groups will only remove all differences by socioeconomic status if there are no differences in outcomes by socioeconomic status within the broad income groups. Given that there are large difference in outcomes by socioeconomic status within broad income groups in the United States, this condition clearly does not hold in reality.

The Problem with Hanushek’s “Time Trend” Critique:

Now that the reader should have a clear sense of our paper and its implications, we now describe the Hanushek “time trend” argument. Hanushek points out that school spending in the United States has increased substantially between 1970 and present day. As such, he argues that, if our results are correct and school spending really does improve student outcomes (with larger effects for low-income children), outcomes should have improved over time and achievement gaps by income should have been eliminated over this time period. He then argues that any improvements between 1970 and today have been small so that it is unlikely that our conclusion that school spending improves student outcomes is correct.

While this “time trend” argument is intuitive, it is flawed for two reasons. The first reason is that it relies on the same flawed understanding of our results outlined above (i.e., that eliminating differences across two broad income groups implies eliminating all differences by socioeconomic status). The second problem with this “time trend” argument is that it is a facile argument based on fuzzy (albeit intuitive) logic. We highlight the problems of his logic below.

To see the problems of Hanushek’s logic, consider the following true statistics: between 1960 and 2000 the rate of cigarette smoking for females decreased by more than 30 percent while the rate of deaths by lung cancer increased by more than 50 percent over the same time period.[1] An analysis of these time trends might lead one to infer that smoking reduces lung cancer. However, most informed readers can point out numerous flaws in looking at this time trend evidence and concluding that “if smoking causes lung cancer, then there should have been a large corresponding reduction in cancer rates so that there can be no link between smoking and lung cancer.” However, this is exactly the facile logic invoked by Hanushek regarding the effect of school spending on student achievement.

While there are several problems with this simplistic argument, to avoid going too deeply into the weeds we focus on the most important flaw in this “time trend” argument. Simply put, the “time series” argument will hold only if nothing else has changed between 1970 and present day. It is important to bear in mind that these spending increases occurred against the backdrop of countervailing influences, such as the rise in single-parent families, more highly concentrated poverty, deterioration of neighborhood conditions for low-income families, the exodus of the middle class to the suburbs, mass incarceration, the crack epidemic, changes in migration patterns, and others. Consider just one countervailing factor: the significant rise in segregation by income between neighborhoods over the past four decades. This increased residential segregation was driven mostly by families with school-age children (Owens 2015), a simple reflection that quality of local schooling options is a key driver of segregation. This significant increase in residential sorting by income among families with school-age children would have likely led to far greater disparities in school resources by community socioeconomic status had SFRs not been an effective leveling tool.

In short, 1970 and 2010 is not an “apples-to-apples” comparison, so there is no reason to expect that the correlation between aggregate spending and aggregate outcomes over such a long time span will yield anything resembling a “causal” relationship. In fact, the observation that using simple correlations over time is unlikely to yield the true “causal” relationship is exactly what motivated us to follow a different methodological approach. Our methodological approach allows for an “apples-to-apples” comparison and allows us to disentangle the effects of school spending from that of all these other countervailing forces. Though Hanushek has chosen not to discuss the methodological advances in our work, they are important, and methods matter.

How We Overcome These Problems to Facilitate “Apples-to-Apples” Comparisons:

We make several decisions in order to facilitate more of an apples-to-apples comparison. First, we use fine-grained data on individual students, rather than comparing the entire United States in 1970 to the entire United States in 2010. With these finer-grained data we are able to account for a variety of other factors that may have changed over time such as family structure, childhood poverty, and neighborhood factors. Using these finer grained data, our main approach is to compare the outcomes of individuals with similar background characteristics born in the same school district but who attended public schools during different years (when per-pupil spending levels may have been different) — i.e., an apples-to-apples comparison. However, this is not all that we do to ensure that our results yield real causal relationships.

In our paper, we point out that even if one can carefully account for several observable factors (as we do), correlating all actual changes in school spending with changes in student outcomes is unlikely to yield causal relationships. We point out that some spending changes are unrelated to other factors that may obscure the real effect on outcomes (i.e., clean spending changes), while other kinds of spending changes would clearly yield erroneous results (i.e., confounded spending changes). We point out that many of the spending changes analyzed in previous studies may have been of the confounded variety. To give an example of such confounded spending changes, consider the following example. The federal Elementary and Secondary Education Act allocates additional funding to school districts with a high percentage of low-income students, who are more likely to have poor educational outcomes for reasons unrelated to school spending. As such, school districts serving declining neighborhoods are also those that are most likely to receive additional per-pupil spending over time. Such compensatory policies generate a negative relationship between changes in school spending and student outcomes that obscure the true relationship between school spending and student outcomes. We avoid this kind of problem by focusing only on clean spending changes. Specifically, we focus on the relationship between external “shocks” to school spending and long-run adult outcomes. The “shocks” we use are the sudden unanticipated increases in school spending experienced by predominantly low-spending districts soon after passage of court-mandated SFR.

As discussed above, by design, very soon after a court-ordered SFR in a state, some districts experienced sudden unanticipated increases in per-pupil spending (i.e., shocks) while others may have experienced decreases. Our analytic approach compares the outcomes of individuals who attended school before these spending shocks to those of similar individuals from the school district after these spending shocks. The validity of our design relies on the idea that districts that experienced sudden increases in school spending right after the passage of a court-ordered SFR were not already improving in other ways in exactly those same years. For this reason, we spend much time in our work showing that the timing of these spending shocks has nothing to do with underlying neighborhood changes or changes in family characteristics, so that changes in outcomes due to these shocks are likely to reflect a causal relationship. We encourage interested readers to consult the full paper for further detail.

Reconciling our results with the Older Literature:

Even though we outline the faulty assumptions in Hanushek’s “time trend” argument, in the interest of good social science it is helpful for us to try to reconcile our findings with the simple time-series evidence. As we explain above, our results do not imply that a 22.7 percent increase will eliminate all differences by parental socioeconomic status. However, they do suggest the much more realistic prediction that one might observe some convergence across groups over time as school spending has increased. Indeed this has been the case. For example, Krueger (1998) uses data from the NAEP and documents test score increases over time, with large improvements for disadvantaged children from poor urban areas; the Current Population Survey shows declining dropout rates since 1975 for those from the lowest income quartile (Digest of Education Statistics, NCES 2012). Murnane (2013) finds that high school completion rates have been increasing since 1970 with larger increases for black and Hispanic students; Baum, Ma and Pavea (2013) find that postsecondary enrollment rates have been increasing since the 1980s, particularly for those from poor families. Contrary to Hanushek’s assertions, outcomes have improved. Importantly, these improvements are consistent with increase in school spending playing a key role.

Finally, Hanushek proposes three reasons why our estimates (if true) may not track the national time trends very well. His ideas are not novel — we considered, tested, and addressed them ourselves in the paper and herein.  First, he says there may be diminishing marginal returns to schools spending. Indeed we find that this is the case in our study. Areas with the lowest initial spending levels were also those for which increased spending had the most pronounced positive effect. The second reason he cites is that spending induced by the courts might have large effects while spending not related to judicial rulings have small effects. Indeed we find evidence of this also. Specifically, spending increases associated with court-mandated reform are much more strongly related to improvement in measured school inputs (e.g., student-to-teacher ratios, length of the school year) than ordinary spending increases. There are a few explanations for this that we explore in our study. Finally, he proposes that our estimates are wrong. We propose an alternative: the time series evidence Hanushek relies on does not reflect a causal relationship. Indeed in our larger study, we show that simple correlations are obscured by a variety of other factors that also influence student outcomes. We also present numerous pieces of analysis in our larger study that support a causal interpretation of our results.

To be clear, we do not think that our study is the final word on the question of whether increasing school spending will improve student outcomes in all contexts. As Hanushek himself concedes “none of this discussion suggests that money never matters. Or that money cannot matter.” Here we will make a similar concession; none of what we show suggests that money always matters. We show that money did matter and that it mattered quite a lot. What our study does is dispels the notion that school spending does not matter, so that one must look only at how it is spent. We find that money does matter and how it is spent matters. Contrary to Hanushek’s claims, our findings do not let policymakers off the hook. Our findings suggest that it is extremely important that money is allocated effectively and also that it is allocated equitably so that all schools have the resources necessary to help all children succeed.

— C. Kirabo Jackson, Rucker C. Johnson and Claudia Persico

C. Kirabo Jackson is associate professor of human development and social policy at Northwestern University. Rucker C. Johnson is associate professor of public policy at University of California, Berkeley. Claudia Persico is a doctoral candidate in human development and social policy at Northwestern University.