Not in the Right Ballpark

By 07/20/2015

1 Comment | Print | NO PDF |

This blog entry is part of a debate over “Boosting Educational Attainment and Adult Earnings,” by C. Kirabo Jackson, Rucker C. Johnson and Claudia Persico, a study which was published in the Fall 2015 issue of Education Next. That article was based on “The Effects of School Spending on Educational and Economic Outcomes: Evidence from School Finance Reforms,” The Quarterly Journal of Economics (forthcoming). Eric Hanushek first critiqued the study in “Money Matters After All?” published July 17, 2015 on the Ed Next blog. The authors of the study responded in a blog entry, “Money Does Matter After All,”  also published on July 17, 2015 on the Ed Next blog. What appears below is a response to “Money Does Matter After All.”

My critique of the paper by Jackson, Johnson, and Persico is very simple and might be lost in the dazzling misdirection of their response.  When I learned computer programming, I was taught to use simplified approximations of results to make sure that my more complicated, and harder to check, programs produced answers that were in the right ballpark.  This step apparently is no longer taught.

Jackson, Johnson, and Persico claim that a 22.7 percent spending increase is large enough to eliminate the average outcome difference in school attainment between the poor and non-poor.  This is their estimate of the causal effect of added spending.  Between 1970 and 2010 we saw real spending increases per pupil of roughly 150 percent, or over six times what they claim is necessary to close the average attainment gap between poor children (those in families below two times the poverty level) and nonpoor children (those in families above two times the poverty level).  Applying their estimates of the causal effect of added spending to the actual increases in spending suggests that the average poor-nonpoor gap in school attainment should have been more than closed – which is not even close to what we observe.  A large and frustratingly-resilient average attainment gap continues to exist between children in poor and nonpoor families.

Separating wheat from chaff is difficult, so a few added remarks are useful.

•Nobody suggests that one should use the time series pattern of spending increases to estimate the causal effect of spending.  The approximation in my critique was introduced only to see if their answer could be in the right ballpark.  It does not look to be.

•There is also no suggestion in my critique that there would not continue to be a distribution of outcomes within poor and nonpoor groups.  If the average poor and nonpoor students got something close to a 150 percent increase in spending, the Jackson, Johnson, and Persico results imply that the gap in average attainment by poverty status should have been more than closed and indeed reversed.  This estimate of the effect on average differences says nothing about whether there is a remaining distribution of attainment, which one would guess that there would be.

•My calculations are not an extrapolation beyond the bounds of their data, because they pertain to the data Jackson, Johnson, and Persico use in obtaining their causal estimates.

•Nobody disagrees that over time the U.S. student population has changed.  There are more children in single-parent families, more children in families where English is not the first language, and more children in families below the poverty level – all factors that might lead to a more difficult to educate student population.  But there are also more educated parents and smaller families over time – factors that might lead to an easier to educate population.  No analysis has accurately netted out these influences. But, back to their causal estimates:  one would have to believe that the net changes in families fully absorb the six-fold spending increase above their mean-equalizing level in order to reconcile actual spending increases with the clear lack of any dramatic outcome improvements.  If the “quality” of our children has really deteriorated so much, we as a nation are in much more trouble than most people believe.

•The investigation of “mechanisms” by Jackson, Johnson, and Persico does not show that reducing pupil-teacher ratios or increasing teacher salaries is causally related to better outcomes.  They show that court-induced funding tends to go more toward these crudely-measured inputs than non-court-induced spending – but they do not show that these factors actually led to the outcome improvements that they identify.  Indeed these findings on mechanisms also fail a simple sniff test:  Between 1970 and 2008, pupil-teacher ratios for the nation fell by over thirty percent.   Indeed this is the biggest driver of the increase in spending over that period, and spending unrelated to court orders followed this policy very closely – but outcomes did not.

•When proclaiming the importance of their causal estimates of the impact of school spending, there are no qualifications: “Our findings provide compelling evidence that money does matter, and that additional school resources can meaningfully improve long-run outcomes for students.”  But, when interpreting their results for policy, they are much more circumspect:  “Spending increases should be coupled with systems that help ensure spending is allocated toward the most productive uses” – a conclusion that is unrelated to their causal analysis but that reflects exactly the conclusion to which most people have come.

The central issue remains:  How do we ensure that added funds to schools are used in ways that improve student outcomes?  Jackson, Johnson, and Persico and I completely agree that just providing money is not sufficient for getting good results.  The time series evidence shows that the existing incentives in schools have not produced consistently better results to go along with dramatically increased funding, even if some funds have been used effectively.  We apparently disagree on the other half – the necessity of spending, but I fail to see how their analysis or other available evidence shows that more money is a necessary condition for improvement.  It may or may not be necessary, but arriving at any conclusion on this depends on specific policies. We need to see the results of a set of policies that consistently improves achievement in order to assess whether (and how much) more funding is really necessary.

– Eric Hanushek

Eric Hanushek is the Paul and Jean Hanna Senior Fellow at the Hoover Institution of Stanford University.

Comment on this article
  • Mark Dynarski says:

    An interesting discussion. It is always useful to remind ourselves, as researchers, that if the econometrics yields results that differ from the facts, the likely answer is that the econometrics are wrong. That is what I read Eric Hanushek to be saying about doing approximations to figure out if one’s computer programming is flawed.

    Jackson et al’s finding is that increasing spending by 23 percent is enough to eliminate the achievement gap. That’s the econometrics. Rick Hanushek is saying that an increase of spending that is six times more than that (the facts) did not eliminate the gap (the facts). So, the econometrics are wrong. Their techniques were done correctly, as far as it goes, but the techniques are yielding the wrong answer.

    The Jackson et al approach of using court-ordered increases in funding as if these dropped out of the sky is an issue. Court cases are not exogenous. They arise when organizations perceiving education-spending disparities can mount a successful legal case challenging them. Using court cases as an ‘instrumental variable’ is clever, but instrumental variables techniques rarely work unless the instrument is something clearly exogenous, like assignment to treatment in an experiment. Winning a court case to increase spending is a long way from being exogenous.

    Here’s a thought experiment. Assignment to attend charter schools is often random, through a lottery process. Charter schools rarely spend more than traditional public schools and often less. Yet some, like KIPP schools, result in higher levels of achievement, when tudents who win the lottery are compared to similar students who lose the lottery. So–spending doesn’t matter, by this logic. But these schools often are open longer and require heavier investments of of time from their teachers and staff. So *resources* matter.

    The answer that resource allocation matters seems trite. But concluding that *spending* on schools improves achievement is flawed if the allocation of resources is the real issue. Spending can increase without any change in the allocation of resources. Both sides seem to agree on this.

  • Comment on this Article

    Name ()


         1 Comment
    Sponsored Results

    The Hoover Institution at Stanford University - Ideas Defining a Free Society

    Harvard Kennedy School Program on Educational Policy and Governance

    Thomas Fordham Institute - Advancing Educational Excellence and Education Reform