New Study on School Funding Assumes Its Way into Trouble

Sometimes I just want to get nerdy. I don’t mean kind of nerdy, like when we throw around phrases like “statistical significance” and call it a day. I mean really, truly nerdy. The kind of nerdy that involves using words like “exogeneity,” which is so obscure a term that Microsoft Word tells me it isn’t a word at all. In response, I proudly push my glasses up my nose and declare that not only is exogeneity a word, it is a word that matters.

What in the world am I talking about? The controversial new Education Next school funding study by C. Kirabo Jackson, Rucker C. Johnson, and Claudia Persico, of course! If you are one of those cool kids who doesn’t spend every morning perusing the latest studies on education, the quick and dirty on the study is this: It upends a great deal of research suggesting that simply increasing public school funding does little to increase academic achievement. Instead, it finds that if one changes the design of the research, large impacts are revealed. The proposed solution? You guessed it: More money.

From the study itself (emphasis added):

Previous national studies have examined the relationship between school resources and student outcomes and found little association for students born after 1950. Those studies, however, suffer from major design limitations. We address those limitations and demonstrate that, in fact, when examined in the right way, it becomes clear that increased school spending is linked to improved outcomes for students, and for low-income students in particular. Investigating the causal effect of school spending increases generated by the passage of SFRs, we conclude that increasing per-pupil spending yields large improvements in educational attainment, wages, and family income, and reductions in the annual incidence of adult poverty for children from low-income families. For children from nonpoor families, we find smaller effects of increased school spending on subsequent educational attainment and family income in adulthood.

This begs an obvious question: What exactly is the “right way” to examine the school funding issue? According to Kirabo et al., previous studies were limited by two major design decisions: The use of standardized test scores as dependent variables (or outcomes), and a focus on school funding as it is traditionally applied. Dr. Jay Greene nicely summarizes both problems and the authors’ solutions to them in a blog post:

[Kirabo et al.] believe that past research was flawed in two important respects.  First, test scores may be a weak indicator of later-life success, so it would be better to look at stronger measures, like educational attainment, employment, and earnings.  Second, they believe that past studies of school spending may suffer from an endogeneity problem.  That is, extra money has tended to go to schools facing challenges.  The failure to find a link between more resources and better achievement may be because schools with a weaker future trajectory are the ones more likely to get more money.  So, the causal arrow may be going in the wrong direction.  Weak performance may be causing more resources rather than more resources causing weak performance.

Jackson, et al solve the first issue by focusing on longer-term student outcomes, like educational attainment and earnings.  They claim to have a solution to the second problem by finding a type of spending increase that is unrelated to the expected trajectory of school performance.  Court-ordered spending, they say, is exogenous, while regular legislative increases in spending are endogenous.

There is some research linking test scores to life-oriented outcomes like educational attainment and earnings, but I’m not opposed to focusing solely on these outcomes in academic studies. In fact, I rather suspect that the results will be the same in most instances. So, fair enough on that design tweak. I do, however, seriously question how “exogenous” court-ordered spending truly is.

For those who don’t know, “exogenous” roughly translates to “external.” In statistics, researchers often need to be sure that a “treatment”—additional funding in this case—is applied to group being studied from outside the relevant system in order to reduce bias. In essence, Kirabo et al. argue that previous research (and there is a lot of it) was unduly influenced by endogenous, or internal, treatments that could not be adequately decoupled from the system being studied to provide good conclusions.

That sounds pretty smart, and maybe it is. The problem is that the authors don’t spend much time justifying the assertion that court-ordered spending back during the 1970s truly provided an exogenous shock effect. As Jay Greene puts it:

It is completely mysterious to me why we should believe that court-ordered spending differs from legislatively-originated spending in the likelihood that it is linked to the expected future trajectory of school performance.  That is, schools facing challenges are just as likely to get extra money if the spending originates in the courts or in the legislature.  If we are concerned that the causal arrow is going in the wrong direction in that weak performance causes more money rather than the other way around, we should have that concern just as much whether the motivation for the money came from the court or the legislature.

This concern is particularly important because deviating from the critical assumption of exogeneity brings the study’s results mostly back in line with previous research on the issue of school funding. Of course, we probably shouldn’t be surprised by that. In a separate response to the study, Rick Hess reminds us that education research is often based on fuzzy assumptions that may or may not reflect real-world truths.

The moral of the story is that you need to be careful about simply taking study results at face value, particularly when they are radically different than almost any other results to date. In the case of this study, I think I’ll stick to my guns (and history) and say that simply throwing money at a problem still isn’t really an effective solution.