New Study Studies Studies on School Choice

Well, friends, the 2016 legislative session is officially a done deal. I’ll have an official wrap-up (autopsy?) for you next week, but for now we can all breathe a little easier knowing that the crush of state-level education politics will recede for the most part until the fall. That leaves plenty of time to nerd it up, and nerd it up we shall.

Let’s get the policy party started today with a new study out of the University of Arkansas’s Department of Education Reform. Written by M. Danish Shakeel, Kaitlin P. Anderson, and Patrick J. Wolf, the study takes a look at the effects of private school choice programs around the world. Or, rather, the study looks at studies on the effects of private school choice programs around the world. That makes it a “meta-study.” Today’s lesson in impenetrable academic jargon: Studying studies yields meta-studies. You’re welcome.

Let’s get something out of the way right off the bat: I have a love-hate relationship with meta-studies. On one hand, comprehensive examinations of previous research are enormously valuable for those of us who swim in policy waters. On the other hand, they can easily fall victim to cherry picking, or the tendency to pick only studies that agree with whatever point you want to make. Then you have the issue of ensuring that the studies you are studying with your meta-study are actually decent—a question that often leads to screening processes that can, once again, easily fall victim to bias. That’s why you so often see meta-studies on the same subject reaching entirely different conclusions.

As a matter of fact, this particular meta-study is largely intended to correct what the researchers see as flaws in previous reviews of school choice research. The full version of the paper even includes a detailed rundown of issues with previous reviews, which it somewhat humorously entitles “A Systematic Review of the Systematic Reviews of Voucher Effectiveness.” Dr. Wolf summarizes the issues he and his colleagues found with previous efforts in a helpful blog post on Education Next:

The clarity of the results from our statistical meta-analysis contrasts with the fog of dispute that often surrounds discussions of the effectiveness of private school choice. Why does our summing of the evidence identify school choice as a clear success while others have claimed that it is a failure (see here and here)? Three factors have contributed to the muddled view regarding the effectiveness of school choice: ideology, the limitations of individual studies, and flawed prior reviews of the evidence.

School choice programs support parents who want access to private schooling for their child. Some people are ideologically opposed to such programs, regardless of the effects of school choice. Other people have avested interest in the public school system and resist the competition for students and funds that comes with private school choice. No amount of evidence is going to change their opinion that school choice is bad.

In an effort to set the record straight, Dr. Wolf and his colleagues selected 19 “gold-standard” studies on the effects of private school choice programs for analysis. For those who don’t remember, “gold-standard” studies are experimental evaluations (in the scientific sense, not the newfangled sense) that use random assignment to evaluate the effects of choice. Rather than selecting choice participants on the basis of academic prowess or parental engagement or anything else, these studies involve situations in which students are randomly assigned to a “treatment” group of students who participate in a school choice program and a “control” group of students who do not. This design allows researchers to mute most of the selection bias that could come from studies in which groups of students self-select into various buckets.

Dr. Wolf asserts in the blog post that the 19 studies selected represent “every experimental evaluation of choice produced to date, anywhere in the world.” The studies cover 11 school choice programs in the U.S., India, and Columbia. After lumping all that statistical goodness together, the study finds that:

The sum of the reliable evidence indicates that, on average, private school choice increases the reading scores of choice users by about 0.27 standard deviations and their math scores by 0.15 standard deviations. These are highly statistically significant, educationally meaningful achievement gains of several months of additional learning from school choice. The achievement benefits of private school choice appear to be somewhat larger for programs in developing countries than for those in the U.S. Publicly-funded programs produce larger test-score gains than privately-funded ones.

Put in non-academic language, school choice works. That shouldn’t really come as a surprise, especially if you’ve read Greg Forster’s literature review on school choice research in the United States. But there were some caveats.

The researchers find significantly larger effects in countries outside the United States, which they hypothesize might have something do with the quality difference between public and private schools in countries less fortunate than ours. That makes good sense, though I will admit I was a little surprised to see that some overall results for U.S. programs were null—a finding that I can’t help but feel may be partially due to recent research on Louisiana’s troubled voucher program.

Even with the caveats, however, this study is yet another piece of evidence that school choice programs can and do help students academically. Does every private school choice program in every corner of the world always help every student in terms of raising test scores? No, not necessarily. Design and implementation matter, and we all know that there are no silver bullets in the world of education.

We also know that school choice can have a wide variety of other positive effects on things outside of raw academic achievement. Graduation and college attendance rates spring immediately to mind, as do less measurable benefits stemming from access to safer learning environments, educational approaches that align with families’ values, and specific programs like Arrupe Jesuit High School’s Corporate Work Study Program or the Havern School for Children with Learning Disabilities’ intervention model.

In short, school choice is good stuff. I think there are perfectly fantastic reasons to believe that philosophically and morally, but there is also substantial (and growing) evidence that we should also be supportive of well-designed programs from an empirical perspective. Here’s hoping Colorado gets the message.