Welcome back, fellow policy explorers. I apologize for my absence these past few days, but the start of the 2016 legislative session and other pressing edu-business issues have kept me away from my keyboard this week. We’re back to work today, and will be looking at some new school choice research out of Louisiana.

First, a bit of bad news. We can no longer say no random-assignment study has ever found that private school choice programs have a negative effect on students. Until recently, there had been 12 random-assignment studies on the topic, of which six found positive impacts for all students, five found positive impacts for some students and not for others, and one found no visible effect.

Enter unlucky number 13. A working paper recently published by the National Bureau of Economic Research examined the effects of the Louisiana Scholarship Program (LSP), which provides vouchers for lower-income kids attending public schools with a C, D, or F grade under the state’s evaluation system. Started in 2008, the program was initially limited to just New Orleans—a place that many of you know I happen to see as something of a proof point in the reform conversation. The program went statewide in 2012, and now serves about 7,100 kids.

Because the LSP uses a lottery system to award vouchers at schools with more applicants than available seats, the researchers were able to easily compare randomly assigned (thus, “random-assignment”) voucher recipients and non-recipients in the program’s first statewide year. I’ll let you work your way through the full paper and its methodology on your own if you are so inclined. For now, we’ll settle for a snippet from the abstract:

This comparison reveals that LSP participation substantially reduces academic achievement. Attendance at an LSP-eligible private school lowers math scores by 0.4 standard deviations and increases the likelihood of a failing score by 50 percent. Voucher effects for reading, science and social studies are also negative and large. The negative impacts of vouchers are consistent across income groups, geographic areas, and private school characteristics, and are larger for younger children.

Ouch. I know not all of you are statistics nerds, but 0.4 standard deviations is pretty serious stuff in the world of academic outcomes. Not surprisingly given these results, opponents of choice and reform were quick to shout the news from the internet rooftops. Even now, I can imagine choice haters leaping from their chairs upon seeing the news and shrieking, “You see. School choice doesn’t work! I knew it.”

I suppose I could write some snarky comment about the silliness of hinging one’s conclusions on a single random-assignment study when there are 11 others showing positive results for at least some students, but that’s too obvious. Instead, I want to talk about what exactly this particular study may reveal.

Enter Cato’s Jason Bedrick, ardent supporter of educational freedom, professional smart dude, and a good friend of mine. Jason put together an excellent blog post taking a look at why the new study may have found what it found. From that post:

Why were the results from Louisiana’s voucher program so different? Although not conclusive, there is considerable evidence that the problem stemmed from poor program design. Regulations intended to guarantee quality might well have had the opposite effect. The LSP’s high level of private-school regulation appears to have driven away better schools while attracting primarily lower-performing schools with declining enrollments that were desperate for more funding.

The LSP is one of the most heavily regulated school choice programs in the nation. Schools that wish to accept voucher students may not set admissions criteria for voucher applicants. Schools may only give preference to siblings of current students or students enrolled in nearby district schools that have received a “D” or “F” grade. If there are more applicants than available voucher seats, the school must hold a lottery for equal priority students. Schools are forbidden to charge families any tuition beyond the value of the voucher, and they must administer the state test to voucher students.

At face value, these regulatory requirements make some sense. They appear to be aimed at making sure vouchers reach the underserved populations they are meant to help, and at making sure participating private schools are doing right by those kids. But as is the case with many well-intentioned government regulations, they fall victim to the Law of Unintended Consequences.

Only about a third of the eligible private schools in Louisiana participate in the LSP. That’s compared to two-thirds of the eligible schools in Florida’s venerable (and huge) scholarship tax credit program, and half of the eligible schools in Indiana’s voucher program, which is the largest such program in the country.

Admittedly, there are a lot of reasons schools may choose not to participate in an educational choice program. Fortunately, a helpful piece of survey-based research by the American Enterprise Institute sheds a little light on the issue in Louisiana. Here’s a chart from that study (hopefully this chart won’t result in me being accused of supporting Common Core):

From “Views From Private Schools”

As you can see, 64 percent of non-participating schools listed “future regulations that might come with participation” as a major factor in their decision not to jump in. The second-largest area of major concern was the alteration of schools’ “independence, character, or identity” (46 percent). Effects on admissions policies (45 percent), difficulty maintaining religious identity (43 percent), and requirements to teach to the state’s “curriculum standards” (43 percent) round out the top five major concerns. Concerns about testing requirements were a close sixth at 42 percent. Many other schools also listed these issues as at least minor concerns.

These data tell us that private school leaders are very concerned about regulatory burdens in school choice programs, and support the notion that schools often choose not to participate in choice programs due to regulatory issues. The issue seems particularly acute in cases where regulations could infringe upon schools’ independence or character academically, religiously, or otherwise. Given that finding, it’s not hard to see how forcing open-admissions policies, requiring state tests, or implicitly or explicitly requiring curricular shifts could make schools uncomfortable.

Aha, says Louisiana’s state superintendent of education in long, surprisingly hostile, and somewhat opaque response to Jason’s blog post. This kind of stubborn behavior is precisely why we can’t trust that silly “market” thing to force schools to do right by low-income kids in admissions policies, academic transparency, and accountability. We have to force them to do those things if we really want this choice stuff to work. (Or at least I think that’s what he said; as I mentioned, the post was a little convoluted.)

Sure, you could try that. But then the best schools will simply choose not to participate. From the Louisiana study’s conclusion:

Survey data indicate that LSP-eligible schools experience rapid enrollment declines relative to other nearby private schools before entering the program. This fact suggests that the LSP may attract a negatively-selected set of private schools struggling to maintain enrollment. Private school selection may also stem from the requirement that schools accept LSP vouchers as full payment of tuition. Voucher payments are capped at the per-pupil expenditure of sending districts, implying that only private school using fewer resources per student than public schools are likely to opt in to the program. Consistent with this argument, tuition at LSP-eligible private schools is typically well below public per-pupil spending (Louisiana Department of Education, 2014a). Our findings imply that these schools also provide lower educational quality.

In other words, there is strong reason to believe that LSP disproportionately attracted low-quality schools that really needed the money and were willing to swallow a stack of regulations to get it. The better-performing schools with fewer financial issues? Well, they appear to have decided to steer well clear of the LSP’s stringent requirements.

In fairness, none of this is conclusive, and there are other proposed reasons for the LSP’s negative results. Douglas Harris opined in an Education Next post that the results may well have been due to the relative effectiveness of surrounding public schools, especially under Louisiana’s tough accountability system. He also raises the possibility that the schools were not adequately aligned with or prepared to teach to the state standards tested by state assessments—though coupled with schools’ fears about having to alter curricula and character, I can’t help but see this as further evidence that mandated state testing doesn’t make much sense in private school choice programs.

Both Harris and Louisiana’s superintendent also offer up the explanation that these results were only from the LSP’s first year, and that they will improve over time as bad actors exit the system and schools adjust to their new reality. The latter explanation is definitely possible. But when it comes to bad actors exiting the system, I do have to wonder whether reducing the number of bad apples in a barrel mostly full of bad apples really shifts the paradigm.

Inconclusive though the regulation theory may be, I think it currently has the most evidentiary support. We will revisit that conclusion later if new evidence emerges. For now, we’d be wise to remember that educational choice programs exist to expand access to high-quality, effective educational options. And we know from existing research that well-designed choice programs can accomplish this goal. Indeed, many states have flourishing, successful educational choice programs.

We now have good reason to believe that heavy-handed regulations—no matter how well intentioned the folks who designed those regulations may have been— can significantly impair a choice program’s ability to accomplish its objectives. Put another way, pushing too hard in our pursuit of the perfect school choice program can lead to outcomes that are diametrically opposed to what we wanted in the first place.

I’m reminded of a famous quote by Friedrich Hayek, arguably the most famous free-market economist in history. In his best-known book, The Road to Serfdom, he wrote:

Is there a greater tragedy imaginable than that, in our endeavour consciously to shape our future in accordance with high ideals, we should in fact unwittingly produce the very opposite of what we have been striving for?

Poignant, right? And in the case of the Louisiana Scholarship Program, right on target.