You know what’s way more fun than debunking silly arguments about charter schools? And almost as exciting as celebrating fair funding for charter students in two of my favorite districts? New research showing huge improvements in New Orleans, which has the nation’s first all-charter system.

Well, almost all charter. Over 90 percent of the city’s students are enrolled in charter schools. For those roughly 40,000 kids, things are looking pretty bright. A new Education Next study by Douglas Harris finds some fairly staggering academic gains in the wake of sweeping reforms that followed New Orleans’ near-total destruction in Hurricane Katrina. Here’s a quick overview of those reforms from the study:

What happened to the New Orleans public schools following the tragic levee breeches after Hurricane Katrina is truly unprecedented. Within the span of one year, all public-school employees were fired, the teacher contract expired and was not replaced, and most attendance zones were eliminated. The state took control of almost all public schools and began holding them to relatively strict standards of academic achievement. Over time, the state turned all the schools under its authority over to charter management organizations (CMOs) that, in turn, dramatically reshaped the teacher workforce …

…. School leaders in New Orleans talk frequently about how critical flexibility in personnel management is to their overall school success. Free of state and local mandates and constraints from union contracts, leaders reopening schools after the storm could hire anyone they wanted, including uncertified teachers, and dismiss teachers relatively easily.

So yeah, the New Orleans reforms were a pretty big deal. They also happen to be rather controversial, so a great many people on both sides of the aisle have been watching the city rather closely. We talked about one piece of research back in January that found some significant transportation and information provision issues in the city’s system.

Harris’s study finds some of the same issues, but he also finds some incredible academic progress. Because a picture is worth a thousand words:

Quite a graph, right? For those who are more statistically minded, those squiggly lines are showing average effect sizes of .2 to .4 standard deviations, which in this case equate to score gains of 8 to 15 percent. Given that we’ve spent billions of dollars on the public education system to see average gains far smaller than that, those numbers are nothing to sneeze at.

Because I can feel my fellow policy nerds looking at me through the computer screen, these estimates are based on two difference-in-difference approaches: One using only New Orleans students who returned to the city after Katrina, and one using grade-based cohorts. Neither approach is perfect, but that is always the case with education research.

And yes, the study makes an attempt to account for things like shifting population characteristics after Katrina, the effects of trauma and disruption caused by the hurricane, and the effects of schools students attended between the hurricane and their return to the city. Harris finds that while some of those factors do diminish the overall average gains to some extent, none of them change the fact that New Orleans students are doing better now than they were under the old system.

What’s especially interesting, though, is that Harris also accounts for “test-based accountability distortions,” which most would call “teach-to-the-test effects.” This argument typically states that test score rises due to teach-to-the-test approaches do not reflect true learning gains. There’s some truth to that, no doubt. However, Harris’s interesting approach to this problem—and the findings it generated—should be mentioned:

To address this problem, we estimate effects separately by subject, recognizing that the stakes attached to math and language scores were roughly double the stakes for science and social studies scores during the period under analysis. Also, the state’s social promotion policy raises the stakes for students in grades 4 and 8. We find no evidence that the size of effects varied systematically with the stakes attached to the subjects or grades. However, it is hard to rule out other potential test-based accountability distortions with our data.

As further evidence, we considered descriptive information on non-test outcomes. State government reports indicate that, relative to the state as a whole, the New Orleans high school graduation rate and college entry rate (among high school graduates) rose 10 and 14 percentage points, respectively.

By nearly any measure, then, things are looking pretty good for charter students in New Orleans. There is still work to do when it comes to providing better information to low-income families about their options, and helping them physically access those options by finding solutions to the age-old transportation issue. And yes, it’s tough to pinpoint exactly which piece of New Orleans’ enormous reforms has had the biggest impact.

Still, I think the quick and dirty is this: A completely non-union, choice-based system that throws off the ancient shackles of the “way we’ve always done stuff” can accomplish some pretty awesome things despite arguments to the contrary. In fact, the success of New Orleans’ reforms raises serious questions about the validity of anti-reform folks’ biggest fears—non-certified teachers, strong accountability systems (including student learning data), a lack of collective bargaining, the presence of large numbers of charters, the involvement of charter management organizations, etc.

Not that they’ll be convinced by this study. It is, after all, hard to believe the boogeyman could actually be a knight in shining armor.