As you probably guessed from the long absence after my last post about two abominable “snowbills,” I’ve been spending a fair amount of time in the shiny hallways of the Colorado Capitol talking about the importance of choice and accountability.

Today, I’d like to take a break from politics and get back to policy. We’re going to do that by taking a look at the new Education Choice and Competition Index ratings from the Brookings Institution in Washington, D.C. Everybody likes ratings, right?

I’d bet the folks at Denver Public Schools are especially fond of ratings these days. Why? Well, because they sort of won. First, they took third place in a Fordham Institute analysis of America’s best cities for choice. And last week, Denver was revealed to be the highest-scoring large district in Brookings’ 2015 ECCI report—a pretty significant improvement from the district’s fifth-place finish in last year’s report. It was also the second-best district overall, surpassed only by New Orleans. In fact, it only lost out to the pretty awesome “Recovery District” by a single point (81-80) on Brookings’ 100-point scale.

First off, congratulations Denver! Woot! Please conduct the obligatory victory dance now. I’ll wait.

With that out of the way, we have to do a little nerding (nope, not a word) and dig into the info. That’s what we do here, after all. Brace yourselves, there be math ahead. 

Let’s start with how Brookings measures this stuff. ECCI ratings are based on the following factors (you can find the more technical scoring guide here):

From Brookings Institution

Denver did better on this scale in 2015 because of improvements made to its already pretty great (though not perfect) choice system, as well as a number of other changes. From the Brookings report:

The school choice environment in Denver was enhanced substantially for 2015 through increased enrollment in alternative schools, the ability of parents to make side by side comparisons of schools on the school assignment website, the elimination of default school assignments for about half the schools in the city, and the reservation of seats at choice schools so that parents could exercise choice 365 days a year.

Admittedly, that stuff sounds pretty good. So why did Denver come in second to New Orleans overall?

The Brookings report lists a couple of issues in Denver related to low choice participation at certain grade levels and among low-income and minority families. But the direct comparison of scores between Denver and New Orleans offered by Brookings’ interactive database paints a clearer picture.

When you look at those tables, you can see that Denver was dinged for not factoring the popularity of schools into funding, not closing schools with declining enrollment under the choice system (i.e., schools that parents choose not to send their children to), and—wait for it—a lack of private school options. Louisiana, on the other hand, does have private school choice programs. Most notably, the state has a relatively large voucher program and a scholarship tax credit program.

So yes, as I’ve said again and again and again, Colorado needs private school choice. But I’m not sure we want the type of private school choice program Brookings prefers. Why do I say that? Because under the current Brookings scale, Louisiana appears to have won by having flawed private school choice programs. The only way for Colorado to win would be to adopt its own flawed program.

Follow me closely on this. Brookings awards a total of three points in category seven for private school choice programs as follows:

From Brookings Institution

Note that the scoring system awards more points for having more heavily regulated programs, with the most points awarded for having a private school choice program in which “Public and publicly supported private schools can be directly compared in terms of their performance because they either participate in the same assessment regimen or different assessment regimens that are normed and standardized and thus directly comparable in terms of percentiles.”

In other words, you get extra credit for making private schools take the state test, or at least something that looks an awful lot like it. I don’t know about you, but I classify forcing private schools to conform to state testing regimens as heavy regulation of private school choice.

How a district does in each category of the Brookings scale is interpreted as a score. That score is essentially number of points earned divided by the total points available. Those scores are added together across categories (some scores are doubled to reflect the relative importance of certain categories), then divided by the total possible score to get a district’s overall rating. To compensate for the fact that some districts do not have private school choice programs, Brookings calculates overall scores using 15 possible points for districts without kids participating in private school choice programs and 16 for districts with kids participating in such programs.

Louisiana’s private school choice programs require students to take the state test. Thus, New Orleans got all three points possible under category seven of the Brookings scale. That’s 100 percent of available points, so it’s expressed as a score of one. But if Louisiana had more lightly regulated programs that only required students to take publicly reported nationally norm-referenced tests not “directly comparable” to criterion-referenced state tests, it would only have gotten two points out of a possible three. So it would have gotten a score of .67 instead of one in this category. After adding that decreased score to scores from other categories and dividing by 16, New Orleans’ overall score would have fallen from 81.25 to 79.1 on the Brookings scale. Meanwhile, Denver got an 80.

So if Louisiana had better school choice programs, New Orleans would have lost. Weird.

But what if Colorado had a private school choice program? Would that have allowed Denver to win out over New Orleans? Not unless the program were as highly regulated as Louisiana’s program. If Colorado’s hypothetical program required kids to test in ways not directly comparable to Colorado’s state tests, Denver’s overall rating would have fallen to 79.3 (thanks to the extra possible point requiring us to divide by 16 instead of 15).

For those of you thinking that Denver would have made up for this drop by getting an extra point under category one for having a private school choice program, think again. According to the data used by Broookings, Denver would have needed to have more than 10,000 kids in a private school choice program to reach the 45 percent alternative enrollment requirement for full points under category one. Not likely.

So, holding all else constant, having a lightly regulated private school choice program in place of no private school choice program at all would have hurt Denver’s score under ECCI. More choice, worse score. Think about that for a second.

On the other hand, if Colorado’s hypothetical program required state testing or directly comparable testing regimens, Denver would have gotten an 81.3, barely edging out New Orleans for the choice-iest district in America.

No matter which way you cut it, the path to victory in the epic struggle between Denver and New Orleans under the ECCI involves more regulation of private school choice. And that’s not a good thing.

Why is less regulation better? As you may recall, I recently wrote about the unintended consequences of overregulation in Louisiana’s voucher program. More specifically, we now have a strong (though admittedly not conclusive) reason to believe that heavy regulation led high-quality private schools to opt out of participation in that state’s voucher program. Meanwhile, the more questionable providers who badly needed the money were happy to swallow extra regulation to get it. The end result of a having providers who fall disproportionately on the lower end of the quality scale? The first random-assignment study on a private school choice program ever to find a negative impact on student achievement. Bummer.

To be fair, the imposition of state testing is not the only concerning factor in Louisiana’s regulatory scheme, so I can’t point my finger directly at that requirement as the sole culprit responsible for the state’s school choice woes. But it is certainly part of the regulatory equation that led many quality schools to refrain from participating in the programs, and therefore bears at least some of the blame.

I agree that the public has an interest in knowing how well private schools are serving kids in private school choice programs, but I most definitely do not agree that the correct way to achieve this is to subject them to the same (or nearly the same) “assessment regimen” as public schools.  Good intentions and the ECCI scale notwithstanding, evidence suggests that you can’t reach maximum choice through maximum regulation of private schools. Go figure.

I see two big lessons here. First, yay Denver! Second, ratings are only as good as the scales on which they’re based. I would love to see Denver—and the rest of Colorado—embrace full-spectrum choice by adopting a scholarship tax credit program. But if the only way for Denver to beat New Orleans in Brookings’ future ECCI reports is for Colorado to saddle any future private school choice program with the same kind of regulation we’ve seen in Louisiana, I think I’ll happily settle for second place.