State education rankings are commonly used in conversations about the U.S. education system. One of the most popular ranking systems is from the U.S. News and World Report.  The Cato Institute conducted a study which focused on the U.S. News and World Report’s education measurement, and set out to assess the bias behind this ranking system. The determined bias was due to the lack heterogeneity (demographic diversity) from state to state and subsequently skewed data.

The education rankings should reflect the performance of each state’s education system with a high degree of accuracy, but Cato points out “The most popular and influential state education rankings fail to provide an ‘apples to apples’ comparison between states.” This failure to find equitable measurements leads to the inaccuracy of some education evaluations.

The education rakings from U.S. News takes the average scores from the National Assessment of Educational Progress exam (NAEP), graduation rates, as well as ACT and SAT scores and assembles them into data that rates states from one to fifty on how well they educate students. The quality rank that Cato developed considers how “students from different socioeconomic and ethnic backgrounds tend to perform differently (regardless of the state they are in)” and “rates states according to how much learning similar students have relative to the amount of resources used to achieve it.” Colorado is a great example. U.S. News places Colorado 30^th in terms of education quality, while Cato’s quality rank places Colorado as 15^th in the nation. Once the heterogeneity of each state is considered in the Cato quality rank, the education rankings differ greatly.

Cato also investigated other factors that impact student performance. Teacher unions were one factor, where union strength negatively related to student performance. The negative relationship is due to unions opposing the removal of under-performing teachers, opposing merit-based pay and implementing stringent work rules. Cato also finds that even after cost of living adjustments, additional spending from states on K-12 education has reached a point of zero return.

Cato concedes their evaluation of student performance still has limitations, even though it is an improved ranking over that assigned by U.S. News. These limitations stem from education being treated as a single-dimensional variable when it is instead multi-faceted. The more one-dimensional an evaluation treats the student population, the greater the skew in favor of states that hold fewer socioeconomically challenged students.