The Transformative Power of Comparison

Here's something that just about every state and local public official is loath to admit: Comparison is a powerful motivator. States don't like being ranked, and localities hate being rated (unless of course, they're sitting pretty in the top five). The fact is, though, those types of rankings and ratings get people's and the media's attention, and they spur action.

If anyone doubts the power of comparison, consider just one example: When former Governor Tim Kaine took office in Virginia in 2006, the state's ranking on finding permanent, stable placements for kids in foster care was 50th, aka dead last. Kaine didn't care for the distinction one bit, which is why the state's poor national standing ultimately led to a sweeping overhaul of how the state does children and family services, a transformation initiative led by his wife, First Lady Anne Holton.

"When we saw the data we realized that what we knew anecdotally to be the problem was in fact persistent and pervasive," says Kaine; "that Virginia was a clear outlier, with fewer discharges from foster care to permanency of any state."

That single stat led to an ambitious, multiyear overhaul of the state's children and family services system resulting in impressive gains in areas that include adoption rates, decreases in incidences of repeat maltreatment of kids and a decrease in kids in foster care overall.

The effort has been so successful that the new Republican administration in Richmond is picking up the standard. "We hope to expand transformation to every agency that touches at-risk children, including juvenile justice, mental health and the office of special education," says Bill Mims, former attorney general and co-chair of Governor Bob McDonnell's transition team. "We would hope that we'll see the same improvement in results in all those areas that we've been seeing in foster care."

That transformative power of comparison is exactly why every state official in the child welfare and public assistance world ought to check out two new reports compiled and published by the Council of State Governments in partnership with the Urban Institute, and funded by an Alfred P. Sloan Foundation grant. By way of full disclosure, I was part of the 24-member steering committee that helped guide the comparative performance measurement effort. The idea was to choose three significant areas of state policy, and then focus on key performance measures in those areas, resulting in state-by-state comparisons of performance. The three areas that the CSG steering committee picked were transportation, child welfare and public assistance.

The first thing that readers should know about the reports is that the primary researcher and writer working on them was Harry Hatry of the Urban Institute. For those who've never heard of Hatry, he's considered the father of the modern public sector performance measurement movement, and he's author of the current bible on the subject, Performance Measurement: Getting Results, Second Edition .

Hatry comes at the whole subject of comparative performance measurement with a level of sophistication and rigor that's beyond reproach. For example, he points out early in both reports that performance data and comparison are very tricky things: exigencies beyond a state's control can impact performance, and states often define "results" in similar program areas in slightly different ways.

Most important, though, Hatry points out that the rankings in the reports should not be used to hammer poor performers or as a reason to slap a blue ribbon on high performers. What the rankings should inspire, rather, is a nationwide conversation about the difference in state numbers and all the factors that influence performance in individual states. "Readers should not base their judgments as to whether particular states are performing well or poorly solely on the data shown in these charts," writes Hatry in each report's "Caveats" section. "Instead," he writes, "the reader should seek reasons for the unusually good or poor results."

What also makes the reports so good is that they really do hone in on a very limited set of fundamental measures that tell policymakers a great deal about how their state is doing in a particular area. For example, rather than report food stamp error rates in cases where people who didn't deserve food stamps got them, the report flips that and reports the reverse: the percentage of applicants who were eligible for food stamps but who were denied. Even more meaningful, arguably, the public assistance report ranks states by the percentage of population that reported "ever being hungry in the last 12 months but didn't eat because there wasn't enough money for food." Now there's a measure that tells you whether an assistance program is having an impact.

The same is true of the handful of key measures that were tracked for child welfare. The first chart in the report covers the number of children per thousand in foster care who were victims of maltreatment by a foster parent or facility staff member over a given 12-month period. In other words, safety first. The second covers the percent of kids reunited with their families less than 12 months after removal - a hugely significant indicator of a child welfare system's overall effectiveness.

So, download the reports and see where you sit on the continuum. Don't pop the champagne corks if you're highly ranked, and don't hang your head if you're in the bottom 20. That's not the goal of comparative performance measurement. Rather, start asking questions about why you rank where you do, and then start sharing information on why top performers are doing well, and what poorer performers could do to step up their game. That's the goal of comparative performance measurement initiatives. Certainly top performers should be recognized for excellence, but primarily, comparative performance measurement initiatives should be used to learn how to improve everybody's performance, and sustain it.