The Difficulties of Measuring Government Success
Take weatherization. It’s harder than advertised to come up with an easy way to assess the program’s success.
Finding a simple way to see whether dollars are delivering a reasonable return on investment can be elusive for many government functions. Take preventive health care. Even when cities and states can effectively calculate the costs of keeping disease at bay, the actual savings may not be delivered for years or even decades, during which time the playing field can change entirely.
Still, we’ve long supported efforts to find the best (if not the perfect) means of using results-based measures to help upgrade performance for most governmental functions. Difficult as this may be, we’ve argued that, at a minimum, governments should go after the low-hanging fruit -- the functions, services or actions that are simple to measure. Perhaps there might even be lessons that can be applied to apples higher up in the tree.
We haven’t changed our minds about this. As time marches on, though, we’ve been forced to acknowledge a somewhat frustrating possibility: There may be very little low-hanging, easy-to-measure fruit out there for the plucking.
Exhibit A is weatherization. With billions of dollars spent on weatherization -- many of them through the 2009 American Recovery and Reinvestment Act -- it’s still unclear just how much good this approach to energy efficiency has accomplished. It seemed to us that measuring the benefits would be simplicity itself. Just take energy savings and divide it by the dollars spent.
We were wrong. It turns out that there are a number of areas of measurement relating to weatherization that aren’t “feasible for the states,” says Michael Blasnik, a weatherization consultant and a member of a team conducting a national weatherization evaluation led by the Oak Ridge National Laboratory. Blasnik cites measures such as broad economic and environmental impacts of weatherization efforts as among those that are far more within the province of the federal government than the individual states.
Turns out, data on many objectives aren’t very well developed in the states but are being pulled together for the national study. “Monetized values of emissions reductions is a complex issue in itself,” says Greg Dalhoff, another member of the Oak Ridge team doing the evaluation. “We are hoping to come up with those kinds of metrics in this national evaluation. If you wanted to have the states do this, it would be very expensive and difficult.”
Even at the federal level, multiple factors can be positively influenced by weatherization efforts, which don’t equate easily into dollars and cents. That’s because federal weatherization dollars are supposed to deliver significant benefits beyond reduction in energy usage. Societal benefits could include improved health and safety, reduced greenhouse gas emissions, reduced water and air pollution, water conservation, higher local employment, and increased local economic activity. An effort to pin this data down is in place, as the evaluation of the Recovery Act’s Weatherization Assistance Program plans to use primary data and a wide range of secondary data sources to estimate total non-energy benefits.
Simplicity isn’t the only issue. There’s a challenge that’s prevalent in a great many performance-oriented efforts: The money targeted to a particular goal may come from a variety of sources and be aimed at comingled missions. Any assessment of the cost-effectiveness of the Weatherization Assistance Program will fall short if it doesn’t take into account other sources of funding, including the Low Income Home Energy Assistance Program, individual utilities’ programs, state benefit funds, and various other national and state programs. In some cases, according to Dalhoff, subgrantee agencies received direct funding which may not be tracked by the state.
“So we have a wide array of funding sources with varying objectives,” Dalhoff explains. These in turn are “measured against a wide array of benefits beyond simple client bill savings -- benefits that are not easily monetized and for which monetized values are not readily available.”
One deceptively easy approach would be to use a methodologically strong model that would allow states or the federal government to forgo gathering actual energy bills from a representative sample of homeowners. But it turns out that experts are highly dubious of even this type of modeling approach.
“Even with really sophisticated energy models,” says Joel Eisenberg, manager of the Oak Ridge evaluation project, “when you have all the information about the energy usage pre- and post-weatherization -- when you plug those into models, the models tend not to reflect reality as much as we might like.” One problem, for instance, is that they tend to overestimate savings. “When you get to things like a residential structure with a changing household occupant population, behavior pattern and weather conditions, you need to be able to look at the real world,” Eisenberg says.
Ultimately, many of these weatherization measurement obstacles apply to a variety of other government programs. But one big challenge stands in the way of using almost all performance information. No matter how well the numbers are gathered and manipulated, the data itself is just a starting point. Appropriate analysis is the key to using this information.
Whatever the field, performance measurement is time consuming and complicated to do at the state level. Take a local agency that turns out to have lower savings in weatherization or any other venture than the rest of the state. That doesn’t mean it was a poor performer. “It could be that it just had fewer opportunities for saving,” says Blasnik, “and maybe other agencies should have saved even more because they had more opportunity for savings.”
We invite you to discuss and comment on this article using social media.