One of the more substantial obstacles to measuring certain kinds of performance is that they are difficult to standardize. For example, say you want to measure the results of disaster relief. Obviously, an ideal goal would be to reduce the loss of life in a flood or a tornado. But it turns out that success is far more likely to depend on the severity of the disaster itself than on the programs in place. It’s like comparing rotten apples to putrid pears. “You can’t standardize an emergency,” says Michael Jacobson, deputy director for Performance and Strategy in King County, Wash. “Over time you’d say that loss of life should be going down, but one flood may come with a [category] 2 hurricane and the other with a [category] 3.”
Some of the areas that are particularly difficult to nail down are in administration, which often rely on analysis or relationship-building for their success. Budget offices fit neatly in this category. You can look at differences between expenditures planned and dollars actually spent, but it’s likely that this varies depending on many other things besides the actual performance of budgeters. Budget office performance, just like measuring the accuracy of revenue estimates, is dependent on external factors, such as hurricanes, the price of oil or unexpected national economic dips.
Beth Blauer, director of GovStat at the company Socrata and former director of StateStat in Maryland, says that economic development was always a tough area to measure because it was all about nurturing relationships. Cities and states have gotten sucked into focusing on process measures -- such as the number of business contacts made, or the number of trade shows attended -- but “it didn’t mean anything,” she says.
Complicating matters still further is an ongoing redefinition of success in economic development. “We’re translating [our goals] from attraction of large businesses to growing our own or supporting small businesses,” says Christine McFarland, the interim director of the Center for Research and Innovation at the National League of Cities. “We haven’t found the answer at all.”
Drummond Kahn, director of audit services in the Auditor’s Office in Portland, Ore., says the toughest things to measure are those where you’re looking for a long-term result. The results you want are in the future and may be affected by multiple factors.
Longitudinal data helps to pin down the relationship between government services and what they accomplish, but such data systems are expensive to build and still quite rare, for example, in the social services/health arena. For most social services, the data you can get is about the population at large and doesn’t provide information on the impact of services on individuals over time. You’d like to know whether homeless services for a particular person put him or her into a more stable living situation, but instead you get a headcount of the numbers of homeless at any one time. “Keeping track of an indigent person is not a simple thing,” says King County’s Jacobson.
It’s also difficult to measure things that involve areas that are hard to count. Again, homelessness is a good example. King County’s efforts are more aggressive than those in many places. City and county representatives head out annually, flashlights in hand, peering under bridges and inside parked cars in addition to getting headcounts from agencies.
Worthwhile? Yes. But there are two problems: Cities with lesser efforts to locate the homeless will appear to have fewer homeless people. Additionally, sampling techniques like this are problematic because they miss people who may be homeless, but reside indoors somewhere -- permanently couch-surfing with friends, say.
Government spending can be designed to get something to not happen -- instead of getting something to happen -- and that’s rather daunting to measure. How do you really know that you’ve prevented a terrorist attack? Or prevented someone from taking that first cigarette? “Prevention is the obvious answer [to many public problems],” says Tryens. “But what didn’t happen as a result of your activities? We measure the negative things about society. Is the absence of a negative a positive?”