Sadly, based on our careful reading of a number of performance audits and press reports, it turns out that even though the public sector has become even more obsessed with using data to make decisions, the quality of the data is still in serious question.
Consider, for example, a discovery near the end of last year that Florida and Texas had been sending the EPA inaccurate data about the status of their lead pipes, which the federal agency didn’t bother to check. Ultimately EPA’s Office of Inspector General discovered the flaws, but “the errors mean some states with the biggest needs may have to wait longer for funds — or will get less than they should have,” according to an Associated Press article. “For example, a single data entry error by Houston caused the EPA to allocate nearly $120 million more to Texas than it probably should have in fiscal year 2023.”
In fact, some experts are concerned that the quality of public sector data may even be getting worse. Some months ago, we quoted Mark Zandi, chief economist at Moody’s Analytics. who said in a podcast called “On the Evidence” from Mathematica that one of his greatest concerns is that “the quality of the data we’re using is starting to erode because a lot of it is based on surveys, and survey response rates are way down across the board.”
He added that “there are concerns around privacy, cyber issues, but for whatever reason the response rates are way down and that’s beginning to affect the data to a significant degree.”
Zandi takes this matter very seriously, and said, “If I were king for the day and I could devote resources to one thing, that would be it. Let’s go make our data sources more resilient, better, more comprehensive, more timely.”
Unfortunately, many people throw statistics around without the context that truly makes them useful. Just a few weeks ago we attended a conference where one of the keynote speakers was justifiably complaining about the problem of gun violence in the United States. She told the audience that guns were the leading cause of violence in the United States, but she missed out on one important wrinkle that put that number into important perspective.
According to the Pew Charitable Trusts, “Though they tend to get less public attention than gun-related murders, suicides have long accounted for the majority of U.S. gun deaths. In 2023, 58% of all gun-related deaths in the U.S. were suicides (27,300), while 38% were murders (17,927). The remaining gun deaths that year involved law enforcement (604), were accidental (463) or had undetermined circumstances (434), according to CDC data.”
Sometimes numbers that are accepted without question simply don’t make sense at all. Many, many years ago, the newspapers were full of stories that claimed that there were one million missing children each year. Milk cartons were full of warnings about the threat that children were disappearing on a regular basis.
At the time — and long before we started to write about state and local government — it struck us that one million seemed like an extraordinarily high figure, particularly since in our experience, we barely read about scores of missing children in the New York papers, with the exception of a 6-year old named Etan Patz who disappeared en route to his Manhattan school bus. We looked behind the “millions” figure and discovered that it had originated with a politician who picked that number out of a hat when speaking to the press. It was picked up from him, and used and reused, and this piece of fiction was widely felt to be fact.
We wondered about this, but never wrote an article about the greatly exaggerated number. Then the Denver Post discovered that “estimates of stranger abductions circulated at the time glossed over the reality that about 95 percent of missing-child reports were about runaways, while most of the rest involved custody disputes.” In fact the reporters won a Pulitzer Prize for this coverage in 1986.
Footnote: We’ve always regretted not having done that story ourselves, which may be a subconscious part of the reason we’re so intent on reporting about bad data every chance we get.
This commentary originally appeared on the authors’ website. Read the original here.
Governing’s opinion columns reflect the views of their authors and not necessarily those of Governing’s editors or management.
