California Finds Predicting COVID Is Difficult to Get Right

For large parts of rural California, the numbers forecast by health officials ended up completely wrong because they didn’t account for social distancing or mask mandates. Many found the data useless.

(TNS) — In mid-July, California’s pandemic forecast painted a bleak picture for El Dorado County.

The state’s so-called “model of models” predicted 45 people with COVID-19 would die within 30 days in the sparsely populated county. With cases surging statewide and more than half of counties on a monitoring list, it was all-but-certain the death toll would soar in the foothills.

But there was a problem. The county hadn’t yet even tallied a single COVID-19 death.

Ultimately, the prediction for El Dorado County was a total bust in the best way possible. The disease caused by the new coronavirus infected just two people who died by Aug. 15, according to state data. California’s pandemic models this summer envisioned a catastrophic scenario in some of the state’s less-populated counties. But those forecasts proved to be wildly inaccurate, prompting several local health officers to dismiss the state’s forecasting website altogether — if they’d even heard of the effort at all, a Sacramento Bee review has found.

Carla Hass, an El Dorado County spokeswoman, said local public health officials did not use the state’s predictions “in any way” when deciding how to respond to the pandemic. The “wild swings” made the tool “not particularly useful.”

Sutter County, where only four deaths had been recorded in mid-July, would have 70 by mid-August, the model projected. Instead, only seven people died. Rachel Rosenbaum, a county spokeswoman, said their senior-living centers fared well during the pandemic compared to other places, which might explain why deaths didn’t spike as forecast. The health team uses models only when they’re “properly validated.”

And Solano County’s deaths were expected to nearly double from 31 to 59 between July 15 and Aug. 15, according to the forecast. Instead, the death count climbed by just 10.

Though the forecasts were much more accurate at projecting statewide totals and forecasts for more populated counties, like Sacramento, the inaccurate predictions cast doubt about whether the state can accurately estimate the course of a pandemic in rural counties.

Dr. Bela Matyas, Solano County’s health officer, said he was unaware of the state’s prediction website, largely because he doesn’t put much stock in modeling forecasts. Disease modeling rarely pans out, he said, because researchers cannot capture the dynamics inside one county, let alone between them. “Garbage in, garbage out,” he said.

“I have come to cynically believe that the only people who value models are modelers and politicians,” Matyas said. “People who work with disease on the streets just know how impossible it is to model what we see.”

Experts: Sparse Data, Changing Behavior Explain Errors

Modeling experts say overzealous predictions from the spring were lost in translation. They proved to be inaccurate because people took unprecedented action by sheltering in place.

California’s Department of Public Health said it stands by the tools. Significant errors in smaller counties should be expected because projections do not account for changes in policy — like the distancing rules or mask mandates.

Plus, a state spokesperson said, prediction models regularly fail to chart the course of the disease in smaller counties because there’s simply less data to work with.

“There is utility to providing estimates for smaller counties even if the estimates are less reliable,” the state said. “CDPH has been in active communication with local health jurisdictions as well as several smaller local health jurisdictions on the reliability of county-level estimates.”

To be sure, California’s “model of models” — an ensemble that uses a blend of forecasts instead of one single tool — has fared much better in recent months at predicting aggregate deaths in more populated places, The Bee review found.

The one-month forecast of deaths statewide was off by only about 1% in mid-August. And with a prediction of 181 deaths in Sacramento County, the model came remarkably close to the reality on the ground of 199.

California’s combination of models now forecasts nearly 1,000 people with COVID-19 will die each week through the end of September, pushing the state’s death toll toward 16,000 by the end of the month.

As for that wildly inaccurate projection for El Dorado County? The models adjusted their forecast. It now predicts four COVID-19 deaths by Oct. 2.

“It’s incredibly frustrating,” said Michael Saragosa, the mayor of Placerville in El Dorado County. “We get it. I want to trust the science. But the numbers are showing we just don’t have the rates or the deaths in our area.”

“The modeling was always going to be modeling that could be wrong.”

$813,000 For Two Modeling Contracts

In the spring, individual models conveyed disturbing projections about outbreaks, hospital shortages and death tolls.

In a letter to President Donald Trump, Gov. Gavin Newsom said models indicated half of the state’s population would be infected within two months. Newsom’s office quickly clarified that the forecast did not take into account social distancing and economic shutdowns.

Models from leading research institutions like Harvard and the University of Washington showed how hospitals would be overrun, ventilators would run out, and bodies would stack up in major cities.

While some of that happened, like in New York City, where freezer trucks parked outside hospitals overrun with bodies, few of those predictions bore any resemblance to the reality on the ground across the U.S.

That’s largely because the models were based on assumptions that did not — and could not accurately — account for widespread and unprecedented stay-at-home orders and the ripple effect precautions like that would have, experts say. The models often lacked context, and in a fast-paced rush for information, became grossly oversimplified.

To better fit what was occurring in the West, California pledged at least $813,000 in modeling contracts to Johns Hopkins University and Stanford, records show. It added another half-dozen modeling efforts to its contingent.

After months of requests from journalists and the public for the information, Newsom unveiled in late June a public website called the California COVID-19 Assessment Tool, or CalCAT, an open repository for anyone to download data about cases and projections and look for data-focused solutions.

“California is home to some of the world’s most accomplished researchers, technologists, scientists, acclaimed universities, and leading technology companies,” Newsom said at the time. “Today, I am opening more California data for them to help inform our efforts in combating this disease.”

The data portal marked the first real look under the hood of how California’s health officials used forecasts to predict the spread of COVID-19.

But since launching the data website, officials in recent months have not cited models in public comments nearly as frequently as in the spring. Turned off by the oversimplified and incorrect predictions from spring, the public seems also to be less interested in the forecasts.

‘Actions Will Change The Course Of The Epidemic’

Many people looked at death and infection forecasts like weather forecasts tracking a storm. When those didn’t pan out, people tuned the projections out, said Inga Holmdahl, a researcher at the Harvard T.H. Chan School of Public Health.

It’s a fundamental problem with public health forecasting, said Holmdahl, whose piece in the New England Journal of Medicine spelled out why modeling cannot be the panacea in public health. People can dramatically alter how disease spreads, leaving researchers scrambling to account for changes to social behaviors stemming from shutdowns or mask mandates.

“We can see an epidemiological model and respond,” Holmdahl said in an interview. “Our actions will change the course of the epidemic.”

Experts now are grappling with a scattershot approach to reopening schools and deciding the fate of indoor dining. “I think it’s just really hard to know what the impact of those is going to be,” Holmdahl said.

Early models oversimplified the situation, looked too far in the future and were miscommunicated to the public. Newer models with more data stand a better chance at getting it right, said Nicholas Reich, a pandemic forecaster and associate professor of biostatistics and epidemiology at The University of Massachusetts Amherst.

“As cases surge again, we have been given another opportunity to let models inform our understanding about the future of this outbreak,” Reich wrote on Twitter. “No model is a perfect crystal ball, but their consensus (or lack of it) can tell us something.”

©2020 The Sacramento Bee (Sacramento, Calif.) Distributed by Tribune Content Agency, LLC.

Special Projects
Sponsored Stories
Creating meaningful citizen experiences in a post-COVID world requires embracing digital initiatives like secure and ethical data sharing, artificial intelligence and more.
GHD identified four themes critical for municipalities to address to reach net-zero by 2050. Will you be ready?
As more state and local jurisdictions have placed a priority on creating sustainable and resilient communities, many have set strong targets to reduce the energy use and greenhouse gases (GHGs) associated with commercial and residential buildings.
As more people get vaccinated and states begin to roll back some of the restrictions put in place due to the COVID-19 pandemic — schools, agencies and workplaces are working on a plan on how to safely return to normal.
The solutions will be a permanent part of government even after the pandemic is over.
See simple ways agencies can improve the citizen engagement experience and make online work environments safer without busting the budget.
Whether your agency is already a well-oiled DevOps machine, or whether you’re just in the beginning stages of adopting a new software development methodology, one thing is certain: The security of your product is a top-of-mind concern.
The World Economic Forum predicts that by 2022, over half of the workforce will require significant reskilling or upskilling to do their jobs—and this data was published prior to the pandemic.
Part math problem and part unrealized social impact, recycling is at a tipping point. While there are critical system improvements to be made, in the end, success depends on millions of small decisions and actions by people.