Technology

Meltdown

How a promising technology mega-project went awry.
by | July 2007

Jeanne Nicholson had high hopes for Colorado's latest big technology project. A new computer system was supposed to streamline the way food stamps, Medicaid and other benefits for poor people are administered. Nicholson loved the idea. She's a county commissioner in Gilpin County, in the mountains west of Denver. And she understood how these programs, which state government oversees but the counties run, had become unwieldy for managers and clients alike. The fragmented old computer system--six systems, really--sent mothers trudging from one county office to another just so that they could feed their children or see a doctor.

As the $200 million project stumbled toward its launch date, however, Nicholson began to feel nervous. The state was in a rush because funding was limited, and there had already been major delays. The system was going to be unleashed statewide all at once, without phasing it in to work out the kinks. And early tests showed many more glitches than Nicholson was comfortable with. She wasn't the only one feeling anxious. All 64 Colorado counties asked the state not to flip the on-switch until fixes were made. State officials who had the final say over that decision decided to go ahead anyway.

It didn't take long for the counties' nightmares to come true. The new system hiccupped and sputtered. Nicholson tells the tale of one Gilpin County mother who came in saying her children were hungry and needed food stamps. She was erroneously turned away by an intake worker who trusted the computer's mistaken calculation to deny her benefits. "Six months later, the client comes back and says her kids have been eating nothing but popcorn and crackers. The technician inputs the same data. This time the family is eligible." Later, due to other glitches, the system repeatedly paid the same client MORE than her family was eligible for. "The system didn't know what the right hand and the left hand were doing," Nicholson says.

It's been exactly two years since Colorado suffered one of the bigger government technology meltdowns in recent memory. As efforts to fix the system continue, the administration of Governor Bill Owens, state legislators and software developers are digesting lessons that their peers in other states would do well to pay attention to. State and local technology managers have become leery of mega-IT projects with nine-digit price tags. That's because when a big project starts heading for trouble, the problems also are grand in scale.

Large IT projects can hit the skids for many reasons. Weak project management. Ignored red flags. Slashed funding. Changes in administration. These projects are complex by nature and require many players, both inside and outside government, to agree on myriad policy and technical decisions. Still, with all the many manuals and best- practice recommendations that have been written on how to implement technology projects, it would seem governments shouldn't keep falling into the same holes.

The good news from Colorado is that a spate of recent tech failures (several other projects also went bust) has left behind a trail of audits, studies and consultants' reports. Those autopsies, combined with the reflections of people involved in the welfare benefits system, offer a unique inside look at the anatomy of one of the state's biggest tech disasters. It's a cautionary tale of good intentions and bad governance, and above all bureaucratic momentum. Colorado's experience indicates that once government gets cranking on a mega-IT project, it can become impossible to stop--even when most of the people involved suspect it's doomed. "I kept thinking, 'There's no way they're going to release this turkey. It's so poorly developed,'" says Tom Mayer, a Boulder County commissioner. "They did it anyway."

"YOU DON'T JUST STOP"

Colorado's saga began with a great concept. The idea was to junk the six antiquated computer systems and develop one state-of-the-art system to take care of welfare clients. It was supposed to save a lot of time and grief, both for the county social services offices and for the 669,000 Coloradans receiving benefits. The system would translate state and federal rules for 80 different programs into simple and consistent answers regarding eligibility. The goal wasn't to save money, per se--the computer was expected to turn up benefits that clients never knew to apply for. Rather, the intent of the Colorado Benefits Management System, or CBMS, was to stretch welfare dollars further by putting more money into the hands of poor families and less into administration.

For such an ambitious project, however, no one person was put in charge. Instead, responsibility fell on two state agencies, the Department of Human Services and the Department of Health Care Policy and Financing. The two agencies had a history of not getting along. Complicating matters further, they had to coordinate with more than five dozen counties that were going to use the system, and which have multiple ways of doing business. It was a weak governance structure that left much to be desired.

The project, begun in 2002, hit snags right from the start. Most had to do with money. After Colorado issued its request for proposals, the economy soured and the project had to be scaled back. The agencies selected a bid submitted by EDS, but in order to save money, asked the vendor to allow state employees to take some of the software development back in-house. After further negotiations, EDS agreed. This shift saved money in the short run. Only later did it seem foolish.

Another important money-saving decision wound up backfiring. Colorado had asked EDS to drive down costs by modifying a similar computer system already used in another state. EDS first tried using a model it had built for Arkansas but later determined that it couldn't be adapted to meet Colorado's timetable or functionality requirements. "Eighteen months into the project, you learn what you were building isn't going to work," says Ron Huston, chief information officer for the Department of Human Services. "You have to start over." The next model put into play was California's. It turned out to be a better fit for Colorado. But the change added 10 months to the implementation schedule.

The problems were obvious to some observers before they unfolded. Maximus, a consulting company required to oversee projects that use federal dollars, reported in January 2004 that CBMS was "flawed from the moment it was initially designed." Huston didn't like being second-guessed by Maximus, which he says he "butted heads with constantly" over matters of business methodology. When Maximus' contract came up for renewal nine months before CBMS was set to launch, the agency heads chose not to renew it. "It was a real tragedy," says Donald McCubbrey, a professor at the University of Denver's business school who taught the CBMS fiasco as a case study for his graduate students. "It's not like there was no one standing there saying you're going to drive off a cliff."

Troubles only seemed to cascade. Some of the tasks the state took back in-house to save money weren't performed well. For instance, state employees developed "decision tables" that form the underlying logic for determining a client's eligibility for a benefit. The software the state designed didn't take into account several important pieces of information, such as federal changes in the rules governing Social Security, Medicaid and cost-of-living adjustments.

Normally when large tech projects roll out, they go live piece by piece in staggered steps so that bugs can be worked out along the way. That was the original plan with CBMS--to phase in the system in a few counties first before going statewide. But as delays mounted and the planned September 2004 launch date neared, Colorado nixed that plan. This enraged county social services managers. They'd seen early tests of the system and knew that CBMS was spitting out a lot of erroneous information. All 64 counties signed a letter asking the state to push back the launch date. The answer was no. "You just don't do that," Professor McCubbrey says, noting that 15 of the 17 graduate students in his class concluded that they would have delayed the project. "In the face of user objections that the system wasn't tested, it struck me as very risky."

Yet state officials felt immense pressure to stick to the schedule. Legislative elections were a few months away, and nobody wanted to poison the political atmosphere with an embarrassing delay. Plus, money was running out. Between state employees and EDS, some 175 people were working on the venture, burning through $3 million a month. Management gurus have a clunky term for the tendency of large organizations to continue pouring money into a project even when it is not going well. They call it "escalation of commitment to a failing course of action." Some would say that's essentially what happened in Colorado. With close to $100 million already invested, CBMS had gained too much momentum to be slowed down. "You don't just stop," Huston explains. "It sends the wrong message. When you're gearing up for the last push, you're at full throttle."

BENEFITS DENIED

Once CBMS launched, the chaos that counties had feared broke out almost immediately. In Gilpin County, one client at first was mistakenly denied food stamps but later received check after check for the back payment when she should have received only one. Other clients were denied benefits they were legally entitled to. As Paula McKey, Boulder County's welfare director, puts it, "The clients were the guinea pigs."

In Gilpin County, computer errors became so common that the county devised a very low-tech workaround: Social services began giving away gift cards. That way, if pharmacists refused to fill prescriptions from poor residents because of computer glitches, Medicaid recipients could still get medicines. Food banks were stocked up in the event that hungry clients were mistakenly turned down for food stamps. "We used this buffering so the client was not harmed," says Nicholson, the Gilpin commissioner. Two years later, she says, "we're still doing a lot of buffering."

If CBMS was supposed to create efficiencies, it hasn't yet turned out that way. If anything, the counties argue that their costs and workloads have increased. For example, data from existing caseloads were never dumped from the old legacy systems into CBMS. That created a backlog for county workers because they had to manually transfer cases themselves. Counties have had to hire more people and do a lot of training that they believed would be done before the project launch, not after. Staffers who work with CBMS every day complain that the system is slow, complicated and non-intuitive--as evidenced by the fat user manuals on their desks.

Huston attributes some of the counties' complaints to the normal resistance to change that accompanies any major technology conversion. If counties are seeing costs go up, he insists, some of that is because CBMS is succeeding at its goal of mining benefits that clients wouldn't have otherwise known they were eligible for. John Picanso, who became the state's CIO in the middle of all the angst last year, calls the project a success. He says it's removed the stovepipes that used to make benefits management fragmented and uncoordinated. "Our focus now is to make it more user friendly," Picanso says.

The vendor, EDS, believes that Colorado could have saved a lot of headaches if the company's proposal for the project were taken as-is, and not modified to save costs. Nevertheless, EDS executive Dick Callahan says he's impressed with Colorado's response to the project's setbacks so far. "With a project of this size and complexity, problems have to be expected," he says. What matters is "how well you react to them."

SOMEONE IN CHARGE

John Witwer sits in his office in Denver's gold-domed capitol, poring over papers that document CBMS's turnaround. Governor Owens appointed Witwer, a former state legislator, as his Mr. Fix-It in May 2005. A new state office was created specifically to oversee CBMS, with Witwer at the helm--finally, someone in charge. So far, Witwer is generally winning positive reviews from county and state officials for getting the project back on track, while using his familiarity with the legislature to hold peeved lawmakers at bay.

Witwer ticks off one statistic after another to prove how far CBMS has come. In January 2005, the state received about 600 to 700 problem calls a day. That number is down to around 50, and during one week in June it was closer to 10. The backlog of pending applications for benefits has dropped dramatically, from 29,000 in November 2004 to fewer than 3,100 now. The number of help-desk tickets has been cut in half. Witwer says he's proud of the system, which is now running 3 million tabulations per day.

Still, the CBMS experience has Colorado restructuring how it plans and administers IT projects. A new law created an IT office and gave the governor power to appoint the CIO. Picanso, the current holder of that title, is taking an aggressive management stance with the tech leaders at each state agency, calling them into his office and grilling them on how projects are going. In addition, he is pushing to air consultants' reports in public, so that mid-course criticism of the sort that Maximus brought to bear on CBMS can't be suppressed within agencies.

Colorado is also learning when to jump from a listing ship that looks like it can't be steadied. In recent months, two Accenture projects were dropped. One was a system to manage unemployment insurance that cost Colorado $35 million. The other was a $10 million elections computer system. "The state has made a statement," Picanso says. It will not allow high-risk projects to continue if it doesn't believe it can ensure success. "At what point do you stay in the game when it's not going well?"

As far as many county officials in Colorado are concerned, these are lessons that came a bit too late. Despite some signs of progress with CBMS, serious problems continue. A state auditor's report in April showed that 72 percent of food stamp payments contained at least one error. So did 25 percent of welfare payments and 41 percent of Medicaid payments. "Maybe the crisis is past," says Tom Mayer, the Boulder County commissioner. "But what we're left with is mediocre system functionality."

Join the Discussion

After you comment, click Post. You can enter an anonymous Display Name or connect to a social profile.

More from Technology