Governing Magazine/November 2001

FEATURE: DISASTER RECOVERY

IT IN THE RUINS

For governments hoping to keep their systems online after a disaster,
the lesson of September 11 is clear: You have to have a plan.

By Ellen Perlman

When New York City spent millions of dollars preparing for the Year
2000 computer meltdown that never happened, there were complaints that
the money and effort were wasted. Those ended on September 11th. The
contingency plans the city had put in place to deal with Y2K provided
a roadmap to recovery after terrorists obliterated the World Trade
Center, wrecking crucial city government data and telecommunications
networks.
 The Y2K effort was an exercise in preparing for the unpredictable.
But then, that is what technology disaster planning and recovery is
about. "It's a set of procedures," says Avi Duvdevani, acting
commissioner of the city's Department of Information Technology and
Telecommunications. "`If I don't have this, I'll do it this way.' It
was all a part of Y2K planning."
 When the skyscrapers fell, a nearby Verizon switching center was
damaged by fire, falling debris and water-main breaks. About 200,000
phone lines were knocked out, including many of those serving City
Hall and other New York City government offices in lower Manhattan.
City workers displaced by the destruction were without phone service
and computers. For Duvdevani, getting IT working again meant 24-hour
days on the job and sleeping on an army cot in his office.
 In planning for Y2K, the city had developed contingency plans for how
to do business if the power went down or computer services were lost.
Infrastructure had been built up and redundant systems had been put in
place. "From a city-wide perspective, there was lots of preparedness,"
Duvdevani says. "People knew how to fairly quickly address how to
conduct business."
 Even with all the disaster planning in the world, however, there are
always unexpected challenges. While Verizon had offered competitive
pricing for city-government phone service, it had not planned for the
possibility that a terrorist attack would take out a main switching
facility. No redundancy existed there. "That was a lesson learned,"
Duvdevani says. "Basically, we were pretty well positioned for
disaster except for this one piece. It was not included in disaster
planning."
 However, there was a secondary solution the city could turn to: a
consortium of telecom providers formed a decade ago. In the early
1990s, the Mutual Aid and Restoration Consortium members agreed that
in the event of an unanticipated critical failure that could not be
fixed within four hours, the consortium would be convened to figure
out which companies might be able to provide alternative routing for
crucial telecommunications services.
 "The objective was to get all carriers set up in a conference call we
coordinate and administer, to see what is the best route, what is
available to restore service," says Thomas Dunleavy, commissioner of
the New York State Public Service Commission, who at that time was
deputy commissioner of a now-defunct city telecommunications and
energy department.
 That collaboration, put in place so long ago, turned out to be
invaluable to Duvdevani. The carriers agreed to provide service at no
cost, except for out-of-pocket expenses, and not to use the
opportunity to market or sell services to the customers they were
helping out. "I was able to quickly marshal them all together,"
Duvdevani says. Otherwise, "we would have spent 10 days trying to find
vendors." One company offered the use of fiber in place right outside
City Hall that could provide a dial tone immediately. In the first few
days after the attacks, Duvdevani was holding consortium conference
calls twice a day. "We get them all on the phone and we start talking
about issues," Duvdevani says. "What the city's priorities are, what
problems they have--for example, if they've had difficulty accessing
certain places because of security."

Lessons learned in New York are likely to be applied across the
country, as state and local governments take a renewed interest in
preparing their IT and telecommunications system for disaster, man-
made or otherwise. But it isn't happening everywhere. A few hundred
miles away from New York City, in Monroe County in western New York
State, there is no disaster recovery plan in place, other than a basic
back-up system for data offsite, and no scramble, even now, to fashion
one. "No one's talked about it in the wake of the attacks," says
county CIO Tim Bortree. Technology employees understand the need for
planning and more sophisticated back-up systems, but they've had
difficulty convincing those who hold the purse strings.
 The technology department even had trouble getting an uninterruptible
power system during Y2K preparations, when there was real concern
about system failures, and when there was plenty of time to plan
specifically for a certain day and time. When the electrical grid goes
down, an uninterruptible system provides battery power to allow time
to power down in an orderly way so data and systems are not lost. The
county did finally manage to get a UPS installed--in 2001. "I'd never
worked anywhere where there wasn't a UPS except here," Bortree says.
 Bortree doesn't foresee anything that would cause a problem of
disaster proportions in Monroe County. But, as he points out, few
could foresee the scale of what happened in New York City. "The whole
purpose of disaster-recovery planning is in case you can't foresee
it," he says. Ideally he'd like someone to come in and work with his
staff to fashion a disaster plan and put it in place. And he'd like a
contract with a vendor to transfer data outside of the county so
county offices could be up and running right away across a high-speed
telephone line in the event of an emergency.
 It's not likely to happen soon. The kind of preparations Bortree
would like to see would cost a few million dollars; the entire county
IT operating budget is $10 million. "It's hard to get money for the
ideal because they say it's not going to happen here," he says.
 Tight budgets are inhibiting disaster preparation elsewhere as well.
Miami-Dade County for years has had some "glaring omissions" in
disaster-recovery plans for its central computer systems, says CIO
Randy Witt. The county has not been able to find the money for a "hot"
back-up site that duplicates its mainframe facility and runs parallel
to it at another location. It does have a signed contract to provide
the mainframe with a disaster-recovery site--not quite a "hot" site,
but a place a team from the county would be able to use to restore
data and get it back online. But systems that don't run on the
mainframe would still be vulnerable. "We have a lot of distributed
systems, networked systems," Witt says. "There's still quite a bit we
need to do." In the wake of the New York City situation, "we'll be
trying to move along faster with back-up facilities for other
departments."
 But it's money that inhibits the Miami-Dade technology department
from completing comprehensive disaster-recovery plans. "It's always
difficult to justify the funds on `what if' scenarios," Witt says.
"This has been going on for years down here."

Even governments that do have detailed emergency plans have to be
ready to improvise when disaster strikes. The 36 inches of rain that
fell in the Houston, Texas, area in six hours one Friday night in June
was nearly as much as it normally gets in a year. Water quickly
collected in a tunnel system housing computer rooms, network switching
rooms, fiber telecommunications lines, and electrical, mechanical and
air conditioning systems.
 Six of Harris County's buildings were evacuated, along with one of
the county's jails. Although no data were lost, county technology
staff had to expand the computer network so 3,000 displaced workers
could connect. The first priority was to keep law enforcement
operational. "One thing that doesn't stop is crime," says Steven
Jennings, executive director of the Harris County Central Technology
Center. Administrative offices were next. Workers were doubling up in
non-waterlogged county offices. The county brought in networking
contractors to help analyze phone and cable capacity.
 Fortunately, the county had three empty buildings that were going to
be renovated. The renovation plans were put on hold, and prior tenants
were relocated back into the buildings. "They're cramped, but it's not
egregious as far as being uncomfortable," Jennings says.
 In buildings that couldn't immediately be brought back with phone
lines, the county provided a wireless voice communications system.
Since the June flooding, the county has constructed six telephone and
data rooms, installed more than 23 miles of fiber-optic lines,
activated two wireless communication systems and added 1,500 extra
telephone lines.
 The county was lucky in that it already had networks set up making it
possible for some employees to telecommute. The county also was able
to get the use of 300 cellular phones normally used for the visually
impaired. "Within a day, those were distributed out," Jennings says.
"Having those phones available was a lifesaver."
 The Harris County technology department actually didn't have a
recovery plan for a disaster on the scale of the one it faced.
"There's no countywide plan saying, `By the way, we're losing six or
five or four buildings, what happens?'" says Jennings.
 He doesn't know how he would have handled the devastation experienced
in New York City, but in his mind it has only underscored the need to
do whatever it takes to be prepared. "It reinforces that you need to
have a plan," he says. "Any plan. You can always make a plan better.
Going in with a blank sheet of paper is not going to help you at all."

----------------------------------------------------------------------

Copyright 2001, Congressional Quarterly, Inc. Reproduction in any form
without the written permission of the publisher is prohibited.
Governing, City & State and Governing.com are registered trademarks of
Congressional Quarterly, Inc.
http://governing.com