Data Defender

On a summer afternoon, Will Pelgrin and several of his staffers gather in a conference room in Albany to conduct a Webcast known as "Info...
by | July 31, 2008

On a summer afternoon, Will Pelgrin and several of his staffers gather in a conference room in Albany to conduct a Webcast known as "Info Security 101." The Internet audience includes 800 people, from 47 states and three countries. Pelgrin, New York's director of cyber security and critical infrastructure, expresses surprise that interest in the bi-monthly Webcasts, which he's been doing for five years, hasn't waned. "I'm just shocked they keep coming back," he says. "It's working. It fuels us. We need to do more."

At another Webcast, Pelgrin launches into an hour-long slide presentation, which includes accounts of 227 million records containing personal information that were involved in a security breach; a city whose tram system was hacked by a schoolboy; and the astonishing stupidity of human beings. Exhibit A: More than 400 people actually clicked on a link asking, "Is your PC virus-free? Get it infected here!" during a six-month advertising campaign on Google.

Down the hall from where the Webcast is taking place, nine computer experts are seated around a table in front of laptops, ready to answer questions posed by participants. When the event is over, the techies go back to watching four large monitors hanging on a wall in their office. One screen shows a 3-D cyber globe indicating where "botnets" are attempting to wreak havoc. A nuclear symbol indicates the worst-case scenario: a nefarious "command-and-control" server taking over "innocent" computers, turning them into "zombies," which do the main server's bidding. Other icons indicate machines that have been attacked by zombies and likely are being used for criminal endeavors.

Late in the day, three members of Pelgrin's incident-management team troop down the hall from the operations center into his office. "When the three of them show up," Pelgrin says wryly, "this is not a good sign." A fourth person steps in, hovering close to the doorway with the others. "Oh, four," he laughs. "This is worse than I thought." The team reports it has discovered a phishing scam: Plotters have infiltrated a popular Internet service provider's site and are capturing user IDs and passwords. It's a crime in progress, just as if police were to come upon masked men in a jewelry store, stuffing gold and diamonds into a black bag. Pelgrin tells them whom to notify with the bad news.

Is it a crisis? No, just one of the day-to-day skirmishes in the ongoing battle that top information security officials face. Over the past decade, hackers and identity thieves have moved from the amateur arena to the top echelons of international crime syndicates. The high stakes mean that cyber experts in both the public and private sectors must race to keep up with increasingly sophisticated and ever-changing schemes for compromising computers and data. In particular, citizens are outraged when personal information held by governments is breached. Pelgrin is Indiana Jones in a tie, fending off the bad guys coming at his networks from all directions. Yet he is the first to acknowledge that he can't guarantee the safety of New York's computer systems. The only surefire way to prevent breaches is to unplug the computers. "If I ever said to the governor, 'We're 100 percent secure,'" Pelgrin says, "he should fire my ass."

New and Mysterious

Just as cyber security is a relatively new field, so is the position of chief information security officer. Responsibilities vary from state to state. Although security management and strategizing skills are in great demand, the profession is in its infancy, "like a wobbly young colt," says Mark Rutledge, a director with McAfee.

The newness and mystery of what goes on in the cyber world makes it tough to persuade lawmakers to fund cyber security at the same level as physical security. "People can't see it, touch it, put their arms around it," Pelgrin says. He tries to explain what it means to the state, for instance, when a Chinese hacker or members of the Russian Business Network take administrative control of a machine: Someone in charge of a Web site or computer system generally has authority to dig into many applications and databases, and if "foreigners" steal access to that authority it could have disastrous results. He hopes that the next generation of lawmakers, those who grew up with technology, will intuitively understand the implications.

Pelgrin himself says he doesn't know his bits from his bytes, except what he's picked up on the job. He spent the early years of his quarter-century in government as a lawyer and criminologist for the New York State Commission of Correction and the Division for Youth. There he learned all about physical security. That gives him a familiarity that helps when he works on strategy and coordination for cyber security.

Pelgrin started thinking about network security vulnerabilities a decade ago, when he was the state's chief technology officer. E-commerce was hot, and financial transactions increasingly were flowing across electronic lines. With the Y2K date change looming, he wrote a white paper advising caution about enterprise security. Then, in 2001, when the two planes struck the World Trade Center towers, state offices in New York City no longer were connected to those in Albany because 2,250 circuits under city streets were knocked out.

The horrific event made Pelgrin ponder what he wanted to do next in government service. That fall, he arranged a meeting with the governor's director of state operations. He suggested the state would benefit from a cyber-security office separate from the information technology office -- a bureau that would be fully focused on protecting the state's data. In those days, cyber-security concerns were buried deeply in technology departments, seldom rising to the level of the governor.

Nevertheless, Pelgrin's new office, overlooking the Hudson River and the Heldeberg Mountains of upstate New York, was up and operating within a year -- an incredibly fast time frame to move a government agency from conception to operation. He reports directly to the governor.

Working Together

In addition to protecting the state's networks from cyber attack and managing whatever damage might arise, Pelgrin and his staff plan for how the state will recover from a disaster and how to keep the state's IT systems secure in an emergency. But they know it would be foolish to concentrate on safeguarding only their own state. Every state has a role in making the whole world safe from cyber evil. So Pelgrin encourages his fellow state CISOs to share what they find while peering into the cyber netherworld. All those extra eyes make security more effective for everyone.

He also joins forces with the FBI, the Department of Homeland Security, local governments, private companies and other agencies. "He's very inclusive and very collaborative," says Andy Purdy, formerly at the National Cyber Security Division of the U.S. Computer Emergency Readiness Team, a partnership between the Department of Homeland Security and the public and private sectors.

Early this year, Pelgrin received information from multiple sources about a "SQL inject" attack that several states were seeing. Such an attack enables hackers to insert code that could help them steal computer information. The New York State staff was able to identify the code and notify states and others to be on the lookout for it in their systems.

They're able to do that through the Multi-State Information Sharing and Analysis Center Pelgrin founded in 2002. It's a central place for states and local governments to report what they're seeing on threats to critical IT infrastructure. The organization started with 10 states but now includes all 50. The MS-ISAC, as it's known (pronounced "eye sack"), compiles security-incident information from others and shares it with all. It circulates early warnings of cyber threats and trend analysis useful for security planning. It amazes Pelgrin that every state voluntarily adopted a common protocol, when individual states often have a hard time getting more than one of their own agencies on the same page.

A lot of that has to do with Pelgrin's perseverance. "The MS-ISAC would not be the MS-ISAC without Will," says Darrell Davis, Alaska's chief security officer. "He has put a tremendous amount of effort into it." Three states are impressed enough with New York's operations to contract with New York to monitor and analyze their own states' network threats.

Alaska's Davis, for example, sends all of his state's "threat traffic" to New York for monitoring and analysis, realizing that after 9/11, the Empire State had developed a good security operation. This was about the time that Alaska had security issues it needed to resolve immediately. New York is able to provide around-the-clock monitoring at a lower cost than Alaska can. In this highly specialized technical field, the partnership allows the states to leverage their resources, reducing costs for both.

The New York-Alaska partnership was the first information-sharing partnership of its kind; Georgia and Montana later teamed up with New York, too. Other states have expressed interest but haven't committed. Typically, organizations shy away from sharing information on security breaches. For a long time, it was something everyone wanted to bury. It felt like failure. They weren't willing to share information about compromises. "They felt they might get bad press," Davis says, "and it would reflect badly on them."

Pelgrin believes in sharing extensively among "trusted" sources. To prevent bad news, New York monitors 171 network "devices" for itself and the other three states. It's the equivalent of 171 undercover agents with high-powered binoculars sitting and scrutinizing cyber highways. The devices search for malicious traffic and suspicious trends. They produce more than 5 billion logs, or pieces of code, a month.

One week in June, New York State techies recorded 652 cyber "events" that created 63 tasks for security experts to follow up on. The to-do list can range from blocking a URL, to patching holes in software to lessen a vulnerability, to taking down a defaced Web site and "cleansing" it.

None of that week's events triggered a high alert. But Pelgrin is ready for emergencies, as is evidenced by the extra clothes that hang on the back of both his office doors. He reaches under his desk and proudly shows off the winter boots that never made it home this springÉor summer. "You just never know" what things will happen, he says. On 9/11, for example, when he was chief technology officer in charge of a 600-employee office, he went straight to an emergency bunker to help with the response.

"I've Had a Breach"

The New York State cyber brigade guards against cyber attacks in two broad ways: One, by keeping an eye on them. The other, through education.

When the Web watchers find something, e-mail alerts go out to 10,000 recipients, including some in Australia, Canada, the Netherlands, Singapore and the United Kingdom. Recipients typically redistribute the alerts, multiplying the effect.

To thicken the cyber walls against breaches, Pelgrin's office has come up with Webcasts, slide shows, guides and talking points to educate business and home users. The more people Pelgrin can bring into the security fold, the happier he is. New York has produced and sent guides on security awareness and training to 38,000 local governments in the U.S. He knows that if he sent out a thick report covering all the possible things to do for cyber security, the massive tome would slide off a pile into the wastebasket without much more than a glance. The short cyber guides, on the other hand, offer a manageable way to get started.

Some of what Pelgrin does is to bring a human touch and provide reassurance on cyber issues. Because it is human nature for people to assume that the worst won't happen to them, Pelgrin often shares his own story: Yes, the cyber guy's home computer was once invaded. Afterwards, he often started speeches with, "Hi, my name is Will Pelgrin and I've had a breach on my computer." The incident happened right after he accepted the appointment as director of the cyber-security office. An e-mail appeared in his inbox with "Congratulations!" in the subject line. "I'm a lawyer," he says. "I don't get congratulations very often." He opened it. ZAP!

Initially he was annoyed that his antivirus program didn't protect him. Then he realized it was his fault. He hadn't renewed it on his home computer when a pop-up window told him to. Although learning the hard way stung, the lesson has been useful in helping others to see how easily anyone's security can be breached. His story usually gets a laugh. But he turns it back on his amused audiences, as well as any individuals who come into his orbit (including visiting reporters and photographers), asking if they have a firewall and antivirus software on their home computers.

Sometimes, it's not clear what wicked thing is bombarding a network. Pelgrin nevertheless prefers to issue timely advisories to his government partners early. Information sharing helped when nine states were hit by a botnet awhile back. Botnets are the bane of security officials because they're difficult to eradicate. They morph and pop up somewhere else. The most vulnerable offices were those running antiquated software that couldn't be patched anymore, and were more susceptible to compromise. But with the notification process in place, at least governments knew what they were up against.

"If we see something, but we haven't fully comprehended everything, we will tell them what we know," Pelgrin says. "If we wait, you're going to read about it in the newspaper or see it on CNN."

Techie Terms

Botnet: Short for robot network; a collection of compromised computers (zombies, see below) controlled remotely. Used to send spam or viruses, or to flood a network with messages so as to slow or shut down a Web site, an activity known as a denial-of-service attack.

Hacker: Someone expert in finding computer system weaknesses and exploiting them, or getting unauthorized access to a computer site or system.

Phishing: Sending an e-mail message or link to a Web site that is sent to try to get users to enter personal information that may be useful to someone trying to steal their identity or gain access to data online.

SQL inject, or injection: A security attack in which someone compromises a computer system by adding code in a computer language called Structured Query Language, or SQL (pronounced "sequel"). The code is slipped into the text fields of a Web form users fill out online, and is used to change data or get access to resources. Automated tools are increasing the risk of SQL injections.

Virus: A computer program that attaches to existing programs, copies itself and infects its host computer, unbeknownst to the user. The virus changes or corrupts a targeted computer's files. It is spread to other computers when the infected computer's user sends it over a network or the Internet, or transports it via a CD or USB drive.

Worm: A subclass of virus that self-replicates without a user having to do anything. It can send copies of itself via a network, hogging a lot of bandwidth.

Zombie: A computer under the control of a hacker, who then commands the zombie to perform a task, shielding the identity of the hacker. A zombie might send a stream of spam or "phishing" e-mails, for example.


More from Technology