The Gathering Storm
Crescent Bank headquarters in New Orleans once overlooked the Superdome from its offices on the 18th and 19th floors. Eric Webb always admired the view from the information technology department’s downtown perch, but he also knew that in the hurricane-prone region, the landscape could change quickly.
In the lazy mid-August days last year, Hurricane Katrina ravaged the bank’s headquarters and many of its local offices. Webb, assistant vice president for technology, quickly learned that his meticulous IT disaster recovery plan had some gaping holes. It’s just one of many lessons learned by businesses and IT executives on the power of nature and the unforeseen snags that arise during natural disasters. Three IT leaders share their lessons learned with BizTech.
A Backup Site for the Backup Site
For years, Crescent Bank had maintained a disaster recovery site in Alexandria, La., about 220 miles northwest of New Orleans. It decided to move operations to easier-to-reach Jackson, Miss. Some $80,000 worth of servers, hard drives and a redundant array of independent disk controllers were in place by mid-August.After Katrina roared ashore packing 125-mph winds, the headquarters “building was pretty much destroyed, but our servers and computer room were basically untouched” because they were housed in an interior room of the building, Webb says. “We still had no access to them.”
Yet, Webb felt comfortably prepared with the bank’s disaster recovery site. That is, until Katrina rolled up to Jackson with 100-mph winds that uprooted the bank’s data service.
“Within six hours of the storm passing in Jackson, we realized we were in trouble,” Webb recalls. His team, already positioned in Jackson, made several tough decisions. First, they loaded all the servers into pickup trucks and headed to Alexandria, where damage was minimal and the old T1 lines at the vacant disaster recovery site hadn’t been canceled. Within two days, the bank was back online for its offices in unaffected areas, such as Virginia and Ohio.
“We were up and operating, but we weren’t 100 percent,” Webb recalls. “Our users could take applications; they could surf the Internet and get e-mail, but there were some other applications they couldn’t hit.”
After the storm passed, Webb faced his next challenge. Several hundred employees needed a place to work. The bank relocated nearly 100 employees to as far away as Chesapeake, Va., and others as close as Lafayette, La., about an hour’s drive away.
Crescent Bank A Year Later …
Crescent Bank lost nearly 100 employees who evacuated and never came back. Its headquarters building remains in disrepair, and the bank has taken up new offices in the downtown Energy Center. The IT department permanently relocated to Baton Rouge, and Webb still makes the two-hour commute three times a week, but plans to eventually move there.
Katrina taught Webb that you can never be too prepared. The bank signed on with a disaster recovery company in Baton Rouge that now houses its servers. “They have triple redundancy on power, phone lines, data lines and triple generators,” Webb explains. The old Alexandria site is still maintained. “It’s the disaster site for our disaster site.”
Webb cautions businesses: “Don’t take anything for granted as far as security of your data and servers. Even if you have a disaster recovery plan, that doesn’t mean it’s foolproof. We found that out.”
Think Quickly and Creatively
The 500 employees at FARA Insurance Services in Mandeville, La., felt pretty confident about their disaster plan. They tested it in 2004 when Hurricane Ivan threatened the region, though it never hit. But FARA, an insurance services firm, still relocated staff to its Houston office — a 14-hour drive on clogged roads.
When they returned, Vice President of IT David Richard and his team decided that they needed a business continuity center closer to home, in Baton Rouge, about 65 miles west of Mandeville and 70 miles northwest of New Orleans. “We felt really good going into the hurricane season,” Richard recalls, “until Friday, Aug. 26.”
Katrina had gone through Florida and was downgraded to a tropical storm. “We went around on Friday as a business-as-usual day and hardly spoke about it,” Richard says. By Friday afternoon, the National Hurricane Center was predicting a course toward Louisiana. “By the time we heard about it on the news, it was 4:30 on Friday; half of our staff had gone home for the weekend.”
The six-member executive team agreed to monitor the storm and reconvene at the office at noon Saturday. “But before we even met on Saturday, there was panic,” Richard recalls. Newscasts warned residents to avoid the storm. “So we talked by phone and agreed really quickly to secure our personal belongings. Everybody was having difficulty finding hotel reservations.”
Richard headed to the Baton Rouge data center and helped get eight essential workers there. “By Sunday, it was chaos,” he says. Nervous drivers jammed highways. Richard and his family took refuge with relatives in Baton Rouge.
On Monday morning, Katrina made landfall. FARA’s Mandeville office suffered damage. But the offsite mission-critical equipment survived. By Tuesday, the main systems were up and running from Baton Rouge, but e-mail service had gone down. “The server was working, but we had dual Internet service providers and lost both of them,” Richard says.
So Richard and two IT workers returned to Mandeville to retrieve and relocate the e-mail server. He was unprepared for what he found: A tree had fallen through his home, power and phone lines lay on the ground, and police trolled neighborhoods looking for looters.
“Fear does set into your mind,” Richard says. “Your world has changed. You see signs reading, ‘You loot, we shoot.’ People were carrying guns, including me, for safety.”
Still, the company continued to answer its 800-customer service lines, applications ran and e-mail was restored. “Our offices that were not affected by Katrina were never without business-critical applications,” Richard says.
FARA Insurance A Year Later …
Though the disaster recovery plan went well, FARA still learned some valuable lessons.
“We were real happy that we went to Web (applications) years ago. We learned that our claims adjusters and our nurses can go (anywhere), connect to the Internet and continue to provide services. But the paper-intensive clerical staff had to be in a central location.”
Richard didn’t anticipate that postal mail service would stop for weeks. FARA prints and mails workers’ compensation benefit checks to thousands of injured workers. “We had to figure out how to find these people and get their money to them,” he says. The company turned to Western Union and wired money to recipients using secret user numbers to verify identities.
Today, FARA has redundant e-mail servers to eliminate future outages. Richard’s team also brought on various cell phone providers using multiple area codes. “We thought we would be able to communicate via cell phone, but they were practically useless if you had certain area codes,” he says. FARA also found additional office space in Nashville to accommodate the executive team during future emergencies and established designated business continuity locations for departmental staff.
“You have to think creatively or you can sit and dwell and wait,” Richard advises. “Certainly you don’t wait for the government. We all learned that.”
Safe Areas Should Brace for Residual Effects
USAgencies Management Services, an auto insurance company in Baton Rouge, only lost power for six hours, but many of its 16 New Orleans offices were badly damaged by flooding, looting and rain.
Shawn Gregory, systems services manager, knew the storm was coming, but he had heard all the gloom-and-doom predictions before. “We’ve had a lot of scares like that, and every time, the storms hit roughly 50 to 100 miles offshore and died. People thought that would happen with this one, too.”
Still, Gregory’s team advised workers to take some basic safety measures, such as unplugging computers and placing them off the floor. After the storm, his team had to locate scattered employees and find them workspace. “It took us a week to a week and a half to track down everybody,” he recalls.
They were also unprepared for the deluge of evacuees taking refuge in Baton Rouge. “A quarter million people moved into Baton Rouge overnight,” he says. “The phone companies couldn’t keep up. Cell phones and land lines didn’t work. One out of every 20 calls would make it through. Many callers couldn’t get through for two weeks.”
Eventually, the company redirected calls through other states with its Voice over Internet Protocol system and gateways throughout the nation.
USAgencies A Year Later …
USAgencies has completely recovered from Katrina and will reopen offices where communities are repopulating. The company now takes its disaster recovery plan much more seriously.
The company switched half of its offices from point-to-point T1 lines to a Multiprotocol Label Switching mesh network that eliminates the hub-and-spoke connectivity of its former network topology. If the central office goes down, connectivity will continue for other offices.
“We’re looking at bringing up a disaster recovery site connected” to the mesh network, Gregory says. “So even if we lose our core servers, we will be able to redirect all offices to the disaster recovery site. They might see downtime of just a few minutes as opposed to a few weeks.”
Gregory advises companies to plan for rerouting phone calls out of the affected areas. “If you’re having a problem with power, expect to have a problem with phones.”
Gregory also advises businesses to contact each of their IT vendors and discuss the vendors’ disaster plans. “Will your vendor, with just a call, ship without purchase orders, without any written authority? Will they ship $100,000 of equipment to a site they’ve never shipped to when UPS won’t deliver there? What are their priorities?”
Finally, get a telecom priority number through the Homeland Security Department, Gregory says. Businesses that rely heavily on telecom services can apply for this number that gives them a higher priority on repairs than other businesses. “If you don’t have a number, you’re lumped in with everyone else with a problem.”
How have Hurricane Katrina and other natural disasters influenced your company’s disaster recovery plans?
| 29% | We had solid plans prior to Katrina |
| 29% | Little impact; our plans are not as airtight as I’d like |
| 18% | We’ve reviewed our plan and filled any holes |
| 17% | It hasn’t; disaster recovery is not a priority |
| 6% | We created a plan as a consequence |
| 1% | Don’t know |
Source: CDW survey of 316 BizTech readers
