Tech Businesses Need to Maintain Services in the Face of Dire Storms
Late last summer, when Hurricane Harvey hurtled toward Texas and Hurricane Irma toward Florida — within weeks of each other — Mariner Finance prepared by boarding up branch offices along each storm’s path, shutting down desktop PCs and servers, and covering the tech with plastic.
As many residents and employees evacuated, the IT department at Mariner’s Nottingham, Md., headquarters tapped into the company’s cloud-based Mitel IP phone system and redirected the closed branches’ inbound phone calls to sister branches to ensure urgent customer service needs were met.
When Harvey and then Irma slammed into the United States, some branches suffered small water leaks, but the company survived relatively unscathed. In Irma’s aftermath, two branches experienced power outages, while one branch lost internet access from one of its two service providers.
SIGN UP: Get more news from the BizTech newsletter in your inbox every two weeks!
The IT staff knew immediately of the outages because they had deliberately kept Cisco Meraki networking gear on at each closed location. When the storms passed, they remotely checked the equipment to help executives assess damage and determine when to reopen branches.
“By leaving networks and circuits up, we could tell business leadership which branches lost power or connectivity,” says Don Wiegner, Mariner Finance’s CIO and executive vice president. “It saved staff members from having to drive to those branches — and in some cases, journey through road closures for two to three hours — only to find out they could not work because there was no power.”
The two recent hurricanes, which devastated parts of Texas and Florida, and the recent wildfires that ravaged parts of California, including its wine country, are fresh reminders that IT leaders must prepare their companies for all types of natural disasters and catastrophes.
Setting up telework capabilities with mobile devices, cloud services or virtual desktop environments ensures that businesses don’t grind to a halt when extreme weather hits or other emergencies occur. IT redundancy is also important, from subscribing to multiple internet service providers to deploying a secondary data center, either internally or through the cloud.
IT Leaders Must Plan for the Worst
To get ready for disasters, IT leaders must conduct a threat assessment and collaborate with other company stakeholders to create a comprehensive DR strategy that encompasses people, processes and technology, says Phil Goodwin, IDC’s research director for data protection, availability and recovery.
“It comes down to examining each of those areas, and asking: ‘How will we recover our technology? What processes need to be in place? And what are the redundancies at the people level?’ ” Goodwin says. “You have to look at redundancy across the organization to figure out what the processes are for different threat scenarios.”
In St. Petersburg, Fla., about two-thirds of the 80-person staff at Taylor Media evacuated for Irma, but many employees were able to continue to work because they had access to mobile and cloud technology.
The startup, which publishes a personal finance website called The Penny Hoarder, equips employees with laptops. A cloud service provider hosts the website, so as long as employees have power and an internet connection, they can publish articles, says Taylor Media IT Director Stephen McDermott.
“We were in a good situation, in that we have no local data center. If a hurricane hits us, it does not hit our site,” he says.
McDermott staggered the evacuation of his eight-person IT staff to ensure someone was available at all times for troubleshooting. Two IT staff members departed three or four days before Irma’s arrival — one going to Texas and the other going to Atlanta. When they were safely ensconced in hotels with Wi-Fi access, McDermott and his family evacuated to Atlanta, where he also worked from his hotel.
In the end, Irma turned east, sparing the Tampa Bay area from a direct hit, but high winds downed trees and caused extensive power outages.
During and immediately after the hurricane, the IT department fielded several requests, but nothing major.
“After the hurricane hit, people traveled home the next day, but most of us worked remotely the rest of the week,” McDermott says.
Keep the Business Running with Disaster Recovery Tools
Mariner Finance, which provides personal, auto and home loans to consumers at 460 branches in 22 states, had a plan in place when the hurricanes formed last year. When Wiegner joined the company in 2015, he spent two years working with his team to incrementally improve disaster recovery by focusing on redundancy and communications, and taking advantage of several cloud services.
“The solutions that are out there have come a long way — to the point that it’s very reasonable and cost-effective, whether for a large enterprise or smaller retail front,” Wiegner says.
Every time he deployed new technology, he and his IT team educated management about the preparedness and recovery benefits. Once all the solutions were in place, the business held a meeting to discuss everyone’s roles and how to use the technology during emergencies.
“That went a long way to making our response to the hurricanes successful,” he says. During the hurricanes, the IT team held conference calls with company executives three times a day. Marketing team members used email and social media to notify customers of branch closures (15 branches in Texas and Louisiana for Harvey; and 50 branches, in Alabama, Florida and Georgia, for Irma).
Local managers in the hurricane’s path used a new cloud-based emergency notification system to text employees, share updates and, after the storm passed, ask if anyone needed assistance.
They could see in real time the staff members’ responses, and fortunately, every employee was safe, Wiegner says. Mariner Finance’s investment in new IT infrastructure also assisted in the company’s pre- and post-hurricane response. Wiegner had replaced stand-alone small-office phone systems in each branch with a companywide Mitel Unified Communications as a Service solution. That let the IT team use a cloud-based management tool to transfer the closed branches’ inbound calls to sister branches throughout the country.
Use Redundant Communications Tech to Keep Branches Online
“For each branch, the business leadership identified who the sister branches should be, and less than 30 minutes after I received the list, my team had used the Mitel administrative portal to redirect those phone numbers,” he says.
A few branches had IP-based security cameras, allowing management to view live video and see the branches’ condition after the hurricanes.
Wiegner had also bolstered redundancy at each branch by subscribing to multiple internet connections, using CenturyLink’s MPLS network and Access Point broadband services.
He installed a Cisco Meraki MX100 Series appliance and a Cisco Small Business SG Series switch at each branch. The Meraki gear comes equipped with software-defined networking capabilities, so if a service provider goes down, it automatically fails over to a second service provider.
The IT team used SolarWinds’ network monitoring tool to analyze the network logs to check for power and network outages. That’s how the company quickly discovered that Harvey caused no outages, but with Irma, two branches lost power and one branch lost an internet connection.
The IT staff handled the internet disruption and reported the power outages to the facilities department, who reached out to utilities to get power restored.
Overall, Wiegner is happy with how the company managed the two emergencies. “It was close to perfect execution of crisis management.”
