3 Tips to Prepapre For An Unexpected Catastrophe
Research that I’ve seen shows that only a tiny percentage of small businesses have business-continuity and disaster-recovery plans. You have a disaster-recovery plan and business-continuity strategies in place, tested and ready for anything, right? Not much could take your business down, and even if the worst hits, you’re backed up and ready to rebuild on the fly. But what about that one thing you haven’t thought of yet?
Take what happened to my company, Phillips Architecture. We run VMware virtualization with automatic fail-over, clustered storage area network (SAN) storage, an autoloading tape library, redundant arrays of independent disks (RAID), uninterruptible power supply (UPS) systems, backup cooling — all the essentials to ensure that we’re covered should a disaster strike. But Murphy’s Law lurks in even the most well-prepared places.
Last October, our IT manager rebooted our secondary domain controller after applying some updates. Unfortunately, when the controller came back up, the entire network went down. No one could log in or access any network resources. After a few hours, we restored the logon, but the network was crippled and sustained irreversible damage. We later discovered that our Active Directory had become corrupted four months earlier. But the problem went unnoticed until one of the domain controllers rebooted and the corrupt database reloaded.
After days of work with hired professionals and 16 hours on the telephone with Microsoft, the Active Directory was declared unsalvageable. Unfortunately, the preceding four months of backups contained the corrupted directory. Our four-month-old backup of the database was useless; all network changes made during that time were also lost, causing more problems than starting over on our relatively small network. We decided that the best option for us was to restart from scratch.
As unlikely as this scenario is, all IT professionals and business owners need to be prepared for an unexpected catastrophe — above and beyond the typical disaster and continuity planning. Use these tips to help you prepare:
Identify the impact to business operations. Understanding the potential cost of a disaster will help both IT professionals and business decision-makers select an appropriate recovery plan. The cost of repairs may differ by an order of magnitude. Faster solutions usually cost the most. If both IT and business management know in advance the cost of downtime per hour, day or week, appropriate funds can be quickly allocated when disaster does strike.
Dust off your long-range IT plans and forecasts. Do any future initiatives provide a solution you need for the current problem? The middle of a disaster may not be the best time for a migration or upgrade. However, new equipment scheduled to be implemented next year might be pulled forward more cost-effectively than alternative solutions. If a core switch fails and you have plans for a $10,000 switch replacement in 12 months, and that switch should last five years, the true cost of replacing it early is only about $2,000.
Look for ways to maximize recovery expenses to the benefit of future technology initiatives. In our case, we wanted to redesign the Active Directory to account for a spin-off company that was created after we implemented the original directory design. Our IT manager used the disaster as an opportunity to add value during recovery by implementing these changes during the rebuild. The final design was much simpler than before, reducing administrative time and improving access and performance for the employees, while adding flexibility should the two companies cease sharing office space.
With a few simple strategies, any disaster-recovery plan can be extended to account for the unaccountable. For most small businesses, the costs to eliminate all known risks are too high to justify.
A balanced approach of adequate investment in redundancy and backup and planning for the unexpected can provide a cost-effective way to assure long-term business delivery.
