lightning strikes
The first mistake companies make when it comes to disaster recovery is thinking the only kind of disaster is a natural one. PHOTO: Clinton Naik

A New York bank had what it thought was a rock-solid plan to recover its non-critical “customer convenience” systems from its backup tapes if a disaster ever struck.

Yet when Superstorm Sandy hit the east coast in 2012, the rush of companies trying to access the third-party tape archive site meant it took weeks for the bank to get the tapes it needed to get its “non-mission critical” ATMs and online banking applications back up and running.

Disasters happen whether you are prepared for them or not. 

Unfortunately, disaster recovery (DR) is one of those uncomfortable topics people tend to avoid. They either fail to plan at all, underestimate the full level of risk their business faces or are simply overconfident in their ability to cope when disaster strikes.

When people don’t think things through, that can truly be a disaster. Companies hit by a disaster resulting in significant service outage often do not survive in the long-term.

Test Your Organizational Readiness Against These 6 Warning Signs

How confident are you in your business preparation? Test your organization against these six warning signs it might be at risk.

1. You think the only disasters are natural disasters

Since 2003, we have tracked the reasons clients declare disaster recovery incidents. What we found was, while natural disasters make up 30 percent of declared disasters, infrastructure failures are the most common, making up 37 percent. 

The most likely disruption to your business comes not from outside, but from within. Utility issues and fires comprise another 20 percent. Cybercrime, terrorism and other civil defense events currently make up 12 percent of our clients’ disaster recovery incidents, but clearly, cybercrime is on the rise and becoming an increasingly relevant threat.

A comprehensive DR plan should include specific strategies for each possible scenario. The way you deal with a hurricane may differ from what you do for a ransomware attack. In the case of needing to restore your business applications due to a natural disaster, you are looking for the most current version of data to restore, while in the case of data corruption following a security breach related, you must go back to a clean copy of the data from before the breach.

2. You haven’t tested your full DR plan in a while

While having a DR plan is an excellent first step, it needs to be tested regularly to ensure it works, is up-to-date and that people clearly understand their roles in the recovery. Such tests can uncover common issues such as data backups that aren’t working as expected, hardware or software incompatibilities between the production and backup systems, or even important new application dependencies that someone neglected to incorporate into the DR plan. 

Your level of confidence in being able to execute a DR plan at Time of Disaster (ATOD) is directly proportional to the frequency and level of success at Time of Test (ATOT).

Think of these tests like the fire drills you had in school, which ensured every student and teacher knew what to do and where to go. You don’t want panic and confusion in your workplace when a disaster happens. You want a calm sense of, “I’ve got this. I know what we need to do.”

Test your plan fully and recognize that partial tests may give you a false sense of security. If you’re not planning to test, then you’re planning to fail.

3. You use the word 'unless' when describing your plan to someone

I recently asked two senior IT leaders I met at a conference what they were doing for DR at their firms. The reply from both was they had multiple data centers, so they felt they were protected because they could easily shift workloads. 

When I asked them, “Where are those data centers located?” one of the two said the data centers were only a few blocks from each other. He immediately admitted that it was not an ideal location, but they had decided it was good enough. 

The second had a better answer. His data centers were about six miles apart. “We’ve done some analysis,” he said, “and, in most cases, we’d be fine unless there’s some severe disaster that took out both data centers.”

Unless! It’s almost as if they are willfully ignoring what would happen if something catastrophic eliminated both their primary and backup data centers located in close proximity. Many companies found themselves in that exact situation after Superstorm Sandy.

4. You only have a DR plan because regulations require it

Regulations require companies in many industries to have a disaster recovery plan in place. This has led companies to think of DR as a compliance issue rather than a survivability one. 

When viewed as a compliance issue, the mentality behind the DR plan unfortunately becomes one of “checking the box.” Disasters don’t care about checkboxes. They are messy, and you need to be prepared at a very detailed level with specifics on how to recover your data and systems.

It’s similar to when you get the federally mandated safety briefing from the flight attendant on an airplane. People have heard it so many times they probably think they could do it just as well as the flight attendant, but how many people would really be able to find that often-mentioned life jacket and put it on or are listening to the instructions and determining how many rows they are away from the nearest emergency exit? 

5. You plan on using test or development systems to take the place of production systems

I sometimes hear from technology leaders that their DR plan involves restoring backups of their production system to the development/test equipment they already have in place. On first blush, that might seem fine, but you really need to go beyond the initial idea to make sure it works. 

What does it take to actually recover to the dev/test system? Is the dev/test equipment ready to go? How long does it take to get it ready? Is that equipment adequate to handle a production workload? Are you positive those test/dev systems are not supporting other production systems?

It’s not enough to have a vague idea of what you might do if disaster struck. You need to validate your assumptions and make sure you have a fully tested plan in place.

6. Your plans are about backups and systems and ignore people and processes

People look at DR as a technology issue: I’ve got my data protected, I’ve got my computer, I’ve got all of the components and pieces. 

But then the big questions become: “Who will actually do the recovery? What are the recovery or post-orchestration steps? What are the procedures? How do we coordinate our efforts and test to ensure we recovered successfully?” 

Organizations tend to postpone figuring out the people and resource aspects of how to put it all together until the time of disaster — only to discover you may have a huge exposure in a critical area.

A major disaster often impacts the very people you would rely on to execute the recovery. In some cases, they may not be able to travel to your secondary site. One company I know used the corporate helicopter to get its key recovery personnel to the backup site, but how many companies have corporate helicopters available on standby? 

It’s not just a question of “Do you have the people?” It’s also about having the people available at the time of disaster. Working with an outside partner can sometimes make all the difference here.

If you recognize any of these warning signs as being “too close to home,” then it is time to take corrective action so your organization does not become one more in the statistic of companies that fail to recover effectively.