Disaster struck the southern United States last month as Hurricane Katrina did major damage to New Orleans and southern parts of Louisiana, Mississippi and Alabama. We don’t yet understand the full impact of the storm in terms of lives lost, families disrupted, and the impact on the American and global economies. But we know that a key part of our responsibility as IT executives is to anticipate disastrous events like Katrina and be ready for them. Here are some of the things I’ve observed about the Katrina experience that are applicable to the IT arena, especially in the areas of business continuity planning and disaster recovery:
No one wants to follow the mediation plan if it’s an inconvenience, but everyone chastises you afterwards for not pushing harder.
If Hurricane Katrina had swerved at the last minute and missed New Orleans, then I can guarantee that the press would be having a field day telling everyone how stupid it was to evacuate so many people. This is one of those “damned if you do and damned if you don’t” situations that make it so hard to be in a position of responsibility. No matter how well you do, it isn’t good enough in the eyes of some people. And if you’re perceived as over-cautious in a situation where nothing happens, then the criticism will be just as fierce.
Pre-disaster exercises don’t help if you don’t apply what you’ve learned.
FEMA (the U.S. Federal Emergency Management Agency) conducted a week-long exercise in 2004 to help Louisiana emergency officials plan for the possibility of a hurricane very much like Katrina. But some of the processes used in the exercise were ignored when Katrina hit, including a process for the large-scale evacuation of people who don’t have their own transportation.
Your contingency plans need their own contingency plans.
Part of the New Orleans contingency plan was to use the Superdome to shelter people who didn’t have anywhere else to go. But the Superdome had to be evacuated when toilets backed up, the air conditioning broke down, and high winds ripped a hole in the roof.
No matter how much you plan, you still have to improvise when the disaster strikes.
There is no amount of planning that will anticipate every possible outcome, and there comes a point where additional planning makes no sense. You have to be prepared for surprises, and make sure that you have the right people in leadership positions to make the on-the-spot decisions that are required.
Insurance policies don’t begin to make up for the loss of business and goodwill, and obviously don’t make up for the loss of life.
Don’t let an insurance company be your disaster plan. Think of an insurance policy as a safety net if everything else in your plan fails.
Contingency plans need to have a defined and published trigger event, and the contingency plans need to be executed when the trigger event occurs.
I believe that more lives would have been saved if each area of the coast had an evacuation plan with a timetable. For example, “If a category x hurricane is headed for this area, then y hours before its scheduled arrival, everyone must be evacuated except designated critical personnel. Here is how that will happen ….” Without a trigger event, everyone holds out a little longer before acting, pushing beyond reasonable limits. This happened on a large scale with Katrina, as both federal and state agencies delayed before taking any action.
Any disaster has secondary and tertiary consequences that are difficult to anticipate.
Katrina caused localized gasoline shortages throughout the Southeast United States as panicked car owners rushed to fill their tanks. It’s still not clear how badly the storm will hurt the U.S. economy, but there is a potential for an economic recession as a result of the hurricane.
Disaster planning is all about compromises.
That’s hard to deal with emotionally; it’s kind of like the idea of “acceptable losses” in an army battle. On the one hand, we don’t want to give up anything if disaster strikes. On the other hand, there is a cost of being ready for a disaster, whether or not the disaster ever occurs. Making compromise decisions is tough.
Risk and Hazard aren’t the same thing, and our business continuity plans have to take the difference into account.
Risk communication consultant Peter Sandman sums up the risk reaction in an equation: Risk = Hazard + Outrage. The idea is that the perceived riskiness of something is not just based on the probability of the bad thing occurring (what Sandman calls “hazard”) but also on the level of outrage that is felt when the bad thing happens. For example, car crashes have higher probability but lower outrage, while plane crashes have lower probability but higher outrage. That’s why planes are considered “riskier” than cars by most people. And that’s why Hurricane Katrina, which destroyed the city of New Orleans and killed hundreds (maybe thousands) of people, is getting so much press coverage: people are outraged that something like this could happen.
When IT people do business continuity planning, or even when we plan for a new system, we typically include a list of risks in our project plan. But being the analytical people that we are, we don’t usually factor in the emotional “outrage” side of the equation. As a result, we focus our attention on the things that are more likely to go wrong, and not on the things that are more likely to get a bad reaction from the public if they go wrong. For example, in the typical systems planning scenario, we spend our time on things like high availability and verifiable financial transactions, and we neglect the relatively low probability ways that bad people can steal data, and the even lower probability ways in which people can be injured or killed. Guess which type of event hurts your company more in the long run.