Dealing with Disaster: How to Plan for the Worst Case Scenario, in Business and Life
A couple weeks ago, our website server crashed. I'm not talking about a crash that is fixed with a quick reboot, I'm talking about a critical, all-systems-down, "Houston, we have a problem"-sort of crash. We host many of our clients' websites on the server and for over 24 hours their sites were inaccessible or garbled.
As the biggest nerd in the office, a position challenged only by Brad "I-make-indie-movies-in-my-spare-time" Rosier, it was obviously my responsibility to get things back up and running as soon as possible.
Believe me when I tell you that those were not good days for me. The level of stress was quite high, especially for someone who used to frequently cry under the pressure of band competitions when I was a young nerd, even when I won. My fragile psyche is just not built for this level of failure.
We ended up resolving the issue with a lot of help from the technical geniuses at MicroIntegration, and things are back up and running, without any permanent data loss. The experience clarified a couple important things, that are not only relevant to this situation, but also to the other disasters in life.
1. Have a backup plan
Because we were doing consistent backups, we didn't lose much (if any) data, even though we lost the entire server. Don't wait until a disaster happens to think about what you will do. Your best chance of surviving a disaster is by planning for it whenever you can.
2. Build a support system of people you trust
It's nearly impossible to survive true disasters completely on your own. It was only because we had built relationships with the server technicians and IT experts at MicroIntegration that we were able to recover from this as quickly as we did. I experienced first-hand the value of personal service. There will always be a cost (time or effort) to building this support system, but it is an incredible resource when disaster strikes.
3. Communicate openly and honestly with those who are affected
Open communication is important for resolving any disaster. When our server went down, and we realized that there wasn't an immediate fix, we sent an email to all our hosting customers explaining the situation and what we were doing to resolve it. We also responded openly and honestly to the customers who called or emailed during the outage. Generally, people are forgiving when they feel that you're being honest with them. All of our clients were supportive and understanding during the outage, which not only speaks to the quality of the clients that we work with, but also confirms our long-standing method of crisis PR, that honesty and transparency are the best ways of dealing with difficult situations.
4. Don't be afraid to give up and start over
This is the hardest lesson of all. After his technicians had spent a day attempting to fix the problems with the existing server, Terry Gour, the president of MicroIntegration, made a remarkable decision that turned out to be as wise as it was counter-intuitive. Rather than spend any more time trying to fix the broken server, he decided to have his team start over with a new server, building it from the ground up.
The wisdom of this particular move is significant, because humans are hard-wired to irrationally place a value on sunk costs. It would have been easy to mistakenly continue trying to restore the broken server, merely because we had already sunk so much time into it. However, the rational choice is not always the intuitive one. Our past time, energy and effort was gone, and was therefore irrelevant to the decision in that moment. All that mattered was determining the quickest way to get our websites back online.
While it may feel natural to keep investing more time, energy and effort trying to revive a lost cause (e.g. a botched product launch, a mediocre brochure design, a website interface that is confusing your visitors), the wise person knows when it's time to give up and start over.
To get our latest articles when they are posted, please subscribe by e-mail or RSS.
