
In an effort to take the positives from what promises to be a four hour commute home (the same it would take me to fly to sunny Tenerife) I thought I would reflect on what this experience has taught me so far.
Disasters (big and small) just have a habit of turning up when you least expect them. The trick is to be prepared to deal with something you weren't expecting. That means having flexibility in your approach. And being prepared for some quick thinking.
Prepare, prepare, prepare and, crucially, expect not to be quite properly prepared. Practise your DR procedures not just by saying "let's try a restore of system x" but by saying "there's no power in the server room, AD is corrupt, the mobile phone network is saturated and there's a rail strike" and take it from there.
Remember what it feels like to be on the outside (of Victoria station in a puddle in the rain) and make sure you communicate. Even if there's no news, communicate. Tell people what the problem is, what's happening to fix it, when that might happen, and what they could do to help themselves in the meantime. It also helps a lot if someone somewhere looks like they're in charge. That helps to reassure people that you're not flustered, you're in control of the situation and working sensibly to get it resolved.
Put aside the why part of the question for later. It doesn't matter why it happened, for the moment all that matters is it did. You can do all the soul searching and "why didn't we think of that" -ting later.
And finally, when you have fixed it, when you've battled outlandish odds and rebuilt the data centre from scratch using bits of twig and a ZX Spectrum, don't expect any applause. Yes you've done a great job but doing your job and doing it well is what your employer pays you to do. Instead expect people to be angry. Their main reaction is likely to be "why did it take so long to fix and what idiot put it together so badly in the first place?" So chin up, make sure you work out how to learn from it and move on.
My train is moving now and I am finally travelling homewards. Am I grateful? Not really. I just want to get my hands on the person who designed the roof of Victoria station.

No comments:
Post a Comment