Thursday, 9 July 2009

In every disaster a little rain must fall


I'm on a train at Victoria station. That's the good news. It's not the train I want to be on admittedly but it is a train. It is going vaguely south and I did manage to get a seat. I have offered to give it up to a pregnant woman but she looked at me a bit funny and now I'm not so sure she's actually pregnant. It has taken me nearly two hours to get here after leaving the office. Did I stop en route for a beer? No, though I wish that I had. So, what cataclysmic problem has befallen the great british transport system to result in a two hour journey so far? Earthquake? Tube strike? Locusts? No. The sorry truth is that it rained. OK, to be fair it rained really quite hard for nearly twenty minutes. And just like we have that one day a year when it snows more than we expected, once a year it rains a lot too. Victoria underground flooded. Victoria station flooded. No trains, buses and taxis full, roads blocked with people and everything grinds to a very soggy halt.

In an effort to take the positives from what promises to be a four hour commute home (the same it would take me to fly to sunny Tenerife) I thought I would reflect on what this experience has taught me so far.

Disasters (big and small) just have a habit of turning up when you least expect them. The trick is to be prepared to deal with something you weren't expecting. That means having flexibility in your approach. And being prepared for some quick thinking.

Prepare, prepare, prepare and, crucially, expect not to be quite properly prepared. Practise your DR procedures not just by saying "let's try a restore of system x" but by saying "there's no power in the server room, AD is corrupt, the mobile phone network is saturated and there's a rail strike" and take it from there.


Remember what it feels like to be on the outside (of Victoria station in a puddle in the rain) and make sure you communicate. Even if there's no news, communicate. Tell people what the problem is, what's happening to fix it, when that might happen, and what they could do to help themselves in the meantime. It also helps a lot if someone somewhere looks like they're in charge. That helps to reassure people that you're not flustered, you're in control of the situation and working sensibly to get it resolved.



Put aside the why part of the question for later. It doesn't matter why it happened, for the moment all that matters is it did. You can do all the soul searching and "why didn't we think of that" -ting later.



And finally, when you have fixed it, when you've battled outlandish odds and rebuilt the data centre from scratch using bits of twig and a ZX Spectrum, don't expect any applause. Yes you've done a great job but doing your job and doing it well is what your employer pays you to do. Instead expect people to be angry. Their main reaction is likely to be "why did it take so long to fix and what idiot put it together so badly in the first place?" So chin up, make sure you work out how to learn from it and move on.


My train is moving now and I am finally travelling homewards. Am I grateful? Not really. I just want to get my hands on the person who designed the roof of Victoria station.

No comments:

Post a Comment