Plan for a Crisis

Posted by & filed under , , .

If you haven’t heard the phrase yet, memorise it now: plan for a crisis! Because if there is one thing certain in the running of each company, that is having a crisis – we all encounter it on one level or another. You might be lucky enough for your crisis to be your project just being delivered one week later and as such dealing with customer expectations for that period or it might be the opposite of the scale where your project can’t be delivered anymore, despite all the time and resources in the world.

How do you cope with it it’s important either way though – from the moment you try to get everything back to being functional again to making sure that won’t happen again. And past that, it does mean you have to plan another crisis where you plan the exact thing happening again and check that when this happens the impact is minimal or even non-existant.

For instance, one of your juniors walks into your datacentre and wrongly wires up a patch panel: the database replication is down, your analytics doesn’t make it from your data centre to the collection point and your server are serving stale data. What do you do? Send all the senior staff there and solve it on the basis that we can still work with stale data for another 24 hours without disrupting the business right? Problem solved – your valuable engineers show their class and get you back online in 2 hours flat! Business was as usual in these 2 hours all is well! Fair enough – but what happens afterwards? 2 days later a similar situation occurs where your junior has to get in there again – so what do you do? I bet you 10/1 most companies will do the same – send the junior in just as before! After all he might have learned from his mistake; though even if he didn’t you now know you can be back online in the case of such disaster in 2 hours right? The junior might not have “got it” but you’re covered! Change some of the choordinates a bit : one of more senior guys fucks up – what do you do? I bet you it’s going to be same as before: some of the people around him can overcome that – be it through good will, hard work or just experience – and I bet you good money that companies will get just as complacent as in the previous case – so what, xyz might have fucked up but we came out alright. (Don’t be in fact surprise if xyz gets some congratulations on the fact that his team is so good to dig the company out of that hole!) And based on that xyz can drop the ball again – after all if he learned anything is that his team can dig him out of the shit!

What about the business though? Such behaviour sends a lot of wrong signals to your staff: to start with it tells your senior guys in the first instance that you don’t care that much about them – and in fact you only hired them to fix what others break and not to help you develop a healthy business infrastructure. If after an incident like the above they voice their concerns as well about a possible repeat of the crisis and you overlook that you’re telling them pretty much once more to shut up, that’s not what they’re there for – you only need them to fix shit, you got other guys working on planning (what the fuck do they know about planning anyway?they’re only fucking IT staff!) Bottom line is your senior team gets a double whammy. God forbid you actually end up with the same crisis on your hands as it only shows them that they’re right and you’re wrong but there’s nothing they can do about it and more to the point there’s nothing you will do about it! And I don’t care how you look at it but there’s zilch reasons why you would stick with a company that doesn’t listen to you, doesn’t react to your advice and makes it clear for you you’re just as valuable as the office stapler. So don’t be surprise if your senior guys start drifting off – some sooner than others, some might actually stick around for quite some times – though you might find out they become simply reactive and treat it like a “pays the bills” job, thus cancelling out their senior skills and knowledge.

A second message you might be sending your team is that you’re potentially clueless when it comes to running certain aspects of the business – which is why you’ve reached the crisis in the first place. The fact that you as a manager/boss are clueless to certain aspects of the business is actually not such a big deal – I am yet to see a Managing Director who knows the ins and outs of the company IT! The fact that you didn’t react to their alarms bell though is a biggie. It either shows that you don’t know how to listen, or you don’t know who to listen to, or that you trust the wrong people – i.e. the layer in between you and your IT staff. While the first to issues can be overcome by setting more regular meetings with your staff, the last one takes a bit more than just that. Simply put your staff feel that whoever is in charge of them is not that good – and that puts you in a bad light because it means you can’t evaluate your staff competencies and trust someone who’s not able to do their job properly. If you’re not acting on this quickly you’ll quite likely find yourself with the boat emptying soon as the staff will be demoralised and quite likely look for a happier place to work. Thing is there is no easy solution to this, however it is still down to you to find it. You might find out sometimes that is just the case that one of the little pieces in the whole organisation is not working properly and as such part with one of your managers who allowed for the repeated failure. Or it might be that rather than firing you need to hire more people. Or simply reshuffle some of the internal structure – distribute the power of executive decision on certain aspects. It might be that you need to get involved personally and talk to everyone involved and find out from the horse’s mouth what went wrong and what needs to be changed to avoid it in the future – and based on these discussions draw your own conclusions and devise a plan of action. One thing you can’t do though is entrust the same person(s) to be in charge without any change on the basis that “this time will be different”.

As I said in the beginning the crisis might have not hit you yet – if you’re one of the lucky ones in that situation why not have a planned crisis to see how ready you are for it. Send one of your staff in the datacentre and ask them to unplug a couple of cables then sit back and observe how you’re coping with it – you’ll have the inconvenience of dealing with it but it’s probably better to have a planned crisis than be hit by a real one and caught totally unprotected.