Forget your RTO*: Real world Disaster Recovery needs garbage bags and bubble wrap If it's Friday morning it must once again be time for On-Call, El Reg's end-of-week meander down memory lane to explore readers' reminiscences of jobs that went bump in the night. This week, reader “Olaf” shares his experience in Thailand's 2011 floods. Yup, those floods that left Western Digital's hard disk factory under …
Ah, our nice (relatively new) office building with a roof that tends to leak any time it rains too heavily - cue water running down the walls/pooling of the false ceilings/coming down through into the comms room straight onto the UPS kit. Aged a bit when I opened the door and saw that I can tell you :(
If, in your career, you've never had to rescue a server or backup media via means of a makeshift boat. You aint lived!
4 times in my career i've watching gleefully as my place of work has sunken beneath the waves of the various rivers of the UK. All of those businesses recovered with very little data loss. All of them continued operation and still operate today.
First rule of IT: Have you backed it up? Do you have 3 copies, in two locations, one offsite?
It's amazing how many people don't plan for the major disaster situations, fire, flood, earthquake, nuclear attack or even war.
Everyone said I was daft to build a datacentre on a swamp, but I built in all the same, just to show them. It sank into the swamp. So I built a second one. That sank into the swamp. So I built a third. That burned down, fell over, then sank into the swamp. But the fourth one stayed up
> It's amazing how many people don't plan for the major disaster situations ...
Especially those tasking others to do it !
With a previous hat on, we got a directive sent down from on high (well OK, from our director) instructing us that we need to "do" disaster recovery plans for the IT stuff. It came out of the blue, and you may note the "isolation" inherent in the "for the IT stuff" bit.
AFAICT, it had come down from above - either our parent company or an insurance company - and so the clueless that led us simply passed it on.
Now, I had a bit of an idea what this might involved - well really I had enough idea to know that I didn't really know all that much. I asked around on some mailing lists, and it transpired that someone had a vacant seat on a business continuity (BC) course the very next week, and since some of the course involved role playing games, he would let me attend the course for little more than what it cost him for our accommodation (it was a 3 day course) - with the proviso that I didn't let on to anyone else how little we were paying of course !
It was something of en eye opener - and confirmed how little I'd known beforehand.
Anyway, I got back to the office, and started asking manglement about the business requirements - because you don't "do" DR for the IT stuff without input from the BC plan about what the business needs are. The conversation was over in two sentences - the second one being "Don't be awkward, just do it" from the director.
As it happens, not long after that we had a fire alarm - not a drill, but it was a false alarm. So there we all are, stood outside in the car park getting very wet - actually, I wasn't getting very wet, I had the presence of mind to grab my car keys as I left my desk and could get a brolly from the car :-) I casually asked one of the directors what their BC plan had to say about welfare of the staff - given that most had left their belongings in the office, they couldn't use their cars to get home, some would not be able to get in if they got home, did they know who held the keys to the village hall, etc etc. Is there a school where upper manglement get to learn that particular expression of "humff" they use when some impudent junior asks an awkward but quite reasonable question they aren't going to answer ?
Just to add, the office was alongside a railway where occasionally nuclear flasks get to travel - so whilst not "nuclear war", an exclusion zone preventing access would not be an unforseen thing to plan for (if, for example, the train derailed).
Our state IT workers were quick enough to move server out of the basement of the state office building in Waterbury, Vermont as Hurricane Irene poured rain into the nearby river. Some of them lost their cars in the parking lot, but they saved the servers....on a weekend.
Much of the central part of the state was paralyzed for weeks and the state workers did not get new digs for several years, but at least they were able to operate.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
