Are all involved...
getting that sinking feeling?
20 posts • joined 14 Feb 2015
....where I went to restore a system from tape backups and found the tapes all useless.
Worked at a company where backups were done it a bit of a strange way. I had to write a header file, followed by all the operational data, then the DBAs would back up the database (using code that they wrote and I had to call), and I would write a trailer file (so we could tell that we had a complete backup for that system).
That's all fine - until one of the DBAs changed the tape device from no-rewind to a rewind without telling anyone. I backed up my stuff, he backed up his (rewinding the tape afterwards), and I wrote the trailer file....
I had a call that was very similar in nature.
Working one afternoon, I had a call from a customer in the US complaining that their document management system was down. Further investigation showed that the database was failing to start (which was a little odd). So, after 15 minutes I finally got to the bottom of the whole sorry state of affairs.....
The original guy looking after the system had left, leaving a very inexperienced person with the root password to the UNIX system, and very basic notes that said...
If the disk space fills up, delete any large files you find.
Guess what happened to the database files.
Oh, and the last back they took was over a week before.
My missus just clicked as usual (telling her about different pop-ups is like explaining quantum physics to a puppy).
Queue an unwanted upgrade, a PC that wouldn't load, and a restore back to factory install. Still, I *very nearly* got her to go to Linux (hers is the only windows machine in the house).....
When I was a young lad, I used to even include a boot floppy I knew worked and had the drivers I needed. All nicely tucked inside a clear plastic folder with the notes I had for the server build, taped to the inside of the case (talking PC towers here).
Every single customer I showed this to gave me an extra few quid for my efforts.
I miss those days.
Then again (being a little old-school), I remember the days when reboot switches stuck out of the front of consoles.
Never forgot the day one of the senior ops caught the switch with the sleeve of his suit jacket and bounced an ICL2972. It was down for 2 days before they could get that beast back up again.
Or the time on an ICL3930 someone managed to toggle the disks to read-only (again a toggle switch that stuck up on the front panel).
And its not only computers - I also remember my boss hitting the emergency stop button on the extremely large aircon unit at the back of the machine room - and then we couldn't get it going again. Sprung loaded twist-to-release button (cleverly made to look like anything but).
Intelligent design for uni-intelligent people...
I think its actually worse than that.
Management are increasingly seeing the infrastructure their services run on as 'expensive' and 'not worth investing in' - despite the fact that its critical to their business. Put on top of that the ever increasing drive to generate profit means that upgrades just keep getting put off because the current solution 'just works' and its 'too expensive' to fix. Add all that into the mix and its a recipe for disaster!
Biting the hand that feeds IT © 1998–2020