Reply to post: When recovery is not an option

College student with 'visions of writing super-cool scripts' almost wipes out faculty's entire system

Trixr

When recovery is not an option

A lot of the time these days, you'd just restore from backup - and it's handy the more sophisticated products let you restore just ACLs if they've been hosed.

However, I learned the hard way in the late 90s that you need to watch your backups as well. We had a HP tape jukebox that held about 24 tapes, I think, and only the database backups were exported and stored in a safe (our mail and user files weren't deemed that important back then, but of course you'd ship them offsite now).

One day I needed to do an inventory to find the tapes that needed exporting. I think it was the second or third time I had to do the job, so it wasn't routine yet. The jukebox control panel had 8 buttons on the front, in two columns, with the actual button names displayed on an LCD panel between the physical buttons (like many ATMs). The INVENTORY button was in the top row.

So I pressed that and waited for it to finish - it normally took about a minute to scan the barcodes and then we were ready to export. Well over five minutes later, it was still "scanning"... in fact, it was loading the tapes and the drive was active for a minute or so after a tape loaded. ...Oh bugger.

Turns out the top row of the control panel had the INVENTORY button, sure enough. But the other button in that row was INITALIZE. The room was dim, the LDC panel was very low contrast, my eyesight is not fabulous, and the words began with the same letters and were about the same length in the fixed-width font...

So yes, I managed to single-handedly wipe 3 months' worth of backups in less than 10 minutes. Thankfully we didn't actually need to restore anything during that interval by the time those backups aged out. Boss was not happy, but at least I'd fessed up straightaway and we did an immediate backup of everything again.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon