Reply to post: Re: This smells like BS.

Sysadmin trained his offshore replacements, sat back, watched ex-employer's world burn


Re: This smells like BS.

Also, how the fuck can you "shut down every data centre in every location in the world"

I work for a large chip company. We use a certain company's software to manage versioning in our chip databases. My company is also notoriously cheap. A decade back or so the team managing the software was reduced to one person (call him 'Chip'). One of the chip designers (call him 'Dave') was a very senior designer and a big proponent of that crappy piece of software given the role of helping out when Chip went on holiday.

Now Chip was a nice enough guy and a wizard at the versioning software, but he wasn't too knowledgeable on remote protocols, and we were using the software on multiple sites under multiple OSes (HPUX, Solaris, Linux, etc) even though it wasn't designed to be networked (think something like rcs vs. git). I'm a chip designer, but I've also had IT administrator experience, so I had written a whole bunch of scripts that Chip used to mirror commands across the company, but having an intense dislike for our crappy commercial versioning software I never learned most of the more dangerous administrative commands and had no checks in the scripts. I had merely handled the networking aspects of the software and told Chip that he was responsible for any checking other than to see that the remote command worked since I had no idea what he needed to do, and he'd never done that since he was too busy doing the work that had been done by six people to implement checks.

Of course, Chip went off on holiday and Dave took over. Dave innocently went to completely reset a private library in the versioning software (delete the old private library and all past versions and recreate a blank one). Only what he thought was his own private library was not only wildcarded, he was doing it with the networked command environment, and included all the production libraries for that silicon generation. And of course he did this at midnight before going to bed since even for his smaller private library the procedure would take some time (I told you it was crappy software).

By early morning, about 800 chip designers all around the globe were howling about being unable to work because the all the production databases had disappeared or were in the process of disappearing. Of course, the Unix backups were only performed at one site because of the sheer volume of data (remember that I said the company was cheap?). And of course, being cheap, backups had not been tested during all the downsizing and they weren't working. All told, the cutting had gone beyond the meat and well into the bone, and now the cost was a company's worth of highly paid designers being idle for two weeks as the databases were recovered. In my case I wasn't affected too much since I wasn't following official procedure and had been working in unmanaged private libraries anyway (remember I said I hated that piece of software?).

I rather suspect that that incident was why Chip was given two guys to help him, and Dave was removed from the administer's group on the software.

So yes, I've seen entire companies' sites all go down at once because someone did something with tools he didn't understand completely.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon


Biting the hand that feeds IT © 1998–2020