the great shutodwn of 2008
Many moons ago as a fledgling junior dev in a guest access solutions startup company:
Friday at beer o'clock and everyone leaves the office just a tad early as the bosses weren't in that day. Got about 2 miles down the M62 before one of the bosses was calling asking why we just had a spike in support calls and logs all claiming login failures of some kind ...
Turning around at the next junction and heading back to the office in time to see our network admin and said senior dev pulling in to the car park too - all got the same call obviously.... worrying
We start looking for logs and find that nothing can talk to auth1 our central authentication server (auth2 was not quite configured as a master-master replica yet but the db cluster was just fine) and there was no response when trying to SSH in to this machine.... panic growing
A call to Rackspace fanatical support to find out more reveals the machine has indeed been shutdown - would we like to start it back up? Yes. Yes we would, very much please
Patiently waiting for said Gentoo production server (ask our network guy) to respond to a network ping eventually it did! However many services are not running ... a few simply needing starting, one or two had issues that were trivial to resolve - got it back running eventually with around 1500 authentication attempts having failed during the ~1hour of downtime, ouch
The logs reveal that it was indeed our senior dev who typed shutdown -h now on the the machine. He was then ridiculed for some time and the resident gentoo expert aliased shutdown to echo "I'm sorry Dave, I can't do that"
This then lead to a series of "hilarious" aliasing wars across various workstations and dev servers. Try to stop someone from being able to fix their alias problem after everything useful is aliased to something funny is quite difficult on such flexible systems. Fun though, for a given range of fun
Biting the hand that feeds IT
