Fat-fingered admin downs entire Joyent data center Cloud operator Joyent went through a major failure on Tuesday when a fat-fingered admin brought down an entire data center's compute assets. The cloud provider began reporting "transient availability issues" for its US-East-1 data center at around six-thirty in the evening, East Coast time. "Due to an operator error, all …
We need to know, so we can lean from his mistak: How fat is his finger?
I will obviously be instituting a company wide lathe based pre-emptive fix for this for my staff. Now that's ISO9000 Preventative Action stuff. It's not Corrective - none of us have ever done something like that before.
Cheers
Jon
Unfortunately, this¹ is exactly what most companies do when faced with this sort of issue. They say "oooh, the <command> is far too powerful - let's remove it, or require an operator to get approval from the change board before it's used in future"
Although Joyent have said they are instigating a full investigation, they will find that their system has so many fundamental holes designed in that fixing them all will require not only a total re-write, but a complete redesign of their software and operational practices. A prospect that is likely (considering how poor the whole discipline of system design is) to introduce as many new problems as it fixes.
So ultimately I fully expect the expedient solutions to be applied: an extra layer of checks that will slow down operations and make life for operators even more exasperating (such as an "are you sure" dialog after every command) and will soon become ineffective due to the pressures of getting stuff done (a 10% decrease in operational effectiveness is never paid for with a 10% increase in staff numbers) and management cuts.
[1] yes, satire: I get it
http://www.theregister.co.uk/2008/08/28/flexiscale_outage/
"As Lucas explained in an email to customers - posted to the Web by CNet - the outage occurred when an XCalibre engineer accidentally deleted one of FlexiScale's main storage volumes."
http://www.theregister.co.uk/2009/05/15/flexiscale_upgrade/
"Nine months after an engineer accidentally deleted its Amazon-like compute cloud - and six months after a second major outage - FlexiScale has finally completed a software overhaul meant to avoid such extended blackouts."
I the days before "cloud", and when Novel were still alive, they had a tool (in beta) which allowed you to run commands on all your NetWare servers at once. It was useful, but then Novel wiped the tool from the face of the planet. I did some asking around, and was told that Novel pulled it because it was too easy to use and too many customers were shooting themselves in the foot with it. I was heard tales of people deleting NDS from every server in the tree with the tool. Ouch!
Patronisingly refering to his staff as "dolphins" is going to make him REALLY popular I'm sure. No doubt another clueless accounting oik parachuted into the CTO job who has no idea what his tech staff actually do and probably thinks its all pretty easy and he could do sooo much better himself if only he had enough spare time away from the golf course.
No doubt another clueless accounting oik parachuted into the CTO job
Hmm, you've no idea who Bryan Cantrill is, have you? He jumped ship to Joyent when Oracle bought Sun, where he was one of the senior Solaris/ZFS designers. You should try and catch one of his presentations one day, he can be a PITA at times but he's far from clueless, and very entertaining.
(I'm the CTO in question. Normally, I could/would resist being trolled, but given that this constitutes special circumstances, I feel it's appropriate to clarify a few things.)
First, I was clearly not referring to our own staff as dolphins, but rather that meting out punishment rarely changes behavior. I can assure you that no one internally took this the wrong way.
Second, as for me being a "clueless accounting oik": to the contrary, I have dedicated my entire career to the understanding and improvement of software operating production systems. Having spent a ton of time on production systems, I have also made by far share of mistakes -- and (like every human who has done something important that requires precision) have had plenty of near-misses that I only found when I double- or triple-checked my own work. So not only do I not think it's easy, I know it isn't.
Finally: I hate golf.
> meting out punishment rarely changes behavior
I wish you'd been one of my bosses.
I've been to so many places[1] where the "important" part of a post-mortem is working out who carries the can. It never helps.
CxOs who understand the need to fix the problem rather than fix the blame seem to be few and far between...
Vic.
[1] Usually as a contractor, called in to sort out the cock-up. Thank $deity...
I was going to post this as a new message, but seeing you've posted here, I'll reply instead.
I found your candidness in your reponse, your openess, and your proposal for moving forward very refreshing.
If I was a customer, I'd have found it most reassuring.
Other companies (and politicians!) should note that bullshit and spin and skirting around the issue impresses no-one.
I'm pretty sure 'transient availability issues' should be the pivotal soapbox issue of the next presidential election. You can play that so many ways: 'Enhanced border protection has resulted in transient availability issues that have caused a spike in food prices, leaving many to starve', for example.
It also works for unsanctioned medical experiments, value priced organ transplants and urban hunting outfitters: 'Transient availability issues have slowed field trial results and publication of Stage IV results of REDACTED will not be available until Q4'. 'Transient availability issues have resulted in an unforeseen shortage of viable Human organs. Until inventory levels have stabilized hunting at inner city housing projects will be permitted between 1-5 AM'.
Also, 'Transient Availability Issues' would be a great band name!
Furthermore, I'll have you know my console is capable of running large numbers of kernels simultaneously. Long before you lot were messing about with your virtual kernels and manipulating kernels with concentrated RF radiation I was commanding THOUSANDS of kernels simultaneously using naught but alternating current, a dead short and a bunch of fucking butter. So suck it.
I suspect Mr Jefe would not be allowed within 25 feet of a console capable of controlling more than one running kernel.
Ah, me glasses again - I read that as "controlling more than one running kennel"....wondering what kind of transients were meant.....
It's a dog's life.....
Paris...her glasses don't seem to be working either....
Pah! amanfrommars uses bendy logic and a very special form of punctuation in his posts. I, on the other hand, utilize the wholly correct method of applying various definitions of a word in a context other than that used in the original statement.
Indeed, others, some from inside asylums for the disturbed, foresaw such intentional mangling of the words of others and even created an alphabetized index of words and their meanings specifically so that context and definitions can be interchanged to great effect. You can make great jokes, create stunning headlines, delight and horrify shareholders or run for public office based solely on understanding what a dictionary is and mastering its use. Dictionary Expansion Packs include Thesaurus, Foreign Language Cross-Indices, The Urban Dictionary and Trade Jargon Indexes.
A colleague of mine used to work for a small company where people had to make quick decisions on the fly. They had a 'mistakes book' where all bad decisions and goofs were recorded. You could make any sort of mistake as long as it wasn't in the book. He said the system worked well.
A new CEO (Marcus Steyn?) of M&S looked at the company's very thick file of accumulated corrective actions. He took it away and returned it slimmed down to a few pages of generic guidelines. He said that the original was too much for most people to have read - therefore its only possible use had been to apportion blame after the event.
"You could make any sort of mistake as long as it wasn't in the book."
In the evening the work surfaces along the computer suite walls were stacked high with card boxes for the night runs. A development programmer was standing waiting for his dedicated hands-on timeslot. He leaned back against the card trays - and they moved backwards. Unfortunately there was an emergency off mushroom button at that point on the wall.
As a result - the work surface cabinets were re-arranged to leave a gap in front of the button. A few days later the same programmer was waiting again. He was now wary of leaning against the boxes. So he positioned himself in the convenient gap in the work surface cabinets and leaned back against the wall - and against the emergency off button.
After that a papertape reel plastic core was taped round the button as a shield - so that only a finger could press it.
Unless it's malicious, you take the person aside and in private figure out what went wrong, and why. Then you gather everyone after everything's back up, publicly thank every one for their response and do an after-action review. What I wouldn't do is to pass the operator's name around. If anyone's ass is going to be chewed on, it;s mine.
I have to get the recipe exactly right since I have the social graces of a male warthog in mating season. I do know how to react in this one.
You're not a real sysadmin until you've made a major cockup (e.g. rebooting the Production system rather than the Test one).
Until then, you're an accident waiting to happen. Afterwards, you have the heightened situational awareness and the little voice in your head that whispers "something's not right" to stop you hitting the wrong button again.
So Joyent now have an experienced sysadmin. Why would they sack him and bring in another overconfident time-bomb?
Anon, because "voices in your head" doesn't sound good!
Absolutely.
And the way to see who is a good sysadmin from a bad is if he admits that he screwed up or tries to blame someone else.
If he blatantly blames someone else, he's a bad one.
Taking the blame is a good one...
Successfully laying the blame on Microsoft... Well, then you have a bona-fide Guru on your hands.
;-)
> e.g. rebooting the Production system rather than the Test one
I've found it useful, in environments where someone might get confused like that, to blacklist shutdown and reboot in the sudoers file.
It won't stop someone with privilege from rebooting the machine, of course, but it does mean the procedure is slightly different, which prevents accidental reboots. It's remarkably effective...
Vic.
>I've found it useful, in environments where someone might get confused like that, to blacklist
>shutdown and reboot in the sudoers file.
>It won't stop someone with privilege from rebooting the machine, of course, but it does mean the
>procedure is slightly different, which prevents accidental reboots. It's remarkably effective...
You're going to have to explain how someone "accidentally" types shutdown or reboot.
>You're going to have to explain how someone "accidentally" types shutdown or reboot.
I'm guessing you're not a Linux admin? Very simply - tab completion. Say you have powermt installed in 98% of your boxes. You'd probably gain a bad habit of typing power<tab>&<enter> to autocomplete powermt and execute the command.
Say one day you hit one of those 2% machines that don't have powermt installed or configured in the path. power<tab>&<enter> suddenly becomes power<off>. Machine gone.
There are lots of examples with other dangerous commands. Personally I'd like to see init, poweroff and reboot etc all require a -y flag by default.
Perhaps someone high up at Joyent has themselves had the fat fingered moment of dread, and recognises the value of experience...
There's a similar thing with discharged bankrupts; they are statistically the least likely people to go bankrupt (yes, again). Despite which, they are also one of the groups that finds it hardest to get bank accounts. The test is of course administered by bankers who have not themselves gone bankrupt.
"You're not a real sysadmin until you've made a major cockup (e.g. rebooting the Production system rather than the Test one"
I have to confess I shutdown a wrong customer server, after being unable to RDP to the server I went on via a KVM and misread o as a in a list of a few thousand similarly named servers. I was on the phone with people in front of the machine and I misread the name four times before hitting shutdown. I then saw the wrong green dot went red on the network screen and realised what I had done!
Definitely true! Only after experiencing that sinking feeling where it feels like the bottom has just dropped out of your world, and then having to tell your boss what you've done, only then can you truly appreciate axioms like "don't assume, check" and "hope for the best but plan for the worst". Until then they're just words that are impossible to put into proper context.
Though you can guarantee he'll be getting the piss taken out of him over it by the other sysadmins for years to come (in a light hearted way of course). My major screw up was about 10 years ago and it still comes up occasionally, and we still mention a friend of mines best screw up now and again some 15 years later.
As an IT executive with in house IT, when sometime goes wrong, you can pull staff from (almost) anywhere to chip in and get things working again. If you treat your staff well, they may even offer to work silly hours to help you and the company out.
As an IT executive with outsourced IT, you can fire off ranty emails (and swear down the phone if you're lucky) and....?
When everything is humming along smoothly, IT always seems like its a service that could be offloaded. When IT issues come crashing down all around you, that's the moment that you realize that your CV isn't updated enough to help you get another job when they sack you for oursourcing IT.
IT is a resource. In-source as much as possible and out-source only those functions that you can't support well for lack of talent (and work on finding the talent).
Let me fix your first sentence, it should read:
"As a IT executive mangler, I love cloud because when something goes wrong I can blame point the finger at someone else, absolving myself in the process of having to take responsibility for a bad decision."
I would append: "Once I have shit all over my current employer, I will fly off to find another place to park my ass, and shit all over that employer in true seagull manglement style."
Oh yes. Re. the comments above, I agree about the 'voices in the head'
I once installed an Oracle RAC cluster in an English university (OK it was UMIST).
I did ask if there was enough power for it before turning it on.... Oh yes I was told.
Sure enough, power up the racks and...... silence. Except for the beeping of UPSes and the running of IT staff feet towards the machien room.
Oh yes. Re. the comments above, I agree about the 'voices in the head'
I once installed an Oracle RAC cluster in an English university (OK it was UMIST).
I did ask if there was enough power for it before turning it on.... Oh yes I was told.
Sure enough, power up the racks and...... silence. Except for the beeping of UPSes and the running of IT staff feet towards the machien room.
That reminds me of the time that "we" were given an old working "mainframe" system. I don't recall which one this was as it was one of several that we were given around that time. A PDP11, I think but it was a long time ago. The machine in this memory was housed in two 19inch racks. We carted the machine's parts home to the apartment complex where we were staying, set it up in the laundry room on the patio and went down the checklist. We used the laundry room because it had a 220 power feed for the dryer and we didn't have a washer and dryer. After we had checked over everything, we threw the switch, applied power and sat back to a facility wide brownout while the system came online and drives spun up. We only started it a few times and it was great fun while it lasted but finally we were forced to break it down. Some of the drive platters became wall clocks. Pretty nice ones, too :)
Currently I have a small system36 sitting in the corner holding up one end of my eight foot long workbench/computer desk. The other end is held up by one of those metal two drawer filing cabinets ;)
Anonymous to protect the innocence :)
Or when you realise that the car keys are in the boot, and that electric boot release that you meant to fix three months ago....oops.
Thank god for spare keys an inside pocket.
I have had that horrible sinking feeling when you realise the wrong server is powering down. And once I tripped over an outlet and pulled out the power to an entire rack. Thankfully, the (massively overspecced) UPS saved my arse on that occasion, but I've always treaded very carefully in server rooms (or broom closets, or the back of the office on the empty desk that subsitutes their server room, etc...
Nothing teaches you how to make things reliable like accidentally breaking stuff early in your career, I'll tell you that. The trick is to stop doing it once you are looking after other peoples gear....!
Steven R
"BOFH keyboard, or BOFH control centre"
A pint for the poor operator he/she could use a drink or several.
First they are upfront about the issue and about the cause of the issue, and they declare publishing the follow-up and not hanging the admin.
That's a far cry from just about everybody else who start by denying any problem until it is absolutely blindingly obvious to everyone that they've got their heads up their ass, then go on to make absurd "only a small percentage of users were impacted" statements.
Joyent sounds like a rather good company to work for, where the entire management chain is capable and takes responsibility. Almost unheard-of these days.
HP, are you noticing ?
I was told these ones years ago, If true my sympathies to all involved
An engineer working in a server room of a big banky\finance outfit leant against a wall, or stretched out an arm, ( we all know how cramped it can get working in a tight space.) and hit the emegency off kill switch for the room. Once order had been restored the great and good were gathered in the room to conduct the inquest and learn from it.
"Right so what exactly did you do..?"
" Well I just stood up stretched out and ..... Oh Shit.."
Outside a big disaster recovery centre Paddy was playing\prospecting in the car park with his company car, a JCB, laying new pipes or summat, As sometimes happens he found the mains duct and broke it killing the power. The UPS's kicked in and everyone waited for the standby gennies to start up, and waited and waited, Then some one realised the control cables and power feeds were in the same duct as the outside mains..
Finally a software one to spread the blame...
Another bank was planning a small mail shot to it's personal clients. aka high value individuals. The vellum was loaded into the printers, The delivery owls were put on standby. Unfortunately the test data used in the mail shot, as created by I suspect a lowly paid junior pogrammer 'leaked' into the production run.
So each letter began Dear RIch Bastard...
The current whereabouts of said programmer unknown.
#1. Someone needed a Mollyguard clue.
(Out of interest, what do the military call Mollyguards? You know, the ones that stop you accidentally launching an ICBM when you sneeze, or blowing the bridge before your army has retreated over it, things like that? )
Tab completion: Yes. To which I'd add,
# rm / tmp/foo/no-more-rubbis [TAB]
and add the -rf at the end of the line, if and only if it does tab-complete, and after you've mentally checked for the very last time, "do I really mean this"?
BTW if your command isn't amenable to this sort of rearrangement, an open-bracket will accomplish much the same for other commands. Add ) CR after you've thought hard.
# ( dangerous_command
>
...is pretty much harmless and hard to type (since windorz reboots all patch Tuesdays), until you create a desktop link with that. Don't. Get that typed on prompt every time and you get used to that, for safety.
The other jewel I remember was something like a chat app... that ran straight off a prompt. Saying the words quit or reboot would cause lulz.... I just can't remember what kind of hardware was that... a phone or pad....
Saw it happen once in a data centre. Company installed some global automation software, agents on every server. Some Wazzock decided to test a reboot job, instead of sending to a small selection of dev servers accidentally sent it to every single agent for execution. Not immediately of course, oh no. They set to to go at 9pm on a weekday, right in the middle of the batch runs! Next thing we hear is callouts to every Windows admin in the company as 450 Windows app servers start rebooting all at once!
I refused to let them install the agents on the production Unix boxes and as the jobs didn't have the right scripting to kickstart reboots on the test/dev servers, they were safe from "fat finger" cock up!
usually, when working on virtual machines and accidently rebooting the host. Easy enough to get confused, they all look the same at the end of a long day.
It's happened to other people I've worked with, usually accompanied by a very quiet "oops" coming from their cubicle.
The child safe translation of the verbiage emitted by a meatbag after hitting the enter key and by mere milliseconds belatedly realizing that the machine minions are dutifully following orders to cause an unintended bitfaust. Or in the case referenced, a cluster bitfaust.
Worse, dead server walking and the pardon is not on the way.
How about a button that says "Reboot All Servers"? When pressed, a dialogue box with "Are you sure you want to reboot YOUR ENTIRE DATA CENTRE???" would appear.
It is hard to call missing a dash or a pike or a space "fat fingering".
Build a GUI or hire a robot. :-)
"Are you sure you want to reboot YOUR ENTIRE DATA CENTRE???"
Followed by another prompt "Are you really, really sure you want to KILL EVERYTHING?"
In the days of Teletype mainframe consoles it was found to be best that dangerous actions should ask a series of questions - with the last one requiring a different affirmative than the standard "Y". Otherwise the operator often went into auto mode and typed "Y" without thinking.
"El Reg would like to commend Joyent for its transparency about the outage and has made one virtual Sorry You Borked A Bit Barn pint available to the operator that caused the error. Interested parties can provide additional pints by selecting the beer icon in the comments below."
A pint for the poor operator, who will be needing to drown his sorrows
And a pint to Cantrill for what may be the only comprehensive, transparent, honest corporate explanation for a screw-up I've read in years. Should be required reading for all clueless executives and companies (eBay, yes, I DO mean you).
Depending on the tool is not sufficient to avoid issues. Partitioning of the datacenter such that any one operation only had a potential scope of impact of, say, 1/3rd of the datacenter is a better approach. That can be accomplished through bastion hosts/gateways. It means that the admin has to log in to a unique gateway for every partition. That 2-step process is often enough of a sanity check to avoid these types of issues entirely.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
