back to article So you accidentally told a million people they are going to die: What next? Your essential guide...

It's a common enough scenario: you are in charge of a critical piece of your company's computing infrastructure, and you make a simple mistake with far-reaching implications. It could be a mail server misdirecting internal documents into journalists' inboxes, or a poorly configured border firewall. Perhaps you accidentally …

  1. Anonymous Coward
    Anonymous Coward

    Fuckwits in charge...

    ...and fuckwits pressing the buttons.

    The End.

    1. Anonymous Coward
      Anonymous Coward

      Consolation

      Remind the million folks that because they've led such empty lives, humanity will be able to carry on just fine without them.

  2. dnicholas

    This piece is why I love el Reg. So few outlets know exactly the likely order of events in such fuck ups.

    We've all done it, just rarely one meeellion people in the loop. One day, when I grow up, I want to get away with something as audacious as this

    1. veti Silver badge

      I hope they've got a good story for the NSA, on how they got their hands on that transcript of the phone call to the governor.

    2. Anonymous Coward
      Anonymous Coward

      Not that rare, based on experience

      A few large cloud provider outages, bank web sites, bank batch schedulers...

      Relatively often, although thankfully not all the same organisation....

  3. Michael Thibault

    In the confusion...

    I wonder whether the re-issuing of the alert at regular -- but short -- intervals would not have served to signal that something was amiss about the signal itself. IOW, if no self-destruct for the message could be found within a minute or two of the realisation that the message was in error, why not make it very obvious what the error was by repeating the message unnecessarily?

    1. Synkronicity

      Re: In the confusion...

      Wtf? No, that just creates more urgency that THE NUKE WE TALKED ABOUT ON THE NEWS IS FINALLY HERE AND IT'S LANDING NOW SEEK SHELTER THIS IS NOT A DRILL. Now imagine that endlessly repeating, hijacking everyone's phones with loud sounds and vibrations for 38 agonizing minutes. That'll solve the panic!

      1. Michael Thibault

        Re: In the confusion...

        "WTF?" is exactly what people would likely respond with -- drawing them out of their inner world and, possibly, encouraging a re-assessment of matters, possibly putting an end to what will end up -- either way -- being a completely pointless flight response. Ideally, there are 0 alerts; very much at the other end of things, there will be 1 alert; what's going on when there are multiple alerts, all identical, over the course of the time it would take for nuclear annihilation to have occurred twice over? Things that make you go "hmmmmm", no?

        1. Anonymous Coward
          Facepalm

          Re: In the confusion...

          You seem extremely confused. Such a system then needs the opposite to what you suggest. An automatic "this was an error" text after 60 seconds. If it is not an error, it requires someone to either send a second statement just before the 60 seconds is up, or in the case the entire office is also evacuated, send a second "reminder" after the first.

          Basically in addition to the (not used in the current case) 2 factor/2 user confirmation it also needs each user to confirm a second time to get the system to also send a second confirmation text.

          Then everyone would get use to the kind of "earth quake/tsunami" warnings, that are just that, warnings, as it is not until it hits that you know how big/small it is. It would encourage everyone to actually take shelter in the case of text 1, but not panic or do anything stupid, as they wait for text 2 to say "stay" or "false alarm/danger passed without damage".

          Sorry, but your example is as much you panicking/grasping at straws as those involved were. :(

          1. Michael Thibault

            Re: In the confusion...

            You have misread what was written. I wasn't suggesting a system for the general case; I was addressing a particular instance -- the one where the system's design proved to have been insufficiently complete in its coverage of the possibilities that it was impossible for participants to deal with the particular circumstances quickly and effectively.

            My original post asked about the wisdom (and likely effect) of using the same signal, in those very circumstances, in an unexpected way -- given that everyone involved should surely have seen that the system was going to be replaced any-fucking-way, and that no more damage could possibly have been done than had been done the instant the erroneous signal was issued and it became apparent there was no pulling it back (or specifically saying "Oops!"over the same channel). IOW, they had nothing else in their quiver...

            1. Anonymous Coward
              Flame

              Re: In the confusion...

              No. Because your solution, though using their existing tech/system/process still does not solve, but adds fire to the problem.

              If the system was at error, it needed an "error" message. Or, if it was life threatening style "we messed up, we gotta fix it with what we got", then send out the "free bagels to everyone" and "alien invasion" and "this is a test message" messages that are no doubt in the system ready to be sent (yes, there are stories of these messages being in such systems, so they exist, it just needs a human to think through and send some type of cancel/error message).

              But without actual access to their system, we don't know what they did have to send. Sending any amount of "missile incoming" messages would never cause people to think it was in error, until the message becomes redundant in all instances. In which case, those ignoring it already were, and those taking action already did, those thinking "is this real or not, that seems too many messages", as you mention already hit the ignoring the message group and thus in a real incident, would not take shelter. :(

  4. Florida1920
    Headmaster

    Management Mantra

    First we fix the blame. Then we fix the problem.

    1. Mark 85 Silver badge

      Re: Management Mantra

      Employee 1 now has a new career path... he/she will be the "government" or "corporate" scapegoat. For a nice fee, he/she will willingly take the blame for any screw up.

      1. david 12

        Re: Management Mantra

        >For a nice fee, he/she will willingly take the blame<

        IT consultant. Notice how much IT consultants get paid? Notice how many external project fail? As the rats desert the sinking ship, failing projects are handed off to external contractors.

        Been there. Done that. Got the paycheck.

        1. This post has been deleted by its author

    2. Hans Neeson-Bumpsadese

      Re: Management Mantra

      I have genuinely worked on a project where an agenda item for the project kick-off meeting was identifying who would get the blame if the project when wrong

      1. Bernard M. Orwell

        Re: Management Mantra

        " who would get the blame if the project when wrong"

        When, not if.

  5. Blofeld's Cat Silver badge
    Mushroom

    Work down the list ...

    "In this case, it was: the police; then the people that operate emergency sirens; then those in charge of the different counties. Then the TV and radio stations."

    But not, for some reason, the person with their finger on the big red button labelled "Retaliate" ?

    1. jake Silver badge

      Re: Work down the list ...

      The "retaliate" button is in DC, not HI. DC was still asleep at 8AM Hilo time. Thankfully.

      1. elDog

        Re: Work down the list ...

        The big dog in DC is in bed by 6:30PM and still snoozing around 10AM. During the awake period, he's not too bright so please don't bring in any critical items, other than USA Toiday and his standings with the fox viewers.

        1. Rich 11 Silver badge

          Re: Work down the list ...

          And don't bring him his standings with the Fox viewers unless they're positive. Show him his inauguration crowd photos again, if you need him in a good enough mood to cope with the day's list of executive signings and photo ops. "Look, Mr President, three million, easy. Everyone says so. Far more than that Kenyan ever got."

        2. Umbracorn

          Re: Work down the list ...

          "Better to let the great man sleep."

          http://www.news.com.au/world/did-a-power-nap-cost-adolf-hitler-victory-in-world-war-ii-staff-were-too-scared-to-wake-him-on-dday/news-story/1b2fa0f13258f79fe4c2ce58013cdab1

        3. Jtom

          Re: Work down the list ...

          Democratically-controlled Hawaii f's up big time, and some deranged people still find a way to insult Trump. Keep it up. You're attitude is what greatly helped him get elected to being with.

          1. Anonymous Coward
            Anonymous Coward

            Re: Work down the list ...

            I believe it's something commonly referred to as 'sarcasm'.

            You must be new here....

            P.s. in where did that statement refer directly to Trump? I'd take it as read it implies they're all as bad as each other.

            1. lorisarvendu

              Re: Work down the list ...

              "P.s. in where did that statement refer directly to Trump?..."

              Well not by name, but the following does prettty much imply the current incumbent.

              "The big dog in DC is in bed by 6:30PM and still snoozing around 10AM. During the awake period, he's not too bright so please don't bring in any critical items, other than USA Today and his standings with the fox viewers."

      2. Antron Argaiv Silver badge
        Thumb Up

        Re: Work down the list ...

        DC was still asleep at 8AM Hilo time....

        Nope. Works the other way. +5, so DC was just finishing lunch.

        1. Anonymous Coward
          Anonymous Coward

          Re: Work down the list ...

          Exactly... DC was still asleep. You don't expect them to be awake and working when they've just finished a hearty meal?

        2. jake Silver badge

          Re: Work down the list ...

          You and I would be just finishing lunch, yes. Capitol Hill was still asleep.

  6. Yet Another Anonymous coward Silver badge

    Cyber-attack

    If all else fails blame a cyber-attack by state sponsored thespians of somewhere to be decided later

    1. Anonymous Custard Silver badge
      Joke

      Re: Cyber-attack

      Quick, let's invade Thespia!

      1. Anonymous Coward
        Anonymous Coward

        Re: Cyber-attack

        I dated a thespian once. I had to end if, everything felt a bit stage-managed...

        1. Michael Thibault

          Re: Cyber-attack

          "I dated a thespian once..."

          Admit it -- it was the lack of encores, wasn't it? No? Performance, then?

  7. Jim Mitchell Silver badge

    I'm wondering what happened at the various military bases on the islands when that emergency text went out.

    1. notowenwilson

      From what I understand initially there was a lot of shouting and running and within a couple of minutes they had been advised that it was all clear and everyone went about their business.

    2. SquidEmperor

      We fragged Lieutenant Neidermeyer and raided the General's booze stash.

  8. Throatwarbler Mangrove Silver badge
    Windows

    NUCLEAR MISSLES NOT ACTUALLY LAUNCHED. NO WW3 AFTER ALL. SAD!

    1. Mayday
      Mushroom

      As opposed to MAD?

  9. jake Silver badge

    So you accidentally told a million people they are going to die: What next?

    "We're going to DISNEYLAND!"

    1. Rich 11 Silver badge

      Re: So you accidentally told a million people they are going to die: What next?

      No, no, no. You only get to go to Disneyland if you eat all your greens and abase yourself before Jesus.

    2. Eddy Ito

      Re: So you accidentally told a million people they are going to die: What next?

      That's just it, they weren't told they were going to die, they were told to "seek immediate shelter" which causes everyone to stampede like cows heading off a bluff on their way to some magic hole in the ground. Oh wait.

      Now I get it, a self burying population means easier cleanup when the Geiger counter ticks a little slower in a few years. Simply genius!

  10. elDog

    Good compendium of possible actions/inactions/mis-actions.

    In real life we don't have a lot of perceived choices. Something about gut reactions vs. conscious decisions.

    If I thought that my life was going to be going up in an instant flare of immense heat, I might thank the lord that I won't have to lie in hospital/nursing-home/hospice beds with tubes running in/out and money flowing out. I think I'd grab whatever is left of a bottle of cheap wine and expensive scotch and toast the new day.

    1. Anonymous Coward
      Anonymous Coward

      Re: Good compendium of possible actions/inactions/mis-actions.

      Considering that I'm facing a fate worse than death, becoming a shut-in? Yeah, I'd guzzle to that.

    2. Phil O'Sophical Silver badge

      Re: Good compendium of possible actions/inactions/mis-actions.

      How long before somone in the Land of the Lawsuit decides to sue Employee 1 for child support on the basis that, facing inbound armageddon, they decided to "go out with a bang", and didn't see any poiint in taking precautions?

      1. Rich 11 Silver badge

        Re: Good compendium of possible actions/inactions/mis-actions.

        According to the usage logs for PornHub, many Hawaiians commemorated the lack of impending doom with a celebratory wank.

        1. 's water music
          Thumb Up

          Re: Good compendium of possible actions/inactions/mis-actions.

          According to the usage logs for PornHub, many Hawaiians commemorated the lack of impending doom with a celebratory wank

          It's the obvious response to a warning of impending fiery death with zero notice. I mean sure, take advantage of situational factors where possible but you need a fall back that requires little preparation and few to no external resources.

          Also, bonus time for the Pronhub marketing department's quick reactions.

          1. Anonymous Coward
            Anonymous Coward

            Re: Good compendium of possible actions/inactions/mis-actions.

            It didn't last though.....

  11. mhoulden

    Anyone can make one mistake. However, if one person kept making the same mistake over and over again on a life-critical system like this, I'd want to know how they were put in a position where they could do so. Why was no one supervising him when he sent the incorrect alert? Did no one check it before it was sent? Did anyone else try using the system to make sure it was foolproof and fail safe? If someone else made the same mistake, what actions were taken to stop it happening again? Scapegoating one person is easy but it looks like there are serious management issues that they'd rather not go into.

    1. kain preacher

      I don;t buy that he was a screw up for a decade/ Public sector unions are not that strong in the US and if you can document a screws ups , poor review over a year the guy is gone. Nothing the union can do.

      1. tom dial Silver badge

        I buy it. At least in the federal civil service the two basic bases for dismissal of non-probationary employees are poor performance and misconduct. it is terribly difficult to terminate an employee for poor performance or even, sometimes, for non-performance.

        I know of a case (this in a US DoD agency) in which a solid foot of paper documentation was diligently accumulated, over a period of over a year, of an employee's incompetence, non-performance of assigned duties, subbasement level annual appraisals, and failure to improve despite detailed counseling from his supervisor. The employee in question previously had been passed on by several other supervisors who noted the behaviour but were unable or unwilling to do the work of documenting it.

        Finally, the I's and T's dotted and crossed, the day came when he was notified of termination on the basis of inadequate performance. Then followed the lawsuit, in which the African-American former employee alleged race discrimination by his supervisor, a white man, and manager, a white woman originally from southern Virginia. I knew both of them well enough to know the discrimination claims were rubbish, but under civil service procedures they still had to be adjudicated fairly. In the end, the employee was, indeed, terminated, but based on disability, complete with a pension based on standard Civil Service Retirement System rules.

        On the other hand, I know of a case in in the same agency where a very competent and generally high performing employee fell into the habit of playing on-line games during working hours when, as sometimes happened in his branch, not busy with specific work-related tasks. Unfortunately, the employee's cubicle was adjacent to one of a few doors from a general work area to the building corridors and in plain view from the deputy department director's office. The employee also had fairly serious personal hygiene issues. Dismissal (based on misconduct) required no more than collection of some Internet traffic showing the online gaming, a few informal counseling sessions, a warning letter and meeting and, when the behaviour was later repeated, a dismissal letter. A sad part is that, after receiving the warning letter, the employee had passed up an early retirement opportunity that included a $25,000 lump sum payment at separation, although I heard later that she had, with legal assistance, arranged to obtain the lump sum as well as the annuity to which she was entitled in any case but otherwise would have been deferred for two or three years due to age and service requirements.

        The Hawaii EMA might have been better off to have "found" porn on the hapless employees work PC.

      2. Anonymous Coward
        Anonymous Coward

        Public sector unions are not that strong in the US

        Must be a different US than the one I live in. Barnacles could take lessons from civil service employees.

  12. Paul Stimpson

    I'm not so sure that Jeff/Jane is the target or only target here. It could possibly be their manager, the one who was late / off drinking coffee / attending to their bathroom duties at the time of the incident who is in the firing line.

    This is an admission they have a 10 year problem with one of their employees that their manager has failed to improve or move them to a role they're better suited to / the unemployment line. It's an admission that not only has their manager not been properly managing them, the manager went on to give someone with performance issues charge of a responsibility where they could do actual harm.

    The slipshod manner this system has been implemented makes me wonder if the department takes this threat or their charge of the system to warn of it seriously.

  13. Paul Stimpson

    one more thing...

    Who on earth puts in a wide-area disaster warning system that doesn't have an "All Clear - The danger has passed" indication to tell the public they are no longer at risk and can go about their business? Such an indication could have been sent to put a false alarm to bed relatively quickly.

    I used to live near a chemical plant that had a siren to warn the local residents of a chemical leak. Continuous siren = Close all your windows and stay indoors. Intermittent siren = Incident contained. Go about your business.

    1. Anonymous Coward
      Anonymous Coward

      Re: one more thing...

      I live in a country that used to have Continuous siren = the Luftwaffe is about to bomb the crap out id us Intermittent siren = If you can hear this you are still alive and they've gone.

      1. Anonymous Coward
        Anonymous Coward

        Re: one more thing...

        I live not that far from Broadmoor "special hospital" where they still have such sirens to warn the locals if a dangerous loony escapes, and test them every Monday at 10:00am - warning siren followed after about a minute by the all-clear siren. Of course they haven't had an escape for over 20 years, and an escape resulting in an innocent civilian being murdered for even longer, so now they want to remove the sirens because obviously they're not necessary any more... Such is the bureaucratic mindset.

    2. This post has been deleted by its author

    3. DontFeedTheTrolls
      Mushroom

      Re: one more thing...

      An "All Clear" is fine as long as the person controlling the messages is:

      a) been given authorization* to issue an All Clear; and

      b) is actually still there to issue the All Clear.

      *English (Simplified) option chosen deliberately.

    4. ICPurvis47

      Re: one more thing...

      When I was in the forces during the Cold war, we had three different siren warning scenarios:

      Wailing up and down - imminent danger of air attack - as per WW2

      Continuous note - All Clear - as per WW2

      Intermittent siren at constant pitch - imminent danger of approaching nuclear fallout

      This was achieved by turning the siren's handle at constant speed and alternately opening and closing the shutters on the front of the siren.

      Thankfully, we never had to actually send out these warnings during the whole 25 years I was in the ROC.

      Unfortunately we were "stood down" in 1991 as there was no longer perceived to be any further threat of nuclear war, but maybe with Mr T and Mr K in charge of their respective countries, we should be "stood up" again?

  14. Long John Brass

    Hey boss...

    8:00am: Hey Boss; not coming in, I've always thought you were an utter bastard; Take this job and shove it!

    8:30am: Yeah I'll be in by 9.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hey boss...

      Seems like everyone should just agree there is a special dispensation for those cases. Evreyone averts their gaze and pretends it never happened.

  15. Anonymous Coward
    Anonymous Coward

    work down the list

    "Start at the top and work your way down."

    So, the firemen, then the math teachers, and so on in that fashion until everyone is warned?

  16. Anonymous Coward
    Anonymous Coward

    Ass protection

    When some donkey takes the fall. Lovely article and classic el reg.

  17. deadlockvictim Silver badge

    I'll bet that Simon Travaglia is fuming with the Hawaiians for the usage of future plotlines. It read like an BOFH article.

  18. Named coward

    DON'T PANIC

    If you have a GUI that has 2 options: "test warning" and "real warning", and you always test the "test warning", how can you be sure that the "real warning" actually works?

    1. Joe Harrison

      Re: DON'T PANIC

      If the "Real Warning" option doesn't work, who cares? You're going to die soon anyway so it's not like you will get the blame.

      1. Michael Thibault

        Re: DON'T PANIC

        Management material, right there!

  19. BinkyTheMagicPaperclip

    I prefer Schlock Mercenary's guide to what to do in the event of certain death

    'Free bar in the mess, let's party'

    If everyone really is going to die, better to let them die happy.

    Of course Yet Another Clickbait Headline, the whole reason for this is because there's a chance that people might not die.

    (also the comic continued on, so it wasn't actually certain death there either).

    It really does beggar belief that no-one ever specified or tested a system for reversing alerts..

  20. DontFeedTheTrolls
    Facepalm

    "No, you misunderstand, I'm not saying it's your fault, I'm saying you're getting the blame"

  21. fedoraman
    Alert

    Expect two spikes

    One, in the birth rate, about 9 months from then, and a second more immediate one in the divorce rate (as a consequence of the first)

  22. iLurker

    Storm in a teacup.

    Reality is this EMA changes nothing - it can’t change the outcome of an attack - real or fictional - and nor will people die as a direct result of it.

    What is more worrying is when people are relying on software to perform actions that - if wrong - may result in many fatalities. Starting with train control systems, and aircraft.

    Almost as bad are situations where humans (such as pilots) rely on software systems to make decisions that will affect the lives of many, possibly hundreds or millions.

    I’m far more worried about the integrity of the software used to inform the the head of state and military in the US, Russia and China of an attack - if any of them gives a false alarm there’s a real possibility a counter strike will be initiated that starts WW III.

    Thankfully they have old-tech telephones and still know how to use them.

  23. 0laf Silver badge
    Childcatcher

    They should have just treated it as an extreme test. If it caused panic and people didn't run to the shelters then the planning has failed and but the test was successful.

    Clearly that sort of testing is high risk but it produces the best data to refine the response plans.

    Someone who creeps around with Teflon shoulder pads might have said something like that.

  24. albaleo

    Pinning the blame can be tricky

    Not so tricky. The proper approach is to blame the youngest person in the team for every mistake. It saves time and lets you get on with solving the problem more quickly.

  25. Arachnoid

    Missed opportunity

    This was Trumps ideal opportunity to say he had been informed a new U.S. laser weapons system had downed the ICBM removing the danger and that any similar launches by Rocket man would be treated the same way.......

    He could then blame Worker 2 for provising misinformation if his bluff were called

  26. anonymous boring coward Silver badge

    "Hawaii's EMA – you’ve never seen crisis management quite like this"

    Guess you haven't seen the current Tory government then?

  27. Neill Mitchell

    Mini population explosion

    I wonder how many Hawaii false alert babies there are going to be.

    Could get awkward in some cases...

    1. jake Silver badge

      Re: Mini population explosion

      So-called Blackout Babies don't exist.

  28. Anonymous Cowerd

    "to be directed before he takes action"

    so who directed him then?

  29. TseTT
    Paris Hilton

    More incoming...

    I wonder if the maternity units are on standby for a sudden burst in numbers in a few months time?

    1. jake Silver badge

      Re: More incoming...

      So-called Blackout Babies don't exist.

  30. ZippedyDooDah
    Mushroom

    Edinburgh 1986 - End of the world

    Way back in 1986 I was a contractor at Bank of Scotland in Edinburgh.

    One morning I was woken in bed by a very loud siren that went on for sometime. I had no idea that such a system existed so I could only guess what it meant. My assumption was that it might be a hazardous chemical leak or just maybe an early warning of a nuclear attack. It stopped after 90 seconds with nothing further happening.

    When I got to work the the consensus was "I haven't got a clue". I was most amused by one of my colleagues saying that he thought that "The Queen has died".

    I found this newspaper "clip" on flickr . Apparently the alarm was controlled by the police and was actually sounded over the whole of Lothian and Borders". "Later, police said BT engineers traced a faulty piece of equipment, which had been taken away for investigation". It doesn't say if the faulty equipment was put in a straightjacket or not.

    https://www.flickr.com/photos/80902862@N05/7626173430

  31. Anonymous Coward
    Anonymous Coward

    So Much Cynicism

    And entirely justified . I'm sure we all recognized those stratagems. I'd claim to have invented some of them if I didn't know there is nothing new under the sun.

  32. Daedalus Silver badge

    The Final Step : Hawaii EMA, My Part in its Downfall

    The guy writes his memoirs and admits that, yes, it was me.

    Just Like Gunner Milligan (RA, demobbed) admitted that it was indeed he who sent the alert about the German invasion fleet without adding "TEST TEST TEST TEST".

    1. jake Silver badge

      Re: The Final Step : Hawaii EMA, My Part in its Downfall

      Any Yanks reading this (and perhaps youngsters in Blighty?), that's a reference to Spike Milligan's 7-part wartime trilogy that starts with the book "Adolf Hitler: My Part in His Downfall". If you haven't had the pleasure yet, get thee to a library, pronto. If they don't have it on the shelf, ask about inter-library loan; they CAN get it in. Should be required reading to graduate from high school on either side of the pond ... and points antipodian. IMO, of course.

      1. Daedalus Silver badge

        Re: The Final Step : Hawaii EMA, My Part in its Downfall

        The epigraph alone is worth the money: "After Puckoon I swore I would never write another novel. This is it."

  33. x 7

    Next time, call on Brian Blessed, he knows what to do

    https://www.youtube.com/watch?v=a8pGS4cWbHo

  34. DeVino
    Joke

    Oops, sure we ain't gonna be nuked ?

    Wonder if there were any of the following since everyone thought they were going to die ?

    1) Punch boss.

    2) Punch spouse.

    3) Invite unsuspecting human/animal/reptile to "Take me you crazy fool" on your desk at work.

    4) Poop in your annoying colleagues in-tray.

    5) Ransack the nearest shop of interest. If pet-shop see item 3.

    6) Down a bottle of something strong and intoxicating (possibly alcohol).

    7) Run aound naked with a bin on your head singing "They can't catch me, I'm the gingerbread man".

    Presumably most of Hawaii just hid in their bedroom the next day.

  35. Joseba4242

    Process allows one individual's mistake to have catastrophic effect. Communication unclear. No provision for unhappy path, or any such rehearsal. Root cause is a system that was systemically incompetently set up. Root cause blamed on one individual.

    The result? La la land announced winner.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021