back to article So you accidentally told a million people they are going to die: What next? Your essential guide...

It's a common enough scenario: you are in charge of a critical piece of your company's computing infrastructure, and you make a simple mistake with far-reaching implications. It could be a mail server misdirecting internal documents into journalists' inboxes, or a poorly configured border firewall. Perhaps you accidentally …

Page:

  1. Anonymous Coward
    Anonymous Coward

    Fuckwits in charge...

    ...and fuckwits pressing the buttons.

    The End.

    1. Anonymous Coward
      Anonymous Coward

      Consolation

      Remind the million folks that because they've led such empty lives, humanity will be able to carry on just fine without them.

  2. dnicholas

    This piece is why I love el Reg. So few outlets know exactly the likely order of events in such fuck ups.

    We've all done it, just rarely one meeellion people in the loop. One day, when I grow up, I want to get away with something as audacious as this

    1. veti Silver badge

      I hope they've got a good story for the NSA, on how they got their hands on that transcript of the phone call to the governor.

    2. Anonymous Coward
      Anonymous Coward

      Not that rare, based on experience

      A few large cloud provider outages, bank web sites, bank batch schedulers...

      Relatively often, although thankfully not all the same organisation....

  3. Michael Thibault

    In the confusion...

    I wonder whether the re-issuing of the alert at regular -- but short -- intervals would not have served to signal that something was amiss about the signal itself. IOW, if no self-destruct for the message could be found within a minute or two of the realisation that the message was in error, why not make it very obvious what the error was by repeating the message unnecessarily?

    1. Synkronicity

      Re: In the confusion...

      Wtf? No, that just creates more urgency that THE NUKE WE TALKED ABOUT ON THE NEWS IS FINALLY HERE AND IT'S LANDING NOW SEEK SHELTER THIS IS NOT A DRILL. Now imagine that endlessly repeating, hijacking everyone's phones with loud sounds and vibrations for 38 agonizing minutes. That'll solve the panic!

      1. Michael Thibault

        Re: In the confusion...

        "WTF?" is exactly what people would likely respond with -- drawing them out of their inner world and, possibly, encouraging a re-assessment of matters, possibly putting an end to what will end up -- either way -- being a completely pointless flight response. Ideally, there are 0 alerts; very much at the other end of things, there will be 1 alert; what's going on when there are multiple alerts, all identical, over the course of the time it would take for nuclear annihilation to have occurred twice over? Things that make you go "hmmmmm", no?

        1. Anonymous Coward
          Facepalm

          Re: In the confusion...

          You seem extremely confused. Such a system then needs the opposite to what you suggest. An automatic "this was an error" text after 60 seconds. If it is not an error, it requires someone to either send a second statement just before the 60 seconds is up, or in the case the entire office is also evacuated, send a second "reminder" after the first.

          Basically in addition to the (not used in the current case) 2 factor/2 user confirmation it also needs each user to confirm a second time to get the system to also send a second confirmation text.

          Then everyone would get use to the kind of "earth quake/tsunami" warnings, that are just that, warnings, as it is not until it hits that you know how big/small it is. It would encourage everyone to actually take shelter in the case of text 1, but not panic or do anything stupid, as they wait for text 2 to say "stay" or "false alarm/danger passed without damage".

          Sorry, but your example is as much you panicking/grasping at straws as those involved were. :(

          1. Michael Thibault

            Re: In the confusion...

            You have misread what was written. I wasn't suggesting a system for the general case; I was addressing a particular instance -- the one where the system's design proved to have been insufficiently complete in its coverage of the possibilities that it was impossible for participants to deal with the particular circumstances quickly and effectively.

            My original post asked about the wisdom (and likely effect) of using the same signal, in those very circumstances, in an unexpected way -- given that everyone involved should surely have seen that the system was going to be replaced any-fucking-way, and that no more damage could possibly have been done than had been done the instant the erroneous signal was issued and it became apparent there was no pulling it back (or specifically saying "Oops!"over the same channel). IOW, they had nothing else in their quiver...

            1. Anonymous Coward
              Flame

              Re: In the confusion...

              No. Because your solution, though using their existing tech/system/process still does not solve, but adds fire to the problem.

              If the system was at error, it needed an "error" message. Or, if it was life threatening style "we messed up, we gotta fix it with what we got", then send out the "free bagels to everyone" and "alien invasion" and "this is a test message" messages that are no doubt in the system ready to be sent (yes, there are stories of these messages being in such systems, so they exist, it just needs a human to think through and send some type of cancel/error message).

              But without actual access to their system, we don't know what they did have to send. Sending any amount of "missile incoming" messages would never cause people to think it was in error, until the message becomes redundant in all instances. In which case, those ignoring it already were, and those taking action already did, those thinking "is this real or not, that seems too many messages", as you mention already hit the ignoring the message group and thus in a real incident, would not take shelter. :(

  4. Florida1920
    Headmaster

    Management Mantra

    First we fix the blame. Then we fix the problem.

    1. Mark 85

      Re: Management Mantra

      Employee 1 now has a new career path... he/she will be the "government" or "corporate" scapegoat. For a nice fee, he/she will willingly take the blame for any screw up.

      1. david 12 Silver badge

        Re: Management Mantra

        >For a nice fee, he/she will willingly take the blame<

        IT consultant. Notice how much IT consultants get paid? Notice how many external project fail? As the rats desert the sinking ship, failing projects are handed off to external contractors.

        Been there. Done that. Got the paycheck.

        1. This post has been deleted by its author

    2. Hans Neeson-Bumpsadese Silver badge

      Re: Management Mantra

      I have genuinely worked on a project where an agenda item for the project kick-off meeting was identifying who would get the blame if the project when wrong

      1. Bernard M. Orwell

        Re: Management Mantra

        " who would get the blame if the project when wrong"

        When, not if.

  5. Blofeld's Cat
    Mushroom

    Work down the list ...

    "In this case, it was: the police; then the people that operate emergency sirens; then those in charge of the different counties. Then the TV and radio stations."

    But not, for some reason, the person with their finger on the big red button labelled "Retaliate" ?

    1. jake Silver badge

      Re: Work down the list ...

      The "retaliate" button is in DC, not HI. DC was still asleep at 8AM Hilo time. Thankfully.

      1. elDog

        Re: Work down the list ...

        The big dog in DC is in bed by 6:30PM and still snoozing around 10AM. During the awake period, he's not too bright so please don't bring in any critical items, other than USA Toiday and his standings with the fox viewers.

        1. Rich 11

          Re: Work down the list ...

          And don't bring him his standings with the Fox viewers unless they're positive. Show him his inauguration crowd photos again, if you need him in a good enough mood to cope with the day's list of executive signings and photo ops. "Look, Mr President, three million, easy. Everyone says so. Far more than that Kenyan ever got."

        2. Umbracorn

          Re: Work down the list ...

          "Better to let the great man sleep."

          http://www.news.com.au/world/did-a-power-nap-cost-adolf-hitler-victory-in-world-war-ii-staff-were-too-scared-to-wake-him-on-dday/news-story/1b2fa0f13258f79fe4c2ce58013cdab1

        3. Jtom

          Re: Work down the list ...

          Democratically-controlled Hawaii f's up big time, and some deranged people still find a way to insult Trump. Keep it up. You're attitude is what greatly helped him get elected to being with.

          1. Anonymous Coward
            Anonymous Coward

            Re: Work down the list ...

            I believe it's something commonly referred to as 'sarcasm'.

            You must be new here....

            P.s. in where did that statement refer directly to Trump? I'd take it as read it implies they're all as bad as each other.

            1. lorisarvendu

              Re: Work down the list ...

              "P.s. in where did that statement refer directly to Trump?..."

              Well not by name, but the following does prettty much imply the current incumbent.

              "The big dog in DC is in bed by 6:30PM and still snoozing around 10AM. During the awake period, he's not too bright so please don't bring in any critical items, other than USA Today and his standings with the fox viewers."

      2. Antron Argaiv Silver badge
        Thumb Up

        Re: Work down the list ...

        DC was still asleep at 8AM Hilo time....

        Nope. Works the other way. +5, so DC was just finishing lunch.

        1. Anonymous Coward
          Anonymous Coward

          Re: Work down the list ...

          Exactly... DC was still asleep. You don't expect them to be awake and working when they've just finished a hearty meal?

        2. jake Silver badge

          Re: Work down the list ...

          You and I would be just finishing lunch, yes. Capitol Hill was still asleep.

  6. Yet Another Anonymous coward Silver badge

    Cyber-attack

    If all else fails blame a cyber-attack by state sponsored thespians of somewhere to be decided later

    1. Anonymous Custard
      Joke

      Re: Cyber-attack

      Quick, let's invade Thespia!

      1. Anonymous Coward
        Anonymous Coward

        Re: Cyber-attack

        I dated a thespian once. I had to end if, everything felt a bit stage-managed...

        1. Michael Thibault

          Re: Cyber-attack

          "I dated a thespian once..."

          Admit it -- it was the lack of encores, wasn't it? No? Performance, then?

  7. Jim Mitchell

    I'm wondering what happened at the various military bases on the islands when that emergency text went out.

    1. notowenwilson

      From what I understand initially there was a lot of shouting and running and within a couple of minutes they had been advised that it was all clear and everyone went about their business.

    2. SquidEmperor

      We fragged Lieutenant Neidermeyer and raided the General's booze stash.

  8. Throatwarbler Mangrove Silver badge
    Windows

    NUCLEAR MISSLES NOT ACTUALLY LAUNCHED. NO WW3 AFTER ALL. SAD!

    1. Mayday
      Mushroom

      As opposed to MAD?

  9. jake Silver badge

    So you accidentally told a million people they are going to die: What next?

    "We're going to DISNEYLAND!"

    1. Rich 11

      Re: So you accidentally told a million people they are going to die: What next?

      No, no, no. You only get to go to Disneyland if you eat all your greens and abase yourself before Jesus.

    2. Eddy Ito

      Re: So you accidentally told a million people they are going to die: What next?

      That's just it, they weren't told they were going to die, they were told to "seek immediate shelter" which causes everyone to stampede like cows heading off a bluff on their way to some magic hole in the ground. Oh wait.

      Now I get it, a self burying population means easier cleanup when the Geiger counter ticks a little slower in a few years. Simply genius!

  10. elDog

    Good compendium of possible actions/inactions/mis-actions.

    In real life we don't have a lot of perceived choices. Something about gut reactions vs. conscious decisions.

    If I thought that my life was going to be going up in an instant flare of immense heat, I might thank the lord that I won't have to lie in hospital/nursing-home/hospice beds with tubes running in/out and money flowing out. I think I'd grab whatever is left of a bottle of cheap wine and expensive scotch and toast the new day.

    1. Anonymous Coward
      Anonymous Coward

      Re: Good compendium of possible actions/inactions/mis-actions.

      Considering that I'm facing a fate worse than death, becoming a shut-in? Yeah, I'd guzzle to that.

    2. Phil O'Sophical Silver badge

      Re: Good compendium of possible actions/inactions/mis-actions.

      How long before somone in the Land of the Lawsuit decides to sue Employee 1 for child support on the basis that, facing inbound armageddon, they decided to "go out with a bang", and didn't see any poiint in taking precautions?

      1. Rich 11

        Re: Good compendium of possible actions/inactions/mis-actions.

        According to the usage logs for PornHub, many Hawaiians commemorated the lack of impending doom with a celebratory wank.

        1. 's water music
          Thumb Up

          Re: Good compendium of possible actions/inactions/mis-actions.

          According to the usage logs for PornHub, many Hawaiians commemorated the lack of impending doom with a celebratory wank

          It's the obvious response to a warning of impending fiery death with zero notice. I mean sure, take advantage of situational factors where possible but you need a fall back that requires little preparation and few to no external resources.

          Also, bonus time for the Pronhub marketing department's quick reactions.

          1. Anonymous Coward
            Anonymous Coward

            Re: Good compendium of possible actions/inactions/mis-actions.

            It didn't last though.....

  11. mhoulden

    Anyone can make one mistake. However, if one person kept making the same mistake over and over again on a life-critical system like this, I'd want to know how they were put in a position where they could do so. Why was no one supervising him when he sent the incorrect alert? Did no one check it before it was sent? Did anyone else try using the system to make sure it was foolproof and fail safe? If someone else made the same mistake, what actions were taken to stop it happening again? Scapegoating one person is easy but it looks like there are serious management issues that they'd rather not go into.

    1. kain preacher

      I don;t buy that he was a screw up for a decade/ Public sector unions are not that strong in the US and if you can document a screws ups , poor review over a year the guy is gone. Nothing the union can do.

      1. tom dial Silver badge

        I buy it. At least in the federal civil service the two basic bases for dismissal of non-probationary employees are poor performance and misconduct. it is terribly difficult to terminate an employee for poor performance or even, sometimes, for non-performance.

        I know of a case (this in a US DoD agency) in which a solid foot of paper documentation was diligently accumulated, over a period of over a year, of an employee's incompetence, non-performance of assigned duties, subbasement level annual appraisals, and failure to improve despite detailed counseling from his supervisor. The employee in question previously had been passed on by several other supervisors who noted the behaviour but were unable or unwilling to do the work of documenting it.

        Finally, the I's and T's dotted and crossed, the day came when he was notified of termination on the basis of inadequate performance. Then followed the lawsuit, in which the African-American former employee alleged race discrimination by his supervisor, a white man, and manager, a white woman originally from southern Virginia. I knew both of them well enough to know the discrimination claims were rubbish, but under civil service procedures they still had to be adjudicated fairly. In the end, the employee was, indeed, terminated, but based on disability, complete with a pension based on standard Civil Service Retirement System rules.

        On the other hand, I know of a case in in the same agency where a very competent and generally high performing employee fell into the habit of playing on-line games during working hours when, as sometimes happened in his branch, not busy with specific work-related tasks. Unfortunately, the employee's cubicle was adjacent to one of a few doors from a general work area to the building corridors and in plain view from the deputy department director's office. The employee also had fairly serious personal hygiene issues. Dismissal (based on misconduct) required no more than collection of some Internet traffic showing the online gaming, a few informal counseling sessions, a warning letter and meeting and, when the behaviour was later repeated, a dismissal letter. A sad part is that, after receiving the warning letter, the employee had passed up an early retirement opportunity that included a $25,000 lump sum payment at separation, although I heard later that she had, with legal assistance, arranged to obtain the lump sum as well as the annuity to which she was entitled in any case but otherwise would have been deferred for two or three years due to age and service requirements.

        The Hawaii EMA might have been better off to have "found" porn on the hapless employees work PC.

      2. Anonymous Coward
        Anonymous Coward

        Public sector unions are not that strong in the US

        Must be a different US than the one I live in. Barnacles could take lessons from civil service employees.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon