back to article UK NHS 850k Reply-all email fail: State health service blames Accenture

The UK National Health Service Digital has blamed IT outsourcer Accenture for its ongoing email Reply-all woes. The cause of the cockup is almost certainly a combination of user error and a trouble-inviting system config update, though. On Monday morning, the NHS' internal email system was brought down by a test message sent …

  1. Kernel

    The usual suspects

    "Irritated folk then began clicking "Reply all,"

    Whether or not there was an incorrect setting in the email system, the actual problem appears to be the usual shower of idiots who can't tell the difference between 'Reply' and 'Reply All'.

    1. Fonant

      Re: The usual suspects

      Not "Reply All" at all (El Reg's reporting is misleading).

      They replied to the message, which had a single sender, which happened to be a "distribution list" containing - server-side - everyone in the NHS.

      1. Paul Crawford Silver badge

        Re: The usual suspects

        Sadly I have seen both issues in use.

        Case in point #1: one club that has 'reply' set to reply to the list because some folk felt it too hard to choose 'reply all' if they really meant it. As a result, you actively have to copy/paste an individual's email address if you don't want to spam to group.

        Case in point #2: Where I work the number of (apparently educated) numpties who 'reply all' to stuff that has no real need of informing the original recipients is depressing. Even worse there were groups set up that allowed a replay-all to everyone, with the expected dumb outcome. At least those distribution lists now only allow a few people to post to them (the actual content is worthless, so its not a great loss).

        1. A Non e-mouse Silver badge
          Unhappy

          Re: The usual suspects

          At my organization, even the people who work in the IT department still haven't worked out the difference between "Reply" and "Reply To All". If the IT bods can't work out the difference, what chance do we have of our users getting it right?

        2. RW

          Re: The usual suspects

          I hope you will allow me to extend your list:

          3. Email recipients who seem incapable of editing their replies and instead quote the entire message replied, no matter now long it is.

          1. Danny 14

            Re: The usual suspects

            Plus not having a sensible max recipients on the email server was a massive fail.

          2. Paul Crawford Silver badge

            @RW

            Worse - those stupid email clients that reply with any attachments also included in the endlessly growing email list.

          3. A Non e-mouse Silver badge
            Facepalm

            @RW Re: The usual suspects

            Email recipients who seem incapable of editing their replies and instead quote the entire message replied

            I've had people complain when I cut-down replies, saying that I'm forcing them to keep all the previous emails.

        3. Hugh_Johnson

          Re: The usual suspects

          ...and that's why I created a rule which moves any mail that I am not on the "To:" line into a folder called CC_Jungle.

          Saves me hours.

      2. Anonymous Coward
        Anonymous Coward

        Re: The usual suspects

        @Fonant

        That isn't what happened though. R sent the message to CroydonPractices DL, with a CC to R also.

        The change to allow people external to the creating org to reply to a distribution list certainly turned what would have been a million mail annoyance into replytoallmageddon, but I also understand why this was a necessary change.

        The NHS doesn't work in isolation of each other, a distribution list of particular specialists all from different organisations is very useful and makes sense in the context of how clinicians work.

        The fundamental issue is that you could even create a Dynamic Distribution List containing every single NHSmail user with no controls to stop it.

        There is no 'what if' report that runs to tell you how many users the query you've built will contain. There was only one way of testing it and they did it.

      3. terven
        FAIL

        Re: The usual suspects

        No, it was from "R", addressed to CroydonPractices, and "R" was in the CC list. A reply would only have gone to "R"

    2. RW

      Re: The usual suspects

      We've all been well aware of the reply-all idiots for a long time now, so if the systems aren't configured to take into account that idiocy, you can hardly blame the users. Note that in this case, the responsible idiot is one of the IT staff.

      Another issue: clearly NHS employees are not given proper training in the use of email. Whose responsibility is that, pray tell?

  2. Mark Allen
    Facepalm

    Reply All Fun...

    Reading this story on the BBC website there are some Twitter posts quoted.

    I like the person who mentions that someone in that email chain hit "reply all and requested read receipts..."

    1. TotallyInfo

      Re: Reply All Fun...

      Yes, that one had me in stitches!

  3. Ken Moorhouse Silver badge
    Pint

    NoRAVirus Symptoms and Advice

    It causes verbal diarrhoea, turns your Inbox into a profuse Outbox, dehydration from lots of hot air. No need to consult your Group Policies, just stay at home and drink lots of fluid.------->

    Any concerns dial 111.

  4. Borg.King
    Coat

    unsubscribe

    This story was presented on my browser by error.

    1. Anonymous Coward
      Anonymous Coward

      Re: unsubscribe

      Please unsubscribe me from your unsubscribe messages AT ONCE!! No, quicker than that!

    2. Anonymous Coward
      Anonymous Coward

      Re: unsubscribe

      I'm Sparticus!

    3. Anonymous Coward
      Anonymous Coward

      Re: unsubscribe

      To unsubscribe from this service you must first purchase a Craft-O-Matic Adjustable Subscription Cancellation Unit.

      The unit can be obtained from most hardware stores and dental clinics.

      Be sure to obtain the proper permits to operate the unit from the Nuclear Regulatory Commission and the Food and Drug Administration in Washington D.C. USA.

      Be sure to carefully unpack the kit and place each component in its accompanying mesh safety bag.

      Mount the Pershing DF4 mesinator on top of the perforated Gerring Mach 77 refibulator and attach them using the eight-millimeter torque fork.

      Be sure that the refibulator is mounted at a 66 degree angle and properly dispersed so that it is flush with the curved section of the Pyrex thistle tube.

      Place the four sections of the triangular separation gear into the posture cylinder and lock them into place using the band aid adhesive strip.

      Insert the wiggling pin into the wobbling hole, making sure that it seated correctly.

      Place the D cell battery and the eleven 9 volt batteries in the power chamber.

      The device should be calibrated before operation using the optional digital corkscrew accessory pack prior to operation.

      Insert the digital corkscrew through the electronic combustion service chamber using caution not to touch the reinforced tungsten igniter control module and quickly turn the inverter drive to 28.6 degrees.

      Turn the Craft-O-Matic Adjustable Subscription Cancellation Unit upside down and hit the bottom plate with a 48-ounce ball-peen hammer while shaking the unit vigorously.

      Force open the door to the incineration valve compartment and set the pressure gauge to 719 psi.

      Close the door and seal it shut with duct tape.

      The unit should now be properly calibrated and ready to use.

      Before activating the Craft-O-Matic Adjustable Subscription Cancellation Unit, you must first elevate it to a height of 229 feet above sea level to insure that the unit receives the proper oxygen level and barometric pressure.

      Point the aerial to 17 degrees north by northeast to within the parameters of the Telstar GS-2 weather satellite and apply pressure to the wing shaft on the southern most section of the modular accelerator.

      Using the special ratchet adapter supplied with the unit, rotate the heater core to the "on" position. The "on" position has been obtained when the green light begins to flash, signifying that the red light is about to go off.

      Once the red light is off, flip the toggle switch labeled "ON/OFF" to the "ON" position and count to 47 before logging on to the system.

      Logon using your username and password and wait for the prompt.

      Once prompted you must check the box with the appropriate action you wish to take and then press the pressure release button and turn off the compressor while turning the hand crank at 231 meters per minute.

      Next, press control, alt, delete, caps lock, shift, number lock, escape and tab simultaneously.

      Press enter.

      You will have one second to complete the procedure.

      If you fail to respond in the time limit allowed, simply purchase a new Craft-O-Matic Adjustable Subscription Cancellation Unit and start from the beginning.

      Please remember that this is the only way we will accept for you to unsubscribe from this service.

      We have made every attempt to simplify the procedure for your convenience.

      WARNING!

      Failure to comply with the unsubscribe policy will result in immediate termination of your subscription so please follow the above directions closely.

  5. Rampant Spaniel

    A similar thing happened at a few companies, virgin media / ntl being one. Somebody sent some idiotic email about thinking of a colour and then an object and the company had 48 hours of people emailing everyone "red hammer", "yellow hammer", or please unsubscribe me. It was hillarious, especially the emails from the chief bin lickers assistant commanding everyone to stop.

  6. Phil Kingston

    On the plus side, at least if was some bod's test-in-prod mail that brought it down in a (hopefully) partially-expected/monitored fashion and not just some external party making a typo at some random point in the future.

  7. TheMadMuskateer

    It took me about 10 seconds to figure out what was going on when I saw the first email. After that it was amusing to watch everyone get more and more irate.

  8. sbt65

    My phone is now off until the morning. I'm not overly convinced I'll be able to get any shut-eye with constant pinging or buzzing tonight. I'll let it catch up tomorrow.

    ps - I'm still getting 09:38 emails at this moment. Doesn't bode well for tomorrow.

    1. Danny 14

      Just add a mail rule with the subject until it blows over.

  9. x 7

    I think this was a deliberate hack. The originating e-mail address was from an individual account at Croydon CCG and somehow that got converted into a mass distribution list.

    There were two problems:

    1) How did the single mail get sent to "all" FROM the CCQ mail account?

    2) How was that CCQ account hacked so that all replies sent to it were then forwarded to "all"?

    Difficult to see how that happened without deliberate tampering

    Its important to note that most replies were NOT sent to "all", but simply back to the originating e-mail address, which by then had been converted into a mailing group. I got several hundred replies (I never got the original) and most were sent to me as a result of the reply just to the originating CCQ mailbox..

    This was the first stage of a spear-phishing attack. Most NHS mails have footers giving the details of the senders.........all those who replied have now had that data harvested. Expect more targeted attacks soon. We already know that there are daily attempts to hack the NHS servers.........

    1. Anonymous Coward
      Anonymous Coward

      1) Either R or a local admin created a dynamic distribution list - I have no clue how the query was formed and I'm not about to test it but it included a lot, if not all, of the users on NHSmail

      2) I don't believe it was, all of the replies I received were sent to the distribution list - if you still have the mail I'd check again, I don't think there is anything more sinister than a user error enabled by a poorly designed feature.

    2. Doctor Syntax Silver badge

      "Difficult to see how that happened without deliberate tampering"

      Never attribute to malice that which can be attributed to stupidity.

  10. Mike Flex

    Reg exclusive?

    "At 1545 GMT, NHS sources were telling The Register that emails with 0950 timestamps were only just beginning to arrive in their mailboxes."

    What, special sources were telling you privately; or you were just reading it off twitter like the rest of us?

  11. Raphael

    reminds me of the time

    when working at a largish Engineering company (with offices in NZ, Aus, UK and Canada) and student in the head office read a (hoax) news item about a virus.

    So he painstakingly went through and sent an email to all staff (all 1000+ of them, he didn't use the mailing lists) warning them of said virus. After the 3rd or 4th reply all to tell him it was a hoax the mail servers crapped themselves and our mail went bye bye for the rest of the day.

  12. cantankerous swineherd

    your medical records are perfectly safe with this shower.

    1. Lotaresco

      "your medical records are perfectly safe with this shower."

      Really? Because my experience with NHS Direct is that their IT people have trouble in the shower finding their derriere with both hands. Oh hang on... this is that irony thing I don't understand, isn't it?

      <FX: wavy lines>

      This takes me back to a conference a few years ago when an NHS Direct bod gave a speech about their "great success" in ensuring that that hundreds of thousands of iToys owned by medical staff were all permitted to access the network within a few hours of the latest Apple product launch. Most of the non-NHS IT bods sat staring and asking if they had thought through he implications of a massive BYOD with access to sensitive personal data. The answers seemed more than a little complacent.

  13. Anonymous Coward
    Meh

    State health service blames Accenture

    I can understand that. It's a fair default position to take and saves the bother of looking for the actual facts.

  14. gh4662

    Disapointed

    Really disapointed that The Reg hasn't caught up with the times, they only have a 'reply' button on their forums, please can we have a 'reply all' button?

  15. meanioni

    Not just the only stupid email thing

    ...like my former boss, the CEO and a peer of the realm. He *always* spelled his email address saying: "lower case" for his name and "upper case" when he got to the name of the Company....

    He also made us do the same on our business cards....

    1. Lotaresco

      Re: Not just the only stupid email thing

      "...like my former boss, the CEO and a peer of the realm. He *always* spelled his email address saying: "lower case" for his name and "upper case" when he got to the name of the Company...."

      I take it he was a PHB?

  16. Norman Nescio Silver badge

    Poorly designed application

    One of the many causes of this particular problem is using a badly designed email application, which attempts to shoe-horn too much functionality into single buttons, and also mixes different classes of email address.

    By having a distribution list look exactly like a normal email address, there is no feedback to the user that sending a message to that apparently single address could, in fact, have bad consequences. I've done it myself: not noticed the distribution list hiding in a long list of recipients in an email. It makes sense that distribution lists are flagged, and handled separately by the user interface.

    For example, if the recipient list contains one or several distribution lists, a pop-up should appear after you have pressed the send button saying "This email will be sent to a distribution list containing <n> members: are you sure you wish to send this email?". If the email is to be sent to more than 100 recipients, then a second pop up after the first should appear saying "Are you really sure?", and if more than 1000, another saying "Are you really, really sure?" - in fact the number of pop-ups should vary as the integer amount of log base 10 of the number of recipients. So sending an email to 800,000 recipients requires navigating through 5 pop-ups. Each pop-up should also not be a Yes/No/Cancel, but require the user to type in the number of recipients they have just been told: just to drive the point home.

    Secondly, Reply All should not have the same behaviour as Reply. Obviously, Reply is to the single sender (which, if it is a distribution list, should trigger the distribution list post-send pop-up), but Reply All should pop-up a dialogue box with radio buttons which explicitly force you to select replying to all recipients that were individual email addresses, or not; all recipients that are distribution lists, or not, and the original sender, or not. in this way the work-flow between hitting Reply and Reply All is different (which is good), and it also allows you to choose sensible options on the Reply All (I have often needed to edit out the original sender from Reply All messages when I'm commenting within the team on a message a team manager has sent; or indeed on a message sent to the team by an outside client - you often don't want the outside client to be included in the inter-team discussion. If anything, an default that doesn't reply-all to any people outside an organisation, whether senders or recipients, can be helpful.)

    The semantics of email addresses are not well defined: there is no standard for when an email address is just an email address or when it is a distribution list - it is actually an attribute of the receiving mailbox. This is flexible, and has some security advantages, but it does make it difficult for email clients to exhibit the behaviour I have described above. In a closed system, you can define special formats of email addresses so that a client can parse them and take action, but in general RFC-compliant land, this is not possible. So this behaviour is not going to change quickly, if at all.

    And note. At no point have I blamed the user. Or the administrators. Well written software helps people do their jobs, and helps to prevent them making mistakes. If people consistently make the same mistake in their interactions, the process needs to be redesigned to help avoid this from happening. It's how the aviation industry works (imperfectly). If you don't take human factors into account, you WILL be bitten by them. Help people to improve and they will tend to. Tell them they are idiots, then you will breed fools.

    1. AndyS

      Re: Poorly designed application

      So, in order to solve an occasional problem, you want to destroy the ease and speed with which email can be used, and break the workflow with multiple pop-ups, radio buttons, text input...

      Maybe running a trial or two (ideally outside the basement) will rapidly show you why this is a stupid idea.

      A warning stating that the email you are about to send will go to X users (where X > a predefined number, probably about 100) would be useful, and is already widely implemented (eg where I currently work). As is a limit on who can email large lists.

      Everything else you suggest is just... ridiculous and nonsense. It's not possible to tell if an address is for a single person or a list. I could set up my gmail address to forward to 10,000 people - how will your lotus notes client know that, when you send an email to an exchange server across the road?

      1. Norman Nescio Silver badge

        Re: Poorly designed application

        AndyS

        Thank-you for making it clear that I committed the cardinal sin of over-estimating the intelligence of my readership. I will try to write more simply and clearly in future.

        1) Standard work-flow of replying to a single sender is unaffected.

        2) Work-flow of reply-all to fewer than 10 individual recipients (not mailing lists) is changed in that you get a pop-up asking you if you want to include or exclude the original sender in your reply-all.

        3) Work-flow of reply-all that includes distribution lists in the recipient lists gets a pop-up with a toggle asking you to choose if you want the mailing lists to be included in the reply-all.

        4) Work-flow of reply-all that includes recipient outside your organisation gets a pop-up with a toggle asking you to choose if you want external recipients to be included in the reply-all.

        5) Work flow of a reply-all that has more than 10 recipients gets a pop-up asking you to confirm by typing in (echoing) the number of recipients the email client has calculated it will go to. More pop-ups may appear according to the order of magnitude of the calculated number of recipients.

        Yes, I am aware this doesn't work across email boundaries as there is no standard for recognising email distribution list addresses, or knowing how many addresses such a list will go to. Within individual domains, it is entirely possible to do this kind of thing without worrying about inter-organisation standards.

        Unless the majority of your emails are of the reply-all type, your standard work-flow is unaffected. Reply-all type emails have further, arguably sensible, checks to help you to not send messages inadvertently to your manager; an external client; or a large population triggering a reply-all storm.

        Sensibly written software would have a way of turning off the checks for those people who reasonably don't need them. You might be one of them.

        1. DryBones
          Coffee/keyboard

          Re: Poorly designed application

          Being able to write simply and clearly shows one possesses uncommon sense.

          Do not reinvent the wheel until you have made educated and effective use of the ones already available.

          1. Norman Nescio Silver badge

            Re: Poorly designed application

            True.

            I find writing simply and clearly difficult. My sense is by no means uncommon.

            As for reinventing the wheel: point taken.

            It would help if people criticising posts actually read and understood the post they are criticising, rather than criticising what they think they read.

            Both the Exchange Server and the Outlook client have a lot of knobs to twiddle, which is good. On the other hand, I don't think anyone would say they are unimprovable. Microsoft claim to put a great deal of effort into making their end-user software easy to use, but their efforts don't seem to be particularly effective in this area. The end result is that users can deploy a foot-gun with depressing ease.

    2. Anonymous Coward
      Anonymous Coward

      Re: Poorly designed application

      Well Exchange does have a thing called MailTips that will usually pop up with the kind of message you are looking for but might have been turned off by policy.

      Certainly agree that Reply All is a scourge.

      Project Manager to Team: Can I have a note of the days off you are intending to take at Christmas please

      All team members except me: Reply All - I'm taking xyz.

      I DON'T CARE!!!!!

      1. 's water music

        Re: Poorly designed application

        Project Manager to Team: Can I have a note of the days off you are intending to take at Christmas please

        All team members except me: Reply All - I'm taking xyz.

        I DON'T CARE!!!!!

        How else will you know what days you can harvest the good stationery, less stained chairs and other good stuff?

  17. Pat Harkin
    WTF?

    WHY does an "ALL NHS STAFF" list even exist?

    I worked in the NHS for years (SWMBO still does) and the only message which I can imagine you'd want to send to all NHS staff is "The NHS has been shut. Don't come to work tomorrow." I worked in histopathology. There's not much everyone in my lab needed to know that all HR also needed to know or psychiatry or community midwifery.

    1. Lotaresco

      Re: WHY does an "ALL NHS STAFF" list even exist?

      "There's not much everyone in my lab needed to know that all HR also needed to know"

      "Someone let picrate dry out in the tissue processor, again. Their remains are now available for collection. May contain traces of glass."

    2. Doctor Syntax Silver badge

      Re: WHY does an "ALL NHS STAFF" list even exist?

      "There's not much everyone in my lab needed to know that all HR also needed to know or psychiatry or community midwifery."

      If you were in HR you'd be convinced that whatever brain-farts you were dropping would be essential reading for everyone.

    3. Ken Moorhouse Silver badge

      Re: WHY does an "ALL NHS STAFF" list even exist?

      A National Emergency would be one reason, I guess. Oh dear, looks like it would fail that test.

  18. Mary Hinge

    'kin users!!

    Either way, if the users hit "reply" and replied to a large DL or hit "reply all" - the bloody users need to be educated, not just in the correct use of "reply" and "reply all" but also on how to identify a message that does not require any response at all!!! Users are too eager in responding to these type of mistakes; "Take me off this list now!!", "Why am I receiving this????", "Do you know you just interrupted me drinking my fifth designer cup of coffee this morning??? - take me off this list at once you cants!"

    Simply ignoring the message and then deleting it not only shows you have a brain cell but also prevents your organisations dirty laundry being broadcast all over the facking internet!!!

  19. Anonymous Coward
    Anonymous Coward

    NHS Connecting for Health -> HSCIC -> NHS Digital.

    Same people, same office, same problems - only the date stamp is different

  20. Missing Semicolon Silver badge
    Unhappy

    "reply all" the basic CYA

    In many organisations, if you don't "reply to all" on anything important, your mail will be ignored - especially if it involves the recipient putting themselves in danger of being responsible for something.

    Without "reply all" you have no email trail in the boss' inbox to prove that the other numpty dropped the ball, not you.

  21. theOtherJT Silver badge

    How does this sort of thing happen?

    We don't use Exchange, so possibly I'm missing some exchangy concepts here but as I understand it someone foolishly sent out a mail to a ton of people on a list that probably shouldn't have existed in the first place.

    Some of those people replied to that email - which, instead of being immediately sent back with an auto reply along the lines of "This address belongs to a mailing list daemon. If you wish to send email to this list, please email $LISTSENDADDRESS" was taken as an instruction to the mailer daemon to send that message to everyone on that particular list - which already feels like a pretty massive configuration cockup right there, but ok, that happened.

    Even assuming that you made the mistake of having the reply-to being the send-out address, why then did every single one of these emails not get the next line of "Your message to this list requires approval. Please wait for this message to be approved by a list administrator" which should always be the case for lists that can hit tens of thousands of people for precisely this reason, and then sit in the mailer's approval queue?

    Then - even assuming that THAT was allowed to happen for some reason, why doesn't the list daemon go "Holy shit, my queue is suddenly full of tens of thousands of messages, that's never happened before. I'd better rate limit those bastards and email my owner to warn them that something weird is happening." at which point it drip feeds messages out a few hundred at a time until someone comes along and tells it that it's OK, no one's account has been compromised and you're not being co-opted into some massive bulk spamming campaign, we really did mean to email the entire organisation.

    Feels like a lot of config level school-boy errors had to be made to allow this to happen in the first place.

    1. Loud Speaker

      Re: How does this sort of thing happen?

      a lot of config level school-boy errors had to be made

      Yep - something must be done - this is something - so it must be done!

      (Pay peanuts: get monkeys).

  22. Anonymous Coward
    Anonymous Coward

    Accenture = Evil Empire

    Well, this is not the first catastrophe by Accenture... there is a whole website dedicated to Accenture's failures and unethical practices, it's called Exposing Evil Empire ( http://ExposingEvilEmpire.com ).

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like