nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

back to article
UK NHS 850k Reply-all email fail: State health service blames Accenture

Bronze badge

The usual suspects

"Irritated folk then began clicking "Reply all,"

Whether or not there was an incorrect setting in the email system, the actual problem appears to be the usual shower of idiots who can't tell the difference between 'Reply' and 'Reply All'.

18
3

Re: The usual suspects

Not "Reply All" at all (El Reg's reporting is misleading).

They replied to the message, which had a single sender, which happened to be a "distribution list" containing - server-side - everyone in the NHS.

17
2
Silver badge

Re: The usual suspects

Sadly I have seen both issues in use.

Case in point #1: one club that has 'reply' set to reply to the list because some folk felt it too hard to choose 'reply all' if they really meant it. As a result, you actively have to copy/paste an individual's email address if you don't want to spam to group.

Case in point #2: Where I work the number of (apparently educated) numpties who 'reply all' to stuff that has no real need of informing the original recipients is depressing. Even worse there were groups set up that allowed a replay-all to everyone, with the expected dumb outcome. At least those distribution lists now only allow a few people to post to them (the actual content is worthless, so its not a great loss).

3
0
Silver badge
Unhappy

Re: The usual suspects

At my organization, even the people who work in the IT department still haven't worked out the difference between "Reply" and "Reply To All". If the IT bods can't work out the difference, what chance do we have of our users getting it right?

0
0
RW

Re: The usual suspects

We've all been well aware of the reply-all idiots for a long time now, so if the systems aren't configured to take into account that idiocy, you can hardly blame the users. Note that in this case, the responsible idiot is one of the IT staff.

Another issue: clearly NHS employees are not given proper training in the use of email. Whose responsibility is that, pray tell?

1
0
RW

Re: The usual suspects

I hope you will allow me to extend your list:

3. Email recipients who seem incapable of editing their replies and instead quote the entire message replied, no matter now long it is.

8
0
Silver badge

Re: The usual suspects

Plus not having a sensible max recipients on the email server was a massive fail.

3
0
Silver badge

@RW

Worse - those stupid email clients that reply with any attachments also included in the endlessly growing email list.

1
0
Silver badge
Facepalm

@RW Re: The usual suspects

Email recipients who seem incapable of editing their replies and instead quote the entire message replied

I've had people complain when I cut-down replies, saying that I'm forcing them to keep all the previous emails.

1
0
Anonymous Coward

Re: The usual suspects

@Fonant

That isn't what happened though. R sent the message to CroydonPractices DL, with a CC to R also.

The change to allow people external to the creating org to reply to a distribution list certainly turned what would have been a million mail annoyance into replytoallmageddon, but I also understand why this was a necessary change.

The NHS doesn't work in isolation of each other, a distribution list of particular specialists all from different organisations is very useful and makes sense in the context of how clinicians work.

The fundamental issue is that you could even create a Dynamic Distribution List containing every single NHSmail user with no controls to stop it.

There is no 'what if' report that runs to tell you how many users the query you've built will contain. There was only one way of testing it and they did it.

3
0
FAIL

Re: The usual suspects

No, it was from "R", addressed to CroydonPractices, and "R" was in the CC list. A reply would only have gone to "R"

2
0

Re: The usual suspects

...and that's why I created a rule which moves any mail that I am not on the "To:" line into a folder called CC_Jungle.

Saves me hours.

0
0
Facepalm

Reply All Fun...

Reading this story on the BBC website there are some Twitter posts quoted.

I like the person who mentions that someone in that email chain hit "reply all and requested read receipts..."

14
0

Re: Reply All Fun...

Yes, that one had me in stitches!

3
0
Silver badge
Pint

NoRAVirus Symptoms and Advice

It causes verbal diarrhoea, turns your Inbox into a profuse Outbox, dehydration from lots of hot air. No need to consult your Group Policies, just stay at home and drink lots of fluid.------->

Any concerns dial 111.

1
0
Coat

unsubscribe

This story was presented on my browser by error.

6
0
Anonymous Coward

Re: unsubscribe

Please unsubscribe me from your unsubscribe messages AT ONCE!! No, quicker than that!

6
0
Anonymous Coward

Re: unsubscribe

I'm Sparticus!

1
0
Anonymous Coward

Re: unsubscribe

To unsubscribe from this service you must first purchase a Craft-O-Matic Adjustable Subscription Cancellation Unit.

The unit can be obtained from most hardware stores and dental clinics.

Be sure to obtain the proper permits to operate the unit from the Nuclear Regulatory Commission and the Food and Drug Administration in Washington D.C. USA.

Be sure to carefully unpack the kit and place each component in its accompanying mesh safety bag.

Mount the Pershing DF4 mesinator on top of the perforated Gerring Mach 77 refibulator and attach them using the eight-millimeter torque fork.

Be sure that the refibulator is mounted at a 66 degree angle and properly dispersed so that it is flush with the curved section of the Pyrex thistle tube.

Place the four sections of the triangular separation gear into the posture cylinder and lock them into place using the band aid adhesive strip.

Insert the wiggling pin into the wobbling hole, making sure that it seated correctly.

Place the D cell battery and the eleven 9 volt batteries in the power chamber.

The device should be calibrated before operation using the optional digital corkscrew accessory pack prior to operation.

Insert the digital corkscrew through the electronic combustion service chamber using caution not to touch the reinforced tungsten igniter control module and quickly turn the inverter drive to 28.6 degrees.

Turn the Craft-O-Matic Adjustable Subscription Cancellation Unit upside down and hit the bottom plate with a 48-ounce ball-peen hammer while shaking the unit vigorously.

Force open the door to the incineration valve compartment and set the pressure gauge to 719 psi.

Close the door and seal it shut with duct tape.

The unit should now be properly calibrated and ready to use.

Before activating the Craft-O-Matic Adjustable Subscription Cancellation Unit, you must first elevate it to a height of 229 feet above sea level to insure that the unit receives the proper oxygen level and barometric pressure.

Point the aerial to 17 degrees north by northeast to within the parameters of the Telstar GS-2 weather satellite and apply pressure to the wing shaft on the southern most section of the modular accelerator.

Using the special ratchet adapter supplied with the unit, rotate the heater core to the "on" position. The "on" position has been obtained when the green light begins to flash, signifying that the red light is about to go off.

Once the red light is off, flip the toggle switch labeled "ON/OFF" to the "ON" position and count to 47 before logging on to the system.

Logon using your username and password and wait for the prompt.

Once prompted you must check the box with the appropriate action you wish to take and then press the pressure release button and turn off the compressor while turning the hand crank at 231 meters per minute.

Next, press control, alt, delete, caps lock, shift, number lock, escape and tab simultaneously.

Press enter.

You will have one second to complete the procedure.

If you fail to respond in the time limit allowed, simply purchase a new Craft-O-Matic Adjustable Subscription Cancellation Unit and start from the beginning.

Please remember that this is the only way we will accept for you to unsubscribe from this service.

We have made every attempt to simplify the procedure for your convenience.

WARNING!

Failure to comply with the unsubscribe policy will result in immediate termination of your subscription so please follow the above directions closely.

9
0

A similar thing happened at a few companies, virgin media / ntl being one. Somebody sent some idiotic email about thinking of a colour and then an object and the company had 48 hours of people emailing everyone "red hammer", "yellow hammer", or please unsubscribe me. It was hillarious, especially the emails from the chief bin lickers assistant commanding everyone to stop.

0
0

On the plus side, at least if was some bod's test-in-prod mail that brought it down in a (hopefully) partially-expected/monitored fashion and not just some external party making a typo at some random point in the future.

0
0

It took me about 10 seconds to figure out what was going on when I saw the first email. After that it was amusing to watch everyone get more and more irate.

1
0

My phone is now off until the morning. I'm not overly convinced I'll be able to get any shut-eye with constant pinging or buzzing tonight. I'll let it catch up tomorrow.

ps - I'm still getting 09:38 emails at this moment. Doesn't bode well for tomorrow.

1
0
Silver badge

Just add a mail rule with the subject until it blows over.

2
0
x 7
Silver badge

I think this was a deliberate hack. The originating e-mail address was from an individual account at Croydon CCG and somehow that got converted into a mass distribution list.

There were two problems:

1) How did the single mail get sent to "all" FROM the CCQ mail account?

2) How was that CCQ account hacked so that all replies sent to it were then forwarded to "all"?

Difficult to see how that happened without deliberate tampering

Its important to note that most replies were NOT sent to "all", but simply back to the originating e-mail address, which by then had been converted into a mailing group. I got several hundred replies (I never got the original) and most were sent to me as a result of the reply just to the originating CCQ mailbox..

This was the first stage of a spear-phishing attack. Most NHS mails have footers giving the details of the senders.........all those who replied have now had that data harvested. Expect more targeted attacks soon. We already know that there are daily attempts to hack the NHS servers.........

2
2
Anonymous Coward

1) Either R or a local admin created a dynamic distribution list - I have no clue how the query was formed and I'm not about to test it but it included a lot, if not all, of the users on NHSmail

2) I don't believe it was, all of the replies I received were sent to the distribution list - if you still have the mail I'd check again, I don't think there is anything more sinister than a user error enabled by a poorly designed feature.

2
0
Silver badge

"Difficult to see how that happened without deliberate tampering"

Never attribute to malice that which can be attributed to stupidity.

3
0

Reg exclusive?

"At 1545 GMT, NHS sources were telling The Register that emails with 0950 timestamps were only just beginning to arrive in their mailboxes."

What, special sources were telling you privately; or you were just reading it off twitter like the rest of us?

3
0

reminds me of the time

when working at a largish Engineering company (with offices in NZ, Aus, UK and Canada) and student in the head office read a (hoax) news item about a virus.

So he painstakingly went through and sent an email to all staff (all 1000+ of them, he didn't use the mailing lists) warning them of said virus. After the 3rd or 4th reply all to tell him it was a hoax the mail servers crapped themselves and our mail went bye bye for the rest of the day.

0
0
Bronze badge

your medical records are perfectly safe with this shower.

3
1
Silver badge

"your medical records are perfectly safe with this shower."

Really? Because my experience with NHS Direct is that their IT people have trouble in the shower finding their derriere with both hands. Oh hang on... this is that irony thing I don't understand, isn't it?

<FX: wavy lines>

This takes me back to a conference a few years ago when an NHS Direct bod gave a speech about their "great success" in ensuring that that hundreds of thousands of iToys owned by medical staff were all permitted to access the network within a few hours of the latest Apple product launch. Most of the non-NHS IT bods sat staring and asking if they had thought through he implications of a massive BYOD with access to sensitive personal data. The answers seemed more than a little complacent.

1
0
Meh

State health service blames Accenture

I can understand that. It's a fair default position to take and saves the bother of looking for the actual facts.

5
0

Disapointed

Really disapointed that The Reg hasn't caught up with the times, they only have a 'reply' button on their forums, please can we have a 'reply all' button?

9
0

Not just the only stupid email thing

...like my former boss, the CEO and a peer of the realm. He *always* spelled his email address saying: "lower case" for his name and "upper case" when he got to the name of the Company....

He also made us do the same on our business cards....

2
0
Silver badge

Re: Not just the only stupid email thing

"...like my former boss, the CEO and a peer of the realm. He *always* spelled his email address saying: "lower case" for his name and "upper case" when he got to the name of the Company...."

I take it he was a PHB?

0
0

Poorly designed application

One of the many causes of this particular problem is using a badly designed email application, which attempts to shoe-horn too much functionality into single buttons, and also mixes different classes of email address.

By having a distribution list look exactly like a normal email address, there is no feedback to the user that sending a message to that apparently single address could, in fact, have bad consequences. I've done it myself: not noticed the distribution list hiding in a long list of recipients in an email. It makes sense that distribution lists are flagged, and handled separately by the user interface.

For example, if the recipient list contains one or several distribution lists, a pop-up should appear after you have pressed the send button saying "This email will be sent to a distribution list containing <n> members: are you sure you wish to send this email?". If the email is to be sent to more than 100 recipients, then a second pop up after the first should appear saying "Are you really sure?", and if more than 1000, another saying "Are you really, really sure?" - in fact the number of pop-ups should vary as the integer amount of log base 10 of the number of recipients. So sending an email to 800,000 recipients requires navigating through 5 pop-ups. Each pop-up should also not be a Yes/No/Cancel, but require the user to type in the number of recipients they have just been told: just to drive the point home.

Secondly, Reply All should not have the same behaviour as Reply. Obviously, Reply is to the single sender (which, if it is a distribution list, should trigger the distribution list post-send pop-up), but Reply All should pop-up a dialogue box with radio buttons which explicitly force you to select replying to all recipients that were individual email addresses, or not; all recipients that are distribution lists, or not, and the original sender, or not. in this way the work-flow between hitting Reply and Reply All is different (which is good), and it also allows you to choose sensible options on the Reply All (I have often needed to edit out the original sender from Reply All messages when I'm commenting within the team on a message a team manager has sent; or indeed on a message sent to the team by an outside client - you often don't want the outside client to be included in the inter-team discussion. If anything, an default that doesn't reply-all to any people outside an organisation, whether senders or recipients, can be helpful.)

The semantics of email addresses are not well defined: there is no standard for when an email address is just an email address or when it is a distribution list - it is actually an attribute of the receiving mailbox. This is flexible, and has some security advantages, but it does make it difficult for email clients to exhibit the behaviour I have described above. In a closed system, you can define special formats of email addresses so that a client can parse them and take action, but in general RFC-compliant land, this is not possible. So this behaviour is not going to change quickly, if at all.

And note. At no point have I blamed the user. Or the administrators. Well written software helps people do their jobs, and helps to prevent them making mistakes. If people consistently make the same mistake in their interactions, the process needs to be redesigned to help avoid this from happening. It's how the aviation industry works (imperfectly). If you don't take human factors into account, you WILL be bitten by them. Help people to improve and they will tend to. Tell them they are idiots, then you will breed fools.

2
1
Silver badge

Re: Poorly designed application

So, in order to solve an occasional problem, you want to destroy the ease and speed with which email can be used, and break the workflow with multiple pop-ups, radio buttons, text input...

Maybe running a trial or two (ideally outside the basement) will rapidly show you why this is a stupid idea.

A warning stating that the email you are about to send will go to X users (where X > a predefined number, probably about 100) would be useful, and is already widely implemented (eg where I currently work). As is a limit on who can email large lists.

Everything else you suggest is just... ridiculous and nonsense. It's not possible to tell if an address is for a single person or a list. I could set up my gmail address to forward to 10,000 people - how will your lotus notes client know that, when you send an email to an exchange server across the road?

3
0
Anonymous Coward

Re: Poorly designed application

Well Exchange does have a thing called MailTips that will usually pop up with the kind of message you are looking for but might have been turned off by policy.

Certainly agree that Reply All is a scourge.

Project Manager to Team: Can I have a note of the days off you are intending to take at Christmas please

All team members except me: Reply All - I'm taking xyz.

I DON'T CARE!!!!!

2
0
Silver badge

Re: Poorly designed application

Project Manager to Team: Can I have a note of the days off you are intending to take at Christmas please

All team members except me: Reply All - I'm taking xyz.

I DON'T CARE!!!!!

How else will you know what days you can harvest the good stationery, less stained chairs and other good stuff?

3
0

Re: Poorly designed application

AndyS

Thank-you for making it clear that I committed the cardinal sin of over-estimating the intelligence of my readership. I will try to write more simply and clearly in future.

1) Standard work-flow of replying to a single sender is unaffected.

2) Work-flow of reply-all to fewer than 10 individual recipients (not mailing lists) is changed in that you get a pop-up asking you if you want to include or exclude the original sender in your reply-all.

3) Work-flow of reply-all that includes distribution lists in the recipient lists gets a pop-up with a toggle asking you to choose if you want the mailing lists to be included in the reply-all.

4) Work-flow of reply-all that includes recipient outside your organisation gets a pop-up with a toggle asking you to choose if you want external recipients to be included in the reply-all.

5) Work flow of a reply-all that has more than 10 recipients gets a pop-up asking you to confirm by typing in (echoing) the number of recipients the email client has calculated it will go to. More pop-ups may appear according to the order of magnitude of the calculated number of recipients.

Yes, I am aware this doesn't work across email boundaries as there is no standard for recognising email distribution list addresses, or knowing how many addresses such a list will go to. Within individual domains, it is entirely possible to do this kind of thing without worrying about inter-organisation standards.

Unless the majority of your emails are of the reply-all type, your standard work-flow is unaffected. Reply-all type emails have further, arguably sensible, checks to help you to not send messages inadvertently to your manager; an external client; or a large population triggering a reply-all storm.

Sensibly written software would have a way of turning off the checks for those people who reasonably don't need them. You might be one of them.

0
6
Coffee/keyboard

Re: Poorly designed application

Being able to write simply and clearly shows one possesses uncommon sense.

Do not reinvent the wheel until you have made educated and effective use of the ones already available.

1
0

Re: Poorly designed application

True.

I find writing simply and clearly difficult. My sense is by no means uncommon.

As for reinventing the wheel: point taken.

It would help if people criticising posts actually read and understood the post they are criticising, rather than criticising what they think they read.

Both the Exchange Server and the Outlook client have a lot of knobs to twiddle, which is good. On the other hand, I don't think anyone would say they are unimprovable. Microsoft claim to put a great deal of effort into making their end-user software easy to use, but their efforts don't seem to be particularly effective in this area. The end result is that users can deploy a foot-gun with depressing ease.

0
0
WTF?

WHY does an "ALL NHS STAFF" list even exist?

I worked in the NHS for years (SWMBO still does) and the only message which I can imagine you'd want to send to all NHS staff is "The NHS has been shut. Don't come to work tomorrow." I worked in histopathology. There's not much everyone in my lab needed to know that all HR also needed to know or psychiatry or community midwifery.

1
0
Silver badge

Re: WHY does an "ALL NHS STAFF" list even exist?

"There's not much everyone in my lab needed to know that all HR also needed to know"

"Someone let picrate dry out in the tissue processor, again. Their remains are now available for collection. May contain traces of glass."

0
0
Silver badge

Re: WHY does an "ALL NHS STAFF" list even exist?

"There's not much everyone in my lab needed to know that all HR also needed to know or psychiatry or community midwifery."

If you were in HR you'd be convinced that whatever brain-farts you were dropping would be essential reading for everyone.

5
0
Silver badge

Re: WHY does an "ALL NHS STAFF" list even exist?

A National Emergency would be one reason, I guess. Oh dear, looks like it would fail that test.

0
0

'kin users!!

Either way, if the users hit "reply" and replied to a large DL or hit "reply all" - the bloody users need to be educated, not just in the correct use of "reply" and "reply all" but also on how to identify a message that does not require any response at all!!! Users are too eager in responding to these type of mistakes; "Take me off this list now!!", "Why am I receiving this????", "Do you know you just interrupted me drinking my fifth designer cup of coffee this morning??? - take me off this list at once you cants!"

Simply ignoring the message and then deleting it not only shows you have a brain cell but also prevents your organisations dirty laundry being broadcast all over the facking internet!!!

3
0
Anonymous Coward

NHS Connecting for Health -> HSCIC -> NHS Digital.

Same people, same office, same problems - only the date stamp is different

0
0
Bronze badge
Unhappy

"reply all" the basic CYA

In many organisations, if you don't "reply to all" on anything important, your mail will be ignored - especially if it involves the recipient putting themselves in danger of being responsible for something.

Without "reply all" you have no email trail in the boss' inbox to prove that the other numpty dropped the ball, not you.

1
0

How does this sort of thing happen?

We don't use Exchange, so possibly I'm missing some exchangy concepts here but as I understand it someone foolishly sent out a mail to a ton of people on a list that probably shouldn't have existed in the first place.

Some of those people replied to that email - which, instead of being immediately sent back with an auto reply along the lines of "This address belongs to a mailing list daemon. If you wish to send email to this list, please email $LISTSENDADDRESS" was taken as an instruction to the mailer daemon to send that message to everyone on that particular list - which already feels like a pretty massive configuration cockup right there, but ok, that happened.

Even assuming that you made the mistake of having the reply-to being the send-out address, why then did every single one of these emails not get the next line of "Your message to this list requires approval. Please wait for this message to be approved by a list administrator" which should always be the case for lists that can hit tens of thousands of people for precisely this reason, and then sit in the mailer's approval queue?

Then - even assuming that THAT was allowed to happen for some reason, why doesn't the list daemon go "Holy shit, my queue is suddenly full of tens of thousands of messages, that's never happened before. I'd better rate limit those bastards and email my owner to warn them that something weird is happening." at which point it drip feeds messages out a few hundred at a time until someone comes along and tells it that it's OK, no one's account has been compromised and you're not being co-opted into some massive bulk spamming campaign, we really did mean to email the entire organisation.

Feels like a lot of config level school-boy errors had to be made to allow this to happen in the first place.

0
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing