from what ive seen its just outlook web access . these "Accenture" people must've built an exchange server and charged , ooh who knows ... 50 million?
..and then failed to config it to "limit the impact one email " etc
The NHS reply-all email fail last year involved 500 million emails being sent across the health service's network in just 75 minutes. A test message sent on 14 November to what an unfortunate "senior associate ICT delivery facilitator" thought was a local distribution list she had created instead went to all 850,000 people …
Perhaps it was intended? FTA: "The report revealed that half a billion emails crossed the NHS network between 0829 and 0945 that day, against the usual traffic volume of three to five million emails per day. It also claimed that the service "did not crash at any point", though it confessed to "significant service delays for the majority of the day"."
Underpaid tech: <Coughs> Sir, the stress test on the new email system is complete.
PHB: Did it pass?
UT: It took more than 100x normal traffic and did not crash at any point.
PHB: Great. Sign it off so we can send the invoice to HMG and get paid.
Back in the days of dial up, I once had an issue where a client sent me a 72Mb attachment, I phoned the isp and they managed to kill off the offending message after some discussion.
After this I then phoned the client to advise that I had not received her message, whilst on the phone, just before the conversation got to the bit about file size and dial up, she hit re-send and I had to repeat step 1 with the isp. FEK!
Years ago, had a contractor (soon to be ex-contractor) who fell for the "we'll give you one share of stock for every two people you send this email to", and sent the email to everybody in the company, individually, not to any "all" group...he selected each person by name, apparently to ensure he got "credit" for each one. My sites (I ran systems for six divisions) happened to be still running Microsoft Mail. The servers were all fine, but the email crashed the client software. This was the days before Windows 95, when you still had to contend with the limited "conventional memory", and MS Mail client had a smaller memory footprint than Outlook. As LAN admin, I had full Outlook, so I had the "pleasure" of cracking every mailbox one by one and removing the offending email so the users' client software would work again. I printed one copy of the email...the "to" line filled five full pages of printout, WITHOUT the headers.
Back in the 90s I worked for a firm where the windows/email admin had made himself SERIOUSLY unpopular through a complete lack of social niceties and utter inflexibility plus dobbing people in to management at what he perceived as the slightest infringement of company email policy.
Anyway, his chickens came home to roost when an email macro virus showed up. It slowly started making its way around the company followed by a worried sounding message from Mr Admin saying please do not forward these emails. And in a stroke he signed his own demise - for that day and half the night anyway. Naturally we forwarded it around the entire company as many times as we could. By the time we got bored the email server was on its knees and so was he. Childish? Probably (we were all in our early to mid 20s), but this guy really had it coming and no sympathy came his way from anyone , including management who considered his warning email a red rag to a bull considering how he was disliked.
Speaking of email size.. ..in 1997 I received my first big spam. It was from a fool in Earthlink who put recipients on the CC: line rather than the BCC: line. It was fecking huge. I had a 56k modem by that time, but it took a long time to DL. More than an hour. I later found a POP3 scanner after that and took to deleting mail first. These were the days when Demon insisted users put their REAL email address in Usenet From: lines. Unbefeckinlievable.
You haven't lived until you have screwed up an M4 macro and set up a sendmail.cf which creates a mail loop without counter.
Hell hath no fury but two misconfigured MTAs firing messages at each other - as a newbie email admin I was glad I could just rip the ethernet cord out of the back. At the 10Mb speeds we had in those days, Linux would run out of disk space FAR quicker than it would run out of resources, and with a mail loop that doesn't take long..
Anyway, those were the lessons we learned before we were allowed to go near production :).
"strict controls must be in place to limit the volume of any one email sent by an individual user or local administrator".
One of these requirements that looks so easy on paper.
"A requirement to put in place compensating controls to limit the impact of fucking idiots who respond with Reply-All to a test message" would be good too.
Where's the tick box in Exchange for that then?
Those "fucking idiots" are too busy saving your life pal to worry about clicking the wrong button on an email.
Gimme a break, they were probably all admin staff.
Oh and FYI, I used to be one of those that does the life saving at the sharp end, and I still know not to click reply-all to a group email.
So every single person who has an NHS email is a life-saving medical professional? There are no clerks, payroll people, people who handle buying more tongue depressors, nobody who...well, you get the idea. For every doctor and nurse there are probably 5 support staff who barely know one end of a stethoscope from the other, and certainly don't spend all day "saving your life".
I seem to recall when we moved from CCMail to MS Exchange many moons ago, something similar happening to our 2,000 users. We deliberately changed the names of the big mail groups so they're slightly trickier to accidentally select and by default there is a default a catchall email group for for anyone clicking on the "To:" button and hitting Send without thinking!
Are you serious?
In Exchange, you can lock down a list to only allow specified senders. It takes about 5 sec, via the GUI or Powershell. We generally control these via another group -e.g. $list-senders are the only ones allowed to send to specified list. Even if one of the permitted senders screws up, the Reply-Alls don't go far.
If you have any smarts, and you're actually allowing end-users to set up email lists, you'd run some kind of script on a schedule to check for email-enabled groups with (recursive!) members > $number and verify that all of those have sender restrictions on them.
For the NHS, the fact the storm went on for that long is appalling - it should have taken approx. 2 mins to lock the list (assuming someone had to logon to a box to set the restriction). Give it 15 mins for someone to verbally raise the alarm... (although, again, if end-users can set up the lists, you'd expect some pretty gnarly monitoring to be in place to actually raise an alert itself, even just seeing if the queues are filling up.)
The problem was caused by people hitting "Reply All" and not by Accenture.The system could cope with many emails sent to everyone. It was those who felt they needed to tell 850K people that they didn't think they should have received that email who caused the problem. Accenture's share of the blame may be the same as the manager - 850K/500Million or 0.17%
If it was part of the brief to prevent mails being sent to large numbers of people, and they failed to deliver the brief, then that would suggest a cock-up on their part, no? I know reply-all is an enraging habbit, but usually it's a rare mistake - just not rare enough in 500 million rolls of the dice.
The issue here was that it was not obvious what had happened and definitely not obvious that it had gone to the whole NHS. All you saw when you received it was a handful of addresses, maybe 7 or 8. So if you did Reply All to that group you would not have expected the maelstrom that followed!
It was only after a couple of hours that the true extent of what had happened became apparent, by which time it was much too late.
The problem was caused by people hitting "Reply All"
And would that happen to be the default choice by any chance?
As an aside, I have seen email groups where reply address is set to be the list, so even if you hit "replay" and not "replay list"/"reply all" you still end up spamming everyone and you have to manually copy/paste the sender's email address if you simply want to reply to them.
The problem was caused by people hitting "Reply All"
Nope, the problem was that people replied to the single email address of the Dynamic Distribution List, which was supposed to be configured to only include a few people but in fact included everyone. Most certainly a problem caused by the system, not the users.
Umm, the article clearly says that the system admin sent the message *to* the problem distribution list which means the sender would have been the system admin and not the problem distribution list. In every implementation I've seen, a "reply" only goes to the sender of the message and it takes a "reply to all" to go to the entire distribution list.
So it seems to me that the issue *was* people doing "reply to all", a problem which is as old as email.
To use an old joke, "We try to make our systems idiot proof but they just keep making better idiots"
all the implementations I have seen.
(That would be reply to list, which would require the client to know it was a list, that would be needed. I'm fortunate not to have to know whether that is implemented in Exchange/Outlook/Web, but it seems to me it would require cleverness to build that in.)
I remember in my early admin days that one user ALWAYS bombed the e-mail server at the private school I worked at. Guess who? The PR lady for the school who was also a student there (so we had to tread carefully when giving a telling off). Every single end-of-term, they'd be a huge circle e-mail of attached raw images (straight from the DSLR) with a copy of the newsletter text to the "All Staff" group (roughly 200 at the time). Then all the OOF replies and externals who always copied back in the image attachments on reply. This was in the age of battling the 16GB limit of the Exchange 5.5 Standard server.
My boss always fumed up, rung me (or visa-versa) and we both go storm the office where the PR lady worked as soon as it happened. Never learn't until we got our new shiny Exchange 2007 server.
Anyway, in light of this piece of the story... "The local admin selected the "only in my organisation" rule, which she thought would restrict the distribution list to her South London clinical commissioning group.
"A software configuration error meant that the system applied an 'All England' rule rather than one including only the administrator's organisation," continued the report on the snafu. "The administrator would not have known that this had occurred."
What happened to competent sys-admins that would test the result first before acknowledging the task is complete to the user? Shouldn't have to put blame up the chain to a more senior admin if they messed up the config. Those things should be caught and dealt with to stop this stuff from happening. Shame NHS won't move on from e-mail either and use chat-based apps or something similar to conduct communications.
Die e-mail DIE! It's a curse to all in IT.
"You'll get me replacing email with chat applications when my manager agrees that I don't have to do any work.
Making it easier to interrupt me at _my_ work to help out with _their_ work is not something I intend doing."
You'll get me to replace e-mail with chat when flying pigs land on the frozen plains of Hell. Chat is a text version of telephoning; e-mail has many advantages over both, such as being able to compose my response and not having to reply immediately.
"Shame NHS won't move on from e-mail either and use chat-based apps"
Hmm. Is there one that works on all sorts of equipment, is easy to use, reliable, and doesn't store loads of shit off-site (and quite likely off country where it can be "analysed for marketing purposes")?
What happened to competent sys-admins that would test the result first before acknowledging the task is complete to the user
From the original article:
The administrator would not have known that this had occurred.
i.e. The administrator was testing what they had setup before handing it off to the users. But the system was so broken, not even the local administrator knew what was going to happen.
Both Exchange and E-Mail are equal problems. Need a lot of expertise to setup in big organisations and keep on top of. Of course processes stop messy setups, but is there any need to spend big bucks on huge e-mail server farms anymore?
E-mail for all its compatibility is awful for inter-company productivity. I'd personally not like to have to deal with them in my line of work. Apart from a few good clients, no-one has written any intelligence around them or organised e-mails better. Outlook is a pile of poop too for it's very small set of good features. Rather not sit behind e-mail either (unlike some who hint at it) as an excuse for not moving my butt to get something done because I've genuinely let the email item rot with a lack of system to come back to it. With better chat/helpdesk/organisation tools out there now, why rely on e-mail?
@heyrick - Zulip, OneTeam, Rocket.Chat - there's plenty of opensource ones for self-hosting that doesn't need reliance on Atlassian/Slack. Although both are very good products.
I personally don't get the dogged determination to defend e-mail. There are ways of stopping this sort of issue happening (although I appreciate get mistakes get made - I've made them myself in the early days).
As well as an open protocol and storage standard, e-mail is better than chat in that more thought has probably been put into the original e-mail and you're not expected to reply right at the moment you receive it. IM just derails your thought train and you're expected to drop what you're doing to answer (which could take half-an-hour of back and forth).
Depends how you use the chat utility. There's no requirement to respond straight away (you can log-off/go to an away status). I love the fluidity of a chat/ticket-system utility over e-mail. For slow inter-3rd-party comms, it's fine. Inside a business? Not so sure now. This is where a business/org like the NHS, there much better ways to distribute global comms and segregate team communication.
I don't care about the thumbs down guys. It's almost a survey of how many of the old-school like e-mail which is fine. As a comms tool for a business, it's abused much so by lack of training on how to use it + it's cumbersome/clunky. Seen it, managed it, done it.
Gutted there aren't many or any who take an opposition to e-mail.
If there are issues with your implementation, fix your implementation. Don't go all ALSA on it.
There's a reason I refused to take phone calls, like IM, it's a jump-to-the-front-of-the-queue-i'm-more-important-than-what-you're-doing-now system
some wellwisher recently sent me a mail message, with a small xml file pretending to be a "meeting.ics", this small file contained a list of 19k+ addresses, all in rsvp-mode
24 Apr 2015 17:26:01 +0200\n
Subject: Adobe Connect - Meeting Invitation to "Meeting Room L2"\n
. . .
removed much stuff that is evil
. . . 19 thousand respondents
TRIGGER:-PT15M\n. . .etc
good to see that it might have only disrupted services for a few hours, I post this info here as I've already described the event widely & openly and I've left the field of crypto/internet-security/balance-of-privacy-vs-security/ for something more peaceful!
As a former Exchange admin I can tell you that mistakes sometimes happen. So I would separate distaste for Accenture from the observation that there but for the grace of god go I. A mis-config is a mis-config, it could just as well have been a screwup on qmail and dovecot as on Exchange.
The real solution is to educate users that when they get a message that is not for them, they either delete it or they reply directly. That rule is drummed into our employees here for good reason. (Second rule: if you want someone to do something, they go in the To: field and not the Cc one. Helps immensely with inbox filtering.)
"The real solution is to educate users that when they get a message that is not for them, they either delete it or they reply directly."
It was pointed out in an earlier comment that the users only saw a single email address to which they replied directly. It wasn't their fault, however well educated, that that one address was expanded by the system into a list of everybody.
And I have to admit a system that can cope with 100x normal load is pretty impressive (although was that a design requirement or is that to allow for expected growth?)
And 850K email addresses. How many organizations come anywhere near that?
TBF I wonder how many orgs have a structure as convoluted (and changing) as the NHS? There must be dozens of admin scripts set up to handle various tasks on this system
This is like a web site hosting company having a site whose traffic jumps 4x in a minute and does not fall over*
*Unfortunately IRL the company I'm thinking of did fall over.
Don't think that is quite correct. At least not according to https://blogs.technet.microsoft.com/exchange/2008/01/02/a-brief-history-of-time-exchange-server-way/
I can tell you that they outsourced some of the X.400 work to third parties, my previous employer being one of them. Back in the day it was considered a reasonable probability that OSI standards like X.500 directory and X.400 messaging would become the global standard. There was even a special version of Exchange for the US DoD, and those guys *loved* X.400/X.500.
I've been reading the Gruaniad and they make it totally clear that this wasn't Accenture's fault at all but the result of those wicked Tories and their completely unjustified funding cuts. You just can't expect the NHS to cope when its funding has been cut from £116 billion in 2015 to just £119 billion in 2017.
This post has been deleted by its author
Our email is provided by a private sector company on a national basis with interesting ideas about reporting spam. If a spam email arrives in your inbox they would like to educate their spam filter, so a user should:
drag the email to the desktop
right click on it and send to a compressed folder
create a new email
attach the zipped spam attachment to your email
address it to the spam reporting mailbox
There is nothing too onerous there. They let themselves down though as they then send you an email to thank you for reporting the spam mail and explaining the procedure to report spam mail, something you have just successfully done. They send you one of these responses every time that you report a spam message.
Remonstrating with them to remove the auto-reply gets you nowhere, because they obviously know better, but they do offer some advice, set up a rule to put their email in the deleted items unread.
Needless to say I have gone from diligently reporting spam emails to just deleting them. I just hope that we are getting the email providers service on the cheap, but working for a government organisation I am only too aware that we will be paying too much for too little.
I would speak to them, but I am not sure if I would understand their accent (ure)
'one of the NHSmail system's design requirements was that "strict controls must be in place to limit the volume of any one email sent by an individual user or local administrator" .. "This functionality is still to be delivered by Accenture'' ref
NHSmail Portal User Guide: 'There is no maximum number of users that can be included in a dynamic distribution list. All distribution lists you create will need to be approved by your Local Administrator. You will receive an email notification as soon as the distribution list has been approved and is live'
Biting the hand that feeds IT © 1998–2021