Cisco email accidentally sent to 1000s of employees causes message list MAYHEM Ever become stuck in a seemingly infinite loop of emails from colleagues hitting the "reply-all" option and asking to be removed from a mailing list containing the addresses of thousands of employees? If the answer to that question is a resounding yes followed by a massive sigh, then you ought to have some sympathy for Cisco, …
Ah, memories.
Working in a london council IT dept in a minor install/support role - somewhere in the region of 5000 users and email accounts. Someone managed to send an email to all staff, and the usual shenanigans as described in the article kicked off. We thought it was funny till the sysadmins pointed out how much bandwidth and storage it was eating up.
We talked to the top man in the support dept, who talked to the CEO/top man in the council. A plan was formed.
All staff email - anyone who continues to respond to the erroneous reply to all email will be terminated for gross misconduct and misuse of company network, or words to that effect that made it sound like a real threat (there was some nuance and explanation of how this was perfectly legal, the bandwidth is affecting the several dozen remote offices on WAN links, no-one can get any work done, you have been warned, etc), signed, the CEO under advisement of IT Operations.
Twenty minutes later, another email - three people have been summarily dismissed for wilful gross misconduct and have been escorted off their respective premises by security.
And about thirty seconds later, all the network monitoring equipment stopped flashing red and making scary flashing lights.
Of course, no-one had been sacked - although some had been noted for written warnings (they had been sending 'yeah right' emails - not big, not clever, probably didn't do their promotion prospects any good). But it gave the network admins time to clear up the mail servers and dump all the crap out of the queues and mailboxes.
I don't believe it has recurred since to such a large degree since (for a start, there is no longer an "easy to acci-click" all staff distribution group - which is how it apparently happened in the first place, this was at the height of the terrorism times just after 7/7 so local govts all had status updates on terrorism alert statuses etc) - and old hats are wary of checking recipients before hitting 'reply to all'.
I work in a far, far smaller operation these days - maybe two dozen users. I can live with reply to all abuse there because I'm the sysadmin, and as such I'm the one who gets to belittle people for doing it ;-)
Anon for a few reasons...
This not only happened a few years ago at HP, it still happens with monotonous regularity to this day. It is unbelieveable the number of numbskulls in the company that still click on "Reply to All" to be unsubscribed from a distribution list - and they are supposed to be IT "professionals".
Not only does this still happen with monotonous regularity within HP, there is also the office sweepstake to see who can complete the 'mandatory training with questions so stupid you would have to have a common sense bypass to fail''. I think the record for the 30 mins on Health and Safety was around 3.
Many years back whilst working for another company we arrived in the office one Monday morning to find the mail servers on the floor and in need of life-saving treatment having become "full". Hours later when normal service had been restored and the full post-mortem began we discovered what had happened.
At the time we had no remote access into email so one Thursday a contractor, prior to going home for a three day weekend, set up an auto-forward rule in his mailbox to forward any incoming mail to his Hotmail account and an account with another ssupplier (can't remember who). This despite a total ban on auto-forwards to outside the company.
Everything was fine throughout Friday; emails arrived and were forwarded to his two private accounts. Unfortunately at some point on Saturday morning his Hotmail inbox became full and so the next time it received a message from his work account it replied saying that it was undeliverable because the mailbox was full.
So our system duly forwarded that to his Hotmail account...
Which replied that it was full...
Repeat this loop until some time later his other account became full...
...and started sending replies that it couldn't accept incoming messages because it was full. Which our server dutifully forwarded to BOTH accounts which BOTH replied that they were full...
And some time on Sunday our mail servers gave up because THEY were full.
Oh how we laughed! Or maybe we didn't. The contractor had his contract terminated that day.
"The guy was fired for that? "
He used his privileges to set up a facility which was specifically banned by the company he worked for. That is very dodgy ground even on the limited information given in the original post. There may be additional circumstances such as lost business due to lack of the email service. Round my way such an arrangement would result in many, many breaches of the data protection act. All of which would probably justify a short, sharp trip across the car park.
Your maths is also bad...
94Kb * 4000000 = 47GB
I assume you meant 94KB average per email, which would be about right.
But note that the 375GB was 'network traffic', which presumably includes the transmission and separate reception stages (e.g. SMTP & POP) and headers/overhead, so each email itself would need to be somewhere around 40KB for that.
... but maybe the email store was on a SAN, in which case that network traffic might also be counted... I'm going to stop here.
@agentgonzo: Maths was correct, holding down of shift was incorrect... I meant 94KB (kilobytes) rather than Kb (kilobits).
Probably you actually meant 94kB!
If I remember correctly the kilo (k) is the only SI prefix for powers of ten great than one (apart from the never-encountered hecto-h) which is in lower case (compare mega-M, Giga-G, tera-T).
Another instance of where a standard is flawed...
</pedant>
nag staff about completing an online training module by the end of the month.
One of the curses of working for a big corporate. I once had to complete a course four times because of various system glitches.
Someone replied to a distribution list that they really shouldn't have been able to send to, something to do with "playing pictionary with the helpdesk team" (it was an article in the newsletter they were replying to).
Cue a storm of reply alls asking to be removed alongside other trying to be helpful. Eventually the DL got disabled.
It made it up to Salford where I then saw pictures on Facebook with printouts on the wall.
I still have the e-mails, I still browse through them for amusement.
The first time was the "Me too" incident in, what, 1998 or 1999? Somebody was testing distribution lists on Exchange, and had simply grabbed a bunch of names and slapped it on the list just to create a really huge list. Someone else, whose name was on that list, sent out an email to the whole list: "Who owns this list, what is it for, and why am I on it?" Someone else replied, "I want to know, too." Then someone else replied, "Me, too!" Followed up with, "Me, three!"
Many inane replies later, and the mail servers had ground to a stop, and took hours to clear. They made up t-shirts to commemorate the event.
I've seen something similar happen a few times since then. Somebody accidentally uses the wrong distribution list, or else sends out an important bulletin, and then some idiot always replies with, "Could you please remove me from this list?" and on it goes.
I remember this happening once. Someone sent one of those chain emails about a missing child in the US. To the entire global address list. The person who replied in a grumpy and abusive manner, of course, hit reply to all... There were many emails in my inbox over a space of about 15 minutes, in which the perpetrator attempted to retract their offensiveding message, their boss then apologising, HR then informed all that the aforementioned person had now 'left the company'. Oh deary, deary me.
I do remember some idiot doing an email to about 40 people, with delivery and read receipts and an attachment to fill out - as these people were notorious for not providing the information. Of course the idiot forgot to include the attachment, and so within five minutes 40 delivery receipts, 10 out of office messages, 10 read receipts and 5 replies appeared in my his inbox, and the thing had to be sent out again with apologies, and 40 more delivery receipts, 10 read receipts, 5 answers... And a partridge in a pear tree.
Ahhhh, you can tell the level of job unsatisfaction at a company, by how readily the proles will take an opportunity like this to enjoy just a few moments of sadistic levity by 'accidentally' or 'helpfully' reply-alling too, thus adding to the carnage and giving them temporary respite from their numbing jobs. Being nagged to complete a (very likely) health and safety, or information handling, online training module? Oh what joy...
He was a very sleazy towards a friend of mine. There was also a bug with the Lotus Mail client which wouldn't lock if you used the Super+M combination, so when we left his desk we sent an email to everyone in Dublin, London, Toronto & Tokyo which requested a receipt (which wasn't optional). His inbox didn't last two minutes!
Ah, good days.
These email storms happen at least once per anum at my workplace (a large recognizable company with a name related to a large South American river). Half the fun is reading all of the responses egging everyone on - like needing to put "unsubscribe" in quotes or other even more silly, extra complex email procedures that have the effect of generating even more fun - especially if the thread forks with a new subject line. Experiencing the joy of others' trolling is secondary to the joy found in the anger of those who don't have the mental capacity to properly use the "rules" feature on their email client to sort out or delete the threads.
And there you have it.
The instant I see a company email come in with over 100+ recipients in CC instead of Bcc, I'm on the phone to the original sender.
Needless to say, this doesn't make my boss happy. (proving yet again that being "proactive" and "taking ownership" usually means you will get fired)
Some companies have LISTS which is just a name and nobody even really knows who is on it, except maybe the list manager, who may or may not be the person listed as such if you can find the subscribe/unsubscribe file. Then other lists are added to lists and you end up with 3 or 4 copies of everything, but you can't unsubscribe from the lower order ones because you DO want the mail that should come on that normally. The last storm we had was small, only about 5000 on the list.
worst one I came across was between 2 mail servers within the same company which had no attachment size restrictions in place.
everything worked perfectly (for years) until an idiot decided to send a 30gb file as an attachment before leaving on a friday evening.
the initial mail server accepted it and started sending it onto the 2nd server.
the second server started to receive the email with no problems and everything appeared fine... for 5 minutes.
5 minutes after the initial sending from one server to the other had started the first mail server went back over its outbound queue and found this 30gb email that hadn't been delivered (because it was still transferring) and resent it.
5 minutes later it found 2 undelivered emails and started resending them
<5 minutes per step>
1 email (30gb)
2 emails (60gb)
4 emails (120gb)
8 emails (240gb)
16 emails (480gb)
32 emails (960gb)
64 emails (1920gb)
128 emails (3840gb)
256 emails (7680gb)
512 emails (15360gb)
1024 emails (30720gb)
2048 emails (61440gb)
4096 emails (122880gb)
<end of 1st hour>
By saturday morning the second mail server had died completely having used up 4tb of previously empty storage while the 1st email server was unable to display how many emails were in its oubound queue.
The sending mail server was exchange (I don't know the version) the receiving server was Mailenable (used internally by that dept.)
the 4tb was 3 2tb drives in a raid5 array.
As Tom pointed out the user had submitted the email to the 1st server and it had submitted fully, and it was the 1st server that was doing the stupid re-send even though the transfer was already in progress.
I bet thats Lotus Domino Server sending between two different Notes Networks!
Ten years ago my work place also used rather silly schedules to send mail from server to server, one server had to have 5 emails in the queue before it triggered the notes agent to send mail to the relevant servers. This meant the last person to send an email from that office was very likely not to have it sent until the next week.
Reply-all is fine, but it is proved time and time again that it's used inappropriately.
I think all email clients should require that when you click "reply-all" you have to individually say "yes" to each of the email addresses that you are responding to.
I can't think of a valid reason to reply-all to more than 20 people, so doesn't seem unworkable.
I used to work for a small division of Honeywell, a company with a couple of hundred thousand employees worldwide.
Last year somebody managed to send an email newsletter to everyone in the company, the all users group address is limited access with only very senior IT personnel allow to use it, so how this message got out is anyone's guess. One person replied to all, asking to be removed from the email list, this prompted other people to do the same, then more and more, it took the best part of a week for the IT bods to finally remove the emails clogging up all the company's email servers worldwide...
Microsoft infamously removed the de-dup feature of MS Exchange in order to improve performance. So, your "self respecting mail server" departmental solution standards are just so '90s. Also, when email is distributed over tens to hundreds of DB backends, the de-dup becomes less viable.
http://blogs.technet.com/b/exchange/archive/2010/02/22/3409361.aspx
Note: a 10,000 user distribution list is a SINGLE email address, so the various "are you sure" schemes only work if the users aren't used to just clicking "OK/Yes/Agree/Continue". If you can't count on them to not try to unsubscribe from the HR system controlled "all-users" or "department x users" email lists - you can't count on them to just click "no".
I accidentally triggered one of these many years ago. Somebody sent a CraigShergold-gram to the building list (about 5000 people) one Friday afternoon, and I thought for a moment about whether I should send a "Please don't reply, especially to the entire list, here's the explanation" reply on not. This was back in the days when most of our mail was on departmental Vaxes, and some of it was on a big Unix-like mainframe system, some running SMTP but some running UUCP, and there wasn't a snopes.com to point people to, just alt.folklore.urban on Usenet.
I decided that it was probably better to send it than not, and of course I started getting bouncegrams from people on vacation, etc. About half an hour later, a friend called, asking if I'd meant to send five copies of the mail, spaced five minutes apart. "Umm. no..." "Thought not, enjoy the rest of the afternoon cleaning things up." Apparently the mainframe in the basement had forwarded out the message to everybody, decided that something hadn't worked, and re-queued it to try again later. I went down to the basement where the building sysadmins lived, apologized, and we spent a couple of hours trying to find the problem and clean up the mess (simply stopping the mail server and clearing out its queue wasn't close to good enough.) Never did find out what was wrong, and of course my bouncegrams were starting to include "Can't deliver message; mailbox full" as various departmental Vax disks filled up. And Monday morning we started with a couple rounds of "Stop sending me this junk at work" "Get me off this mailing list" "Don't send that to the entire list, dummy, just the sender (sent to the entire list, of course)", but none of them broke the mail relay this time.
For a one-off I use the delete key. For more persistent irritating stuff like "your timesheet is now (over)due" I set up a mail forwarding rule to auto-delete. That's how I would have dealt with a storm like that described, although I might have gone back and laughed at some of the emails in the deleted items mailbox later.
During the last international mail-storm here , I watched hundreds of "unsubscribe" emails being sent to the entire list. My first thought was that many colleagues were foolishly hitting Reply-all. In fact, they were actually (still foolishly) using Reply.
Most list-servers in my humble experience simply treat a Reply as a Reply-to-list; you need to use a different admin address if you want out. List members tend to forget or lose this vital piece of info. At work, entire departments may get signed up to new lists, with little warning.
Some list-servers at least may trap any unsub requests, but it doesn't seem to be default.
Still, that's my observation as a user, not an IT admin of any sort.
I worked for a subsidiary of a certain Blue Bank and some noob sent an email to the "all Company" group (which covered every UK branch, head office, etc) asking a question (something like "does anyone know where the printer ink is"). Cue much "why is this coming to me?", "please stop emailing me" emails, which was stopped very, very quickly by an email from the CEO's PA, which simply introduced herself and stated that everyone should stop responding, the email was obviously an error and to leave it alone. It was a very clever email, it came across as a school teacher scolding children but with the added backing of "carry on and you'll be dragged in to see the CEO" but managing to leave the threat heavily implied without being threatening. kudos to her the trail stopped dead and everyone got on with their lives.
Or the person at a Scottish financial institution who somehow managed to import the entire corporate directory into their Facebook contacts and then sent everyone a friend request. The recipients were surprisingly unfriendly.
The Management subsequently insisted that the directory should be hidden from everyone.
Some management type managed to send a message to every email address in the entire company - using OrificePower - so the X.400 email address list was so big that servers keeled over all over the place attempting to deliver them all.
Once the storm was finally over, said manager decided to send out an apology - to the exact same list.
I don't know if he was ever allowed to use email again after that...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
