back to article Reply-all email lightning storm STRIKES TWICE at Cisco

Cisco is tackling another email storm today, just weeks after its servers were deluged with millions of "reply-all" messages being sent by the networking giant's staff - with many requesting they be removed from the list. The Register has heard from Cisco workers who are currently in the, er, eye of the storm, which is …

COMMENTS

This topic is closed for new posts.
  1. jason 7

    Ahh this is always funny when that happens.

    I remember it happening at my firm (about 15000 staff) loads of people going "Please stop!!!" and not getting the fact that they were not helping.

    Then the wags started going "I'm off out for coffee anyone want anything?"

    That was a fun morning.

    1. Cliff

      Even better when it includes a large attachment and has been sent to the mailing list by somebody who isn't even at the company because a friend told them it got technical support (and some twunt hadn't locked the list to accept posts from internal addresses only).

      1. Stevie

        Even better when it includes a large attachment

        I once had the good fortune to work for a large operation that connected many smaller satellite operations to its e-mail directory, and it turned out that there were suddenly two of me. The real me in NYC, and someone with the same name in Upper Bumfuck, NY.

        I ended up having to write a robot to deal with the floods of misdirected time sensitive and legally sensitive mail people would send me that was intended for the other one. I tried engaging the other guy to get the word out, but he remained willfully thick and insulting, until I changed the robot to stop forwarding the mail to him and simply bin it with a notification to the sender that they had the wrong guy. Things reached a head when I got notification that my brand new computer was scheduled for replacement. I managed to sort that out with his manager and I think I got his replacement machine delayed a month too. Hope so.

        One particular manager had a group who always felt the need to say something when she mailed out stuff, even if it was just "Me Too" (Reply to all of course). One Monday I arrived to find my mailbox had buckled under multiple copies of huge PDFs and Word docs as everyone replied to all about each others replies about how wonderful it was. I also had sixty four notifications from the admins that my e-mail was turned off because of misuse.

        So I wrote a special robot just for that manager. It recognized every member of the claque and forwarded all mail sent to me by one of them to her (which meant she got two copies for each reply), along with her original (making three copies) and an explanatory cover note ("Wrong Stevie again, Ms Manager"). Three misdirected mailings and she finally got the message (I calculate she got it a minimum of 192 times in fact).

        Matters weren't helped by shirt-thick mail admins who would not grant update permissions to people so they could fill out their address book entries in a more helpful way than the minimalist default of name, phone number and SMTP mail. One of the admins offered to change MY email address when approached to help solve the issue. When I pointed out it wouldn't solve the problem but would break the address books of everyone I had a legitimate business relationship with outside our mail directory, they grudgingly agreed to add "NYC" to my address card for me.

      2. Goat Jam

        Re: large attachment

        Reminds me of the time I was in Tokyo during the late 90's with dial-up email and some moron at US HQ who I had never heard of sent out an email to everyone in the entire company with some nice pictures of some other person who I had never heard of who had apparently just had a birthday party in an office I had never been to.

        Outlook at the time could not be told to not download that email so it sat their trying to download megabytes on a 28K dial up link.

        Yeah, thanks for that.

        1. Tom 13

          Re: large attachment

          Even with broadband that still happens. My roomie works in an office with a very limited mail space. They are expected to monitor mail, archive/download and delete for the server. Periodically secretaries for high level mucky-mucks send out "All Hands" type messages with html wallpapers, flashing GIFs, or overly large PDF fliers for upcoming celebrations. In fact just the other day her "funny for the day" comment when she got home was one such message had taken more than a fifth of her mail space that morning. Mind you, there are cases in which she is legitimately expecting largish images to be coming in so she can look at photos of stuff to determine failure modes of equipment.

  2. Anonymous Coward
    Anonymous Coward

    "I can't find my friend Spartacus in the phone book"

  3. Infidellic_
    Trollface

    Troll nostalgia

    Reminds me of something classic I did at school:

    In outlook web access three friends and I set up an e-mail rule each,

    If a received e-mail had "SPAM" in the subject then forward it to the other three and then delete it

    one person then sends one e-mail and sits back. The admin realised after a few 10's of minutes why the servers were bogged down and tried to *e-mail* us to ask us to stop. Naturally given the rules we didn't even get that e-mail for 5 hours.

    Not sure how I didn't receive any punishment. Nor when we used a similar technique to bombard a different friend with 15,000 e-mails in 10 minutes

    1. Florida1920
      Mushroom

      Re: Troll nostalgia

      A troll on alt.www.webmaster tried the email-bomb tactic on me once. He worked for an ISP and left a trail. What he didn't know was, though I'd moved away, I knew his boss from when we were younger. Last laugh = Best laugh.

    2. PGregg

      Re: Troll nostalgia

      If Outlook existed when you were at school then you are not permitted to use the word nostalgia you young whipper-snapper. :p

  4. AndyS

    $600,000?!

    Anyone care to enlighten me as to how a flood of emails add up to that much damage? I mean, that's some fairly hefty pocket change for what, a few minutes pressing delete one morning? A mild nuisance?

    Or is this one of those silly-bugger accountant sums from never-never land where you say 35,000 employees (citation needed) each had to spend (citation needed) an 30 minutes (citation needed), at an average cost to the company of $35/hour (citation needed)...

    1. rurwin

      Re: $600,000?!

      If you count the lost productivity from the entire workforce being without email for at least half a day, you'll probably get somewhere around that figure. Then if you add any on-call workers and their expenses claims that will add a bit more. It's not so much that everyone had to delete a thousand emails, but that the servers had to cope with 34 million emails within a few minutes and promptly crashed in flames.

      1. Wize

        Re: $600,000?!

        Reminds me of oil companies and their claims they have lost several million in production for being down for a few hours. That oil is still in the ground, its not been lost. You just have a small team sitting about idle for a few hours.

        1. SteveK

          Re: $600,000?!

          Particularly since they'd probably point to lost production as a reason to push the price up a fraction for the next week, generating 10 times that in extra profits.

          Cynical? Me?

    2. L05ER

      Re: $600,000?!

      Service providers always refer to lost productivity as an "inconvenience".

      Surely this number comes from something else...

  5. Slacker@work
    Facepalm

    I used to work at a large international firm...

    One of the guys accidentally sent out a message to all 50k+ employees and even worse had read receipts turned on (he was finance and they always have them on by default)....So within seconds he realised what he had done so he tried to do a recall.

    Then followed about 2 days of him sat there unable to do any work as every couple of seconds up would pop a notification of his email being read (followed by a "fail to recall").

    OK so I could have killed the response messages and turned off his notifications etc. but no 'cos he wouldn't learn anything that way... :)

  6. Ralph B

    Certification Programme, anyone?

    Obviously someone should set-up an enormously expensive Certified Emailing Professional scheme, requiring regular renewal, and get everyone at Cisco to enrol in it.

    It's the only solution, Shirley?

    1. Anonymous Coward
      Anonymous Coward

      Re: Certification Programme, anyone?

      ...with Bronze, Silver and Platinum level certification, required annual training hours, 3-day refresher courses in exotic locales like Kalamazoo, Michigan, proctored testing...

      1. Anonymous Coward
        Anonymous Coward

        Re: Certification Programme, anyone?

        Don't knock Kalamazoo, some of the finest small Jazz on the planet and really good coffee

        1. Michael Wojcik Silver badge

          Re: Certification Programme, anyone?

          Yeah, and Kalamazoo is on I-94, just a couple hours from Chicago1 and a direct shot from Toronto, via Port Huron. And it's not far from the junction with I-69.

          If you want out of the way, try holding the exam in Marquette.

          1Or a couple of days, depending on traffic, since this is Chicago we're talking about.

  7. Anonymous Coward
    Anonymous Coward

    "Just as night follows day, some of those staffers committed the mortal sin of hitting reply all to the reply-all message requesting to be removed from the list."

    LOL The office trolls, they just cant resist an opportunity like that.

    "Other staffers than piled in by berating the "idiots" who failed to realise they were creating yet another email storm."

    LOL^2 A combination of The Gullable people that dont know about the office trolls yet, and the office trolls who (only barely) managed to resist the first round opportunity temptation of "please remove me from this mailing list, me too, and me" in wait of the inevitable 2nd round of "dont reply all!".

    If you're lucky, it all happened in another timezone while you were asleep, or drinking by yourself at home infront of the tv after dinner, and you get to review the whole shitstorm over your morning coffee to pick whether you want to be part of LOL or LOL^2, with the innocent defence of "oh sorry i didnt see those other e-mails yet, my INBOX was full of ...".

    But seriously, think of all the innocent photons and electrons needlessly excited across ocean floors and continents to deliver all this rubbish. No one ever thinks of the poor photons and electrons....

  8. Anonymous Coward
    Anonymous Coward

    Blame the originator

    Whoever sent out the original email is at fault. When you send an email to a large list you should use the Bcc: field for the list name and not the To, From or Cc fields. That way a "reply all" will never include the list.

    1. Anonymous Coward
      Anonymous Coward

      Re: Blame the originator

      Thank god for Captain Obvious :^\

      1. Anonymous Coward
        Anonymous Coward

        Re: Blame the originator

        Obviously it wasn't obvious enough

    2. Anonymous Coward
      Anonymous Coward

      Re: Blame the originator

      "Whoever sent out the original email is at fault. When you send an email to a large list you should use the Bcc: field for the list name"

      That would depend on what you're trying to achieve. If your intent is to broadcast information then you're right. If, however, your intent is to ask something from a large group of people, and the answers may prompt others to answer with more detail then it's entirely appropriate to not only use the To field but also to use the reply all function. Sometimes this is the only way to get useful and complete info, and at work I often resend a mail reply back out to the group to get further feedback because people did a direct reply to me.

      Anonymous because this post is going to be deeply unpopular - it is at work generally because it's nerds I send them to who don't like to communicate.

      1. Stevie

        Re: Blame the originator

        For God's sake man, if you are going to write something like this at least take time to proof read!

        You used "you're" correctly and had no spelling errors at all as far as I can see.

        Intolerable laxity for someone aspiring to Anonymous Coward credentials.

      2. Marvin the Martian

        Re: Blame the originator

        "That would depend on what you're trying to achieve": NO.

        If you want to share info with a lot of people and get their input: REPLY, with the list as said in BCC.

        If you want the above, but share the input with all the others: GET A WIKI/FACEBOOK/FORUM/whatever. You can even stick in a poll. It's like the future, you know.

      3. Intractable Potsherd

        Re: Blame the originator @ "what you are trying to acheive" AC

        As pointed out, you are wrong. You are one of the arseholes that cause these problems - there are much more effective tools designed for multiple inputs that have been around for a long time now.

        (The trouble is, you modus operandi is strangely familiar - do you work for a British university?)

    3. Dude in Columbia, SC

      Re: Blame the originator

      Folks, "Blame the originator" is absolutely correct... this was like leaving a series of landmines for others to step in.

  9. Roger Greenwood

    Need some free publicity?

    Use the Cisco Reply All Procedure.

    or CRAP for short.

    1. Merlyn

      Re: Need some free publicity?

      You can only use CRAP after you have received Special High Intensity Training of course

      1. ecofeco Silver badge
        Thumb Up

        Re: Need some free publicity?

        Classic.

      2. Anonymous Coward
        Anonymous Coward

        Re: Need some free publicity?

        "You can only use CRAP after you have received Special High Intensity Training of course"

        Don't you mean the Special High Intensity Training Simulation Tracking Override Remedial Mailers?

        1. Anonymous Coward
          Anonymous Coward

          Re: Need some free publicity?

          "You can only use CRAP after you have received Special High Intensity Training of course"

          or unless you've gone through the course to setup an autoresponder, the (S)He's Out of Office Training thing.

      3. Anonymous Coward
        Anonymous Coward

        Re: Need some free publicity?

        Once you SHIT and CRAP, don't forget to engage in a Waste Immobilisation Process Experiment

  10. Anonymous Coward
    Anonymous Coward

    @AC " Blame the originator"

    You can bet they'll never make that mistake again. Ever.

    Or the sender was an innocent victim, having stepped away from their desk to fill their waterbottle for their drink-4L-of-water-a-day-and watch-the-pounds-dissappear-from-your-gut diet , forgetting to lock the screen in their enthusiasm for the aqua aquisition, and an opportunistic office troll took advantage of their oversight and sent the email while they were away...

    (You can bet they'll never make that mistake again. Ever.).

  11. Anonymous Coward 101

    I once had email accounts with different university departments. I decided to save hassle by forwarding emails from account A to account B. And also forwarding from account B to account A.

    I recall I got to about 2 or 3 thousand emails before everything fell over.

    1. poopypants

      @AC 101

      Upvoted because I like silly people.

  12. Suricou Raven

    Legend

    There's an old story of the 'email laser' said to have occured at IBM, back in the early days of email.

    Employee A went on holiday, setting an email rule to forward all email to his co-worker, Employee B.

    Employee B, however, was also away - and set an email rule to reply to all incoming email with an out-of-office message.

    As the rule was enforced by the email server, there was no network delay - the rate of back-and-forthing was limited only by the email server's own processor speed. The problem was noticed when it ran out of disk space.

  13. Me19713
    Happy

    Handling that volume of email must be why they came out with UCS! gr

  14. Anonymous Coward
    Anonymous Coward

    Back in school some chums figured out you could send modal pop up messages to every computer on campus. Turns out this included both student and staff computers, so after some particularly raunchy messages the admins locked that down.

    1. squigbobble

      yeah...

      'net send' lasted about a fortnight of my first year at Uni. At least we had the consideration to target them :P

  15. Kruzman

    My favorite is when you get someone in IS do a reply all to the original replay all explaining why you never do a reply all....

  16. Anonymous Coward
    Anonymous Coward

    Bloody annoying

    I'm a Cisco employee and I could have used my phone as a vibrator for all of the action it got today.

    There were two mail storms - the first was about some mundane technical question emailed to 5 people as well as (for some reason) a mailing list which is for everyone who works in engineering under one of the VPs - 99% of it was the usual crap like "how am I on the list' and "please unsubscribe me" as apparently no one in Cisco has the mental capacity to look at the list of recipients and see that one of them has "mailer list" next to it.

    The second one was started when someone emailed a blank email, subject "unsubscribe" to the same list, followed by hundreds of other people doing it thinking it'd amount to anything. You can't unsubscribe from this list, it is automatically generated and permanent for as long as you work under this VP, and even if you could this is not how you do it.

    I haven't got an email for a while so I guess someone in San Jose has got to work and finally put an access control list on the damn thing. Not sure why the average employee is theoretically able to email about half of the company.

    Anon just bcuz.

    1. Lusty

      Re: Bloody annoying

      " You can't unsubscribe from this list, it is automatically generated and permanent for as long as you work under this VP"

      Not using modern Outlook and Exchange then? I think it was 2010 they put the "ignore" function in to block whole threads in your inbox.

      1. Lusty

        Re: Bloody annoying

        In fact, come to think of it, for several years Outlook has also warned people that they are about to mail a lot of people if they use a big group. Presumably because Microsoft found it cheaper to change the software than watch this kind of nonsense. Cisco ought to upgrade :)

      2. Anonymous Coward
        Anonymous Coward

        Re: Bloody annoying

        This multi-billion dollar giant only finally upgraded to the latest Exchange as of last year. Before then it was on good old 2003.

        The ignore button was mentioned by someone but it was drowned by the sea of idiots (and by people changing the subject which would probably defeat the filter anyway). I counted probably close to 500 emails before they stopped flowing.

        There are rumors that the original email that started it all didn't have the mailing list CC'ed when the sender sent it, and that Outlook or Exchange mangled it to include it. I can believe that, the original email was certainly not worthy of being sent to a VP's mailing list.

    2. Anonymous Coward
      Anonymous Coward

      Re: Bloody annoying

      I'm also a cisco employee and can confirm the ordeal lasted from UK morning time till about 4pm. I see it starting with a very innocent email on a technical discussion. As for the ignore button in outlook. A lot of staff have their emails forwarded to work phones. I don't have a blackberry myself but I have the feeling that it doesn't have the 'ignore' button. One email to the list from a US staff complained about being woken up at 3am by a non stop vibrating phone. Also many of the replies changed the subject like to 'PLEASE STOP THE SPAMMING' or 'I DONT WANT TO BE ON THIS LIST'. I ended up creating a rule in outlook to ignore everything sent to that VP's mailing list.

      1. Richard 26

        Re: Bloody annoying

        "One email to the list from a US staff complained about being woken up at 3am by a non stop vibrating phone"

        Which illustrates how totally screwed up modern communications technology is: forwarding your email to a phone, then leaving it on all night. How messed up is that?

  17. Anonymous Coward
    Anonymous Coward

    A final tweak if its your last day there anyway

    Change the subject line

    Add confirm read receipt

    Add a forwarding rule to your home email.

    The read confirmations were showing up FOUR MONTHS LATER

    More fun than putting two telephones together on an all staff conference call to make it nice and echoey

  18. Anonymous Coward
    Anonymous Coward

    Got a good one years ago, saying someone had left their car lights on.

    From Tokyo.

    1. Anonymous Coward
      Anonymous Coward

      Had one like that too...

      "Per the Florida Wildlife Dept.: Don't feed the alligators in the pond, it habituates them to human contact".

      Our facility had a pond as well... but we were in Minnesota. Any gator that could live in our pond would have been truly terrifying!

      1. Michael Wojcik Silver badge

        Re: Had one like that too...

        Don't feed the alligators in the pond

        What an unhelpful message. Where am I supposed to feed them?

        it habituates them to human contact

        In much the same way that McDonalds has habituated me to cheeseburger "contact", I suppose.

        Our facility had a pond as well... but we were in Minnesota. Any gator that could live in our pond would have been truly terrifying!

        Tonight on SyFy: SnowGators!

  19. Brian Miller
    WTF?

    Why is this news?

    Stuff like this happens all the time in large companies, e.g. Microsoft. Yes, it bogs things down. No, it's not a cataclysm.

    To start a storm: send an email out to a distribution list.

    Someone replies, "I don't want this email. Please remove me."

    Someone else replies, "Me, too!"

    Someone else replies, "Me, three!"

    And then really stupid comments follow.

    Solution: fire people who can't rationally deal with email. Also, fire the people who have flagged themselves as idiots.

  20. Tracy Nelson

    We had something like this happen once, but the original message had a .BMP file embedded -- all 8MB of it. And of course "reply all" conveniently included the original message. I think I only got five or six of those messages before the disks wedged tight.

    1. Michael Wojcik Silver badge

      Personally, I'm always amazed that the mail servers aren't configured to reject messages with large attachments. I have a number of ... helpful ... coworkers who insist on sending emails with multi-megabyte attached files that are already available on internal web servers. So they've gone through the trouble of downloading the file and attaching it to their email, rather than just, y'know, putting the URL in the message. (Or letting me find it myself, which would take me approximately two seconds.)

      If I were BOFH around here, the MTAs would be locked down tight against this sort of thing. And messages to lists with attachments would be filtered, too. And messages to lists with empty bodies, or with bodies that just say something like "unsubscribe". (Of course, if I were in charge of corporate IT, there would be many changes indeed. Oh, the wailing and gnashing of teeth that would ensue. But then what's the point of tyranny if you don't enjoy it?)

      People won't get in the habit of using appropriate tools until you provide the incentives, and that includes making it more difficult to use the wrong tools.

  21. Anonymous Coward
    Anonymous Coward

    Oh, the irony!

    "MAIL THE AUTHOR

    Sorry, we experienced an internal error trying to handle your request."

    I noticed in today's email storm that the "Subject: Unsubscribe" chain later turned into "Subject: UN-UNSUBSCRIBE". Peeking back in the message bodies showed why. Someone dropped this gem:

    ---

    Subject: Re: Unsubscribe

    Ah, but you can't unsubscribe like that.

    You need to use:

    UN-UNSUBSCRIBE

    In the subject line. Capitalization is important.

    For the especially bright people it might be worth digging out a recipe of casserole

    ---

    The previous storm included some casserole recipes from snarky recipients making fun of the clueless.

    Productivity killer? Yeah, because I keep laughing at the stupidity.

  22. Anonymous Coward
    Anonymous Coward

    30 comments in and no one has asked

    30 comments in and no one has asked why Exchange and/or Outlook (it will be them presumably, right?) don't have a capability to have a rule that says "if recipient_count > (say) 50 then (make extra checks before continuing)"

    That used to be possible back in the 1990s, albeit probably via local customisation, but presumably Outlook and Exchange have improved since then and IT departments have got worse since then.

    My "world class" employers Communications Department love to send out "All Employees" messages, typically relating yet another reorganisation of "senior leadership" which is few tens of KB of PDF. And a few hundred KB of distribution list, which would be trivial to "reply all" to. Maybe IT should have a word with them. Maybe IT don't know how to do it properly?

    1. MachDiamond Silver badge

      Re: 30 comments in and no one has asked

      "That used to be possible back in the 1990s, albeit probably via local customisation, but presumably Outlook and Exchange have improved since then and IT departments have got worse since then."

      Actually, the IT department has been downsized (what a polite way for management to say that they sacked all of the the smart wonks so they could have bigger bonuses come the holidays) since it's so simple now for people to configure their own mail programs (on the computers they had to bring with them).

      There is an email training course at Lynda.com. The instructor covers the scenario that lead to Cisco's mess and many of the other mistakes that MBAs make when trying to use anything more complicated than a biro.

    2. david 12 Silver badge

      Re: 30 comments in and no one has asked

      Start, System Manager, <organisation>, Global Settings, Message Delivery, Properties, Defaults,

      Recipient Limits, "Maximum (recipients)"

      That's why no-one asked.

      1. Anonymous Coward
        Anonymous Coward

        Re: 30 comments in and no one has asked

        Easy way to stop it, remove permissions to "send to" the mail list from everyone except administrators.

        1. Tom 13

          Re: remove permissions to "send to"

          That's a bit too limited. What you actually need is a way to specify the people who can send the list, not that they be administrators. Assuming of course when you say "administrators" you mean mail admins instead of secretaries. Oddly enough my work place has some ancient Unix system for the LDAP service which provides exactly that functionality. And I mean ancient as in the current real* mail admins are getting tired of trying to keep it running.

          *I'm shown in the system as a mail admin, but I consider myself a fake mail admin. I create user accounts and mailing lists. I don't know the first thing about standing up an actual mail server.

          1. david 12 Silver badge

            Re: remove permissions to "send to"

            >. What you actually need is a way to specify the people who can send the list,

            What I posted above is the setting for the DEFAULT value. If you want to specify a different value for a group, you have define and select the group. It's no more difficult, but it would be kind of boring what with explaning the names and groups. I didn't think anyone here would want to read it.

      2. An ominous cow heard

        Re: 30 comments in and no one has asked

        "Start, System Manager, <organisation>, Global Settings, Message Delivery, Properties, Defaults,

        Recipient Limits, "Maximum (recipients)"

        That'll be " IT departments have got worse since then." won't it.

        Or in Cisco's case, got non existent?

  23. Stevie

    Bah!

    Wasn't this a Dilbert cartoon about ten years ago?

    Netadmin to all users: Stop using "Reply to All" as it clogs the network with unnecessary traffic.

    Dilbert Reply To All: I agree.

  24. Henry Wertz 1 Gold badge

    What surprises me...

    What surprises me is that a modern e-mail system would still buckle from this kind of thing. I mean, with postfix, one portion expands the "all@cisco.com" or whatever into 36,000 addresses, a chunk at a time, into the send queue. This chunk size is lower than the send queue size, so normal e-mail still gets into the queue and gets processed. This makes e-mails to some ill-advisedly large list like "all@cisco.com" (or whatever) slow but ordinary e-mail is not too impacted. Of course you still have the excessive number of messages to deal with then.

    1. Jim 46

      Re: What surprises me...

      Last place where i worked anywhere near the email stuff, the mail servers were well resourced & tuned for high volumes and peak loads, unfortunately the same attention wasnt given to the virus/spam scanners that sat infront of them. The virus/spam filters were pedantic, buggy as hell and borked under load. And when they borked, the mail stopped flowing...

  25. Kevin McMurtrie Silver badge

    >OOTO >>OOTO >>>OOTO >>>>OOTO

    No work will get done with that storm going on. They all should activate the "I'm currently out of the office" auto-reply and go home early.

  26. Greem

    Oh, the memories

    In $former_job in commercial web hosting, customers had the ability to create simple mailing lists. One of them ran a mailing list for a large, disparate charity organisation - and also ran several of said organisation's branch mail systems.

    For some unknown (still) reason, they used Microsoft's SBS product with the dumbest POP3 receiver code I ever saw. The following happened:

    1. Mail gets sent to list

    2. Mail gets expanded to recipients

    3. Mail goes to one of the aforementioned SBS POP3 connector widget, which looks at the incoming message and thinks 'this isn't for one of my addresses, it's been sent to some-list@some-domain...'

    4. SBS POP3 widget sends message to mailing list

    5. GOTO 1.

    The list had thousands of subscribers. I lost count how many of these SBS machines were involved - at least 4 - but the resulting storm saw our customer attempt to claim £250k in damages from us for lost business! The last I knew about it, they'd backed down and as far as I know it never got to court.

    Hilarious at the time, but the aftermath was bloody irritating.

  27. fLaMePrOoF

    Hasn't something gone wrong in their admin when they fail to use BCC when sending out emails to large lists? DUH!

    1. Tom 13

      Re: Hasn't something gone wrong in their admin

      No, I'd put this one on failure to have a restricted send ability on the mail group.

      For the BCC you are depending on humans to be infallible as well as having unlimited willpower. Restricting who can send to the mail group/list is a technical solution that IT can implement. Others have noted other mitigation strategies IT can implement to put breakers in the system, but for a cold hard stop, it has to be a restricted mail group. Restricted send should be automatic on ANY large mail group (I'd say 50+ recipients but YMMV) and should be a consideration for any mail group. Yes it is more of an annoyance to the mail admin to maintain (but that can be delegated from real mail admins to fake ones like me), but not nearly the annoyance of dealing with a mail storm.

      1. Anonymous Coward
        Anonymous Coward

        Re: Hasn't something gone wrong in their admin

        There is the ability to lock down the mailing lists quite well so that only owners or approved people can send email to the list, including with escrow where messages have to be approved by someone before they are fired off. I suspect that this, and the actual mailing list infrastructure runs on some shonky old Sun boat anchor as there's still a surprising amount of it at Cisco.

        I have just got three "unsubscribe" emails from people who are emailing the *same* list so clearly someone still hasn't used any of the tools that are at their disposal to lock it down. I wonder if that person saw this article and wanted to try to go for three in a week.

        The list is basically used by the VP / high up execs to send their usual feel good crap to the slaves in the engine room, I can't see any reason at all why someone like me should have the ability to send mail to it.

  28. Necronomnomnomicon

    Makes me wonder though

    Is there any mail server that does clever de-duplication stuff that'd stop this being such a problem?

    As in, an individual email and content is only stored once, and then the notifications and read monitoring is managed in the background. Shirley the big email providers would like something like that.

    1. Necronomnomnomicon

      Re: Makes me wonder though

      Not excusing the brain donors who actually cause stuff like this, mind.

    2. Anonymous Coward
      Anonymous Coward

      Re: Makes me wonder though

      "an individual email and content is only stored once, and then the notifications and read monitoring is managed in the background"

      DEC's ALL-IN-1 (tm) office automation and email system did that, if I remember rightly.

      On VMS, on VAX

      In the 1980s.

      When 450MB was a big disk (RA81).

      There's been lots of progress since then. Disks are bigger and cheaper and faster. Software is bigger and cheaper. IT departments are bigger.

    3. david 12 Silver badge

      Re: Makes me wonder though

      >As in, an individual email and content is only stored once, and then the notifications and

      Yes, that is what mail stores do. But when you respond to a mail message, you create a new and different mail message with new content. If you don't, then wtf are you doing pressing the send button? And your email client is modifying the message anyway -- adding that "in reply" or "forwarded by" text to the top of the message.

      1. Necronomnomnomicon

        Re: Makes me wonder though

        I mean, say I emailed n people internally. As I understand it, Exchange (or similar) will put one copy of that email into each of the n mailboxes. Each of those emails is identical, so the server is using n -1 times more disk space than it needs to.

        The AC above points out that I'm not having an original thought as it was done years ago, but I'd have thought that at least the giant-scale mail providers like Google and Microsoft would have thought about something similar because at their scale the level of duplication must be huge.

  29. El Limerino

    The good old days

    When I worked at Cisco in the 90s and the same thing happened, Don Listwin replied and said he'd catapult the next person to reply over the light rail track on Tasman Drive. The replies stopped.

  30. shmendel
    Mushroom

    Cisco Employee

    At some point some prankster replied with the following:

    ------------

    Ah, but you can't unsubscribe like that.

    You need to use:

    UN-UNSUBSCRIBE

    In the subject line. Capitalization is important.

    For the especially bright people it might be worth digging out a recipe of casserole

    ------

    Immediately a ton of e-mails started arriving with "UN-UNSUBSCRIBE" subject line!

    I think the stop to the whole madness was put by a guy in Israel who wrote the following:

    -----------

    Maybe this link will help unsubscribe: http://bit.ly/16t2VYr

    -----------

    The link contained (and I believe still contains) an HTML form that in Q and A format convinces people that the way to stop the storm is just to stop replying to the thread.

  31. Stretch

    This used to happen weekly back at Accenture, and always made much worse by the poorly trained muppets they use to write the code out in countries to stay nameless.

  32. Vociferous
    Trollface

    I used Spot back in the day. It was fantastic, it allowed editing of all headers, so one could create completely untraceable mails.

This topic is closed for new posts.

Other stories you might like