back to article Oh sure, we'll just make a tiny little change in every source file without letting anyone know. What could go wrong?

The week is almost done, and Friday sees the start of the delightful tumble that ends in the weekend. Start your roll with a tale from those unfortunates forced to deal with the foolishness of others in The Register's On Call feature. Today's story comes from a reader we will refer to as "Brad" in order to spare the blushes of …

  1. hmv

    Ratelimiting

    So many similar stories that I can't remember them all.

    I did implement rate limiting with Exim so that a bulk spewer would end up with messages being 'frozen' in the queue and not delivered; as a result 'exiwhat -zif root@someplace | xargs exim -Mrm' is hardwired into my fingers. I seem to recall a common issue was web→mail forms being spammed by scanners.

    1. swm

      Re: Ratelimiting

      Our sysadmin at college had a tar pit for spam mail. For selected senders he would set the maximum packet data length to one byte (which really slowed them down). After receiving the email it would be rejected with a code to resend it.

      His theory was that he was tying up resources of the sender as a public service.

      Talk about rate limiting.

      1. W.S.Gosset
        Happy

        Re: Ratelimiting

        I like your story but I hate your sysadmin.

      2. EarthDog

        Re: Ratelimiting

        sounds like a real bastard

    2. W.S.Gosset

      Re: exim

      Your mailserver can get jammed before you have time to react, if a dedicated spambot injection is kicking off hard on a connected webserver This leaves exim etc incapable of responding usefully, if at all.

      We got stuck in this situation at an ISP+MgdSvcs company a few years ago. No response from anything but shell built-ins or nearly-so primitives, and even THEY were sluggish (!). (Unix, to be clear.) We went from fine to fkd in less than 60secs from the alarm, terabytes tempdir jammed while we were trying to work out what was happening.

      Finally decided to risk wiping some client emails and just wipe enough of recent tempdir to allow server load to drop enough for us to get some control.

      Problem: rm crashed. Too many files.

      Problem: even piping ls -t to it crashed for the same reason.

      Solution: take advantage of oldskool lowlevel unix shell design intelligence of a pipe being a parent, which closes all its children as soon as the terminal process throws an exit value. Combined with intrapipe interprocess streaming, ie character by character. So despite ls still running like a loony, it's not hitting capacity limits because it's just vomiting what it reads, and as soon as head hits its parameter (streaming-from-0 rather than read-all-and-count-back like tail), it terminates, so pipe terminates, so pipe terminates all children --> shell can continue.

      while true

      do

      FILELIST=`ls -t | head -1000`

      rm -f $FILELIST

      done

      ls|wc on another terminal showed the numbers dropping. Race Condition, and we were winning.

      4-5hrs later we regained control, sorted it.

      We'd contacted clients to warn them. As luck would have it (it was right on Close-Of-Business), no-one had to resend anything.

      Yee. Haw. And. Phew.

      .

      Stern words the next day with the culprit website owner company, then their devs.

      1. W.S.Gosset

        Re: exim

        In retrospect:

        * the simplistic backquoting does not competently handle filenames with internal spaces. I knew they had none by the mailserver's design, but, still, looking back, I get a gut-clench.

        * woulda been MUCH better if I'd grepped between ls and head for characteristics of the originating website. Excuse: panic. 400+ websites just on the one upstream burgled VM. Lots of other stuff and people upstream. Centralisation is brilliant, until it's not. And this machine was LOCKED. In retrospect: idiocy, fail vs potential risk.

        [EDIT: wait, hang on, there weren't any. Due to the chaining/structure, the filenames etc had by that point no source-identifying info.

        OK, I feel a bit better about #2

        But still, if YOU'RE in the same situation, take 1sec to think if you can carve out anything non-relevant. Go go gadget grep -v!

        ]

      2. Anonymous Coward
        Anonymous Coward

        Re: exim

        @W.S.Gosset

        "Your mailserver can get jammed before you have time to react..."

        We had a similar experience three to four years ago - completely self-inflicted.

        After having used Spiceworks for a couple of years, mainly to track a large client's IT inventory and using it internally to handle tickets, the admin who looked after Spiceworks decided it was time to implement an e-mail ticketing system that the client could use to raise tickets (we used to open tickets as faulty equipment came in, but the burden on us was becoming increasingly more onerous).

        So, one particularly slow Friday, late in the year when everyone was in relaxed mode, he set up the rules and we did a couple of tests. After ironing out some minor niggles, we were ready and the system went live.

        The next step was to send out a company-wide (to the client) mail, from Spiceworks, advising them of the improvement to service, as well as a short run-down/instruction manual on how to log calls/raise tickets.

        About ten minutes later we received the first call about e-mail not going through, as well as people having problems accessing mail.

        A short discussion determined it had to have something to do with implementing the ticketing system, but what? The feeling of dread started sinking when we discovered that we could not log on to either the mail server or to Spiceworks. The sessions we had open had also become unresponsive.

        We were able to log on to both servers via ssh (albeit extremely sluggish), only to find that the mail queue was brimmingly full and new mails being added in copious amounts.

        I was given the task of clearing the mail queues as fast as possible, whilst he struggled with trying to stop Spiceworks sending mail. Two or three hours later we managed to wrest control back of both servers and then stopped the mail server (the end of the day was imminent in any case, and nobody could send mail in any case).

        We still had no clue as to what had happened, but we were reasonably certain it had to have been a rogue rule or two that caused the mayhem. The first clue came when I noticed that most of the messages had been between Spiceworks and a very small number of users. Inspecting two or three of those mailboxes, brought to light that they were on leave and had set up out of office messages, as per policy, advising clients of their absence and who to contact instead.

        When I mentioned that to my colleague (at the time I was not privy to all the rules he had set up in Spiceworks), the awfaul truth dawned; he had a rule that Spiceworks would send an auto-response to the person who had raised a ticket, acknowledging receipt of same, et cetera.

        So when Spiceworks received the out of office message, it would send out its acknowledgement, generating an o-o-o message, which... You get the message.

        Handling that particular scenario was not easy nor straightforward, but by the next Tuesday we had it sorted.

        Anon for obvious reasons.

        1. A.P. Veening Silver badge

          Re: exim

          Perfectly predictable pitfall ... in hindsight ;)

      3. Old Shoes

        Re: exim

        If I may introduce you to `find | xargs` it may save you some time at a later date.

        `find . -type f -delete`

        That will delete all files in this directory and sub-directories.

        `find . -type f -print0 | xargs -r0 rm -v --`

        That will delete all files in this directory and sub-directories, using `rm` which I've also set to announce which files it is (or isn't) deleting.

        1. Stoneshop

          which I've also set to announce which files it is (or isn't) deleting.

          find . -type f -print -delete

  2. This post has been deleted by its author

  3. chuBb.
    Facepalm

    Too soon to tell I got a good one this week

    Wish I could retell my tail of epic face palm, with no specifics massive escalation by a customer, threatening to pull contract see us in court etc. With an equally rapid climbdown once root cause of their admin didn't rtfm, grok the permitted request rates or understand what an amplification attack was, was identified.

  4. Anonymous Coward
    Anonymous Coward

    Mail Storm

    Back in the day I was responsible (with others) for maintaining an Exchange 2003 clusterfuck. We were a Microsoft shop and made use of AD to control access to applications and shares and other security stuff. One of the applications we used had some AD permissions groups to control access to the app.

    One morning I arrived at my desk, slurping my coffee I noticed I had email, lots of email. Me and over a thousand other peeps. A user had discovered he could send mail to all the apps AD groups. Something only the app admins were supposed to able do.

    The original mail was a rant and a thousand users responded to a thousand users. Some saying don't mail me this crap, others ranting back. Because this happened out of hours we didn't see the storm growing until it was so bad Exchange slowed to a crawl, right in the middle of the day. By now there was nothing we could do except lock out the AD group's mail privileges and wait it out.

    Best bit was the main Exchange admin sending a stroppy mail to the whole app mailing list saying "stop using sending mail to this group" before locking the groups down. Yup mailstorm 2, the return.

    Anonymous to protect the guilty.....

    1. chuBb.
      Pint

      Re: Mail Storm

      Have a pint for using the correct exchange cluster terminology

    2. Anonymous Coward
      Anonymous Coward

      Re: Mail Storm

      Similar position with Exchange 2003. The entire email system went slow around midday on Friday. Big backlog of email that cleared after a little while. The next week it ground to a complete halt with massive mail queues....

      The eventual root cause was found to be a new HR Inclusivity email - the first was a text only introduction, the second contained several rather large images making the email just under the 10MB limit.

      1. Tom 7

        Re: Mail Storm

        We had a mail tsunami on EX2003 due to a virus. While the PHB and systems manager did the headless chicken thing on MSN and our well paid support* I wrote a VB app that could open each mail in the DB (bit of a posh name for it really) and scan the mail for signs of infection. I got the OK to unleash it on the DB where it deleted 99% of the emails in it and things settled down once a few other patches were in place. It was a useful bit of code I could use to build a new "DB' when it got corrupted as it occasionally did. And also looking at peoples emails while bored on a long weekend waiting for some upgrade to finish.

        *We paid a company for support which then never seemed to manage before we fixed things ourselves - largely because when MS fucked up they fucked up all their customers at the same time. I used to enjoy getting phone calls in the early hours where they would tell me the steps to fix whatever upgrade had damaged and almost chant along with them and then give them further tips and trix. They were never grateful!

        1. PM from Hell
          Facepalm

          Vendor Support

          We had similar issues with Oracles Service Desk. They would cycle through new recruits until they knew enough to deliver 'consultancy' this meant my DBA's had to suffer an hour of list ticking while they went through their script before they could admit they didn't have a clue and pass us on to a 2nd line engineer.

          I'd put up with this for something minor but used to have to call my account manager to escalate anything serious to second line as soon as we'd been given a ticket number. We were pretty leading edge and working on a joint development at the time so were usually on beta or just released versions for the tools.

          It was exciting at the time but developing business critical apps using new Oracle products is not good for your blood pressure.

      2. EVP

        Re: Mail Storm

        ” text only introduction, the second contained several rather large images making the email just under the 10MB limit.”

        Stock photos of shiny happy people working, I presume.

        1. Aitor 1

          Re: Mail Storm

          All minorities represented and smiling. At least that is the standard I receive.

          I mean, why not put pics, please don´t fill everyone's ssds!

    3. chivo243 Silver badge
      Pint

      Re: Mail Storm

      Haven't seen one of those in our shop for years... but we did have a good one many years ago. 30,000+ messages(each) later, we figured out it was due to most of the staff being on holiday with an out of office responder, and belonging to many listserv... who replied when they received a group message, who got an out of office reply, rinse, repeat ad nauseam.

      1. Martin
        Happy

        Re: Mail Storm

        I've often thought that if senior management in a large company wanted to make some cutbacks, they should send out an email to everyone "by accident" which is clearly not intended for everyone. Then anyone who replies to all with "Please remove me from this mailing list" goes on the potential cutback list, as they have demonstrated a lack of sense and consequential thinking.

        1. Yet Another Anonymous coward Silver badge

          Re: Mail Storm

          Simpler just send out emails saying "you're fired".

          Anyone who reads emails from HR is obviously bored and stupid

          1. swm

            Re: Mail Storm

            As in "Desk Set".

          2. FatGerman

            Re: Mail Storm

            I often wonder if the purpose of HR emails is to detect people who don't have enough work to do.

    4. Trygve Henriksen

      Re: Mail Storm

      Why are you writing 'Exchange 2003 clusterfuck'?

      Isn't that like writing 'Fat grease'?

      1. MGJ

        Re: Mail Storm

        Better than the NT4 first cluster, along with Exchange 5.5. Wolfpack it was called. We called it Wolf Pair given the limits to it. All the service passwords on our Compaq cluster were clusterfuck though. Rebuilding at 3 in the morning a configuration designed by compaq and MS that had shared fibre storage that the German support team admitted was never likely to work as shared fibre storage wasnt supported, and how had we ever got it working in the first place?

    5. MrBanana

      Reply All

      Ah, the classic Reply-All email storm. "Has anyone seen Bob's chair?" sent to all 10,000 employees in Europe, instead of Floor 3, building 2 where it should have been targeted. After the "who's Bob" avalanche, the follow up "why did you sent this to me" storm, there is the "stop sending me email" and the "unsubscribe" complaints - all sent as Reply-All. The "unsubscribe" can develop into a regular tsunami as it looks like the official way to stop the onslaught. I've once tried to head off this issue with a Reply-All of my own, to tell people to stop hitting Reply-All... but... stupid people.

      1. Emir Al Weeq

        Re: Reply All

        I've seen a situation where the email came from the "everyone" group, and so even just hitting "reply" as opposed to "reply to all" had the same effect. I did neither and just sat back with the popcorn.

      2. Psmo

        Re: Reply All

        Me too!

    6. Stevie

      Re: Mail Storm

      https://dilbert.com/strip/1995-05-10

    7. ecofeco Silver badge

      Re: Mail Storm

      I once had an Exchange admin call us (helpdesk) wanting help with... Exchange.

      I had a little chat with my boss asking him why the admin couldn't do their job and explained there was nothing we could actually do.

      The look and response I got told me I wasn't going to be there much longer.

    8. Anonymous Coward
      Anonymous Coward

      Re: Mail Storm

      Another anon email storm story.

      I was in super computing, back then, and my colleague was having a hard time with calculations being incorrect in very rare circumstances on his program.

      So, easy peasy, he decided to shoot an email if calculations were incorrect.

      Yes, the "very rare circumstances" became very common on the next run, causing thousands of email being shot to the SMTP relay, killing it off. Angry sysadmin ...

  5. Anonymous Coward
    Anonymous Coward

    I loved Groupwise

    Loved the collaboration bit, was easy to manage comms for various projects in groupwise, all project correspondence was sent within the project group keeping everything neat and tidy.

    i hate gmail for work, somehow its more of a mess than normal free gmail, hides my email for me & meet always starts with camera enabled with no option to default to camera off.

    1. Anonymous Coward
      Anonymous Coward

      Re: I loved Groupwise

      Dont use gmail for anything but forwarding to a better solution ;)

  6. Anonymous Coward
    Anonymous Coward

    I fondly recall just a few years ago #ReutersReplyAllGate - the emails just kept on coming! Baffled why people kept responding to them!

    https://blogs.wsj.com/cmo/2015/08/26/reuters-employees-bombarded-with-reply-all-email-catastrophe/

    1. TeeCee Gold badge
      Facepalm

      Because "Baaaaaa". Obviously.

      Well, that's what ever instance of "Why did you send this to me?" I ever saw read as to me.

  7. GlenP Silver badge

    Not quite on the same scale but...

    We use SharePoint (yes, I know but I inherited it and we're tied to being an MS site anyway). I had one user a few years back who set up email alerts on a Library for All Changes and Send Immediately (contrary to advice otherwise).

    He then bulk uploaded a few thousand files (it was the ISO 9000 Library) and wondered why his email on the laptop and Blackberry (I said it was a few years ago) were going mad.

    1. Yes Me Silver badge
      Happy

      Re: Not quite on the same scale but...

      Anyone who downloads the ISO 9000 library deserves whatever happens to them, unless they are doing so under orders from a person with a gun.

  8. Sequin

    I maintained a system on Netware that processed applications for government assistance, which eventually lead to Giro cheques being printed and posted to the applicants. These were on tractor-feed stock and were printed on an impact printer to prevent the cheques being altered by scraping off laser toner, which does not penetrate the paper.

    After building up a backlog of cases, the management decided to have a blitz and instituted lots of overtime and brought in temporary staff. They then hooked up a box of paper and ran the print job. Unfortunately, the size of the job was larger than the disk space allocated to the print queues. The cheques started printing, but after a while the system crashed and restarted. This started the the print job from the start, not from where it left off, and led to hundreds of duplicates being printed. The temporary staff in the mail room were not aware of this and just separated the cheques, plonking them in window envelopes, and put them in the postal system.

    By the time it was noticed, they were too late to recall them. I got a call in Liverpool and had to get a hire car and drive down to Birmingham with my foot hard on the pedal - I even got pulled by the boys in blue on the motorway, but was luckily let off with a warning after being breathalysed.

    It cost them thousands of pounds in the end as people cashed in their windfalls.

    1. A K Stiles
      Pint

      Cheque runs

      Shudder!

      I had the joy some years ago of being involved in setting up a new cheque printing system where the monthly cheque data would be auto-generated by an overnight routine on the AS400, then someone in the finance department had to fetch the cheque stock (separate A4 sheets, pre-printed with serial numbers) from a locked store, load it into the cheque printer (which used magnetic ink for the automated cheque readers in the bank and had locked paper drawers) *the right way round and up*, then key in the first cheque serial number to the AS400 which would then store the SNs against the cheque transactions on the system and dump the data to a shared directory, which was auto-monitored by a separate system running some flavour of Crystal Reports to take that data and transform it into a cheque page layout (address and transaction breakdown at top of page, physical cheque perforation-attached at the bottom).

      Depending who was doing the cheque run they either got exactly the right number of cheques out, or grabbed more than enough and returned the extra to the store after the run. But my <deity> the number of times I had to help sort out that cursed printer deciding to eat pages, or them being loaded the wrong way round. It was deemed necessary to take the messed up cheque pages to the Finance Director, so the numbers could be logged in a manual binder and signed for, before being put through the chipper shredder in his office while he watched then signed off the signature sheet, and the pain to change the recorded cheque numbers *one by each* manually on the system which would send each one to the printer again separately.

      Mind you, if you think normal ink / toner is expensive, keep away from the magnetic stuff!

      Icon, 'cos that's what the end of the day with a cheque run really required.

      1. jake Silver badge

        Re: Cheque runs

        "Mind you, if you think normal ink / toner is expensive, keep away from the magnetic stuff!"

        Oh, I dunno ... I bought a 1 kilo can of magnetic ink from Valley Litho for about 80 bucks a couple months ago. That's on par with other offset inks.

        1. Zarno

          Re: Cheque runs

          That's insane cheap, if it was HP/Epson/ETC inkjet ink, you'd be looking at a few thousand buckazoids a gallon...

      2. Tom 7

        Re: Cheque runs

        If you were doing fairly low numbers of non-A4 prints we found having special feed trays with customised idiot sheets glued to the bottom so in theory they couldn't get the cheques or whatever the wrong way round.

        It was worth a try.

        1. A K Stiles
          Facepalm

          Re: Cheque runs

          Doesn't work so well with face-down trays though.

          Many of the printers I've had the joy of responding to ended up with signs attached to the drawer front saying something along the lines of "Headed Paper Only. Header this end, face down".

          It reduced the incidence of incorrect paper loading by approximated 50%...

      3. Ken Moorhouse Silver badge

        Re: pre-printed with serial numbers) from a locked store

        Reminds me of when I was working for the Signals Computing department of London Underground.

        One day we had a delivery of stationery, which we chucked in the stationery cupboard without any further thought. A while later, someone opened the box to find that it was pre-printed cheques. They were intended for the Payrolls department.

        Actually, security was pretty tight in our building. One day there was a bit of a queue to get in. A rather irate guy was arguing with the security guard on the door. "I cannot let you in as you don't have the correct pass, I'm going to have to report you to the building manager", to which the reply was "b b but I AM the building manager".

        Email? Oh yes, this is about email. I liked Groupwise. You did however have to watch out for how ISDN was configured for these systems though as repeatedly dropping and picking up the line to process emails could be costly. Once an ISDN line is up it should be kept up for the minimum charging period.

        Moved on to becoming a dealer for Mdaemon which addresses most of the issues others are including in this thread.

        1. Snapper

          Re: pre-printed with serial numbers) from a locked store

          The REAL killer with ISDN was the two channels being set to auto switch instead of 'Use 1 or two channels as default', in other words switch from one 64k channel to using two due to the amount of data being used. Used to switch on and off many times a second and the end-user would get a thick ream of invoices because each connection was a separate charge for the minimum period IIRC 60 seconds. BT made a fortune and never coughed up what was causing the issue.

          And don't get me started on the near hopeless quality of the BT engineers sent out to install ISDN sockets! Complete Cockwombles the lot of them!

          1. JimC

            Re: ISDN bills

            Back in the day the story was told of an expensive satellite link and an early snmp implementation. The unwise network admin thought it would be a good idea to confiure all his routers to send an snmp trap if their links changed state. You can tell what's coming, can't you. Yes, he forgot to exclude the satellite link. And better yet, he didn't have a minimum call duration, so as the story goes the line dropped, snmp sent a trap, the line went up to transmit it and dropped again. And better yet, it continues, he hadn't allowed for the minimum period, so, like your ISDN bill, the satellite bill was for many times what having the link up 24 hours would have cost. And in those days satellite bills weren't cheap like ISDN...

  9. cosymart
    Megaphone

    Most Dangerous

    The most dangerous button ever created "Reply to all" not helped by idiots who use the shotgun approach to sending out emails.

    1. Pascal Monett Silver badge

      Re: Most Dangerous

      Indeed. That thing should have had an admin password lock on it from the start.

      Since Office 2010, I've worked with a few companies who actually removed it from the ribbon. You could, of course, plop it back in, but woe to the guy who tries that. I know of one who got hauled right up to the CEO's office. I don't think he tried that again.

      1. Charlie Clark Silver badge

        Re: Most Dangerous

        Exchange makes it easy to do this and difficult to prevent abuse.

        1. Antron Argaiv Silver badge
          Joke

          Re: Most Dangerous

          Yet another example of Microsoft's tailoring of product features to customer needs.

    2. Anonymous Coward
      Anonymous Coward

      Re: Most Dangerous

      I often get e-mails that specifically say ‘please reply to all’ makes the blood run cold!

      1. diodesign (Written by Reg staff) Silver badge

        Reply all

        When a tech vendor or some other organization writes to us to complain about an article, they like to CC in their lawyers and insist they stay CC'd, presumably to scare us into thinking this is a Potentially Big Problem.

        And, depending on the tone and weight of the complaint, sometimes I enjoy replying with those uninvited guests flicked off the CC list.

        C.

        1. Steve D
          Mushroom

          Re: Reply all

          I do hope you use the responce given by Private Eye in Arkell v Pressdram.

          1. jfm

            Re: Reply all

            I've always liked the response by the Cleveland Browns' lawyer to a silly complaint:

            Dear Mr. Cox:

            Attached is a letter that we received on November 19, 1974. I feel that you should be aware that some asshole is signing your name to stupid letters.

            Very truly yours,

            CLEVELAND STADIUM CORP.

            James N. Bailey,

            General Counsel

        2. Old Shoes

          Re: Reply all

          CC'ing in the legal counsel also means they can't be subpoena'd for those emails later because they're client-lawyer confidential.

          This is some stupid loophole that a dodgy company tried to use in America. I remember reading about it here on El Reg but I can't recall their name.

          1. Anonymous Coward
            Anonymous Coward

            Re: Reply all

            "CC'ing in the legal counsel also means they can't be subpoena'd for those emails later because they're client-lawyer confidential."

            It doesn't actually hold up, though. "[T]o be protected, the communication must be for the purpose of providing legal services."

            https://www.mcneeslaw.com/copying-legal-counsel-doesnt-create-privileged-communication/

  10. Mark192

    "spewing email like a teen discovering cider for the first time."

    :-)

    1. heyrick Silver badge
      Happy

      Yeah, suddenly I feel like my teenage years were a disappointment. I too could have discovered cider and emailed gibberish to everybody in sight...

      1. Anonymous Coward
        Anonymous Coward

        The spewing was overrated.

  11. Hans Neeson-Bumpsadese Silver badge

    rm * was Brad's friend that day

    That sounds like the old disaster movie plot device...a closing line that just sets everything up for a sequel

    1. Anonymous Coward
      Anonymous Coward

      that just sets everything up for a sequel

      Yeah. I would probably have written a script intended to only delete the relevant (useless) messages rather than the whole queue. Of course, I say "intended", ... who knows what it might have actually done instead :-) [1]

      .

      [1] i.e. Exactly what I coded it to do.

  12. Anonymous Coward
    Anonymous Coward

    Amarillo.....

    Is this the way to crash a network?

    Dial up modems really don't like it

    Stop resending that bloody video

    I will surely kill you all!

    I was a poor sysadmin at an office where everyone received and sent that video (the one from Iraq)... our connection to [redacted] was a lowly modem so, try as we might, we couldn't receive/send email for a while...

    Its not always an automatic SNAFU that borks one's coffee breaks.

    1. Zippy´s Sausage Factory

      Re: Amarillo.....

      Reminds me of the time a former employer decided to install a proxy server to their single 56K modem.

      Within days me and the rest of the IT team found out about so many dodgy sites. Still, at least we found that the block function on the proxy server worked really well, so that was a bonus.

    2. MJI Silver badge

      Re: Amarillo.....

      Am I the only person who would like to kidnap Tony Christie. Put him into a car in Chicago and tell him to drive 1270 miles and then he should be there.

      1. J.G.Harston Silver badge

        Re: Amarillo.....

        Hasn't Tony Christie admitted that he can't drive?

  13. DailyLlama
    FAIL

    Early 2000's

    We had an exchange server that pulled from the Cluster in head office. One day I was asked to install some updates, and shove some more RAM in the box, while they did the same to the cluster. I warned my site that we'd be without email for an hour or so, and that if they wanted to stop seeing errors, just close Outlook. SO the hour came, everyone closed Outlook, and I installed all the updates on the box, then powered down to install the memory. Once done, I turned it back on, and carried on with my normal work.

    About 6 hours later, someone came in to my office and asked if it was ok to use email yet, as they needed to send something out to customers. It occurred to me that I hadn't heard from head office, so I rang one of them. Who told me "Yes, we sent you an email to say it was working again"

    1. Anonymous Coward
      Anonymous Coward

      Re: Early 2000's

      I've experienced a couple of issues like this being emailed to tell me we can use emails again and another when we were not told that the system was ready to use again after an emergency outage to replace a pair of dodgy power supplies (one failed completely and the second started logging errors before it could be replaced(. I had over 1000 users waiting to log in again but never got a call from the IT PM. When I finally found someone who knew what was going on it turned out the psu's had been replaced then they went to the pub with the engineer.

    2. heyrick Silver badge

      Re: Early 2000's

      The number of times I've had various techies finish their emails with words to the effect of "please feel free to contact me again if you don't receive this message".

      Um.

      1. the spectacularly refined chap

        Re: Early 2000's

        Something similar was a running joke in lectures at uni:

        Put your hands up if you can't hear me!

    3. EVP

      Re: Early 2000's

      Like when my friend needed a password reset, helpdesk helpfully sent it to him by email. At least they did it with no delay, it turned out. Great service.

      1. TSM

        Re: Early 2000's

        I once reported to our helpdesk that I wasn't receiving new emails. They worked out what the problem was -- and emailed me the instructions to fix it.

  14. Michael Hoffmann Silver badge

    If this were the DailyWTF...

    ... the story would have ended with Brad getting fired for daring to simply delete the emails instead of going through each and every one. It so happened that that the CEO's extremely important golf invite was among them!

    1. Anonymous Coward
      Anonymous Coward

      Re: If this were the DailyWTF...

      After our Chief Exec missed a 'critical' email from the Cabinet Office we had to train his PA to look n his trash file every few hours as he could not understand how spam filters work or be bothered to add the cabinet office address to his 'never block' list.

  15. gnasher729 Silver badge

    A harmless case

    I started at a company apparently exactly on the day when someone had managed to put the "All <location>" mailing list into the "All <location>" mailing list. Which meant anything sent to "All <location>" would be sent to everyone at the location, then to the "All <location>" mailing list. Which sent it to everyone at the location, and then to the "All <location>" mailing list. Which sent it to everyone at the location, and then to the "All <location>" mailing list...

    Their IT was very good, they managed to stop it after just four rounds.

    1. Hero Protagonist
      Pint

      Re: A harmless case

      “ they managed to stop it after just four rounds.”

      Then off to the pub for another four rounds! (see icon)

      1. Anonymous Coward
        Anonymous Coward

        Re: A harmless case

        Then off to the rifle range with the ID10T who made the recursive mailing list for another four rounds!

  16. druck Silver badge
    Pint

    Bugzilla - emailzilla

    A few years back I used to dread the Friday morning bug triage session, as Bugzilla would generate a constant flow of emails. But at least deleting them all was something which could be done after lunchtime at the pub.

    1. Joe W Silver badge
      Pint

      Re: Bugzilla - emailzilla

      Yeah, our Kanboard does that as well - serves me right for enabling the notifications. But I need those, when working remotely I cannot access the board, due to... (yeah, we all know those stories, I don't want to think about that now). So I check progress by looking at the incoming mails - and I can even add tasks by sending an email to the board :) (still cannot modify "my" tasks, but some colleague has to push them my way by putting me in as the person in charge... f'ing annoying, but works)

      Except our meeting is Wednesdays...

      icon ->>

      /me needs that. Now. /me been fighting with sharepointless *shudder* for some time.

  17. Anonymous Coward
    Anonymous Coward

    Not quite a 'reply all'...

    once had an urgent 'please delete email subject xxxxx sent dd/mm/yy hh:mm UNREAD' that was sent from on high after someone had managed to send a sensitive attachment to a large part of the company.

    The 'recall message' had already done the work for my email account, so I never got to see it, but I heard from others that the attachment detailed the CEO's pay and allowances

    (the CEO has since moved on but I assume there are still a few copies kept by people for potential blackmail)

    1. Antron Argaiv Silver badge
      Thumb Up

      Re: Not quite a 'reply all'...

      Nothing will get that email read more quickly...

      1. Version 1.0 Silver badge

        Re: Not quite a 'reply all'...

        These days the attachment is CEO_pay_details.docx.exe

      2. Anonymous Coward
        Anonymous Coward

        Re: Not quite a 'reply all'...

        Or when an email was sent out that shouldn't to just about everyone...it got recalled and deleted from Exchange but a follow up email saying don't read and just delete it was a reply all of the original message!

  18. Anonymous Coward
    Anonymous Coward

    Don't forget the NHS!

    We did it too. It was the first and only time my job was mentioned on Have I Got News For You.

    1. Anonymous Coward
      Anonymous Coward

      Re: Don't forget the NHS!

      Many of us are in jobs where a mention means something has gone wrong.

      We had a mention in thr press when a botched contract resulted in a public service getting a large bill reduction.

  19. disgruntled yank

    GroupWise

    I thought GroupWise was fine, but my recollection is that WordPerfect Office's email was something quite different. A government contractor where I worked used WP's email on Data General minis, then the successor contract used GroupWise with Solaris servers. I don't recall there being much similarity between the two.

  20. MarthaFarqhar

    Back in the days of Napster, Kazaa, students setting up their own workgroups in halls of residence and them not being isolated from the real campus network, there was a lot of network traffic purely just for evaluation of copyrighted material *ahem*.

    One bright spark decided to use our mail server to email his mp3 collection from his office computer to his own account. Via an old Sun SPARC running our mail service with a spool of around 70MB.

    People kept wondering why mail was patchy that day. Of course, the logs were being filled with disk full, and then once the non-delivery reports started to fill the logs, /var started to rapidly fill.

    Drastic action was needed. Short of using the scissors method of network disconnection, it was a small enough building to find the (ab)user and inform him how we applauded him on his understanding of the concept of attachment size limits, however, flooding the mail queues with multiple mails exceeding the mail queue for the department wasn't a good thing.

    He then opened Outlook Express (his choice of client). His mail queue was about a third of the capacity of the entire University mailqueue. At that point, we advised him it was probably quicker to invest time in learning about sftp or scp, or investing in a CD-RW.

    Reply all storms seem to be cyclical. Every so many years, these happen, until a personal attack/information disclosure gets sent to all, is memorised by everyone as an example, then people move on. New people move in. Wait about two-three years. Off we go again.

  21. logicalextreme
    Coffee/keyboard

    I've certainly DOSed my own inbox over the weekend on numerous occasions after either being pulled to the pub on a Friday afternoon while an every-sixty-seconds diagnostic script was still running, or by a monitoring script that I neglected to set a state flag on so got an email per $checkinterval of an outage (or several outages).

    1. Anonymous Coward
      Anonymous Coward

      I've been having (DSL) modem problems this week. When I finally got reconnected, my server sent me all the "I have a network problem" emails it had helpfully queued up. On top of that, my external IP address changed, of course, but it had been a while since the last change, and apparently the script that updates my domain name with the new IP had been updated and didn't work... which the server happily informed me of, via email and text message, once an hour until I could get around to fixing it.

      Watchdog scripts that send messages are a wonderfully useful thing, until an extended issue. I *REALLY* need to rewrite them to see if "now" is the first notice of an issue so tell me, or if it's been bad for a bit so don't tell me.

      1. logicalextreme

        That's something Nagios is really good at — telling you once per actual incident, and configurable as all get-out. It also has a concept of "Flapping" being a state, which is incredibly useful for not being hoist by your own notifications.

        1. Anonymous Coward
          Anonymous Coward

          I briefly considered Nagios, but it looked like a cannon-against-mosquito kind of solution. The "server" itself is only a "mini" computer (Zotac Zbox, about the same size as the router), and isn't running much - simple webserver for static pages, internal-network-only fileshare, etc.

  22. Cynic_999

    Cost me £millions

    Once my admin stopped sendmail and deleted the queue. As a consequence my reply to a Nigerian prince who wanted to send me £10 million was lost. Can I sue?

    1. Imhotep

      Re: Cost me £millions

      Too late. Last I heard the Nigerian prince had died of starvation, surrounded by mountains of cash he had been unable to give away.

      Tragic, really.

  23. Stuart Castle Silver badge

    I think I've posted this before, but can't find it. So, here goes..

    When I was a green young student, too many years ago to count, we had just been given access to the email system. The Uni ran a netware based network, with Pegasus Mail on the clients, and, I think, Mercury mail running on the server (Mercury is the server side software by the authors of Pegasus). We'd just discovered mailing lists, and had one called "All Users". So, I emailed a Happy Christmas message. Unfortunately, the setup defaulted to requesting read receipts, and I didn't disable the option.

    So, I sent that "Happy Christmas" email to 17,000+ students and likely well over 1,000 staff. I got read receipts from every one, which clogged up my mailbox for weeks (it seemed like no matter how main I deleted, more came back), then there was the abuse and complaints I got. While I was aware that not everyone celebrated Christmas, I was genuinely surprised that it could be considered offensive to send a goodwill message to someone.

    In terms of actions, the University did not take any real action against me. Yes, I got a telling off from the lab manager (who could be scary, and a bit of a BOFH but was actually a good bloke), but beyond that. the only action the Uni took (that I know of at lease) was to remove the All Users mailing list, and restrict who could send mails to others.

    1. Jou (Mxyzptlk) Silver badge

      Your University did the right thing. Trying to put the blame on a single person for such a trivial "exploit" doesn't help, and others will do the same.

  24. well meaning but ultimately self defeating

    And Godzuki too

    Yay email

  25. Jou (Mxyzptlk) Silver badge

    "Ever got The Call only to discover that somebody else was doing something silly?"

    That's part of the job. Getting blamed for shit others do too. You need some experience in how to handle the blame part, I usually avoid finger pointing.

  26. Imhotep

    Opportunity is fleeting

    "....the early part of the century"

    I would have gone for the early part of the millenium.

  27. herman

    Any half decent email system will only keep ONE copy of each unique email, so that send/reply to all will not bog the system down.

    1. A.P. Veening Silver badge

      True, but that is only a relatively recent thing due to email storms. Besides that, every reply to all will generate a new, unique email as every single one has a different sender (and the likelyhood of exactly identical content is also negligible).

    2. Jou (Mxyzptlk) Silver badge

      That is the theory, until you come across a case where it does not apply. Welcome to the real world.

  28. Colin Bain

    Close call

    https://thenewstack.io/dr-torq-rm-r-and-other-linux-command-line-gotchas/

    Lists an almost on-call using rm *

    I blush because I am a super user fixing things at the user end at my work which is non computer related and I had to look rm * up

    1. Caver_Dave Silver badge
      Angel

      Re: Close call

      If I didn't use a very dangerous command multiple times a day, then I would look it up every time I wanted to use it. If nothing else, it might suggest a less dangerous version to use.

  29. steviebuk Silver badge

    Ricoh

    I was on the print server having a look at some settings in Ricoh's Follow Me printing solution during a quiet time, way back in 2015 I think. Notice the "Purge print jobs once logged out" option. That would be useful for security. Ticked it, applied. I'm aware of change control, just wasn't thinking and as I was the only one that properly knew the system, I doubt the idea would of come up on why it wasn't a good idea to apply.

    All was quiet. Then I noticed calls coming in for failed print jobs. Jobs failing half way through etc.

    Then it hit me as I'd witnessed it many a time. The users swipe their cards and choose their print job and instead of waiting for it to finish printing, log out. That of course meant their job purged before it finished. Or their job was so long, the timeout would kick in and log them out (because they all kept leaving themselves logged in so had to put a timeout on the printer login).

    Oops. I grabbed all the calls, quickly turned the purge print jobs on logout option off and quietly closed all the calls. No one noticed so didn't get moaned at for not following request for change.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like