back to article What evil lurks within the data centre, and why is it DDoS-ing the ever-loving pants off us?

Welcome to another in The Register's series of stories from those receiving calls for help and slightly passive-aggressive helpdesk tickets. Start your Friday with a helping of On Call. Today's tale comes from "Jon", who found himself having to deal with the fallout from an ill-considered corporate emission. In the middle of …

  1. GlenP Silver badge

    TomTom Updates

    Not in house but...

    Not long in my current IT Manager role, and with an inherited 2MB leased line to the 'net we had similar issues with our entire connectivity grinding to a standstill. This included the RD connections for two factories so was a critical failure.

    Frantic sorting through traffic statistics and logs from the main Cisco device then checking round the building traced it to a TomTom device update. I figured out roughly what was happening, the SatNav would request an update packet from the servers at Akamai; if it didn't receive the response in a certain time it would assume packet loss and resend the request. The update server would happily service both packet requests though and with a slow connection neither would be received in time so the device would send another request for the same packet, slowing things down more; repeat, rinse and recycle ad infinitum. Once the device was disconnected and the multiple update packets had worked through the system everything returned to normal. I was able to then justify adding a backup ADSL line to use for any updating.

    The same user had a habit of leaving the Sky home page running on his computer, the video playing constantly also impacted on our traffic levels until I told him to stop. More recently we've had an issue with the standard IE* home page hitting the CPUs on the Parallels servers but a GPO change to force the homepages to an internal site cured that.

    *We have to use it for legacy apps.

    1. Hubert Cumberdale Silver badge

      Re: TomTom Updates

      Ah, the days when things were written specifically for IE. I remember having to manually tinker with pages to get them to work properly in it. Isn't it great now that every browser behaves in exactly the same way and conforms exactly to the same standards. Isn't it? Okay, well, maybe we're not quite there yet. But it's getting better.

      1. Ogi

        Re: TomTom Updates

        > Ah, the days when things were written specifically for IE. I remember having to manually tinker with pages to get them to work properly in it.

        Yeah, we are now in the days when things are written specifically for Google Chrome.

        Unfortunately if Chrome decides to do something unilaterally, and others don't follow, you just get breakage to which peoples most unhelpful responses are "Use Chrome". It is like we are going back to the days of "IE Only" websites, although a bit better because Chrome is at least cross platform, although more spyware infused then I remember IE being.

        1. Hubert Cumberdale Silver badge

          Re: TomTom Updates

          I continue to be surprised at the relatively low market share of Firefox – it's not perfect, but I much prefer it to Chrome (not least because of the spyware aspects you mention). But yes, I still have to keep Chrome around for those dumbass sites that don't work in FF.

          1. logicalextreme Silver badge

            Re: TomTom Updates

            Chrome's stranglehold basically put the final nail in the coffin for Presto and Opera. The whole "but websites look funny in Opera" thing back in the day was more often than not down to the websites not being coded to the standards, which Presto was pretty rigorous about and thus aced the Acid3 test. Chrome basically went and threw all of that out of the window by allowing any old shit to display nicely.

            I toyed with Firefox again for a while but adopted Vivaldi immediately when it landed, Chromium and all. I just want the thing that works these days.

          2. Doctor Syntax Silver badge

            Re: TomTom Updates

            "I still have to keep Chrome around for those dumbass sites that don't work in FF."

            I usually take the view that if they can't take the trouble to make sites that work properly the site itself isn' worth me taking the trouble to look at.

          3. Bruce Ordway

            >> market share of Firefox

            Firefox.... hmmn, they really lost me the way they switched over to Quantum.

            (I do still run FireFox 56 on a couple systems for some old extensions).

      2. Stuart Castle Silver badge

        Re: TomTom Updates

        I used to have a universal remote control that used software that required IE. I had previously had a Logitech Harmony universal remote control that had failed, so I decided to go for the competition. the new remote had a full colour display, and looked lovely. The one downside is that it had a crap management program that installed on Windows XP and used IE to provide it's awful UI,

        The Harmony had its' faults, but the software did at least run on a recent version of Windows, and did not require IE. After a few weeks of putting up with the competition, I bought a replacement harmony.

      3. The obvious

        Re: TomTom Updates (2025 update)

        “ the days when things were written specifically for Chrome.”

        Chrome: the new Internet Explorer.

        Meet the new boss, same as the old boss.

  2. JassMan Silver badge

    I haven't chuckled so loud in ages

    I feel guilty about the schadenfreude but I really enjoyed this one. I think that reading the Reg should be part of the employment contract for everyone who works in the industry, so that there would be less of these clangers. No wait, I want more because more laughter makes you live longer. Oh, I don't know or care as long as all the good stories appear here.

    1. logicalextreme Silver badge

      Re: I haven't chuckled so loud in ages

      I'd rather it wasn't in the employment contract because non-readership of El Reg or The Daily WTF have been excellent signifiers to me that I'm probably not dealing with somebody that's competent and/or has a sense of humour.

      1. The Oncoming Scorn Silver badge
        Holmes

        Re: I haven't chuckled so loud in ages

        Working things out.

        I used to put little things into my teams chatroom during quiet times or if a discussion was going on & going a little off topic (Me).

        One day I got a side message to the effect of :

        "Are you oblivious to what you do or do you like stirring the pot deliberately?"

        "Deliberately, I have never met anyone on this team with one exception, when I went to Fargo, I put in the comments in to see whose of a like mind, in humour, technical ability & wanting to get the job done, more importantly who I can fall back on & whose the company men".

        (OK some of the team had been on training together, before going to their hubs.

    2. bombastic bob Silver badge
      Devil

      Re: I haven't chuckled so loud in ages

      Yeah El Reg as "required reading" for I.T. (and related) staff...

      just 'On Call' and BOFH and the occasional IT security-related article would be enough for "basic require dreading", but I'm sure (like any other IT-related person) they'd be spending a significant period of time during the day looking at the rest of the site...

      1. Anonymous Coward
        Anonymous Coward

        Re: I haven't chuckled so loud in ages

        "basic require dreading"

        Now THERE's an apropos typo! Especially for reading BOFH.

  3. The commentard formerly known as Mister_C

    Makes sense both ways

    The way I misread it made more sense

    ... "although the developers refused to admit it at the time," Jon added. This was normal practise, but ...

    1. dak

      Re: Makes sense both ways

      Did they admit it ever, or was the awareness their culpability beaten into them with a clue stick?

      As is usual practice.

      1. Zippy´s Sausage Factory
        Facepalm

        Re: Makes sense both ways

        Speaking as a developer, I personally put my hand up as soon as I realise it's my bad.

        Mainly because I've noticed that, in job security terms, it tends to be a lot safer than the standard "I don't do anything / Well yes but but I didn't change anything / OK there was a change but it wasn't me / I mean yes I checked in the change, as per policy / And I did do a build, to make sure the build didn't break / Ah well yes now I did deploy it but.." sequence.

        1. Zippy´s Sausage Factory
          Headmaster

          Re: Makes sense both ways

          *"didn't do anything".

          Wish I'd spotted that sooner.

          1. Control Phreak
            Trollface

            Re: Makes sense both ways

            Although the original statement also wouldn't be great for your job security.

    2. bombastic bob Silver badge
      Meh

      Re: Makes sense both ways

      checking for updates every 4 hours, though - that's ALREADY ridiculous (nevermind the DDoS'ing 4 minute setting)

      How about every 4 weeks? 4 months? 4 YEARS? or, my favorite: when _I_ want to check.

  4. JeffB
    Pint

    Update (mis)-scheduling

    I used to work on a helpdesk at one of the large outsourcing companies, there were a number of teams ranged across a large open plan office, each team servicing a number of clients. There were a number fo sites across the country, so their internal IT support was centrally managed (in India). Microsoft roll out their major patches on a Tuesday (good ol' patch Tuesday...) and there's a reason for this that was obviously slightly lost on out internal IT support.

    They were in the habit of downloading the patch Tuesday updates, running their own tests, then uploading them to WSUS and timed to go out over the weekend, all sounds fine and dandy. However, in the UK we were taught to turn our computers off over the weekend to save power, so we all came in on Monday morning, the busiest day of the week, and turned our computers on. Needless to say, within half an hour, when the phone lines were red hot, you could hear a wave of curses spreading across the floor as people's computers started rebooting to install the updates, putting the entire helpdesk operation into meltdown.

    After about a year of this mayhem the Indian boys finally got a rap across the knuckles and re-timed the WSUS release for 2pm UK time on a Friday

    Because that's beer o clock, innit??

    1. Anonymous Custard Silver badge
      Facepalm

      Re: Update (mis)-scheduling

      Similar experience here (this time from the userland perspective) back in the day where our lot used to schedule a full monthly anti-virus scan in the same way.

      Problem was, that due to the *ahem* spec of our laptops and how things were configured, the AV software basically grabbed as much CPU capacity as possible and everything ground to a halt whilst it trawled through the hard discs of each machine poking its nose in everywhere.

      Most people had reasonably large drives (for the time, we're going back a bit here) and all were spinning rust, so it wasn't unusual for the scans to take all morning, and on some where the disc was majorly full the whole day. And whilst it was going on, basically all you had to work with was the phone and the coffee mug...

      1. Antonius_Prime
        Devil

        Re: Update (mis)-scheduling

        "And whilst it was going on, basically all you had to work with was the phone and the coffee mug..."

        If that occurred more than twice to me, my phone would... [checks BOFH excuse server] ...suffer an Unreportable Transmission Override Warning... and be unusable for the day...

        Modern.Problems.Dave.Chappelle.gif

        1. Anonymous Custard Silver badge

          Re: Update (mis)-scheduling

          It didn't last too long, as senior manglement weren't too impressed by more or less the whole building having an enforced four hour coffee-break on the first Monday morning of every month.

          A new AV product was fairly rapidly procured, and this one more carefully configured not to gorge on processor cycles.

          We humble minions on the other hand generally put on a stoic face and got busy doing nothing except coffee and gossip about the weekend/football/telly/whatever...

          1. John Brown (no body) Silver badge

            Re: Update (mis)-scheduling

            "We humble minions on the other hand generally put on a stoic face and got busy doing nothing except coffee and gossip about the weekend/football/telly/whatever..."

            That sounds like an excellent team building exercise. Far better than anything [dis]organised by HR.

            1. A.P. Veening Silver badge

              Re: Update (mis)-scheduling

              That sounds like an excellent team building exercise. Far better than anything [dis]organised by HR.

              The only team building exercise that works even better is the defenestration of HR.

          2. Anonymous Coward
            Anonymous Coward

            Re: Update (mis)-scheduling

            Yep, blame the product and just have the same bastards configure the new product differently...

      2. shedied

        Re: Update (mis)-scheduling

        It was around that time that you found out that you had your own version of the bottomless cup of coffee

      3. Jou (Mxyzptlk) Bronze badge

        Re: Update (mis)-scheduling

        I meet that type of "scheduled scan, high priority, every file, and every packed file nested-endless unpacked and scanned" from time to time. I usually convince them quickly to change that to a reasonable setting, since I count the hours of people not being able to work, and multiply it with the number of working people to show how expensive such a little setting can be.

        Not that I take into account that all those people are actually working, not only those who complain, thus ignoring those who don't even notice a difference. A type of employee every company has.

    2. bombastic bob Silver badge
      Pint

      Re: Update (mis)-scheduling

      beer-o-clock time (for updates), assuming you left your computer on during the liquid lunch and came back sober enough to shut it down after the updates had already installed...

      Or, you were too intoxicated to come back, and just left it on over the weekend.

      For that reason, I'd suggest noon UK time. It'd give you a chance to get out of the office earlier, because "ooh, my computer needs to update. I'll just start that now..." followed by "Hey Phil, you want to go to 'lunch' a bit early today? yeah, the computers need to update.... better invite the rest of the company, too, it could take an hour or three!" "Oh, it's on YOU? Thanks, Boss!"

  5. Anonymous Coward
    Anonymous Coward

    SMTP ddos

    A long time before, in a global 50 000 employees company, I had a LOT of issues with the GDC email system.

    Turned up I had shitloads of SMTP connections, in the thousands, while the site was only 200 users plus apps.

    I deployed SMTP blacklisting and things went OK.

    And then, this indian dude came to my desk asking why his deployed app on 10 000 desktops worldwide was no longer able to send emails !

    I asked: "have you by any chance hard-coded our local GDC email relay instead of all 50 company's relays ?". There was a blink. Of course he had. Without asking us.

    Turned up he was not even able to treat error conditions on SMTP ... Idiot.

    1. Jamesit

      Re: SMTP ddos

      "And then, this indian dude came to my desk asking why his deployed app on 10 000 desktops worldwide was no longer able to send emails !"

      Does the dudes race have anything to do with the story? If not why mention it?

      It makes you sound racist.

      1. A.P. Veening Silver badge

        Re: SMTP ddos

        Indian is not a race, it is a country of origin, so it can't be racist.

        And yes, there is a bit of confirmation of a cliché in there, but clichés become clichés just by their frequency of occurrence.

        1. Mooseman Silver badge

          Re: SMTP ddos

          "Indian is not a race, it is a country of origin, so it can't be racist."

          Without getting into the whole "ooh that's racist" about your comment, saying "x isnt a race so it cant be racist" is the number one go-to excuse for, erm, racists. I'm not saying you are, just suggesting you're careful.

          1. EagleZ28

            Re: SMTP ddos

            "__________ is the number one go-to excuse for racists."

            How does the person ORIGINALLY asserting this claim (probably someone at some Uni)

            know that the person using the "excuse" is a racist?

            Telepathy? Confession?

            1. Francis Boyle Silver badge

              Re: SMTP ddos

              Because the person making the statement general follows it up with a shitload of other obviously racist statements. #evidence – how does it work

          2. A.P. Veening Silver badge

            Re: SMTP ddos

            Frankly, I don't think I am racist as I judge every single person on his or her own merits (and lack of them), but I am well and truly fed up with extremely melanin deficient woke idiots pulling the racist card on behalf of melanin challenged persons who don't give a fart about it.

        2. BrownishMonstr Bronze badge

          Re: SMTP ddos

          How do you know he didn't mean the chap's ethnicity? He could have very well be born in the UK.

          Most people refer to ethnicity as "race", and "Indian" would generally refer to people from the Indian sub-continent. So depending on what was meant, "Indian" could refer to the race.

      2. Doctor Syntax Silver badge

        Re: SMTP ddos

        I think it relates more to business practices and maybe training practices in India.

        Back in the day my then client did a good amount of work with one of the Usual Suspects. Like many at the time and, no doubt, much later the Usual Suspect subbed all development out to one of the Indian Usual Suspects who would - I think for visa reasons - rotate staff from India (or Indian staff if you're prepared to tolerate the adjectival form) through their UK office. These ranged from great* to just out of some training establishment. Needless to say it was the latter who got thrown into the deep end of actual coding. The consequence was periodic bouts of receiving not-quite XML files and having to explain to one of these staff-newly-arrived-from-India (and presumably just out of some training establishment there) how to get names such as O'Neil into well-formed XML.

        So the fact that the dude was Indian speaks volumes about the general business environment.

        * And a distinct improvement on the initial definitely not Indian "consultant" who initially arrived to brief us about one project.

      3. low_resolution_foxxes Bronze badge

        Re: SMTP ddos

        Groan.

        If you're going to make a scene and pull the whole "Racist!" card, it does help if you say it to people who actually say derogatory things. Rather than just implied subconscious bias. Otherwise, you end up looking like an asshat.

      4. John Doe 12

        Re: SMTP ddos

        Funny thing is I find Indians tend to be very tech savvy - at least the ones that are in the I.T. industry. But I do get the point being made that the guy's race is a bit too prominent in the original post - even though I admit I would most likely use such language in conversation. I guess some things look worse when written down and then can be read more than once.

        1. Anonymous Coward
          Anonymous Coward

          Re: SMTP ddos

          "Funny thing is I find Indians tend to be very tech savvy - at least the ones that are in the I.T. industry."

          Working a lot with offshore I think the competence of Indian IT varies just as widely as here in Blighty. However there is a certain class of dev I've only seen in India - the dev who can develop a whole heap of software and use advanced concepts or features in frameworks, but they have no clue about computer science basics. It freaks me out when I have to explain how computers work to people who are able to churn out reams of code.

      5. Anonymous Coward
        Anonymous Coward

        Re: SMTP ddos

        It makes me shudder every time I call a company, especially tech support, and get someone with an Indian accent. I don't have a problem with people who live in India, are from India, are descended from people from India, etc. - it's that calling a US company and getting someone with an Indian accent nearly always means "outsourced call center", which typically also includes "lowest bidder", and I've been trained Pavlov-style to expect to get someone who doesn't know what they're talking about and will waste an hour of my life before transferring me to someone who can handle the call in 5 minutes. They very rarely disappoint.

      6. Jou (Mxyzptlk) Bronze badge

        Re: SMTP ddos

        Would you have complained that loud if he'd written "this UK dude" or "this US dude"? Probably not.

        His choice of words might be clumsy, but it was the truth. And nobody would have complained about "this UK dude", because by PC-Principal, you have to reflect your intolerance onto others thinking they would be even more as offended as you are, without actually knowing what they think.

        1. Anonymous Coward
          Anonymous Coward

          Re: SMTP ddos

          Hi there,

          Original AC OP of this thread here, I'm normally not much commenting, but from the volume of comments, allow me to do it.

          Of course, I'm not racist, I still have great indian colleagues in my linkedin profile ! I met all of them multiple times and we had a blast, personal and professionnal.

          Second, the indian IT staff market is split in 2 parts, and this is not coming from myself but ... from an indian colleague:

          - the brilliant people, that went to university, understand things very quick, will build and run any IT services

          - the people who purchased their certifications, dozens of them, on whatever technology. Like my indian colleague told me, this is systemic in India, unlike other countries. I once met a dude with 10 Cisco certifications who was unable to put up an IP interface on a router !

          As I said, it's always great to work with the first part of the population, but clearly, on the original story, the dude in question was from the second group !

          1. Jou (Mxyzptlk) Bronze badge
            Pint

            Re: SMTP ddos

            Upvote from me!

  6. Admiral Grace Hopper Silver badge
    Mushroom

    Kaboom

    Let she who has never shot herself in the foot fire the first bullet. There is a proud tradition of auto-mutilation in IT.

    1. Antron Argaiv Silver badge
      Thumb Up

      Re: Kaboom

      Update the code remotely, do network reset, try to log in again and discover....your code isn't as good as you thought it was.

      Cue long drive at night, out to the tower site to fix things.

      1. stiine Silver badge

        Re: Kaboom

        This is why backdoors are so prevalent, yet so prohibitted.

  7. Dave K Silver badge

    Updates are important!

    A good one this week, and yet another tale of developers believing that the latest patches are so unbelievably great that everyone must have them immediately. Obviously 4 hours is far too long to wait for the new shiny-shiny, must... patch... now!!!

    1. JeffB
      Facepalm

      Re: Updates are important!

      Or of developers believing everyone has the same shiny new kit that they do...

      Developing complex teaching materials on a 24" monitor, then wondering why students are struggling to make sense of them on an old 17" monitor

      1. G.Y.

        Re: Updates are important!

        or developing on a fast internal network, in the days were users lived by 56K modems if they were lucky

        1. EagleZ28

          Re: Updates are important!

          LOL! My local land-line is doing well... REALLY well... to get 19.2K...

          and that, by the way, is Kb not KB.

    2. John Brown (no body) Silver badge

      Re: Updates are important!

      "Obviously 4 hours is far too long to wait for the new shiny-shiny, must... patch... now!!!"

      That jumped out at me too. WTF? Checking for updates, licencing etc every 4 hours? And a bastardised version of Chrome to do it?

      1. Richard 12 Silver badge

        Re: Updates are important!

        My money is on it being an Electron so-called app.

        Which makes this on-call pretty recent.

    3. Doctor Syntax Silver badge

      Re: Updates are important!

      "yet another tale of developers believing that the latest patches are so unbelievably great"

      Or possibly doing what they'd been told to do by some mangler who insisted they needed this level of checking right now.

  8. tip pc Silver badge

    Why is the IT manager deploying HA Proxy?

    "Jon was the IT manager of the whole show and responsible for all the gizmos from the CEO's mobile phone to the receptionist's desktop and everything in between."

    "In the end, "I spent a good few hours spinning up new VMs to have all traffic route through a layer of Linux servers running HAProxy." He was able to carve out the application manager traffic and therefore allow a small percentage of requests to succeed and the fixed code gradually rolled out."

    He could of just rate limited the inbound traffic to a lower amount or perhaps limited the number of sessions each server could muster, both likely would have been quicker and cheaper than spinning up new gear and deploying extra stuff, assuming the actual bandwidth they where handling required proper network gear that could actually do rate limiting etc.

    I'd be really worried if my IT manager started installing & spinning stuff up.

    1. GlenP Silver badge

      Re: Why is the IT manager deploying HA Proxy?

      I'm an IT Manager and I do things like that, although we're a medium sized company in turnover terms we're low staffing levels so it's a small department. If he was the most experienced person there why shouldn't he carry out the necessary fixes. Better than the sort of manager who is clueless and just stands there shouting.

      1. Anonymous Coward
        Anonymous Coward

        Re: Why is the IT manager deploying HA Proxy?

        This doesn't only happen on medium or small sized companies. My position at -ahem- a B*g B**e company has "manager" plastered all over my signature, but the only things I get to manage are automation scripts, with zero human resources and a shit ton of coding/scripting. The Powers at Being™ wanted to have someone capable but at the same time reassure the customer the work is being -ahem again- managed the right way, with the minimum resource investment.

      2. This post has been deleted by its author

    2. MatthewSt Silver badge

      Re: Why is the IT manager deploying HA Proxy?

      If a job's worth doing, it's worth doing right!

      I'd be worried if my IT manager didn't at least know how to spin stuff up, even if they delegated it most of the time.

    3. doublelayer Silver badge

      Re: Why is the IT manager deploying HA Proxy?

      The culture of having managers and nonmanagers, where managers are defined as people who don't know how to do any of the things that need doing but direct people to do so is harmful. A good manager understands what the people they're managing do, and they understand how to do those things in a pinch. The team members themselves may be better at doing it, but if the manager doesn't know, they're not competent. In addition, the word manager can be applied to people who manage things other than people; it's pretty generic.

  9. ColinPa

    Football team web site throttles business

    I heard of a company where they found 10% of the traffic was from the local football club's web site. The football web site was sending traffic to web browsers every few seconds/minutes(I forget), even though the browser was minimised, or hidden behind real work. A quiet word with the club, and they fixed it so it only sent data when the window was active

    1. Pete B

      Re: Football team web site throttles business

      You're blaming the wrong party there - need to take it up with the idiot users who leave pages open when they're not doing anything with them.

      1. MatthewSt Silver badge

        Re: Football team web site throttles business

        Yes, because if there's one thing I've learnt as a dev, it's that users can be trusted to use anything you've written correctly! What's the point of having tabbed browsing if, as a user, I'm meant to close them as soon as I flick to another page?

        With a few rare exceptions, it's not the user's fault for how they use your website.

        1. Anonymous Coward
          Anonymous Coward

          Re: Football team web site throttles business

          This, most emphatically, this. Chrome and Firefox have decided that no-one needs accurate data on any tab that's not being displayed full time, Where I used to be able to have a dozen tabs open actively working, I now have to use my 2nd monitor as a scratch pad so that each process can run in the only tab in the browser window...fuckers.

      2. Zippy´s Sausage Factory

        Re: Football team web site throttles business

        Actually if anything, that's a good way of dealing with it. The football club gets good feedback and the users stay happy. Instead of the ban hammer being deployed and smashing off a chunk of worker morale on its way down.

      3. doublelayer Silver badge

        Re: Football team web site throttles business

        "need to take it up with the idiot users who leave pages open when they're not doing anything with them."

        That's not a good approach. Why shouldn't most pages be left on? They're documents. Mostly static documents. If I want to have them open, it should do little harm other than using some memory. Same thing is true of most nonstatic pages, because if they need to update, they probably don't need to thrash the CPU for it. After all, there are tons of other apps and applets whose entire job is to remain in the background and update information so it's always available; they seem fine at it. A few pages may do something else, like livestreaming video, but even those should be coded to stop streaming when they're not in focus. Leaving a properly-written livestreaming page open in another tab is simply a better way of returning to it when needed. None of these things are unusual, and users shouldn't be blamed for not knowing that a bad developer has made a site that wastes CPU time or network bandwidth.

  10. Bronek Kozicki

    "deploy HAProxy in front of all web traffic"

    It's a good lesson to have learned.

  11. deadcow

    ISP DDOSes self.

    I used to work for a major ISP. We had development teams working across several different departments. One morning I came into work, booted up my VM and noticed it was super chuggy, eventually hanging completely. I had a poke around going on, to find out the site was making thousands of requests to the server - specifically requesting a timestamp. "That's odd", I thought and sent out a call asking if anybody knew what this timestamp request was to see if I could find out what was happening.

    It turns out the marketing department, in an effort to create the most granular tracking I have ever seen in my life, had decided that they wanted to know exactly what time users were clicking on interactive elements on the website. Note this was any element on every page of the site: links, accordions, show/hide buttons, popups, everything. Now - they had also decided that they required this with such extreme precision that they didn't want to rely on the user's own system time - they wanted it synchronized with the server's timestamp. So they set up a script that pinged the server for its current timestamp every single time a user clicked on anything. I watched with growing horror as I started to repeatedly open and close an accordion on the homepage, every single click resulting in a server call.

    I enjoyed raising that P1 to a red-faced development team. We also had a long chat with the marketing department about DDOS-ing our own website in order to collect completely useless user data.

    1. Anonymous Coward
      Anonymous Coward

      Re: ISP DDOSes self.

      they didn't want to rely on the user's own system time - they wanted it synchronized with the server's timestamp

      Hells, hells, hells! You've just triggered my PTSD.

      [AC for obvs]

  12. diver_dave

    That said

    I've been headscratching with an old Meridian switch before.

    Getting multiple capacity drop outs that gave every indication of a card fault. Very very intermittent...

    So after hours myself and Big Dave from BT have the panels off and start testing. Nothing. Anywhere.

    Fault keeps reoccurring on and off.

    Eventual diagnosis and all round circular arse kicking.

    We had two sites. North and South.

    One team split across the two sites. North forwarding phones to South hunt group. And.. Yep Vice versa.

    Telephony equivalent of the old mail out of office auto response.

    Nightmare to chase down. Only found it after some very careful traffic analysis.

    Pint o'clock.

    Dave

    1. Anonymous Coward
      Anonymous Coward

      Re: That said

      ahhh the good 'ol mail loop. I started work at a Lab back in 98 and one of the first sh1t storms we had was caused my a mail loop. We ran Netware 4.11 and Groupwise 4.X (on the same box) Oh the fun we had when a boffin left and forwarded his email to his uni account which in turn forwarded stuff back to his lab account. The server sh1t itself trying to cope with all his email, which as groupwise was on the same box as Netware meant users couldn't login etc, utter sh1t show!

      1. diver_dave

        Re: That said

        Indeed.

        Strange as this was behaviour that technically shouldn't have been possible.

        I've always sworn the damb thing was Maliciously sentient!

      2. This post has been deleted by its author

      3. dak
        Headmaster

        Re: That said

        Please use the correct past tense form - "sh@t itself".

        Thank you.

        1. John Arthur
          Joke

          Re: That said

          Shirely the final 't' is redundant. Sh@ should be sufficient!

      4. PPK
        Alert

        Re: That said

        Recipe for disaster:

        - Late 90's

        - Bonded ISDN - mighty 128kbps connection!

        - Lotus Notes

        - Manager sets OOO email then sends out a message to let customers know

        - Customer has OOO enabled....

        A few thousand emails later I went down and pulled the ISDN cable out of the wall. Job done!

        1. diver_dave

          Re: That said

          IIRC Jones Cable /VM disconnected an office block in centre London.

          By basically driving a JCB through the incomer shed.

        2. Anonymous Coward
          Anonymous Coward

          Re: That said

          Back around then I managed a Lotus Notes/Domino network across roughly 15 European offices ... cue hilarity ... done? OK.

          So while LND had it's detractors for our company (pre-proper-internet) it was a revolution in communication of service/sales information. However, inherent in Lotus' design was the principle of infinite bandwidth. So as our network expanded and use expanded it just swamped our shared international links and ISDN links. Many times I had to placate office managers for extreme ISDN bills and we were constantly at odds with IT over network use. So much so that we installed bandwidth limiting software on the main replication hub.

          Eventually one of my guys moved to IT, set up a mesh network of our internet connections and our traffic was migrated. Everyone happy yes? No, whenever there was any network outage or slow down LND got the blame ... even when the bloody thing had been deprecated for years.

          Anon cos that rep f'kin follows you man.

    2. swm Silver badge

      Re: That said

      On the old telephone 4A crossbar office (a telephone router with no subscribers) they had metal cards to control the routing of calls in and out of the exchange. They once had incorrectly punched some metal cards and caused a routing loop that tied up all of the lines from Chicago to Milwaukee.

    3. Trevor Everything is Trevor

      Re: That said

      I remember that well with Meridian switches, 2 companies, 2 meridian switches, private PRi connection between both. We used to call it tromboning, user in company A, forwards to company B, who send it back to company A. Result complete collapse of the phone system.

  13. Anonymous Coward
    Anonymous Coward

    That fix seems a bit extravagant.

    I think a quicker and easier solution would be to have the router or the webserver's iptables drop all traffic except for a manageable amount, and replace the code that checked for an update with a static document saying "yes, this fixed version here: " so apache* could serve it instantly.

    Then expand the allowed traffic a block at a time until everything is allowed.

    Also fixing the apache config to not allow too many simultaneous connections would help.

    * or whatever

  14. Stevie Silver badge

    Bah!

    Why didn't this "Jon" simply reboot the internet and flush out all the ddosses and 4chans?

    1. The Oncoming Scorn Silver badge
      Pint

      Re: Bah!

      Jen was holding it at the time & demonstrating it at Reynholm Industries to upper management.

      So Roy couldn't try turning it off & on again.

  15. Stuart Castle Silver badge

    Ah DDOS..

    I've probably said this, but..

    A few years ago, i worked in a University computer lab. Every so often, we arranged open days where school kids could come in and sample some of our tutorials..

    One of the lecturers was teaching computer security. She hadn't notified us technicians of what she was doing, so about an hour and a half after the open day started, we were confused when the network suddenly slowed and ultimately hung. By this time, the other lecturers in the lab were starting to complain, and we agreed with the lecturer in charge of the open day (not the one teaching security) that we would sort the problem and she sent everyone else out for lunch.

    After an hour, we discovered the problem. One of the kids had logged into a few PCs and was using them to run a DOS attack on the switch connecting the lab to the rest of the network. After we'd discovered this, and traced the PCs concerned, we approached the lecturer teaching security. She apologised. Apparently, she had told the students not to run DOS attacks on anything but the machine she'd set aside for that purpose, because, of course students always do everything you ask them to..

  16. Claptrap314 Silver badge

    What was that DDOS protection doing again?

    Seriously, would it not have been simpler and faster to contact them and say, "We've got a thundering heard. We need you to block 99% of the traffic with this user agent.

    And let the fix change the user agent stream. When the herd thins enough, you can drop the block percentage.

    Of course now, with SRE, the rollout is paused by the system at the 15% level because of the increased network traffic driving up system load.

  17. NogginTheNog
    Devil

    Devs

    And THIS is why developers are the spawn of Satan, worse than any other class of 'users'!

  18. hmv

    "the developers refused to admit it at the time"

    Do they all do that?

  19. Archie Campbell

    4B or not 4B

    I got an AS in statistics.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021