The Register Home Page

back to article Google engineer unplugged every fiber they could see and – surprise! – took down a chunk of the G-Cloud

Google Cloud has revealed the embarrassing reason for a partial outage at its us-central1-b region this week: An engineer unplugged it. The incident spanned 07:41 to 11:52 PT on September 1, and saw a portion of the G-Cloud region experience “severe network degradation and resource isolation.” “Traffic flow drop rates for …

  1. simonlb Silver badge
    FAIL

    Procedural Error?

    Having an engineer unplugging active fibre cables in a datacentre that are handling live internet traffic without a second set of eyes to confirm each action is pretty reckless. I would also expect whoever was planning the work to have triple checked just which cables were to be unplugged due to the potential for mishaps fuckups like this.

    Don't be evil clueless.

    1. Jou (Mxyzptlk) Silver badge

      Re: Procedural Error?

      > Don't be evil clueless.

      And I thought it was "Do the right thing" since 2018...

      1. EdSaxby

        Re: Procedural Error?

        AI wrote the detailed procedure?

      2. RM Myers Silver badge
        Facepalm

        Do the rght thing

        Evidently the engineer thought the motto was "Undo the right thing, and while you're at it, the left thing also".

    2. blu3b3rry Silver badge
      Facepalm

      Re: Procedural Error?

      Sounds like someone's been taking pages out of Elon Musk's style of datacentre management

      1. Anonymous Coward
        Anonymous Coward

        Re: Procedural Error?

        I wonder what Alex from Uzbekistan is doing now. Hopefully ICE haven't caught up with him.

      2. DJV Silver badge

        Re: Elon Musk's style of datacentre management

        Sounds like right out of the Stockton Rush school of safety management.

        1. Anonymous Coward
          Anonymous Coward

          Re: Elon Musk's style of datacentre management

          Only if he was there to control backups with a logitech controller.

      3. that one in the corner Silver badge

        Re: Procedural Error?

        > out of Elon Musk's style of datacentre management

        My favourite quote from that is

        >> “This is making my brain hurt,” he said.

        The Musk genius is stumped by simple numbers about how one DC is physically different to another DC.

        1. This post has been deleted by its author

    3. Filippo Silver badge

      Re: Procedural Error?

      Grown-up companies check everything, because it's how you prevent fuckups.

      Unfortunately, when they grow into publicly-traded megacorps, they stop checking everything, because the stockholders always want more more more, and why do you need to have two engineers triple-check something that never fucks up anyway?

      Just have one (junior) engineer check it once. It saves money, and you risk nothing, because it never fucked up before. Right?

      It's the same kind of reasoning that leads to people stop getting vaccinated against X, because nobody has ever even heard anyone getting X in decades, so what's the point?

      I know they teach cause-and-effect in school, but maybe they should teach it harder.

      1. james 68

        Re: Procedural Error?

        It almost certainly wasn't an actual Google engineer.

        From experience they rely on contractors, some of whom are very good, others... not so much. It boggles the mind that they don't even use local (and by local I mean based in the same country as the data centre) contractors. Here in Japan for example a UK company has the contract for all their cabling, contractors are flown out almost daily and put up in accommodation at great expense just to run a few pre-made fibres, then before they even leave to go home another contractor is flown out to put some more fibre in the same damn data hall, but because it's a separate work order a separate contractor is required even though the first one is still onsite. It's ridiculous, but if you've got money to burn I guess you do ridiculous things.

        The actual Google engineers show up once in a blue moon, watch the contractor for five minutes and then wander off to God knows where, and nobody has a clue what anyone else is doing.

        I'm not bitter and jaded... honest.

        Seriously, worst way to run a company I have ever seen, hemorrhaging money for no good reason.

        1. Anonymous Coward
          Anonymous Coward

          Re: Procedural Error?

          It won't have been a contractor at the DC. Way too expensive.

          If it had been, I doubt the same problem would have occurred...most contractors are highly experienced. You get this kind of error with junior hiring and no shadowing (because there is nobody to shadow).

          We're probably all guilty of a junior cock up like this because of lack of experience / senior person giving vague instructions.

          For example, when I was a budding junior, in my late teens a very, very long time ago...I was shadowing a guy on maintenance visit, a very good engineer, a bit slow and methodical to the point of tooth pulling agony, but very good none the less...I'm more senior now than he was then and I'm still nowhere near that slow, he reeked of fear...disk space was low on the file server, so he asked me to free up some space. I'd never been there before, didn't know how the business operated...hadn't a clue about the file/folder structure etc...I found the biggest folder, organised by date and deleted everything over 12 years old. I free up nearly 200GB of space. I was very pleased...I chose 10 years because I knew there was no legal requirement to keep things older than 10 years old in most cases...I added on 2 years for safety...that's junior thinking.

          Turns out I'd deleted half of a long term archive containing old firmware, drivers, source code etc etc for old products of theirs...luckily it was backed up and could be recovered.

          Both myself and and the senior learned an important lesson that day. I learned to ask questions, he learned to be more specific.

          In at least the last 10 years, could be longer because I haven't really thought about it...I haven't seen many truly senior engineering types in any flavour of tech in permanent roles in any company ranging from tiny sub 100 employee firms all the way up to massive corporates like O2...on pretty much every project I've participated as a contractor, I've either been the only contractor there and the most senior in terms of experience and knowledge, or I've been one of a few contractors with a team of relative internal juniors / mid-career (wish I was freelance) type of folks.

          On your "hemorrhaging money for no good reason" comment.

          They actually aren't with contractors. Yes they charge a lot per day, but they probably aren't working on that contract every day. A £500 a day contractor probably costs less than a £160k a year permie. A contractor you generally only pay a small retainer, plus a day rate when you use them. They aren't being paid for every day they hold the contract...that's not how it works. Some people can swing that, if their rate is low enough...I do, I typically work on a retainer only that lands somewhere in the middle...because I can't be bothered with complicated billing and I prefer not to negotiate on every single piece of work that comes down the pipeline...I prefer to just crack on...you get more interesting work that way without it being tendered to other contractors etc...I am way cheaper than a permie and most other contractors...and I come with 25 years plus of experience and a skill set that would make a lot of engineers choke...it's broad but also very deep in almost every area that matters because I've used almost every skill extensively bouncing around hundreds and hundreds of different businesses for over a decade.

          You get incredible value for money out of a contractor, because you're getting applied knowledge and the experience of working with tons of different companies of all shapes and sizes. An institutionalised permie that became senior rising through the ranks of one business over a decade is no substitute for a good contractor. The depth and range of experience is beyond comparison for the same scale.

          1. james 68

            Re: Procedural Error?

            You missed my point. the hemorrhaging money thing comes from hiring from half the world away, they're not just paying for the skills of the contractor, they're paying for flights both ways and accommodation when there are perfectly good contractors based IN the relevant country.

            1. Softsuits
              Alien

              About those procs and policies

              Working an IBM 360 with a stack a punch cards. The documentation was a Gorilla proof book flowchart type trouble shooting guide. Pretty visual. Knocking a Cert Server non redudant outside the observable universe out because pokey couldn't figure out the correct connetor is ludicrous. This is AI practicing Lean Sig Sigma. It looks like the great simplification is occurring as we speak.

        2. MachDiamond Silver badge

          Re: Procedural Error?

          "then before they even leave to go home another contractor is flown out to put some more fibre in the same damn data hall, but because it's a separate work order a separate contractor is required even though the first one is still onsite. It's ridiculous, but if you've got money to burn I guess you do ridiculous things."

          What boggles the mind is that some exec sees this outsourcing as a cost saving measure over having in-house staff. Yes, there may not be enough things for them to do all of the time, but they are much less expensive to have around than flying people in from other countries.

          With in-house staff, there can be requirements that things are done in a particular way, work is documented, the staff has been thoroughly vetted and those people have some skin in the game. I've been able to bring in mates when a job needed more bodies as contractors for things like running cables or shifting heavy items for a couple of days. I got them a fair wage, but it wasn't hundreds/hour plus travel and expenses.

        3. Potty Professor
          Flame

          Re: Procedural Error?

          Makes my blood boil when these monkeys are referred to as "Engineers".

          Have any of them actually ever spent several years at University studying for an actual degree in any branch of Engineering? I would surmise not, so they are not entitled to be called "Engineers".

          In some countries, Germany for example, it is illegal to use the term "Engineer" unless you have a degree or at least a diploma in your field, but over here, and, it would seem, the left side of the pond, anyone who gets their hands dirty occasionally is entitled to use that moniker.

          I have complained to the Advertising Standards Agency about this, but they just shrugged their collective shoulders and said that it had become accepted usage.

          Grrrrr!

    4. breakfast Silver badge
      Facepalm

      Re: Procedural Error?

      I hope that we'll get to see the Who Me? story on this at some point in the future.

      1. james 68

        Re: Procedural Error?

        I can't even name my employer (former) because I've got NDA's up the ass, but... let's just say that you can get a kind of kids spaghetti with a similar name.

        In Japan:

        Sudlows (great bunch of guys TBH, hanging out with them was fun) have the contract for Google's cabling in Japan.

        DB Schenker have the contract for hardware (this includes plugging in the cables that Sudlows installs).

        Google engineers just sign off on it and do software.

        1. Maximus Decimus Meridius
          Trollface

          Re: Procedural Error?

          "you can get a kind of kids spaghetti with a similar name"

          Would that be the type that Dave Lister can do amazing things with to Rimmer's mum?

          1. james 68

            Re: Procedural Error?

            Oh if only polymorphs were real... I could be a much saner bloke with a few negative emotions sucked out of me.

            Rimmer would indeed be very unhappy to be employed by them, both for naming reasons and for a completely illogical work environment.

            1. ShadowDragon8685

              Re: Procedural Error?

              Let's be fair, Rimmer's going to be unhappy pretty much anywhere short of going back to Wax World and getting to order armies of Waxdroids around all day and give them the most asinine orders that will get them all killed.

              At least until they mutiny and rise up...

              1. james 68

                Re: Procedural Error?

                He might enjoy working there after all then, so long as he could land a management position.

    5. cmdrklarg

      Re: Procedural Error?

      Probably an intern who was given dodgy instructions to unplug some cables.

      1. Anonymous Coward
        Anonymous Coward

        Re: Procedural Error?

        Or an internal techie unplugging a load of shit to get *his* job done knowing that a contractor would be sent in to fix his mess if he made one on someone elses kit.

        This is what happens when you compartmentalise everything. People stop giving a shit about other peoples stuff.

    6. Anonymous Coward
      Anonymous Coward

      Re: Procedural Error?

      They probably sent a junior, most senior techs are contractors now and used sparingly.

    7. theblackhand

      Re: Procedural Error?

      My assumption is that the "procedural error" related to the location of the equipment rather than the work that was carried out. I.e. data hall V/row W/rack X/rack unit Y/device Z had a mismatch between the data centre techs orders and the network engineers orders.

      While I maybe giving Google too much credit, I would assume that techs need some form of hot work approval to disconnect a device and that has been approved by multiple parties and that equipment location details are very clear.

      So by the time an engineer is about to remove cables, there have already been multiple failures (paths and routing sessions disabled in preparation for maintenance work on incorrect equipment and verification of equipment being worked on failing to spot the mismatch), I'm not sure a second pair of eyes helps as most DC techs don't have larger view of the network beyond what they are told to do.

      Admittedly, a second tech to laugh at you while you soil yourself is always.....something.

      1. MachDiamond Silver badge

        Re: Procedural Error?

        "I would assume that techs need some form of hot work approval to disconnect a device and that has been approved by multiple parties and that equipment location details are very clear."

        This may sound horribly inefficient, but if you had two techs with one looking at the documentation (you have docs, right?) while the other unplugs.

        In aerospace, it can seem monotonous, but everything is done to a checklist and there will be a person with a clipboard looking over shoulders and making sure that 20-digit part number thingamabob is connected to 20-digit part number widget with 20-digit part number wiring harness. All double checked. Been there, got the plateau trace stare. It's the same sort of thing. An enormous amount of damage can be done and so many of the items look nearly identical. I once had to re-spec a connector as an adjacent one was close enough that with enough force, it could be connected wrong. It was two days of tight space wiring, but important.

        1. tip pc Silver badge

          Re: Procedural Error?

          In aerospace, it can seem monotonous, but everything is done to a checklist and there will be a person with a clipboard looking over shoulders and making sure that 20-digit part number thingamabob is connected to 20-digit part number widget with 20-digit part number wiring harness. All double checked.

          sadly that process is not always followed in aerospace

          https://www.ntsb.gov/investigations/Pages/DCA24MA063.aspx#:~:text=due%20to%20Boeing’s,were%20properly%20reinstalled.

          1. MachDiamond Silver badge

            Re: Procedural Error?

            "sadly that process is not always followed in aerospace"

            Yes.

            With spacecraft, the stakes can be higher since you likely can't fix it later and you'd not only lose the cost of the craft, but all of the money it would have earned while you build another one (if you can) and by that time, your competition has launched theirs and hoovered up the best part of the market share.

      2. M.V. Lipvig Silver badge

        Re: Procedural Error?

        You'd be surprised at how often documentation is incorrect. Half my career is spent in a room with 4 doors, there is a map in a foreign language and a key that opens no doors. I've become a pretty good detective over the years as a result. In my world, the documentation on where stuff is simply represents a snapshot in time, and sometimes a circuit isn't even in the same building the layout shows it in. Occasionaly a circuit will be down for a few days and we'll tell the customer that spares failed and replacements are on order, but really we're waiting on this one tech to come back from vacation because he was the one who ran the circuit 10 years ago and nobody else can find jt because it wasn't documented. When this happens on one I'm working on I get the documentation fixed but next week, who knows?

  2. Jou (Mxyzptlk) Silver badge

    Reoccurring Traffic Failure Management

    Isn't that what RTFM means?

    1. Anonymous Coward Silver badge
      Megaphone

      Re: Reoccurring Traffic Failure Management

      Reconnect The Fibre, Martin

      1. Antron Argaiv Silver badge

        Re: Reoccurring Traffic Failure Management

        Who, Me?

    2. Doctor Syntax Silver badge

      Re: Reoccurring Traffic Failure Management

      It's possible, of course, that the R should stand for Re-write.

      The manual might have said something like "Disconnect each fibre in turn. After disconnecting a fibre do $MAINTENANCE" and never said anything about when to reconnect the fibres or even whether to reconnect them. Some people can be very literal. It was once said that a good tester was someone who could look at the system menu button on the top left of the then Windows ~3 window and see a minus sign.

      1. FirstTangoInParis Silver badge
        Black Helicopters

        Re: Reoccurring Traffic Failure Management

        The best recovery procedures you write are ones you come back to after a few months and have to do the recovery yourself because someone broke your system. You know this because you've read the logs and identified exactly who did what and when, but they deny it anyway. You've forgotten some of what you wrote and why. You're scratching your head going 'that can't be right' but you have to trust what you wrote. If the system recovers after you finished, the procedure is good.

  3. Sorry that handle is already taken. Silver badge

    I saw the headline in my feed and thought it was a bit early for a Who, Me?

    In a few years we'll have a Regomised name to add to the story, at least.

    1. ssharwood

      I was thinking that too - it would make a fine Who, Me?

      If you're out there, Google chap - maybe suddenly former Google chap - you know where to find us!

  4. BartyFartsLast Silver badge

    "Down No Evil"?

    I thought Move fast and break things was Microsoft's credo?

    1. Anonymous Coward
      Anonymous Coward

      Re: "Down No Evil"?

      It was Sun Tze who said "move fast and break things" in a now-deleted tweet. In his defence, at the time there were no optical fibres in China.

    2. MachDiamond Silver badge

      Re: "Down No Evil"?

      "I thought Move fast and break things was Microsoft's credo?"

      Like many bad ideas that people try to spin in a good way, the person that said it first will be working hard to erase some history/attribution.

      There are some cases where it's faster to bodge something and give it a go rather than spending weeks trying to model it on the computer only to find the error bars (if the person knows how to handle that) are greater than the range of answers. For that to be a useful approach, it has to be fast, cheap and be low impact. Building an entire orbital rocket and seeing if it works by launching it can be a poor choice. Breadboarding a circuit and applying power, not usually a big problem even if magic smoke comes out.

  5. sedregj
    Gimp

    RLY?

    What sort of issue do you attempt to fix by unplugging everything you can see?

    1. AtomicDog

      Re: RLY?

      AI becoming sentient?

      1. NetMage Bronze badge

        Re: RLY?

        HAL says “hi!”

    2. The Oncoming Scorn Silver badge
      Mushroom

      Re: RLY?

      What sort of issue do you attempt to fix by unplugging everything you can see?

      I give you J***** Peters.

      Who complaining that his test jig wasn't working correctly, I suggested he might put a spare into play . He did not note the connections or work by simple one to one transfer.

      Then complained that it still wasn't working & now the Voltage readings were still not right, I urged him to recheck his connections as I could see the colour disparity between Voltmeter & test jig output wires, to which he pulled two banana plugs out at random & swapped them over.

      Complaining yet again that didn't solve the problem to me while I was losing my patience at this level of his fuckwittery, I explosively pointed out he had red & black wires going into his (GPIB controlled) Voltmeter with blue & yellow wires connected to the jigs Voltmeter connection.

      He stomped off around the factory, came back & angrily stated "If you insult me one more time, I'll hit you!" to whit I responded "You're fucking ugly too!". He failed to make good on his threat.

      Management did get involved & they listened to my side of the story & moved him off to another test department for a while.

      1. MachDiamond Silver badge

        Re: RLY?

        "I explosively pointed out he had red & black wires going into his (GPIB controlled) Voltmeter with blue & yellow wires connected to the jigs Voltmeter connection."

        I've had similar things happen. I had a co-worker ask about some electronics I designed and built that was having trouble (DC-DC converter batch didn't meet spec and shut off unexpectedly). He response to an answer I gave him was that I was lying. I told him to sod off and stop bothering me and trace the damn thing out on his own. I would up solving the issue shortly after that and ordered new parts by a different manufacturer. One expects that brand new parts being operated well within spec aren't the problem. Welllll, sometimes they are.

    3. MachDiamond Silver badge

      Re: RLY?

      "What sort of issue do you attempt to fix by unplugging everything you can see?"

      Remaining employed?

      I expect the open position notices went out before EOD.

  6. Bebu sa Ware Silver badge
    Facepalm

    Perhaps Reading is now optional for Network Engineers

    Rather takes the R out of RTFM — FTFM ?

    Actually now I think about the networking types I have dealt with over the years there was a definite over·representation of dyslexia and related literacy disabilities not to mention a good many who didn't speak or undestand English (and my Korean was rather rusty /s) and were from a telco background - try explaining netmasks and why you need to change one on a router for your subnet. (Inherited a non contiguous (sub)netmask.)

  7. Fonant

    While that was going on, technicians “identified the disconnected optical links and physically reseated the fibers.

    The "reseated" word is interesting. Not "reconnected". Perhaps the technician accidentally knocked a set of fibres which disrupted the network due to small physical misalignments, and didn't actually completely disconnect them?

    IANAFNE

  8. Lee D Silver badge

    One of the things I insisted on at a previous employer was a redundant line to the Internet, and redundant servers. We had so many power-outages, etc. that you couldn't rely on any one building or path to stay up. We couldn't get the money for redundant switches, but I compensated for that in other ways.

    So I double-looped the site. Every switch was connected to each of its neighbouring switches, a minimum of two fibres one way to one switch in one building, two fibres another way to another switch in another building. Then, with LACP and STP, even if you break any single connection, nothing happens. Chop through any given fibre, no matter how many cores, and nothing happens. And at the end, the first switch was connected back to the last switch, and vice-versa. If you were to completely disconnect any building in the middle, the traffic just goes around the other way, or even takes a shortcut via another intermediary switch if it can find one, and you don't even notice. STP just handles it.

    You know, the way the Internet was designed to work. But on my local network.

    And that also meant that we had two paths to the Internet, two server cabinets, and so on. We were only little, but we had great redundancy (and, as a bonus, it means that your switches are only ever really dealing with half the traffic no matter where they are, so you get a lot more out of your hardware too).

    But one part of that was... not just relying on having "switch stacks" or two redundant routers one of top of the other, and so on. Part of it was physical separation. To cut us off from the Internet, you would have to sever one bunch of cables at one end of the site, and then go to the exact opposite end of the site and sever another bunch, that doesn't run anywhere near the first bunch. Same for servers, routers, etc. You'd have to turn off two disparate sets of hardware that were physically separated by a long distance, a distance you couldn't cover without someone going "Has something gone off?" on your transit between those two.

    I'm both amazed (that they don't) and not surprised (because large companies are often incompetent) that someone like Google doesn't have the same. How is it even possible to sever two independent connections from the same cabinet? It means that THEY'RE NOT INDEPENDENT, surely?

    That site, with some of the cheapest and oldest hardware, had some of the best networking uptime (servers were a different matter for other dumb reasons unrelated to this) I've ever seen in my life. Power would go on and off across the buildings, especially during maintenance periods and things like electricial and fire checks, and... it didn't matter. Nothing of it was visible. It all just worked. Just by joining everything possible with as many SFP modules and fibres as the switches would allow, connecting them to everything else possible, and letting STP etc. do its job.

  9. J.G.Harston Silver badge

    Have you tried plugging it back in?

    1. Francis Boyle

      At least

      we know where Roy ̶i̶s̶ was working now.

    2. herman Silver badge

      Did you turn it on after plugging it back in?

  10. that one in the corner Silver badge

    Traffic flow drop rates for resources hosted in the affected area reached 100%

    Or:

    Traffic flow there stopped.

    You couldn't possibly be using negatives in order to stick in a "100%" in the hopes someone glancing at it will say "100%, that must be good"?

    1. Yet Another Anonymous coward Silver badge

      Re: Traffic flow drop rates for resources hosted in the affected area reached 100%

      The data center was 100% protected from all hackers.

  11. Andy the ex-Brit
    Facepalm

    RTFM? What FM?

    Bold of you to believe there was a manual for him/her to read.

    1. herman Silver badge

      Re: RTFM? What FM?

      I used to have a coffee mug with RTFM on it and it was interesting how many clearly didn't know what it meant.

  12. Arty Effem

    Engineer != Technician

    The terms 'engineer' and 'technician' are not interchangeable. One writes manuals for the other to follow.

    1. blu3b3rry Silver badge

      Re: Engineer != Technician

      Not always. I'm a technician and have written more documentation than some of the engineers I work with....

      1. The Oncoming Scorn Silver badge
        Facepalm

        Re: Engineer != Technician

        I had a design engineer sit next to me, to see if he could investigate the root cause of product failing in the field.

        We got to the point where I had to make coarse & fine adjustments to the VCO*'s, as I made my adjustments, in he swooped in his best Nelson Muntz voice

        "Ha HA! That's why they are failing, you are setting the VCO's at the very edge of the upper limit".

        I promptly took the wind out of his sails by putting the screening can lid on (From the next assembly stage) & it fell to dead center of the expected value.

        Oh he said I hadn't counted on that, I'll have to change the limits. I replied "& while you are at it can you please change the spelling from Course to Coarse Adjustment".

        Then I followed up with & if you really are looking for the root cause of field failures, he's sat to your left, whereas I set the VCO's at the upper limit, J***** Peters always sets them at the lower threshold & did the screening can removal & the Voltage flew back up to the higher upper limit.

        *Voltage Controlled Oscillators

        1. bazza Silver badge

          Re: Engineer != Technician

          Sounds like the circuit was being influenced by capacitive coupling with the screening can. That's probably an engineering design mistake!

    2. nonpc

      Re: Engineer != Technician

      The Eng Soc teeshirt at university read 'Munce ago I cudent even spel 'injenier' but now I are wun'

    3. M.V. Lipvig Silver badge

      Re: Engineer != Technician

      Yes - the engineer writes the manual, the technician corrects the manual.

  13. LessWileyCoyote

    "Make something foolproof and the universe will build a better fool"

    1. IanRS

      Some people are just incompetent

      I once had a plumber come around to fit a new sink. It turned out he had previous been a (Cisco) network engineer, but had given that up. It also turned out he could not fit sinks either.

      He had some bits left over when he thought he had finished and fortunately he left them with me when he left. They were for connecting the overflow outlet to the main drainage pipe leading down from the plughole, so if the water level had ever gone too high it would have just poured into the cupboard. To cover the opening in the drainpipe he had bodged a piece of plastic sheet to try to seal the hole.

      1. Unoriginal Handle

        Re: Some people are just incompetent

        I had a heating engineer put a tin can and mastic over a hole in the boiler sump, caused ( although we didn't know it then ) by his previous service.

        Less than impressed, especially when he didn't want to fess up to his original error even when presented with the evidence.

      2. Fido Bronze badge

        Re: Some people are just incompetent

        After maintenance at home I often have to "reseat" the networking cables. This is because the clips broke off from the RJ45 plugs.

        I didn't know hyperscalers had the same problem with fiber.

      3. Potty Professor
        Facepalm

        Re: Some people are just incompetent

        I went to visit my Brother in Law on Christmas Eve. As I walked into the flat I could smell gas. I commented on this and was told that it wasn't gas, it was fumes coming from Rugby Cement Works just down the road. I could hear a faint hissing sound coming from the gas fire in the lounge, so I said to open all the doors and windows because there was definitely a gas leak. I asked whether they had had any gas work done recently, and was told that the Corgi "Engineer" had been that morning to service the central heating boiler. I found a screwdriver and removed the front cover of the gas fire, and the hissing got louder. I touched the union nut on the inlet valve, and it was less than finger-tight. I went out to my car and came back with an adjustable spanner, and tightened the union nut. No more hissing. The gas fitter had neglected to tighten the union after servicing the gas fire, which he wasn't supposed to be doing anyway. If that had been allowed to continue, who knows whether my BiL and his wife would have survived, either by poisoning or by being blown to kingdom come!

    2. MachDiamond Silver badge

      "Make something foolproof and the universe will build a better fool"

      The universe already has a large supply that are kept hidden by the ranks of fools that have yet to be knocked out of the way. Under every deep, another deep unfolds.

  14. wsm

    It takes a janitor...

    to accidentally knock over a mop and bucket to disconnect one connection. It takes an "engineer" to really screw things up.

  15. IGotOut Silver badge

    WTF?

    "within 13 minutes. The nature of the error, combined with the speed of the action, prevented warnings of incorrect action reaching the engineer"

    Thirteen minutes? Thirteen fucking minutes?

    If we lost a link for more than a few seconds we got alerted (we had multiple redundancy so there would be zero outage). If we lost two out of three , pretty much our systems would be spammed with alarms; email, text and big visual displays.

    But I guess that's Telecoms, rather than the network kids (we always had to tell them a link was down).

    1. Richard 12 Silver badge

      Re: WTF?

      It means the fall guy in the datacentre didn't answer the phone, and it took the security guard too long to go yell at them.

      Most likely the real cause is lone working. Had there been two technicians they might have noticed the mistake or taken the phone call before everything got unplugged.

  16. Ambivalous Crowboard

    The way they've written this is so Doublespeak

    Traffic flow drop rates

    [traffic rates]

    for resources hosted in the affected area reached 100%

    [for resources hosted in the affected area dropped to 0%]

    Shit went down, not up. Why can't they be honest?

    1. Richard 12 Silver badge
      Boffin

      Re: The way they've written this is so Doublespeak

      Um actually, the drop rate is the one most places normally measure.

      Describing it as the percentage "successfully delivered" would be misleading, because you don't know if the receiver actually accepted and processed the packet.

  17. sarusa Silver badge
    Trollface

    'engineer'

    'engineer'

  18. BILL_ME

    Change Management no longer a thing?

    Up until 2021 when I was a network engineer, everywhere I worked we had Change Management.

    Changes scheduled via SAP/CRM, Peer Reviewed and looked at by at least 2 levels of CM/SME's.

    Bio-tech, Pharma, Medical, Manf's, etc.

    How does The Internet 800# gorilla get to a point in 2026 where their Ops are this slack?

    Any bets this was one of their contract vendors a la Tata, Infosys, Accenture?

    Unless the Engineer purposely deviated on his/her own from doing 1-at-a-time to en masse, there should be at least 3-4 others getting the chop and likely a Service Agreement Contract break.

    1. Excused Boots Silver badge

      Re: Change Management no longer a thing?

      ‘Change management procedures’ are one of those things that many, many organisations claim to have but in reality.....

      It going to suggest that a) the person involved was woefully lacking in basic understanding of what would happen if they pulled all of the connections - in which case the company is guilty of sending the wrong (not qualified*) person; or b) they did understand it but was ordered by a PHB to ‘just do it’ regardless!

      Either works!

      * and by qualified, I don’t mean someone with a load of paper certification, I mean someone who actually does understand how this shit works - ie, if i pull this cable then xyz company drops offline, so I’m not doing it, unless you instruct me in writing (email will do) to proceed, oh and could you cc my lawyer into the email chain? Just in case!

  19. Anonymous Coward
    Anonymous Coward

    RTFM... ?

    Of course, in this person's defense, we are all assuming the "FM" exists, is readable and is correct. That may be the case, but it may not. I've seen site manuals that claim full rack and route redundancy, while utterly neglecting to indicate SPOFs. If (s)he failed to read the manual, shame on that former employee. But if the manual wasn't right, even though common sense would cause one to question pulling all the cables (and a second set of eyes, as suggested above, certainly ought to be SOP) then there might be an argument for shared blame.

  20. Anonymous Coward
    Anonymous Coward

    Let's be gentle with assigning blame

    It's quite possible that this technician was simply following the step-by-step instructions typical of a telecommunications cabling work order, and that work order itself instructed the technician to unplug all the cables. Such errors are simply made due to the nature of telecommunications cabling database systems: it's very easy to highlight all the cables rather than one cable, and the automated creation of the work order has so much detail that it is difficult to spot such errors in long works, just an extra few pages within hundreds. Disconnections typical require less space in the work order: they don't need photographs of fibre faces or other proofs of quality work.

    Why would a technician follow such an instruction? Often a third party technical officer won't have the context which a in-house technical officer will have.

  21. ChrisBedford

    RTDB?

    But that's boring

  22. Lazlo Woodbine Silver badge

    Who me?

    I'm guessing we'll be reading about this in a Who Me? in a few years...

  23. Anonymous Coward
    Anonymous Coward

    a bit harsh blaming the engineer when other resilient paths where present

    It therefore seems like the engineer who pulled the cables failed to observe one of the most basic rules of tech: RTFM

    Once Google noticed the situation, it moved traffic away from the routers that the engineer disconnected. At that point its resilience provisions kicked in as traffic automatically shifted what the company described as “healthy capacity elsewhere in the region.”

    i'd expect an organisation like google to have an automated system that would have instantly rerouted the traffic towards the “healthy capacity elsewhere in the region".

    BGP with bfd will do that for you, just put all your resilient routes in and weight them for preference, once the main preferred path disappears (or preferably starts to loose its aggregated bandwidth ) BFD would ditch the routes from going to the affected router and go to the alternates already in the table. A tiny amount of traffic will be lost but defo less than 1 seconds worth & wouldn't require human intervention to resolve.

    i very much suspect that the instructions to the engineer where checked by multiple AI & humans before they where even given to the engineer to do what they where told to do. If the engineer did something they where not instructed to do then they should be dismissed, but i doubt that was the case here. I would bet that everyone had assumed traffic was migrated before the engineer did what they did. Lots of times traffic migration is automated so that between x & y time, which should be slightly larger than the change window, the requested physical operation should be safe to perform which should include the possibility of a router instability causing total outage to that system.

    there is defo more to this story & google have very much blamed the lowest hanging fruit which is the hands on engineer, shame on them for deflecting the blame to the one with the least information to counter the claim,

    1. M.V. Lipvig Silver badge

      Re: a bit harsh blaming the engineer when other resilient paths where present

      That's funny. I deal with the Googler quite often. They are always Indians, and they are always arrogant. I can believe that one would yank all the cables, 100 percent confident that whatever they do is automatically correct and anything else that happens is someone else's fault.

  24. Medixstiff

    So they tried to build it to be idiot proof?

    “The system is designed to be resilient to all single device or fiber-path failures, and most double-or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power sources,” the document explains.

    But it’s possible to defeat all that redundancy by pulling out all the cables." <- Better idiot?

  25. What? Me worry?
    Go

    Is this Who me? or a On Call article fodder?

    See title, Looking forward reading this on a Friday afternoon in the (hopefully) near future. @Simon, c'mon use your sources and get this one!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon