back to article Lufthansa flights grounded by major IT snafu, 'construction work' blamed

German airliner Lufthansa Group is working to restore services after an unspecified IT glitch – which it says was caused by a sliced broadband cable – forced it to delay or cancel flights. The company confirmed to The Register: "Fiber-optic cables belonging to a telecommunications service provider were damaged during …

  1. Paul Crawford Silver badge

    Probably multiple "redundant" fibres in the same cable/duct, maybe even used by different ISPs. Very, very common I'm afraid and if you ask your ISP about this you will often not get far..

    1. Alex Brett

      Indeed, even when redundant ducts are installed, often the contractors who then get tasked with pulling the fibre into them end up taking the lazy approach and pull both into the same duct and leave one empty.

      The other classic one is you have redundant ducts properly taking geographically diverse routes to a facility, but they then come together where they enter the building presenting a lovely target for a JCB...

    2. Norman Nescio Silver badge

      Yup. Entirely normal.

      You buy service from two ISPs to get 'redundancy'.

      Super-Duper ISP - Yes, you can have service from us between Frankfurt and Munich

      Excellent ISP - Yes, you can have service from us between Frankfurt and Munich

      Super-Duper ISP then calls Little-Horse Backbone fibre company - I say, can we have a path lit up between Frankfurt and Munich? Answer - of course!

      Excellent ISP then calls Little-Horse Backbone fibre company - I say, can we have a path lit up between Frankfurt and Munich? Answer - of course!

      So Little-Horse Backbone fibre company lights up a couple of wavelengths on their fibre between Frankfurt and Munich.

      Little-Horse Backbone fibre company will not tell Super-Duper or Excellent which other customers are on that fibre (commercially confidential, obviously)

      Along comes a JCB (backhoe for Americans), and Whammo!, your redundant solution fails.

      ***

      You do not get guaranteed redundancy simply by buying from separate service providers, when the services are layered like filo pastry in baklava.

      It's also surprisingly difficult to get from a single provider, but that's another story.

      1. M.V. Lipvig Silver badge

        Not really hard, just more expensive. All telecoms will offer geographic diversity, but usually the last mile there is no way to be diverse. Buildings matter as well as the building will need to have two separate entrances. All this, and we'll find the customer uses one mutliport router to save a buck instead of two separate routers.

        Telecoms have other failures as well, such as a ring might have a node drop and be down for months and nobody even looks at it until the other side of the ring drops. Yes, they're alarmed, but traffic outages ALWAYS come before loss of diversity outages. When $Telco says that your circuit was down due to a fiber cut along with a card failure, you can be certain that one or the other happened a while back. And yes, a fiber cut may not be addressed immediately if it's a single fiber break or no other circuits on a fiber span (and this happens often.) If they use the term "collapsed ring" it means both working and protect were on the same fiber cable. A collapsed ring protects against equipment failures but not fiber cuts.

    3. Flak

      There is 'redundant' and then there is diversity and physical separation

      The latter can be very hard to prove, particularly if you work with two organisations for the two 'redundant' routes. Sounds counter-intuitive, but having one organisation providing two physically diverse fibre links means that they (assuming their records are accurate) can actually prove that physical diversity. Otherwise you have the nested selling/reselling issue - often without the visibility you would require.

      You need to know what you are asking for and then insist on evidence from the underlying fibre providers, regardless of your chosen service provider.

      1. Yes Me Silver badge

        Re: There is 'redundant' and then there is diversity and physical separation

        You could also do some high precision ping testing, and if the two paths give essentially the same latency, ask some very direct questions.

        1. M.V. Lipvig Silver badge

          Re: There is 'redundant' and then there is diversity and physical separation

          That won't work to find out whether two separate carriers use a common 3rd party provider. Latency won't be the same, but will be close, geodiverse or not, unless you specify your route. If you specify a route (and you can, for extra money) ping tests won't matter. A traceroute won't matter either, because that will only pick up on Layer 2 devices, not Layer 1.

          .

          And, if you're so far in the sticks that you're using Joe's General Store and Telecom, you may not have a choice about geodiversity. That carrier may be the only one there, and he doesn't have more than the one run.

    4. Hans Neeson-Bumpsadese Silver badge

      Ironic that this happened in the airline industry, which is all about resilience through carefully designed redundancy.

      1. DevOpsTimothyC

        Not really. Most airlines know very little about IT with it still being seen as a major cost centre

        1. John Brown (no body) Silver badge

          ANd now they get to compare the cost of the loss of business for a full day, and the knock time into the following days against having a proper geographically different backup route which the infrastructure team have probably requested many times in the past and had knocked back by the beancounters. I hope they are relishing the "I told you so" moment in the post-disaster meeting.

          There will, of course, have a been a financially positive business case put forward by Infrastructure demonstrating why the more expensive redundancy option was a viable solution against the losses incurred by exactly this scenario, but of course the beancounters buried their heads in the sand (that the JCB just dug up) because "it'll never happen to us".

          1. DevOpsTimothyC

            Ah the old risk & impact questions.

            Low risk, high impact never gets the cash unless it's something the c-levels want.

      2. Flak

        Do you mean like the Boeing 737-8 Max Angle of Attack Sensor?

        Resilience is great until it bows to (short term) economics.

    5. Anonymous Coward
      Anonymous Coward

      I used to work in IT in a US airport that LH flies into.

      LH were one of the few carriers to bother getting two separate circuits into the airport, into two separate routers...in the same comms room. Where we then split one circuit to go over our internal backbone to our redundant data centre.

      So you're left at the mercy of a single cable duct which could take out most of the airline connectivity at that airport.

      They did look at the option of getting a circuit directly into the redundant data centre, problem was it was in the middle of nowhere about 3 miles away from anything, and the extortionate amount the telcos wanted to put a circuit for LH directly into that data centre probably would have wiped out their profit margin on that route......

      1. Flak

        That reminds me of...

        Die Hard 2

    6. CrazyOldCatMan Silver badge

      Probably multiple "redundant" fibres in the same cable/duct

      We have redundant lines - one exits the back of the building and one out of the side in a completely separate duct.

      But, as we discovered later (when the weight of a JCB collapsed a duct that was too close to the surface), both those 'separate' ducts went into the same conduit under the next-door car park. And, when said JCD ended up nose-down in the duct we discovered that both our lines were in that duct and, both got severed at the same time.

      Opps.

      Our landlord had some explaining to do (it was their ducting) and we had specified that out outgoing circuits *had to* travel in serapate ducts. They tried to blame BT but the GIS systems clearly identified that they had built the ducts and that they were on their land. Cue a rebate on the rent..

      The lines now (genuinely) do take diverse routes. As does the 3rd line..

      1. Yet Another Anonymous coward Silver badge

        They were, one line was in duct 'A' and the other was in duct '1' - the two ducts just happened to be superimposed in exactly the same place in "real world" space, on the network chart they are totally separate

  2. This post has been deleted by its author

  3. sebbb
    Facepalm

    Fiber... shred

    Deutsche Telekom confirms too

    https://twitter.com/deutschetelekom/status/1625587220785598464

    https://twitter.com/deutschetelekom/status/1625824840924950528

  4. Yet Another Anonymous coward Silver badge

    German engineering

    >Where is the redundancy?

    As soon as the fibre was cut a team was dispatched to cut all the other fibres. For some reason this was in the operations manual and so 've vere only following orders'

    1. Neil Barnes Silver badge

      Re: German engineering

      Aye well, they've got a month to get it sorted before I'm flying :)

      1. Charlie Clark Silver badge

        Re: German engineering

        Don't worry: strikes at the airports will start on Friday.

    2. yetanotheraoc Silver badge

      Re: German engineering

      1. Engineering == over-engineering.

      2. Bean counting == maximum efficiency, just in time logistics, minimum spec parts, etc.

      3. Outage == Engineer's problem, not the bean counter's problem.

      I think we can see the root cause is in #3.

      >Where is the redundancy?

      Seems to be in the Deutsche Bahn.

  5. Marty McFly Silver badge
    Joke

    Redundancy

    Meanwhile, in other news...

    "Lufthansa is actively working to align other redundant systems with their newly established Information Technology standard. Going forward, all aircraft will only operate a single engine. And the flight deck will only be staffed by a single pilot.

    A company spokesperson reported this is not a cost savings measure: 'This is strictly to align standards of redundancy across the company. If a single point of failure is good enough on the ground, it should be good enough in the air.'"

    1. Strahd Ivarius Silver badge
      Joke

      Re: Redundancy

      they should align on the "IT Coffee Machine" standard

  6. Anonymous Coward
    Anonymous Coward

    This is Lufthansa getting their retaliation in early...

    Ahead of their staff going on strike later this week.

    What are the chances of flying between Britain and Germany these days if there is the chance of strikes at one or both end, coupled with incompetence at the infrastructure level?

    1. Anonymous Coward
      Anonymous Coward

      Re: This is Lufthansa getting their retaliation in early...

      But Global Britain is one step ahead of the Germans

      - they offered a replacement high speed train ticket, we engineered a coordinated rail strike

      1. John Brown (no body) Silver badge

        Re: This is Lufthansa getting their retaliation in early...

        I did think it curious that they offered a rail replacement service and not alternative airlines as an option.

        1. Yet Another Anonymous coward Silver badge

          Re: This is Lufthansa getting their retaliation in early...

          It was for internal flights. Imagine a country where a high speed rail service would be comparable to flying between cities. It must mean they have really slow planes.

          1. Strahd Ivarius Silver badge

            Re: This is Lufthansa getting their retaliation in early...

            if you take into account to total travel time from city center to city center, I am pretty sure that even Concorde would loose against ICE...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon