back to article London's Gatwick Airport flies back to the future as screens fail

London Gatwick Airport’s shiny new cloud-based flight information display system had a hard landing this morning as its vision of the future was brought down to earth with a bump. While collecting the Cloud Project of the Year award at the Real IT awards in May, the airport proclaimed its new screens were "an innovative, cost …

  1. Ochib

    So why was there no redundancy in the internet link?

    1. Dan 55 Silver badge

      Because oddly enough it doubles the cabling costs and that wouldn't do.

      1. Mike 'H'

        But...VODAFONE...

        Where the bloody hell was the LTE backup - 3 megabits on their Network must be child's play.

        Point a directional YAGI or dish antenna towards an off-site tower to bypass on-site DAS/macros, likely afflicted with the same fibre cut, and bobs yer uncle.

        Or maybe the airport should spend a bit more for a competing LTE provider for 3-megabit backup services...

      2. Lotaresco

        "Because oddly enough it doubles the cabling costs and that wouldn't do."

        It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.

        This is, quite frankly, poor practice on Vodafone's part.

        1. Wayland

          Of course it doubles the cable cost. Just because some of the fibres are dark does not mean you get to use them for free.

        2. ButlerInstitute

          That wouldn't really be redundancy.

          "Pulling a multi-pair cable"

          But then you've got two cores in a single cable, thus failing to provide any resilience when a digger goes through the cable. Ok for a fault with the cable/core itself maybe, but not for a physical break.

          Your alternative core needs to come into the site via a different route, So wouldn't be cut by the same digger. See for example major BBC facilities where there are redundant power and signal cables coming in from opposite sides of the site.

          1. Anonymous Coward
            Anonymous Coward

            Re: That wouldn't really be redundancy.

            A very big site I worked at had multiple connections, unfortunately someone emptied a truck of waist tar down the manhole cover out the front of the building where all the connections went through, business stopped for a couple of days.

        3. qwertyuiop
          FAIL

          Two failures?

          "...It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution..."

          Ah, I see. We increase resilience by using a multi-pair cable but immediately reduce it by using... a multi-pair cable. When the digger goes through the cable it doesn't matter how many pairs are in it! (and yes, I know that in this case it wasn't a digger)

          The only way to truly increase resilience is to have two cables which arrive by completely different routes - and that will increase the cost.

          1. jmch Silver badge

            Re: Two failures?

            "The only way to truly increase resilience is to have two cables which arrive by completely different routes"

            Or have one cable connection and one wireless connection

        4. Anonymous Coward
          Anonymous Coward

          Ummmm.. yeah, it does. Multi-pair cables don't help if the cable's cut (since all the pairs are cut too). If you want to be truly fault-tolerant, you need a second cable, geographically isolated, going preferable to a second ISP but in any case at least to a different access point for your ISP.

        5. jimbo60

          > Pulling a multi-pair cable is a sensible precaution

          Sorry, multi-pair cables don't offer any protection against diggers slicing the entire thing. You need to have different physical routes to avoid that problem.

        6. Mark Dempster

          >It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.<

          That probably wouldn't have prevented the issue, though, as pulling 2 cables from the same location to the same destination means they're separated by millimeters at best - so both would have been cut through at the same time.

          If you're going to this trouble (and for something so important why wouldn't you?) then you need 2 completely different runs of cable, preferably from 2 different providers, coming into the building at different locations.

      3. G_Man

        3G/4G failover

        Forgive my ignorance, but given they are saying that normal operations only requires a 3Mbps link, couldn't they save on truly diverse, resilient fibre by using 3G/4G in a failover event?

    2. macjules

      The only redundancy they now have is the moron who failed to QA before pressing the big green 'go' button.

      1. Doctor Syntax Silver badge

        "The only redundancy they now have is the moron who failed to QA before pressing the big green 'go' button."

        No way. Have you any idea how many senior managers would insist on getting their names on the sign-off of a new big shiny? They're going nowhere.

        1. Anonymous Coward
          Anonymous Coward

          @Doctor Syntax

          A few were spotted checking the Departures whiteboards, with a grimace

    3. Pascal Monett Silver badge
      FAIL

      Re: "no redundancy in the internet link"

      Exactly that. Over 1000 monitors tied to a mission-critical component and nobody ever asked "what if the cable was cut" ?

      Complete, utter and total project specification failure.

      1. Version 1.0 Silver badge

        Re: "no redundancy in the internet link"

        "Complete, utter and total project management failure." - but this is exactly how business works these days. Expect a big bonuses all round at the end of the year.

        1. yoganmahew

          Re: "no redundancy in the internet link"

          Outages happen. The question for me is why there was no local cache? It would have grown stale over time, but a well installed local cache with a GUI for updates could put everything into manual mode with zero impact.

          Is this the Internet of Tripe future? One failing link and your IoT belt unbuckles and your trousers fall down exposing your single-point-of-failure-arse?

          1. Captain Scarlet Silver badge
            Trollface

            Re: "no redundancy in the internet link"

            As they are on Vodaphone why don't they use their mobile network instead in the event of a fibre cut.

            (Ok below the belt as its likely nearby equipment may also be affected by the fibre cut)

          2. TkH11

            Re: "no redundancy in the internet link"

            Caching is not the answer to a fundamental failure of resilient network design.

            1. Flakk

              Re: "no redundancy in the internet link"

              This tiny whiteboard is the only departures information in Gatwick Airport right now; every screen is offline. Utter chaos. This is a signature flourish at the end of a short trip that’s been full of reminders of how badly the UK’s infrastructure is crumbling. pic.twitter.com/6r7CDVheLf

              — Rob Fahey (@robfahey) August 20, 2018

              To be sure, that the info boards went down due to a network outage is pretty bad. But utter chaos? Really? It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website? Failing that, maybe the airline's website?

              1. Anonymous Coward
                Anonymous Coward

                Re: "no redundancy in the internet link"

                Sadly at 17:10 all departure gate data is missing from the website.

                The point and beauty of cloud based computing is you never have to worry about a site going down.

                Just monitored my site and it's now beer o'clock.

                Good luck.

              2. Efer Brick

                Re: "no redundancy in the internet link"

                was it not the same system?

              3. JohnG

                Re: "no redundancy in the internet link"

                "...the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website?"

                Reports elsewhere indicated that updates to Gatwick's mobile app were affected by the same fibre outage. It seems that the design was engineered to fail comprehensively.

              4. David 45

                Re: "no redundancy in the internet link"

                Potential of a smart phone? No earthly good if the information is not there to start with. I was there today, dropping off passengers, then attempting to pick up others. Website arrivals flight information was sketchy, with some flights missing or no information against the flight numbers (so no use for one's shiny smart phone there!). Gatwick's auto. phone information system just went dead the second I entered the flight number (so no use for one's shiny smart phone there!). I had no idea of the status of my incoming passengers' flight and felt that Gatwick's main number would probably be inundated, so I didn't bother trying that (so no use for one's shiny smart phone there!). Arrivals concourse info. screens were also all over the place, with my flight number also not showing on there at all. My passengers also said that the baggage reclaim section was also not working. The main complaint I have is that there was no information whatsoever in arrivals that there was a problem - not even any public address announcements and certainly no whiteboards. There were lots of baffled-looking folk with furrowed brows looking at useless screens, obviously wondering what the devil was going on. In fact, as far as I could see, the only information that something was amiss was a rider on the flight information website saying that information screens were not working properly. A bit more communication (in more ways than one!) wouldn't have gone far amiss.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: "no redundancy in the internet link"

                  use a proper system like flightradar24 for any inbound flight they have real time up until the plane gets to the gate. For outbound the schedule will show that minus any delays.

                  There ar millions of ways to fins out whats happening without the airport on screen system if you travel enough you will soon get the hang of it

                  1. WallMeerkat

                    Re: "no redundancy in the internet link"

                    Flightradar24 is nice for viewing the route your flight took, but for arrival data it can be hit or miss, for example it doesn't deal with diverted flights nicely.

              5. tin 2

                Re: "no redundancy in the internet link"

                "It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding."

                Seems that information was not making it out of the airport to whatever cloudy goodness was updating the website and app either.

              6. Lotaresco

                Re: "no redundancy in the internet link"

                "maybe look up the gate information on Gatwick's website? "

                Have you tried to do that? Good luck trying it. If you're lucky you'll get departure gate information in time to watch your flight depart. If it's working as usual you will get the information 24 hours later.

                I have the Gatwick app. It's never told me a gate number before the flight has departed.

              7. Anonymous Coward
                Anonymous Coward

                Re: "no redundancy in the internet link"

                International travellers without data roaming perhaps?

            2. Hans 1
              Happy

              Re: "no redundancy in the internet link"

              It is pure and utter design failure.

              Why have 100's of systems download flight information from the cloud ? <---- that is already brain-dead, somebody in the design team has never heard of multiplications!

              No resilient internet connection ? <---- that is brain-dead

              Caching is not the answer to a fundamental failure of resilient network design.

              It reduces internet traffic significantly and buys you time for the connection fail-over..

              1. Anonymous Coward
                Anonymous Coward

                Re: "no redundancy in the internet link"

                Welcome to the future, at least the future if the current "Cloud" fad continues.

                Single points of failures all round.

                Someone took a digger to your fibre? Bad luck. Maybe your backup link will take up the slack, but as these are usually lesser specification it will soon grind to a halt.

                Someone took the cloud server down? No local caching or redundancy? Bad luck. Maybe someone will turn it off and on again soon. Maybe the script kiddies will get bored and stop DDOSing it.

                Ironically the internet was developed to survive a nuclear war.

                1. jmch Silver badge

                  Re: "no redundancy in the internet link"

                  "Welcome to the future, at least the future if the current "Cloud" fad continues."

                  the thing is, a properly implemented cloud solution gives many benefits one of which is redundancy. Except management only got the memo where cloud = cost savings and don't realise that you still need to spend money on redundant network links. Every piece of the puzzle has to have redundancy, not only the cloudy bit

                2. Martin Gregorie

                  Re: "no redundancy in the internet link"

                  One thing nobody seems to have forgotten - BT and other wonderful network providers currently operating in the UK have been known to engineer their own single point of failure. It happens this way:

                  • The system design team specifies a disaster recovery site and a high speed connection to it
                  • Their network design requires separate dual redundant links from the operations centre (LGW in this case) to the main ops site and to the disaster recovery site via at least two paths which are required to leave the building via separate ducts and then follow different routes.
                  • These specs get handed to the network provider, whose contractors promptly ignore all the fancy separate routing details and put all the cables through a single duct so they can trouser all the money they saved by skipping all that costly separate routing nonsense.
                  • The local council puts a digger through the cable duct....

          3. Lotaresco

            Re: "no redundancy in the internet link"

            "The question for me is why there was no local cache? It would have grown stale over time"

            It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. What is needed is resilient comms and that is standard provision for systems like this. There should be no SPOFs in a real time system.

            1. yoganmahew

              Re: "no redundancy in the internet link"

              @Lotaresco

              "It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. "

              Not really, the scheduled departure and arrival times are well known days in advance. The gates are usually well known, but at lest could be manually updated (so at least people stand a chance of finding their gate). The amount of data that tranmits by FLIFO for FIDS updates is vanishingly small, being essentially designed in the 1960s. Never mind LTE, you could run it on dial-up...

        2. Quentin North

          Re: "no redundancy in the internet link"

          Not a project management failure, possibly a service specification failure due to "value engineering" by the service provider or customer. The PM probably did exactly what they were asked to do.

      2. Anonymous Coward
        Anonymous Coward

        Re: "no redundancy in the internet link"

        Exactly that. Over 1000 monitors tied to a mission-critical component and nobody ever asked "what if the cable was cut" ?

        (Insert Wiley Coyote gag here)

      3. bpfh

        Re: "no redundancy in the internet link"

        So Gatwick generates the raw departures/arrivals data, posts that data to somebody else’s computer, then the screens pull that back to Gatwick. And it only uses 3 megabits...

        So 2 questions:

        0. Did they really need Somebody Else’s Computer to do this (especially if the computers that run this are still there, but now offsite - out of sight out of mind?)

        1. 3 megabits... cut fibre... nobody thought of setting their mobile phone into WiFi hotspot mode - or dare I say it, backup 3/4G router ? Or is mobile internet that crap there?

        1. Anonymous Coward
          Anonymous Coward

          Re: "no redundancy in the internet link"

          No techies on site as all were offshored a long time ago?

        2. yoganmahew

          Re: "no redundancy in the internet link"

          @bpfh

          "DId they reall need someone else's computer"

          No, but, generally you stick the flight information data on a server somewhere so you can access it from multiple sources. Of course, if you build that as PUBSUB and there was a local server serving coax to the airport... (why coax? Coax can be fixed by anyone with pliers and a piece of tape... Bring back coax!).

          It's a bit weird, that they are having problems still suggests it's not just a fibre cut?

          1. defiler

            Re: "no redundancy in the internet link"

            I'm just going to lob this on the end here, since everyone's screaming about having a second cable (expense of installation, potential proximity to first), and others are screaming about having a cellular backup (potential proximity to cable because Vodafone).

            Nobody's saying satellite link. If all you're looking for is 3Mb/sec, it's not expensive. Oh noes - 200ms latency? Who cares? If the power stays up, the data stays up. And if the power drops, the airport is closed anyway.

    4. Anonymous Coward
      Anonymous Coward

      Situation Normal: Outsourced Data

      When LGW was run and managed by BAA, BAA IT provided the information screens in co-operation with CAA/NATS in the tower.

      Now, with GIP owning the Airport, VodaFone providing the data and the tower outsourced to a German company with an unfinished website - hardly surprising really....

      Too many layers of outsourcing....too cheaply....

    5. Multivac

      They didn't need it, everyone has a smart phone right? Just go to https://www.gatwickairport.com/flights/?type=departures .

      If everyone is carrying their own screens around then you you have an airport full of redundancy, different phones, different OS's using different network providers.

    6. GrumpyOldGeek

      Because : time to market, at all costs. Blockers be damned.

    7. jamesdagger

      Diverse routing

      Fully diverse routing can cost a fortune in civils.

      Almost certainly why it wasn't signed off, or was signed off to a pinch point where the break occurred south of that.

  2. Anonymous Coward
    Anonymous Coward

    According to http://www.mediacentre.gatwickairport.com/press-releases/2018/18_05_18_it_awards.aspx

    "We have also just future proofed our entire IT network for the next decade so that we can take advantage of the latest technologies, while also making the network more resilient and tolerant to disruption."

    Bullsh*t...

    1. Hans Neeson-Bumpsadese Silver badge

      take advantage of the latest technologies

      That's no bullsh*t. I've seen some photos, and let me tell you that's one modern-looking whiteboard that they're using.

      1. m0rt

        Wouldn't it be funny if they used permanent markers by mistake...?

      2. Anonymous Coward
        Anonymous Coward

        Interestingly, the Gatwick whiteboard photo here:

        https://twitter.com/raulmarcosl/status/1031496898375507968

        (linked from BBC news article) shows a flight BA9256 to Heathrow.

        A flight number that doesn't exist, on a route that (supposedly) doesn't exist.

        The whiteboard reveals things that are not supposed to be known...

        1. Anonymous Coward
          Anonymous Coward

          Its not exactly secret.....

          This is a positioning flight number - there are quite a number of them depending on the destination and airline.

          Its partially a paperwork excersize - allows flight planning and gives a callsign.

          It probably shouldnt be on the whiteboard - but when you dont know what you are doing - and you are in meltdown mode....mistakes happen.... this just isnt secret - it just provides absolutely no useful information to passengers ....

        2. Michael H.F. Wilkinson Silver badge

          Don't knock whiteboards!

          A whiteboard is an essential software development tool! It requires no internet connection, doesn't use power, is multi-user, multi-tasking, and zero start-up time. Bliss.

          When we had to move to a new building, I (successfully) insisted that my old, really big whiteboard be transferred to my new office, even though they claimed it couldn't be done. It was too heavy for the idiotic wire-suspension system the architect insisted on using, and we were not allowed to attach anything directly to walls (architect's orders, again). After some delays, my whiteboard magically attached itself to the wall, and there it still is, helping me design new (parallel) algorithms for image analysis, amongst many other tasks.

          1. Anonymous Coward
            Anonymous Coward

            Re: Don't knock whiteboards!

            Also, whiteboard surprisingly modern for Gatwick ...

          2. keithpeter Silver badge
            Facepalm

            Re: Don't knock whiteboards!

            A roll of 'magic whiteboard' can be handy at times when improvising. Care is needed when erasing near the edges of the pieces though...

      3. shedied

        You almost forgot the must-have accessory for the modern whiteboard: the whiteboard backup tool that can save what was written, and save the information in a highly compressed format. It is commonly called the whiteboard eraser so that the nontechnical people can also use it, without having to resort to a "HOWTO manual" but they find the entire set very easy to use (whiteboard+marker pens+eraser), especially in a tight situation like the system failure at Gatwick's.

    2. a_yank_lurker

      @AC - They forget that all idiot proofing does is generate better idiots.

    3. fidodogbreath

      We have also just future proofed our entire IT network

      Sadly, they did not present-proof it.

      1. m0rt

        They certainly future proofed it.

        It has been proofed against working in the future.

  3. ibmalone

    "Passengers at "London"** Gatwick Airport"

    Of course, depending where you are in central London it takes about the same amount of time to get to any of the big three (LHR, LGW, STN). LCY is definitely in London and a bit quicker, though ideally you'll want someone paying for your flight.

    1. Anonymous Coward
      Anonymous Coward

      I see that Lydd airport in Kent is still calling itself "London Ashford Airport" on its website. It takes a while to drive there from Ashford, let alone London.

      1. katrinab Silver badge

        And Kidlington Airport calls itself "London Oxford". You can get to Birmingham quicker.

      2. TkH11

        London Stansted is not in London. It's all a marketing strategy to convince foreigners they are landing in London.

        1. Anonymous Coward
          Anonymous Coward

          Ryanair will be pleased with all airport names having London as the prefix.

          1. Anonymous Coward
            Anonymous Coward

            Ryanair probably has flights to London Le Touquet as part of it's Brexit strategy, though ....

    2. Korev Silver badge

      Gatwick Express recently got its knuckles rapped for misleading adverts as their service is frequently slower than advertised

      1. Lee D Silver badge

        Come on people.

        LONDON SOUTHEND.

        Since when?!

        1. Anonymous Coward
          Anonymous Coward

          LONDON SOUTHEND.

          Since when?!

          I'm not sure, but definitely some time after Luton was advertising itself as a London airport

          1. Chris G

            Back in the '90s. The old RAF Manston was threatening to become a 'London' airport, London Biggin Hill is actually within the London Borough of Bromley but the infrastructure needs a bit of work, although I have seen a few interesting aircraft when I worked there in the '70s.

            1. The humble print monkey

              You fly by helicopter or spitfire to Biggin Hill, then off on your Learjet.

        2. ibmalone

          Come on people.

          LONDON SOUTHEND.

          Since when?!

          If nothing else, it's quite a fun airport. You can pretend it's your own private terminal.

      2. ibmalone

        Gatwick Express recently got its knuckles rapped for misleading adverts as their service is frequently slower than advertised

        To be fair to Gatwick (and to be fair to Gatwick Express, but that's not really to their advantage), it's not that the journey can't be done it in 30 minutes, it's that it's not infrequently delayed. When you consider the fact that the southern service on the same route is cheaper and only takes a few minutes more (assuming not delayed...) and leaves at intervals between the express departures you find yourself wondering about the value of the "Express".

    3. Admiral Grace Hopper

      I'm still waiting for my local aerodrome to be renamed "London Ringway".

      1. TkH11

        You must be in Manchester then.

  4. Anonymous Coward
    Coffee/keyboard

    Best Airport information infrastructure is at...

    ...Mulu International airport.

    There are four flights a day, all going to Kota Kinabalu. To display this busy schedule, there is one departure screen and one arrivals screen, high on the wall in the joint departures/arrivals hall, fixed neatly next to each other. On a wooden shelf near the screens, are two identical generic tower PCs. One is labelled 'departures screen', the other 'arrivals screen'. The cabling appears to be exemplary. I suspect neither PC is particularly taxed.

    Perhaps the cloud gurus at Gatwick could learn an thing or two from this.

    1. Doctor Syntax Silver badge

      Re: Best Airport information infrastructure is at...

      "Mulu International airport"

      The army helipad in S Armagh used to be styled Bessbrook International Airport. I'm not sure they were supposed to be international on a regular basis.

  5. JeffyPoooh
    Pint

    I like Frankfurt Airport's flipboard display system...

    Technically called "Split-Flap Display".

    Years ago, since I had time between flights, I stood there for a while and watched it noisily refreshing each minute. Hilarious.

    (They'd still have cables that might be cut...)

    1. Yet Another Anonymous coward Silver badge

      Re: I like Frankfurt Airport's flipboard display system...

      But the advantage of cloud-based is that if you ever had to move Gatwick airport to a new location you wouldn't need to re-locate the server

      1. Anonymous Coward
        Anonymous Coward

        Re: "if you ever had to move [...] airport to a new location "..

        "if you ever had to move Gatwick airport to a new location "

        Didn't the Germans try something similar to that, when Munich airport moved overnight?

        Ah yes, back in 1992, see e.g.

        https://en.wikipedia.org/wiki/Munich_Airport

        That would be back in the day when "industry standard" didn't mean "mediocre" (or worse), and when "failure is not an option" still meant something even at board (ie Director) level.

        The introduction of compulsory MBA [1] training instead of MBWA [2] put an end to almost all of that.

        [1] MBA: managemement bug***ing about.

        [2] MBWA: management by wandering about.

        1. J.G.Harston Silver badge

          Re: "if you ever had to move [...] airport to a new location "..

          One of my nephews* moved Hong Kong airport overnight.

          *not on his own - took about 2000 peeps IIRC. Lovely bit of overtime.

          1. Anguilla

            Re: "if you ever had to move [...] airport to a new location "..

            Re: "if you ever had to move [...] airport to a new location "..

            One of my nephews* moved Hong Kong airport overnight.

            *not on his own - took about 2000 peeps IIRC. Lovely bit of overtime.

            Well, as a looooong time resident, I recall that it didn't go entirely smoothly - and was renamed [for a while] as Chek Lap Kok-up airport by the frustrated passengers & staff.

            Nowadays - it is MUCH BETTER.

    2. GSTZ
      Happy

      Re: I like Frankfurt Airport's flipboard display system...

      ... as by design it contains a mechanical cache that would help greatly at least during short outages of the flight information system. And there is no dependency on external clouds, no chance to cut fibre cables leading there, and no impact by all those many other cloudaches so often interfering in today's wonderful marvellous Internet world ...

  6. Anonymous South African Coward Bronze badge

    Oh dear.

    So which one's the more resilient?

    The oldstyle split-flap display (which cannot display advertisements or lewd pictures *ahem*), or the newfangled one with wide-screen monitors (which ne'er-do-wells can have fun with by displaying pr0nz and other unsavoury stuff, not to mention boring advertisements)?

    Not to mention displaying bluescreen errors for world+dog to see and comment on (preferably on El Reg).

  7. Julian 8 Silver badge

    4G

    What LGW should have done is go with an an ISP who have a brilliant tie in with a mobile provider and could offer a decent 4G service incase the redundant cables to LGW were cut.

    1. James 51

      Re: 4G

      Would cost more and it's tempting to think that redunacy is just costs that should never be needed.

    2. katrinab Silver badge

      Re: 4G

      Not as resilient as you might think. Where I live, the 4G tower and my landline are both connected to the same green box. If there was a problem in the 10 meters or so between the green box and my house, then I could switch to 4G, if it is the 1.5km or so between the green box and the exchange, then both would be taken out.

    3. dubious

      Re: 4G

      I'd suspect that what they really should have done was run the screens off a pair of on-site servers preferably split over server rooms at opposite ends of the airport, but CxOs don't get lauded for not racing to (public) cloud.

      1. Chris G

        Re: 4G

        I live 5.5 Km outside of my village, when it rains the 4G and 3G disappear quite often the signal goes because nobody can be bothered to wind it up. For something like a major airport all comms other than ground to air should be multiple redundancy hard wired.

        If I flew regularly I would use the Flight Board app.

      2. Anonymous Coward
        Anonymous Coward

        Re: 4G

        >I'd suspect that what they really should have done was run the screens off a pair of on-site servers preferably split over server rooms at opposite ends of the airport, but CxOs don't get lauded for not racing to (public) cloud.

        They could probably have tied a couple of Raspberry Pis (or similar) to the back of each screen and created a Python application to update them all. It can't exactly be challenging to update flight information.

        1. Peter X

          Re: 4G

          NEC can supply Raspberry Pi equipped displays... honestly, I'm pretty sure a particularly sharp 9 year old child could've probably manage to set something workable up and running.

    4. DonL

      Re: 4G

      "What LGW should have done is go with an an ISP who have a brilliant tie in with a mobile provider and could offer a decent 4G service incase the redundant cables to LGW were cut."

      Which is exactly the type of connection we ordered from Vodafone (not in the UK though). When the fiber cable gets cut, the IP block automatically gets routed to the 4G connection.

      For our branch offices we use 3 WAN connections from 3 different providers (2x VDSL2 and 1x 4G), so the IPSEC tunnel is automatically rerouted over another connection when the current connection goes down. This all for €150 p.m. connection costs in total with a € 300 Ubiquiti Edgerouter Pro. (per branch)

      I guess airports lack the creativity and freedom to implement such rather simple/cheap but very effective solutions.

      1. Alan Brown Silver badge

        Re: 4G

        "I guess airports lack the creativity and freedom"

        It's not that so much as manglement being dazzled by shiny shiny being shown to them by some east end barrow boy and telling the underlings it shall be done THIS way.

        They've given up telling $bossage that "THIS WAY" is guaranteed to fuck up, and merely ask for that order in writing, for when the inevitable fingerpointing happens.

      2. Alan Brown Silver badge

        Re: 4G

        "Which is exactly the type of connection we ordered from Vodafone (not in the UK though). "

        Voda do offer this kind of connection in the UK.

        It was amusing when they offered us a 1Gb/s ethenet connection with 3G backup.

        "Our average never falls below 100Mb/s. Can your fallback handle that?"

        "Each radio link can go about 7Mb/s" (we'd already measured it and knew it was slower)

        "7 is a lot less than 100. If we need more, we can run a bunch in parallel?"

        (Some days later).... "Um.... no. We don't have that capacity"

        "How much total capacity can you offer on the backup?"

        "7Mb/s"

  8. Ian P

    "All flights are currently leaving on time." Much easier to do this when the passengers cannot find their planes.

    1. Rusty 1

      Or they could do a TFL and not specifically state that everything is on time, but instead, just running normally.

      1. Korev Silver badge
        Joke

        Or they could do a TFL and not specifically state that everything is on time, but instead, just running normally.

        That'd be a sneaky, underground thing to do...

        1. Yet Another Anonymous coward Silver badge

          Public transport here solved the problem by removing "delayed" from the displays.

          Services upto an hour late show "on time", after that they just don't show the "on time" label

  9. Elmer Phud

    Blah Blah

    ""an innovative, cost effective system that are easily scalable, more flexible and resilient, and require considerably less infrastructure and maintenance"."

    Someone has got a fat bonus at the costs of several workers.

    1. Anonymous Coward
      Anonymous Coward

      Re: Blah Blah

      Especially now you can run each monitor out of simple and cheap Pi-like SoC, which, with the proper software and data, can also keep on working even if there is a relatively brief outage... and maybe even switch to WiFi if needed...

      But maybe they forgot to give their IT team an IFR rating to teach them how to go through "the cloud" safely?

      1. Dan 55 Silver badge

        Re: Blah Blah

        There's at least one manufacturer which does large public display monitors with Pis built in, forgot who it is though.

        1. Dan 55 Silver badge

          Re: Blah Blah

          It's NEC.

      2. SplitBrain

        Re: Blah Blah

        I used to work there many years ago, the FIDS system ran on Sun v240's running Solaris 9, there were about 6 of them split airside and landside, redundant power and networks and SAN. BAA used to be a great Unix shop, would have taken a nuclear event to take it offline.

        "Cloud" eh....

  10. Anonymous Coward
    Anonymous Coward

    'Pointed a finger at Vodafone'

    No - Point the finger at CloudFog and raining multiple-points-of-failure!

    #Cloud_Hype - #Laughable_Cloud_Redundancy- #Fat-bonus-for-execs

  11. andy 103

    Cost

    It really is as simple as this:

    Install a redundant fibre cable, or other systems to ensure if a loss of connectivity happens there is a backup

    vs:

    Get a few members of staff - on less than 50k a year (accounting for London salaries) - to write the flight information on a whiteboard, on only 1 - 2 days of the year when the situation actually occurs.

    The people who oversee and are "in charge" of these things want Range Rovers, fat pensions and country houses. Putting in redundant systems takes away from that, and the spleen venting on Twitter is a small price to pay for them being able to have the spare change.

    And yet, people are still confused and shocked as to why these situations occur. It comes down to money, like everything else. It's not because it can't be done, it's because it "can't" be done.

    Airports don't go out of business due to some customers being pissed off about whatever they say on Twitter. Don't like it? Fly from a different airport - where you may experience the same or worse - or shut the f**k up. What are the other options?

    1. Richard 12 Silver badge

      Re: Cost

      The cost is far higher than that.

      Gatwick want to expand, but this (and many other) events will be used to prove that they simply could not cope with any expansion whatsoever, as they are already having trouble keeping track of the flights they currently have.

      Doesn't matter whether it's true of course, but it's another self-administered nail in their coffin.

    2. Alan Brown Silver badge

      Re: Cost

      "Airports don't go out of business due to some customers being pissed off about whatever they say on Twitter. "

      They _do_ however sack manglement when travel insurers start turning up wanting their pound of flesh for payouts caused by this kind of cockup.

      And I'm quite sure the technical staff will have helpfully provided anonymous access to the paper trail where the vulnerabilities were pointed out some time ago, so Gatwick's liability insurers will be reluctant to pay out.

      That's the kind of thing that brings down seagull management.

      1. Doctor Syntax Silver badge

        Re: Cost

        "when travel insurers start turning up wanting their pound of flesh for payouts caused by this kind of cockup."

        Travel insurers, payout. Does that sort of thing happen?

  12. Stevie

    Bah!

    THE cable??????

    1. Anonymous Coward
      Anonymous Coward

      Re: Bah!

      To be fair, it makes problem diagnosis so much quicker if you only need to check one failure point. Just think of the time saved

  13. James 51

    hmmm sounds like business case for some massive e-ink screens (low power consumption so if need be could run for a while on batteries) hooked up to a mesh wi-fi network (or maybe GSM). Without the mesh could still go war driving on those little golf cart things and updated them from a laptop.

    Need a daydream icon.

  14. Crypto Monad Silver badge

    Nobody has yet asked the obvious:

    Does Gatwick have an online departures board? You know, the sort of thing that people could access with those mobile screens that they carry around with them?

    And was it affected by this outage, or not?

    1. Hans Neeson-Bumpsadese Silver badge

      Re: Nobody has yet asked the obvious:

      Does Gatwick have an online departures board? You know, the sort of thing that people could access with those mobile screens that they carry around with them?

      And was it affected by this outage, or not?

      A good question but even if it was unaffected and available, I don't think it would have been a viable alternative.

      As individuals, it's really easy for us to think rationally and come up with alternatives when things go wrong. However, one thing I've learned from years of air travel is that once you get more than a few people together in an airport they become collectively stupid.

      1. Dan 55 Silver badge

        Re: Nobody has yet asked the obvious:

        However, one thing I've learned from years of air travel is that once you get more than a few people together in an airport they become collectively stupid.

        But enough about airport security.

      2. JulieM Silver badge

        Re: Nobody has yet asked the obvious:

        It works like resistors in parallel: The inverse of the combined IQ of a group of people is equal to the sum of the inverses of each individual's IQ.

    2. TkH11

      Re: Nobody has yet asked the obvious:

      From information I have seen elsewhere, Gatwick provides an app for mobile phones but this was affected too.

    3. Anonymous Coward
      Anonymous Coward

      Re: Nobody has yet asked the obvious:

      https://www.gatwickairport.com/flights/?type=departures

      For a cloudy system it doesn't seem to be particularly resilient.

    4. djsds

      Re: Nobody has yet asked the obvious:

      Even if the website/app went down, they could have set up a Google spreadsheet (or similar) and had someone updating the information from whatever data source they were using for the whiteboard, then put up a short URL to the sheet on the whiteboard for those with smartphones to consult.

  15. Lee D Silver badge

    Flew from Gatwick last week.

    Best bit - their app just pops up and tells you the gate number when you've selected your flight, no faffing.

    In fact, went Gatwick-Spain-Stansted with only a passport and a smartphone (no tickets, boarding cards, etc.) and it all worked amazingly well.

    The BA app is also quite good, especially as they can notify people at each end if you're delayed and it does a countdown to online check-in and your flight.

    I absolutely detest flying, I'd like to point out, not because of a fear but because of the faff. However they managed to sort it out this time round and I barely queued at all at any of the airports.

    But they still need to sort out departure lounges (i.e. a less humungous duty free to walk through and ignore everything, and more seating!), timings (if I need to be on-time, so does my plane), etc.

    Shame that they've messed up, but it actually worked really well for me the other day and I didn't need to bother to check the physical boards at all.

    1. Dan 55 Silver badge

      Obviously not Ryanair then. It's like they asked themselves what passengers would need of their app in an airport and implemented the exact opposite.

  16. 0laf
    Childcatcher

    Yeah but the senior bod who signed off on this thinks he saved money (he probably didn't) and got to tick the Cloud/Digital/Agile/Transformation/Disruptive buzzword tick-boxes with this so it's all good.

    1. Korev Silver badge
      Joke

      Well, it's certainly disruptive....

  17. Anonymous Coward
    Anonymous Coward

    Fair enough wanting to avoid having a PC behind each screen, and thus using a 'dumb terminal' approach, but there is absolutely no reason why this should have required connectivity *external to the airport*.

    I presume the gate allocation etc is managed on-site, so why not have the exact same screens, but talking to an on-site pair of redundant servers (perhaps with failover to a cloud based server if you're worried about the on-site ones not being resilient).

    It's this obsession with "moving to the cloud" people seem to have, I've worked with several companies who are doing it despite a simple calculation showing it is going to cost them far more (as their applications are not normally designed in a 'cloud first' way, so they require quite a lot of permanent infrastructure to be running), and give them much less control / ability to fix things if/when there are outages (as to the cloud provider they're just another number, whereas before they had their own techies able to deal with things).

    1. Lee D Silver badge

      How do you know if a flight coming in is going to be delayed?

      You have to connect to something - the airline, the air traffic control (unlikely outside UK airspace), the other airports, etc. Without that you have no idea if a flight is delayed, thus no idea if the next one has to be bumped down, etc.

      You have to have a live connection for this to be anywhere near vaguely useful. That they can't get a redundant connection to Gatwick is ridiculous. They should never fall over, there should be multiple fibres km's apart from each other connecting to different towns at minimum I imagine. Can't fathom what Vodafone is doing running that, to be honest. Unless they're complaining that a 4G backup didn't work (but then... if that's the case... surely that 4G connection just runs from a local leased line, etc.?

      That they need a connection isn't the problem. That they haven't got a suitably redundant connection is the problem. It looks immensely like the boards aren't even connected as well as, say, the cell tower in the terminal, or the free wifi.

      1. cantankerous swineherd

        the with the walkie talkie at the whiteboard had a connection.

      2. Anonymous Coward
        Anonymous Coward

        "That they need a connection isn't the problem. That they haven't got a suitably redundant connection is the problem"

        Also .... didn't BT Wholesale have a large facility at City Place, round the corner from the airport? Think they've moved, but surely there will be at least some fibre in the area .....

    2. Alan Brown Silver badge

      "wanting to avoid having a PC behind each screen"

      When that entails _literally_ a desktop box behind each screen (which was the case not so long ago) you can understand it.

      That doesn't excuse having the important bits offsite though. Airlines and airports are complex IT operations (core business) that happen to run logistics and aircraft. Outsourcing your core stuff is a bad idea.

  18. Allonymous Coward
    Joke

    Obligatory XKCD

    Including the requirement for "caching":

    https://xkcd.com/908/

  19. Aristotles slow and dimwitted horse

    It might all be in the cloud...

    But it still needs physical cables!!!

    1. Yet Another Anonymous coward Silver badge

      Re: It might all be in the cloud...

      >But it still needs physical cables!!!

      If only vodafone had a way of transmitting data without cables, some sort of cableless network

      1. Anonymous Coward
        Anonymous Coward

        Re: It might all be in the cloud...

        Vodafone use lots and lots of physical cables. So many physical cables it would make your head spin.

  20. nuked
    Thumb Up

    I love that a cloud service has brought an airport to its knees due to a cabling issue.

    1. Anonymous Coward
      Anonymous Coward

      just wait until there's "fog computing" to cope with as well .....

    2. Yet Another Anonymous coward Silver badge

      >I love that a cloud service has brought an airport to its knees due to a cabling issue.

      And the service being managed by a mobile phone company

  21. Anonymous Coward
    Anonymous Coward

    London Gatwick, Obviously...

    ...not.

  22. Anonymous Coward
    Anonymous Coward

    Use a camera

    placed in front of a single white board in the office and broadcast the image to all the screen via HDMI cable or whatever inputs are available on the display screens.

    Staff could stand in front of said camera with an advertisement hording from time to time too.

    Get it right and nobody would know the cloud was down.

  23. Anonymous Coward
    Anonymous Coward

    To all system designers.

    When creating the design, be redundant or be redundant.

  24. Anonymous Coward
    Facepalm

    Cloud-based flight information display system ..

    The Flight Information Display System (FIDS) at Gatwick Airport was the result of a project that kicked off back in 2015 to replace legacy systems that required a separate PC running behind the majority of the airports’ screens

    Running the screens from the one 'cloud' solution through a single fibre cable sounds like some bean-counters idea of saving money. Multiple devices running multiple screens with multiple data paths is the correct solution. You could do it with Raspberry Pis connected in a banyan-tree topology.

  25. JimC
    Big Brother

    Pedantic Geography

    I contest that West Sussex adjoins London. There's still a few miles of Surrey and Kent that the GLC hasn't subsumed...

  26. Duffaboy
    FAIL

    When will they learn

    Seriously cutting costs never ever saves money in the long term

  27. Dwarf

    SPOF

    So if there is one Single Point of Failure thats been discovered, I wonder how many others are lurking waiting to be found.

    Resilient platform design isn't difficult or new, so its worrying that this made it through design, design review, network 3rd party assurance (their design and review processes), implementation and testing. The obvious question here - where's the governance and who's going to get the chop because it was not sufficient ?

    1. Anonymous Coward
      Anonymous Coward

      Re: SPOF

      Where's the governance?

      https://www.gatwickairport.com/business-community/about-gatwick/company-information/ownership-management/executive-management/

      Who's gonna get the chop?

      aka

      Who's gonna pay for this?

      No one at corporate board level ever pays for their mismanagement (not in the UK anyway). Staff and customers will pay, as is traditional.

      1. Anonymous Coward
        Anonymous Coward

        Re: SPOF

        At least one of those responsible has already moved on... I know this as they are now lending us the benefit of their experience, which apparently includes not worrying about old fashioned operational standards and resilience, and instead being dynamic and embracing risk. Looks like there was a bit too much of that during their tenure at Gatwick!

        1. 0laf
          Childcatcher

          Re: SPOF

          My commiserations. That person talks exactly how your management expect 'dynamic go getters' to talk. All your talk of resilience and planning will just introduce a cloud of negativity into projects and you'll be sidelined. Mr/Mrs 'Dynamic' will shortly be shifted into a nice senior role with Cyber, Digital, Evangelist, Solutions, Architect or some other vacuous pish in the job title.

          They will then move on to the next gullible gobshite loving exec team and you can enjoy picking up the pieces and taking the blame for the failure of governances in controlling Mr/Mrs Dynamic.

          Something for you to look forward to.

          1. Anonymous Coward
            Anonymous Coward

            Re: SPOF

            I wish someone had told me this before, you've just explained why I haven't been promoted to one of those roles. I'm still digging these idiots out of their holes.

    2. Doctor Syntax Silver badge

      Re: SPOF

      "design review, network 3rd party assurance (their design and review processes)"

      The management view: These reviewers, they're so negative about things. If we ever took notice of them we'd never get anything done.

    3. 0laf
      Stop

      Re: SPOF

      I suspect form personal experience (and stories from many other security bods) that the group with the project of updating the info boards probably specifically hid the project from anyone who might have pointed out the lack of resiliency and therefore made them do things properly.

      I know security are not popular with projects but id we get spoken to early on we can indicate the howlers that might happen. It's when we find out about things late on that the big 'stop' notices come out.

  28. cantankerous swineherd

    resilience at its best. most services just shrug their shoulders when the digger hits the fibre optic. regular tweeting of the whiteboard would have been a bonus!

  29. Cynic_999

    3Mbps?

    Seems a bit of overkill, and implies that each screen is fully refreshed individually whenever the slightest thing changes. If the displays had a tiny bit of intelligence so as to all use the same raw flight information which they format and scroll themselves, a 500Bd link (at the most) would be more than sufficient. I shouldn't think that all the information needed to be displayed on any screen would take more than 50 bytes per flight, so even at 10 bits per character (e.g. 8 data, 1 start, 1 stop), 500Bd would be able to handle 1 new flight per second, which is at least 2 orders of magnitude faster than aircraft arrive & depart, so historical information can easily be interleaved between new flight data for initialisation of freshly powered-up screens. Thus already powered screens would update within 2 seconds of new flight information becoming available, and a freshly powered-up screen would be fully populated with a list of flights in under 2 minutes.

    Compared to the rest of the hardware costs, a simple CPU and a few KB of RAM would be an insignificant cost. Add a radio receiver and a 500Bd radio link could be implemented that needs no cabling to the screens other than plugging into the mains.

    1. Anonymous Coward
      Anonymous Coward

      Re: 3Mbps?

      You're forgetting that they're probably running Windows and therefore pulling gigs of patches and AV updates down daily to each screen... and if there's no resilience in the design there's probably no WSUS etc either...

  30. Anonymous Coward
    Anonymous Coward

    Great Scott!!!

    ONE POINT TWENTY ONE GIGABYTES?!

    This is heavy, Doc!

    This event caused a cataclysmic chain reaction which caused dozens of people to miss their flights. The entire annihilation of the space-time continuum can be traced back to this one key event!

    1. Aladdin Sane
      Trollface

      Re: This is heavy, Doc!

      Do you even lift?

  31. Fruit and Nutcase Silver badge
    Happy

    Gatport Airwick

    was how Ray Moore used to refer to Gatwick in the travel update on the early morning show on BBC Radio 2

    1. Intractable Potsherd

      Re: Gatport Airwick

      I prefer Gitwack - it seems appropriate (one of the worst airports I have ever used).

  32. Anonymous Coward
    Anonymous Coward

    no redundancy in the internet link

    Thanks heavens (or whatever deity you prefer) that the people who run the Gatwick airport displays don't actually operate the planes that fly from the place.

    Left to them, each airliner would have one socking great engine and the plane would be controlled by a pilot based in the cheapest possible location remotely linked by a single non-resilient radio link. No allowance would be made for holidays, sickness etc.

    The profit motive is essential in running a business, but it mustn't be the only motive.

    1. Anonymous Coward
      Anonymous Coward

      Re: no redundancy in the internet link

      "Ladies and gentlemen, this is your captain speaking. The aircraft is losing it's connection to the Cloud and so you're about to fall out of them."

    2. Yet Another Anonymous coward Silver badge

      Re: no redundancy in the internet link

      But each aircraft would contain a branch of sports direct, 3 bars and a raffle for a sports car - so would never get off the ground anyway

  33. OzBob

    So was William Sadler found in charge of some mercenaries

    in a church on the outskirts of the airfield, with a lot of air traffic control equipment? I've seen this before.

  34. Anonymous Coward
    Anonymous Coward

    Re. Die Hard VI: Departed

    Yeah, that would be Bruce Willis.

    Also IIRC he took over *after* the bad guys did some damage.

    To be honest, whoever is responsible for this complete quadroclusterfsck of Biblical proportions should do the honorable thing and fall on his/her/its sword.

    Maybe they can get a job doing failure analysis, plenty of prior experience is needed.

  35. Anonymous Coward
    Anonymous Coward

    Total Design Failure

    I once worked for a company that builds the old type of system that LGW replaced with this shiny new Cloud System.

    Our systems had dual newtork connections. Physical not virtual. These went right back to the two totally separated DC's. You could take out on DC and everything would fail over.

    The FIDS Display had totally separate network cable runs apart from the last 5m or so. The runs went off in different directions through the building wiring. Those sort of decisions need to be made really early in the design of the terminal. LGW-South dates from the late 1950's.

    LGW-North is at least 20 years old. Just like LHR-T1 you get to a point when you need to knock it down and start again. LHR-T3 should be next for the demo team. Airport Terminals are really one huge great system. Most systems are connected to others in many ways. Changing one is just the same as putting sticking plaster on a cut.

    There is a reason why I avoid LGW if I can and I was brought up in Crawley.

    Sorry, this is IMHO clearly a cost cutting move by the people who now run LGW that may well come back and bite them really hard.

    1. Anonymous Coward
      Anonymous Coward

      Re: Total Design Failure

      Yeah but.... if you put all your stuff in the cloud you don’t need all that expensive dual redundant diverse routing shit... oh wait... actually you do.....

  36. DarkLordofSurrey

    ** And technically not in London at all, but in the adjoining home county of West Sussex.

    NOT adjoining London at all I think you'll find Surrey in between....

  37. Anonymous Coward
    Anonymous Coward

    The Cloud...

    Other people computers that prevent you flying up into the clouds.

  38. J.G.Harston Silver badge

    When they said they were using their standby manual whiteboard display boards, I pictured something like this - like the big display board you see in WW2 RAF movies with WAAFs scurrying up and down ladders. Not a poxy conference room flip chart.

    1. Intractable Potsherd

      I had that exact scenario in my head, but knew it would be far more prosaic. I like the world in my head.

  39. Anonymous Coward
    Anonymous Coward

    Where are the Whiteboard AI / Bots going to come from?

    https://www.bbc.co.uk/news/business-45240758

    https://www.bbc.co.uk/news/technology-45219902

    ~~~~~~~~~~~

    The hype of Cloud / AI / Automation / Algos is amazing.

    What happens when it goes titsup, as it will sometimes.

    Will the whiteboard human robots all have been laid off?

    The Gatwick clusterfck is the real Disruptive-Technology.

  40. eldakka

    TITSUP

    * Tottering Information Terminals Seriously Upset Passengers

    Total Inability To Support Uninterrupted Processing.

    (I.e. single point of failure, no HA).

  41. anonymousI

    Some targetted voting?

    Has an IT droid from LGW been through the comments section? It looks like nearly every post in this thread has attracted one downvote.

    No, of course not; they would all be too busy "future-proofing".

    1. Mark Exclamation

      Re: Some targetted voting?

      I was about to post the same thing. At the time of posting this, only 3 posts have not received the obligatory downvote. There are some very strange people around, who seem to have very boring lives.

      1. Anonymous Coward
        Anonymous Coward

        Re: Some targetted voting?

        Those down votes.

        That will be Ms McGillicutty, the Airport Managers Secretary...

  42. SAdams

    Everything is just magic when you put it in “the cloud”... no need for any designs, you can have lots of agile applications without any designs and they will just work fine forever.

  43. Ben1892

    Really no excuse, some of the most prolific and redundant dark fibre in the country runs through Crawley heading for London

    Having said that, the redundant bit of the connection is probably in "phase 2" of the project as they were in a hurry to deliver something, anything - I mean what are the chances of someone digging throu......

  44. SteveTM

    Absolutely no failover consideration when this job was speced? These job sites will have consultants and project managers all earning decent money and none of them thought about the "What if..". Its a complete joke.

  45. DuncanLarge Silver badge

    Basic HA

    We have 2 connections to the internet. On opposite sides of the building, provided by different providers. One is Virgin, the other is BT.

    We can switch between them with ease. It pays to think ahead Gatwick..

    1. yamahaga

      Re: Basic HA

      Having worked for one of those ISP's I can confirm this a very common solution for businesses a lot less reliant on their connection than Gatwick airport. I almost spat my coffee out when I heard that a fibre break caused this problem. Everyone involved in this project should be sacked but will be promoted, given bonuses or suspended on full pay for 6 months...

  46. TonyH000

    Expected

    Many years ago my company had a contract to supply all the PCs for Heathrow where they functioned impeccably. Each year we bid for Gatwick but were never chosen. They were only interested in the cheapest price. You get what you pay for.

  47. steviebuk Silver badge

    Ironic

    "an innovative, cost effective system that are easily scalable, more flexible and resilient, and require considerably less infrastructure and maintenance"

    Although it wasn't the cloud host that was the issue, part of cloud is the link to it that can also be an issue. Greedy companies relying to much on "the cloud" is what annoys me.

    I still don't understand how London airport can run it's air traffic control 80 miles from the airport. Yes they claim to have redundancy in place. But the whole idea of it being at the fucking airport is when John McClane is around, if the anything happens you can look out the fucking window with binoculars.

  48. Anonymous Coward
    Anonymous Coward

    ONLY 3Mb/s?!

    I see a lot of people on here talking about how the data connection requirements are so low that they shouldn't a problem. And yes, a simple bridge to a nearby homeowner's WiFi would suffice. Or 'appropriate' a satellite connection from a parked 'plane. Or any number of other solutions.

    But 3 Megabits per second of WAN required just to update a bunch of boards that are all working from the same data source and don't need to update quickly? That's a surprisingly high amount of traffic for something with so little actual data throughput, especially for a small dataset that's being generated and consumed in the same facility.

    Is it just horribly inefficient (as well as being unreliable) or is there more to these boards than just displaying flight information?

    1. Anonymous Coward
      Anonymous Coward

      Re: ONLY 3Mb/s?!

      Were the advertising hoardings down too? 2kb for departure data, the balance is for advertising.

      1. Anonymous Coward
        Anonymous Coward

        Re: ONLY 3Mb/s?!

        They're individually streaming adverts live to each screen rather than having a cached bank of them to cycle through?

        There's no resilience in this built in at all, it's idiotic.

  49. sanmigueelbeer

    The Flight F*cked Information Display System (FIDS) at Gatwick Airport

    Fixed that for you.

    1. Anonymous Coward
      Anonymous Coward

      FUDS seems more appropriate.

  50. herman

    ADS-B

    Hmm, where is my RTL SDR dongle and adsb program when I need it?

  51. Joe Gurman

    I must be missing something

    ....not that that would be unusual.

    This is still 2018, right? When world + dog carries a vibrating, light-up Internet fondletoy with them at all times? Is there any airline passenger who cannot consult their airline's app for a check on flight status, including gate? Or, if their phone was stolen during security screening, ask another traveler politely if they could use their phone to check?

    In the last three months, I've been faced with at least on instance with the larger flat-panel displays now at every gate (in US airports, at least) displaying hilariously incorrect information that conflicts with the information on the display at the gate 10 meters away. The app always had up to date correct info.

    1. Anonymous Coward
      Anonymous Coward

      Re: I must be missing something

      But who do the airlines get their data from? I suspect there was more wrong here than just a cable break, otherwise other data sources would not also be down.

      Perhaps Vodafone can publish their report.

  52. GSTZ

    Reliable Infrastructure ...

    Back in the 70's, some company invented fault tolerant (=failsafe) computers which by design had no single point of failure. They even extended that fault tolerance into their system software - if one CPU tripped over some sporadic software bug it was immediately halted, and parallel CPU's took over. Applications continued flawlessly without loss of data or any impact to the end users. By the way, that product line still exists and you can buy such fault tolerant computers today.

    Back at that time, large airports ran their critical operations locally and did link up their devices to their own local computers, not to someone else's cloud. Nor did they outsource any critical IT task that they could run locally by their own staff. Those were the days of reliability ...

  53. Luiz Abdala
    Megaphone

    Ouija Boards

    Congratulations to the staff that had a whiteboard and marker pens stashed in the back and could still provide information in a nearly fail-safe, albeit EXTREMELY sub-optimal manner.

    Just like Ouija boards* aboard Aircraft Carriers relaying the status of every aircraft on the deck, using literally nuts and bolts.

    It is better than having NOTHING to show.

    *Not literal Ouija boards, just a synoptic representation of the Aircraft Carrier deck drawn over a table.

  54. Anonymous Coward
    Anonymous Coward

    Cloud

    The problem with clouds is that you can't see through them.

    1. GSTZ

      Re: Cloud

      Another problem with today's clouds is that they are designed, built and operated for utmost cost optimization, not for high reliability and running critical applications. Such reliable clouds might be possible, but it is not very likely that they could become economically successful as a standalone offering - the beancounters would shy away from the price premium. Another obstacle is that cloud providers cannot know enough about their customer's individual businesses to provide them with the right scope of reliability they need.

      So cloud providers do promote various options that *theoretically* allow their customers to achieve the reliability needed for critical applications themselves. In real life that approach doesn't work particularly well, as customers are not deep enough into the complex art of making applications really failsafe. This would also require to have at least some control over the infrastructure of the not-so-reliable clouds availble today. More common than hard failures (like the defective fibre cable in the Gatwick example) are temporary overload situations causing application timeouts thus making services unavailable.

  55. A_Melbourne

    Departure Control

    This brings back the memories. I helped install the departure control system at Gatwick for British Caledonian Airways - some 35 years ago. I was working for Raytheon at the time. BCal had lots of pretty girls working the desks. I was on crutches due to a recent car accident.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon