So why was there no redundancy in the internet link?
London's Gatwick Airport flies back to the future as screens fail
London Gatwick Airport’s shiny new cloud-based flight information display system had a hard landing this morning as its vision of the future was brought down to earth with a bump. While collecting the Cloud Project of the Year award at the Real IT awards in May, the airport proclaimed its new screens were "an innovative, cost …
COMMENTS
-
-
-
Tuesday 21st August 2018 05:11 GMT Mike 'H'
But...VODAFONE...
Where the bloody hell was the LTE backup - 3 megabits on their Network must be child's play.
Point a directional YAGI or dish antenna towards an off-site tower to bypass on-site DAS/macros, likely afflicted with the same fibre cut, and bobs yer uncle.
Or maybe the airport should spend a bit more for a competing LTE provider for 3-megabit backup services...
-
Tuesday 21st August 2018 09:12 GMT Lotaresco
"Because oddly enough it doubles the cabling costs and that wouldn't do."
It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.
This is, quite frankly, poor practice on Vodafone's part.
-
Tuesday 21st August 2018 10:12 GMT ButlerInstitute
That wouldn't really be redundancy.
"Pulling a multi-pair cable"
But then you've got two cores in a single cable, thus failing to provide any resilience when a digger goes through the cable. Ok for a fault with the cable/core itself maybe, but not for a physical break.
Your alternative core needs to come into the site via a different route, So wouldn't be cut by the same digger. See for example major BBC facilities where there are redundant power and signal cables coming in from opposite sides of the site.
-
Tuesday 21st August 2018 12:37 GMT qwertyuiop
Two failures?
"...It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution..."
Ah, I see. We increase resilience by using a multi-pair cable but immediately reduce it by using... a multi-pair cable. When the digger goes through the cable it doesn't matter how many pairs are in it! (and yes, I know that in this case it wasn't a digger)
The only way to truly increase resilience is to have two cables which arrive by completely different routes - and that will increase the cost.
-
Tuesday 21st August 2018 14:17 GMT Anonymous Coward
Ummmm.. yeah, it does. Multi-pair cables don't help if the cable's cut (since all the pairs are cut too). If you want to be truly fault-tolerant, you need a second cable, geographically isolated, going preferable to a second ISP but in any case at least to a different access point for your ISP.
-
Wednesday 22nd August 2018 14:44 GMT Mark Dempster
>It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.<
That probably wouldn't have prevented the issue, though, as pulling 2 cables from the same location to the same destination means they're separated by millimeters at best - so both would have been cut through at the same time.
If you're going to this trouble (and for something so important why wouldn't you?) then you need 2 completely different runs of cable, preferably from 2 different providers, coming into the building at different locations.
-
-
-
-
Monday 20th August 2018 12:55 GMT yoganmahew
Re: "no redundancy in the internet link"
Outages happen. The question for me is why there was no local cache? It would have grown stale over time, but a well installed local cache with a GUI for updates could put everything into manual mode with zero impact.
Is this the Internet of Tripe future? One failing link and your IoT belt unbuckles and your trousers fall down exposing your single-point-of-failure-arse?
-
-
-
Monday 20th August 2018 14:27 GMT Flakk
Re: "no redundancy in the internet link"
This tiny whiteboard is the only departures information in Gatwick Airport right now; every screen is offline. Utter chaos. This is a signature flourish at the end of a short trip that’s been full of reminders of how badly the UK’s infrastructure is crumbling. pic.twitter.com/6r7CDVheLf
— Rob Fahey (@robfahey) August 20, 2018
To be sure, that the info boards went down due to a network outage is pretty bad. But utter chaos? Really? It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website? Failing that, maybe the airline's website?
-
Monday 20th August 2018 19:44 GMT JohnG
Re: "no redundancy in the internet link"
"...the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website?"
Reports elsewhere indicated that updates to Gatwick's mobile app were affected by the same fibre outage. It seems that the design was engineered to fail comprehensively.
-
Monday 20th August 2018 20:09 GMT David 45
Re: "no redundancy in the internet link"
Potential of a smart phone? No earthly good if the information is not there to start with. I was there today, dropping off passengers, then attempting to pick up others. Website arrivals flight information was sketchy, with some flights missing or no information against the flight numbers (so no use for one's shiny smart phone there!). Gatwick's auto. phone information system just went dead the second I entered the flight number (so no use for one's shiny smart phone there!). I had no idea of the status of my incoming passengers' flight and felt that Gatwick's main number would probably be inundated, so I didn't bother trying that (so no use for one's shiny smart phone there!). Arrivals concourse info. screens were also all over the place, with my flight number also not showing on there at all. My passengers also said that the baggage reclaim section was also not working. The main complaint I have is that there was no information whatsoever in arrivals that there was a problem - not even any public address announcements and certainly no whiteboards. There were lots of baffled-looking folk with furrowed brows looking at useless screens, obviously wondering what the devil was going on. In fact, as far as I could see, the only information that something was amiss was a rider on the flight information website saying that information screens were not working properly. A bit more communication (in more ways than one!) wouldn't have gone far amiss.
-
Monday 20th August 2018 21:48 GMT Anonymous Coward
Re: "no redundancy in the internet link"
use a proper system like flightradar24 for any inbound flight they have real time up until the plane gets to the gate. For outbound the schedule will show that minus any delays.
There ar millions of ways to fins out whats happening without the airport on screen system if you travel enough you will soon get the hang of it
-
-
Monday 20th August 2018 22:56 GMT tin 2
Re: "no redundancy in the internet link"
"It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding."
Seems that information was not making it out of the airport to whatever cloudy goodness was updating the website and app either.
-
Tuesday 21st August 2018 09:17 GMT Lotaresco
Re: "no redundancy in the internet link"
"maybe look up the gate information on Gatwick's website? "
Have you tried to do that? Good luck trying it. If you're lucky you'll get departure gate information in time to watch your flight depart. If it's working as usual you will get the information 24 hours later.
I have the Gatwick app. It's never told me a gate number before the flight has departed.
-
Tuesday 21st August 2018 08:00 GMT Hans 1
Re: "no redundancy in the internet link"
It is pure and utter design failure.
Why have 100's of systems download flight information from the cloud ? <---- that is already brain-dead, somebody in the design team has never heard of multiplications!
No resilient internet connection ? <---- that is brain-dead
Caching is not the answer to a fundamental failure of resilient network design.
It reduces internet traffic significantly and buys you time for the connection fail-over..
-
Tuesday 21st August 2018 08:30 GMT Anonymous Coward
Re: "no redundancy in the internet link"
Welcome to the future, at least the future if the current "Cloud" fad continues.
Single points of failures all round.
Someone took a digger to your fibre? Bad luck. Maybe your backup link will take up the slack, but as these are usually lesser specification it will soon grind to a halt.
Someone took the cloud server down? No local caching or redundancy? Bad luck. Maybe someone will turn it off and on again soon. Maybe the script kiddies will get bored and stop DDOSing it.
Ironically the internet was developed to survive a nuclear war.
-
Tuesday 21st August 2018 13:17 GMT jmch
Re: "no redundancy in the internet link"
"Welcome to the future, at least the future if the current "Cloud" fad continues."
the thing is, a properly implemented cloud solution gives many benefits one of which is redundancy. Except management only got the memo where cloud = cost savings and don't realise that you still need to spend money on redundant network links. Every piece of the puzzle has to have redundancy, not only the cloudy bit
-
Tuesday 21st August 2018 18:58 GMT Martin Gregorie
Re: "no redundancy in the internet link"
One thing nobody seems to have forgotten - BT and other wonderful network providers currently operating in the UK have been known to engineer their own single point of failure. It happens this way:
- The system design team specifies a disaster recovery site and a high speed connection to it
- Their network design requires separate dual redundant links from the operations centre (LGW in this case) to the main ops site and to the disaster recovery site via at least two paths which are required to leave the building via separate ducts and then follow different routes.
- These specs get handed to the network provider, whose contractors promptly ignore all the fancy separate routing details and put all the cables through a single duct so they can trouser all the money they saved by skipping all that costly separate routing nonsense.
- The local council puts a digger through the cable duct....
-
-
-
-
Tuesday 21st August 2018 09:15 GMT Lotaresco
Re: "no redundancy in the internet link"
"The question for me is why there was no local cache? It would have grown stale over time"
It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. What is needed is resilient comms and that is standard provision for systems like this. There should be no SPOFs in a real time system.
-
Wednesday 22nd August 2018 07:59 GMT yoganmahew
Re: "no redundancy in the internet link"
@Lotaresco
"It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. "
Not really, the scheduled departure and arrival times are well known days in advance. The gates are usually well known, but at lest could be manually updated (so at least people stand a chance of finding their gate). The amount of data that tranmits by FLIFO for FIDS updates is vanishingly small, being essentially designed in the 1960s. Never mind LTE, you could run it on dial-up...
-
-
-
-
Tuesday 21st August 2018 09:54 GMT bpfh
Re: "no redundancy in the internet link"
So Gatwick generates the raw departures/arrivals data, posts that data to somebody else’s computer, then the screens pull that back to Gatwick. And it only uses 3 megabits...
So 2 questions:
0. Did they really need Somebody Else’s Computer to do this (especially if the computers that run this are still there, but now offsite - out of sight out of mind?)
1. 3 megabits... cut fibre... nobody thought of setting their mobile phone into WiFi hotspot mode - or dare I say it, backup 3/4G router ? Or is mobile internet that crap there?
-
Wednesday 22nd August 2018 08:03 GMT yoganmahew
Re: "no redundancy in the internet link"
@bpfh
"DId they reall need someone else's computer"
No, but, generally you stick the flight information data on a server somewhere so you can access it from multiple sources. Of course, if you build that as PUBSUB and there was a local server serving coax to the airport... (why coax? Coax can be fixed by anyone with pliers and a piece of tape... Bring back coax!).
It's a bit weird, that they are having problems still suggests it's not just a fibre cut?
-
Wednesday 22nd August 2018 09:23 GMT defiler
Re: "no redundancy in the internet link"
I'm just going to lob this on the end here, since everyone's screaming about having a second cable (expense of installation, potential proximity to first), and others are screaming about having a cellular backup (potential proximity to cable because Vodafone).
Nobody's saying satellite link. If all you're looking for is 3Mb/sec, it's not expensive. Oh noes - 200ms latency? Who cares? If the power stays up, the data stays up. And if the power drops, the airport is closed anyway.
-
-
-
Monday 20th August 2018 14:35 GMT Anonymous Coward
Situation Normal: Outsourced Data
When LGW was run and managed by BAA, BAA IT provided the information screens in co-operation with CAA/NATS in the tower.
Now, with GIP owning the Airport, VodaFone providing the data and the tower outsourced to a German company with an unfinished website - hardly surprising really....
Too many layers of outsourcing....too cheaply....
-
-
Monday 20th August 2018 11:53 GMT Anonymous Coward
According to http://www.mediacentre.gatwickairport.com/press-releases/2018/18_05_18_it_awards.aspx
"We have also just future proofed our entire IT network for the next decade so that we can take advantage of the latest technologies, while also making the network more resilient and tolerant to disruption."
Bullsh*t...
-
-
Monday 20th August 2018 13:33 GMT Anonymous Coward
Interestingly, the Gatwick whiteboard photo here:
https://twitter.com/raulmarcosl/status/1031496898375507968
(linked from BBC news article) shows a flight BA9256 to Heathrow.
A flight number that doesn't exist, on a route that (supposedly) doesn't exist.
The whiteboard reveals things that are not supposed to be known...
-
Monday 20th August 2018 14:39 GMT Anonymous Coward
Its not exactly secret.....
This is a positioning flight number - there are quite a number of them depending on the destination and airline.
Its partially a paperwork excersize - allows flight planning and gives a callsign.
It probably shouldnt be on the whiteboard - but when you dont know what you are doing - and you are in meltdown mode....mistakes happen.... this just isnt secret - it just provides absolutely no useful information to passengers ....
-
Monday 20th August 2018 14:41 GMT Michael H.F. Wilkinson
Don't knock whiteboards!
A whiteboard is an essential software development tool! It requires no internet connection, doesn't use power, is multi-user, multi-tasking, and zero start-up time. Bliss.
When we had to move to a new building, I (successfully) insisted that my old, really big whiteboard be transferred to my new office, even though they claimed it couldn't be done. It was too heavy for the idiotic wire-suspension system the architect insisted on using, and we were not allowed to attach anything directly to walls (architect's orders, again). After some delays, my whiteboard magically attached itself to the wall, and there it still is, helping me design new (parallel) algorithms for image analysis, amongst many other tasks.
-
-
Tuesday 21st August 2018 14:48 GMT shedied
You almost forgot the must-have accessory for the modern whiteboard: the whiteboard backup tool that can save what was written, and save the information in a highly compressed format. It is commonly called the whiteboard eraser so that the nontechnical people can also use it, without having to resort to a "HOWTO manual" but they find the entire set very easy to use (whiteboard+marker pens+eraser), especially in a tight situation like the system failure at Gatwick's.
-
-
-
Monday 20th August 2018 12:24 GMT Korev
Gatwick Express recently got its knuckles rapped for misleading adverts as their service is frequently slower than advertised
-
Tuesday 21st August 2018 14:15 GMT ibmalone
Gatwick Express recently got its knuckles rapped for misleading adverts as their service is frequently slower than advertised
To be fair to Gatwick (and to be fair to Gatwick Express, but that's not really to their advantage), it's not that the journey can't be done it in 30 minutes, it's that it's not infrequently delayed. When you consider the fact that the southern service on the same route is cheaper and only takes a few minutes more (assuming not delayed...) and leaves at intervals between the express departures you find yourself wondering about the value of the "Express".
-
-
Monday 20th August 2018 12:03 GMT Anonymous Coward
Best Airport information infrastructure is at...
...Mulu International airport.
There are four flights a day, all going to Kota Kinabalu. To display this busy schedule, there is one departure screen and one arrivals screen, high on the wall in the joint departures/arrivals hall, fixed neatly next to each other. On a wooden shelf near the screens, are two identical generic tower PCs. One is labelled 'departures screen', the other 'arrivals screen'. The cabling appears to be exemplary. I suspect neither PC is particularly taxed.
Perhaps the cloud gurus at Gatwick could learn an thing or two from this.
-
-
-
Monday 20th August 2018 17:18 GMT Anonymous Coward
Re: "if you ever had to move [...] airport to a new location "..
"if you ever had to move Gatwick airport to a new location "
Didn't the Germans try something similar to that, when Munich airport moved overnight?
Ah yes, back in 1992, see e.g.
https://en.wikipedia.org/wiki/Munich_Airport
That would be back in the day when "industry standard" didn't mean "mediocre" (or worse), and when "failure is not an option" still meant something even at board (ie Director) level.
The introduction of compulsory MBA [1] training instead of MBWA [2] put an end to almost all of that.
[1] MBA: managemement bug***ing about.
[2] MBWA: management by wandering about.
-
-
Tuesday 21st August 2018 04:43 GMT Anguilla
Re: "if you ever had to move [...] airport to a new location "..
Re: "if you ever had to move [...] airport to a new location "..
One of my nephews* moved Hong Kong airport overnight.
*not on his own - took about 2000 peeps IIRC. Lovely bit of overtime.
Well, as a looooong time resident, I recall that it didn't go entirely smoothly - and was renamed [for a while] as Chek Lap Kok-up airport by the frustrated passengers & staff.
Nowadays - it is MUCH BETTER.
-
-
-
-
Tuesday 21st August 2018 16:17 GMT GSTZ
Re: I like Frankfurt Airport's flipboard display system...
... as by design it contains a mechanical cache that would help greatly at least during short outages of the flight information system. And there is no dependency on external clouds, no chance to cut fibre cables leading there, and no impact by all those many other cloudaches so often interfering in today's wonderful marvellous Internet world ...
-
-
Monday 20th August 2018 12:06 GMT Anonymous South African Coward
Oh dear.
So which one's the more resilient?
The oldstyle split-flap display (which cannot display advertisements or lewd pictures *ahem*), or the newfangled one with wide-screen monitors (which ne'er-do-wells can have fun with by displaying pr0nz and other unsavoury stuff, not to mention boring advertisements)?
Not to mention displaying bluescreen errors for world+dog to see and comment on (preferably on El Reg).
-
-
Monday 20th August 2018 12:31 GMT katrinab
Re: 4G
Not as resilient as you might think. Where I live, the 4G tower and my landline are both connected to the same green box. If there was a problem in the 10 meters or so between the green box and my house, then I could switch to 4G, if it is the 1.5km or so between the green box and the exchange, then both would be taken out.
-
-
Monday 20th August 2018 14:00 GMT Chris G
Re: 4G
I live 5.5 Km outside of my village, when it rains the 4G and 3G disappear quite often the signal goes because nobody can be bothered to wind it up. For something like a major airport all comms other than ground to air should be multiple redundancy hard wired.
If I flew regularly I would use the Flight Board app.
-
Monday 20th August 2018 16:37 GMT Anonymous Coward
Re: 4G
>I'd suspect that what they really should have done was run the screens off a pair of on-site servers preferably split over server rooms at opposite ends of the airport, but CxOs don't get lauded for not racing to (public) cloud.
They could probably have tied a couple of Raspberry Pis (or similar) to the back of each screen and created a Python application to update them all. It can't exactly be challenging to update flight information.
-
Monday 20th August 2018 21:05 GMT Peter X
Re: 4G
NEC can supply Raspberry Pi equipped displays... honestly, I'm pretty sure a particularly sharp 9 year old child could've probably manage to set something workable up and running.
-
-
-
Monday 20th August 2018 18:22 GMT DonL
Re: 4G
"What LGW should have done is go with an an ISP who have a brilliant tie in with a mobile provider and could offer a decent 4G service incase the redundant cables to LGW were cut."
Which is exactly the type of connection we ordered from Vodafone (not in the UK though). When the fiber cable gets cut, the IP block automatically gets routed to the 4G connection.
For our branch offices we use 3 WAN connections from 3 different providers (2x VDSL2 and 1x 4G), so the IPSEC tunnel is automatically rerouted over another connection when the current connection goes down. This all for €150 p.m. connection costs in total with a € 300 Ubiquiti Edgerouter Pro. (per branch)
I guess airports lack the creativity and freedom to implement such rather simple/cheap but very effective solutions.
-
Monday 20th August 2018 21:08 GMT Alan Brown
Re: 4G
"I guess airports lack the creativity and freedom"
It's not that so much as manglement being dazzled by shiny shiny being shown to them by some east end barrow boy and telling the underlings it shall be done THIS way.
They've given up telling $bossage that "THIS WAY" is guaranteed to fuck up, and merely ask for that order in writing, for when the inevitable fingerpointing happens.
-
Tuesday 21st August 2018 08:14 GMT Alan Brown
Re: 4G
"Which is exactly the type of connection we ordered from Vodafone (not in the UK though). "
Voda do offer this kind of connection in the UK.
It was amusing when they offered us a 1Gb/s ethenet connection with 3G backup.
"Our average never falls below 100Mb/s. Can your fallback handle that?"
"Each radio link can go about 7Mb/s" (we'd already measured it and knew it was slower)
"7 is a lot less than 100. If we need more, we can run a bunch in parallel?"
(Some days later).... "Um.... no. We don't have that capacity"
"How much total capacity can you offer on the backup?"
"7Mb/s"
-
-
-
Monday 20th August 2018 12:43 GMT Anonymous Coward
Re: Blah Blah
Especially now you can run each monitor out of simple and cheap Pi-like SoC, which, with the proper software and data, can also keep on working even if there is a relatively brief outage... and maybe even switch to WiFi if needed...
But maybe they forgot to give their IT team an IFR rating to teach them how to go through "the cloud" safely?
-
-
Monday 20th August 2018 14:38 GMT SplitBrain
Re: Blah Blah
I used to work there many years ago, the FIDS system ran on Sun v240's running Solaris 9, there were about 6 of them split airside and landside, redundant power and networks and SAN. BAA used to be a great Unix shop, would have taken a nuclear event to take it offline.
"Cloud" eh....
-
-
-
Monday 20th August 2018 12:25 GMT Anonymous Coward
Cost
It really is as simple as this:
Install a redundant fibre cable, or other systems to ensure if a loss of connectivity happens there is a backup
vs:
Get a few members of staff - on less than 50k a year (accounting for London salaries) - to write the flight information on a whiteboard, on only 1 - 2 days of the year when the situation actually occurs.
The people who oversee and are "in charge" of these things want Range Rovers, fat pensions and country houses. Putting in redundant systems takes away from that, and the spleen venting on Twitter is a small price to pay for them being able to have the spare change.
And yet, people are still confused and shocked as to why these situations occur. It comes down to money, like everything else. It's not because it can't be done, it's because it "can't" be done.
Airports don't go out of business due to some customers being pissed off about whatever they say on Twitter. Don't like it? Fly from a different airport - where you may experience the same or worse - or shut the f**k up. What are the other options?
-
Monday 20th August 2018 21:06 GMT Richard 12
Re: Cost
The cost is far higher than that.
Gatwick want to expand, but this (and many other) events will be used to prove that they simply could not cope with any expansion whatsoever, as they are already having trouble keeping track of the flights they currently have.
Doesn't matter whether it's true of course, but it's another self-administered nail in their coffin.
-
Monday 20th August 2018 21:11 GMT Alan Brown
Re: Cost
"Airports don't go out of business due to some customers being pissed off about whatever they say on Twitter. "
They _do_ however sack manglement when travel insurers start turning up wanting their pound of flesh for payouts caused by this kind of cockup.
And I'm quite sure the technical staff will have helpfully provided anonymous access to the paper trail where the vulnerabilities were pointed out some time ago, so Gatwick's liability insurers will be reluctant to pay out.
That's the kind of thing that brings down seagull management.
-
-
Monday 20th August 2018 12:43 GMT James 51
hmmm sounds like business case for some massive e-ink screens (low power consumption so if need be could run for a while on batteries) hooked up to a mesh wi-fi network (or maybe GSM). Without the mesh could still go war driving on those little golf cart things and updated them from a laptop.
Need a daydream icon.
-
-
Monday 20th August 2018 13:13 GMT Hans Neeson-Bumpsadese
Re: Nobody has yet asked the obvious:
Does Gatwick have an online departures board? You know, the sort of thing that people could access with those mobile screens that they carry around with them?
And was it affected by this outage, or not?
A good question but even if it was unaffected and available, I don't think it would have been a viable alternative.
As individuals, it's really easy for us to think rationally and come up with alternatives when things go wrong. However, one thing I've learned from years of air travel is that once you get more than a few people together in an airport they become collectively stupid.
-
Tuesday 21st August 2018 11:08 GMT djsds
Re: Nobody has yet asked the obvious:
Even if the website/app went down, they could have set up a Google spreadsheet (or similar) and had someone updating the information from whatever data source they were using for the whiteboard, then put up a short URL to the sheet on the whiteboard for those with smartphones to consult.
-
-
Monday 20th August 2018 12:54 GMT Lee D
Flew from Gatwick last week.
Best bit - their app just pops up and tells you the gate number when you've selected your flight, no faffing.
In fact, went Gatwick-Spain-Stansted with only a passport and a smartphone (no tickets, boarding cards, etc.) and it all worked amazingly well.
The BA app is also quite good, especially as they can notify people at each end if you're delayed and it does a countdown to online check-in and your flight.
I absolutely detest flying, I'd like to point out, not because of a fear but because of the faff. However they managed to sort it out this time round and I barely queued at all at any of the airports.
But they still need to sort out departure lounges (i.e. a less humungous duty free to walk through and ignore everything, and more seating!), timings (if I need to be on-time, so does my plane), etc.
Shame that they've messed up, but it actually worked really well for me the other day and I didn't need to bother to check the physical boards at all.
-
-
Monday 20th August 2018 13:42 GMT Anonymous Coward
Fair enough wanting to avoid having a PC behind each screen, and thus using a 'dumb terminal' approach, but there is absolutely no reason why this should have required connectivity *external to the airport*.
I presume the gate allocation etc is managed on-site, so why not have the exact same screens, but talking to an on-site pair of redundant servers (perhaps with failover to a cloud based server if you're worried about the on-site ones not being resilient).
It's this obsession with "moving to the cloud" people seem to have, I've worked with several companies who are doing it despite a simple calculation showing it is going to cost them far more (as their applications are not normally designed in a 'cloud first' way, so they require quite a lot of permanent infrastructure to be running), and give them much less control / ability to fix things if/when there are outages (as to the cloud provider they're just another number, whereas before they had their own techies able to deal with things).
-
Monday 20th August 2018 13:59 GMT Lee D
How do you know if a flight coming in is going to be delayed?
You have to connect to something - the airline, the air traffic control (unlikely outside UK airspace), the other airports, etc. Without that you have no idea if a flight is delayed, thus no idea if the next one has to be bumped down, etc.
You have to have a live connection for this to be anywhere near vaguely useful. That they can't get a redundant connection to Gatwick is ridiculous. They should never fall over, there should be multiple fibres km's apart from each other connecting to different towns at minimum I imagine. Can't fathom what Vodafone is doing running that, to be honest. Unless they're complaining that a 4G backup didn't work (but then... if that's the case... surely that 4G connection just runs from a local leased line, etc.?
That they need a connection isn't the problem. That they haven't got a suitably redundant connection is the problem. It looks immensely like the boards aren't even connected as well as, say, the cell tower in the terminal, or the free wifi.
-
Monday 20th August 2018 18:53 GMT Anonymous Coward
"That they need a connection isn't the problem. That they haven't got a suitably redundant connection is the problem"
Also .... didn't BT Wholesale have a large facility at City Place, round the corner from the airport? Think they've moved, but surely there will be at least some fibre in the area .....
-
Tuesday 21st August 2018 08:21 GMT Alan Brown
"wanting to avoid having a PC behind each screen"
When that entails _literally_ a desktop box behind each screen (which was the case not so long ago) you can understand it.
That doesn't excuse having the important bits offsite though. Airlines and airports are complex IT operations (core business) that happen to run logistics and aircraft. Outsourcing your core stuff is a bad idea.
-
-
Monday 20th August 2018 15:21 GMT Anonymous Coward
Use a camera
placed in front of a single white board in the office and broadcast the image to all the screen via HDMI cable or whatever inputs are available on the display screens.
Staff could stand in front of said camera with an advertisement hording from time to time too.
Get it right and nobody would know the cloud was down.
-
Monday 20th August 2018 16:30 GMT Anonymous Coward
Cloud-based flight information display system ..
“The Flight Information Display System (FIDS) at Gatwick Airport was the result of a project that kicked off back in 2015 to replace legacy systems that required a separate PC running behind the majority of the airports’ screens”
Running the screens from the one 'cloud' solution through a single fibre cable sounds like some bean-counters idea of saving money. Multiple devices running multiple screens with multiple data paths is the correct solution. You could do it with Raspberry Pis connected in a banyan-tree topology.
-
Monday 20th August 2018 17:05 GMT Dwarf
SPOF
So if there is one Single Point of Failure thats been discovered, I wonder how many others are lurking waiting to be found.
Resilient platform design isn't difficult or new, so its worrying that this made it through design, design review, network 3rd party assurance (their design and review processes), implementation and testing. The obvious question here - where's the governance and who's going to get the chop because it was not sufficient ?
-
Monday 20th August 2018 17:34 GMT Anonymous Coward
Re: SPOF
Where's the governance?
https://www.gatwickairport.com/business-community/about-gatwick/company-information/ownership-management/executive-management/
Who's gonna get the chop?
aka
Who's gonna pay for this?
No one at corporate board level ever pays for their mismanagement (not in the UK anyway). Staff and customers will pay, as is traditional.
-
Tuesday 21st August 2018 10:35 GMT Anonymous Coward
Re: SPOF
At least one of those responsible has already moved on... I know this as they are now lending us the benefit of their experience, which apparently includes not worrying about old fashioned operational standards and resilience, and instead being dynamic and embracing risk. Looks like there was a bit too much of that during their tenure at Gatwick!
-
Tuesday 21st August 2018 12:25 GMT 0laf
Re: SPOF
My commiserations. That person talks exactly how your management expect 'dynamic go getters' to talk. All your talk of resilience and planning will just introduce a cloud of negativity into projects and you'll be sidelined. Mr/Mrs 'Dynamic' will shortly be shifted into a nice senior role with Cyber, Digital, Evangelist, Solutions, Architect or some other vacuous pish in the job title.
They will then move on to the next gullible gobshite loving exec team and you can enjoy picking up the pieces and taking the blame for the failure of governances in controlling Mr/Mrs Dynamic.
Something for you to look forward to.
-
-
-
Tuesday 21st August 2018 08:16 GMT 0laf
Re: SPOF
I suspect form personal experience (and stories from many other security bods) that the group with the project of updating the info boards probably specifically hid the project from anyone who might have pointed out the lack of resiliency and therefore made them do things properly.
I know security are not popular with projects but id we get spoken to early on we can indicate the howlers that might happen. It's when we find out about things late on that the big 'stop' notices come out.
-
-
Monday 20th August 2018 17:16 GMT Cynic_999
3Mbps?
Seems a bit of overkill, and implies that each screen is fully refreshed individually whenever the slightest thing changes. If the displays had a tiny bit of intelligence so as to all use the same raw flight information which they format and scroll themselves, a 500Bd link (at the most) would be more than sufficient. I shouldn't think that all the information needed to be displayed on any screen would take more than 50 bytes per flight, so even at 10 bits per character (e.g. 8 data, 1 start, 1 stop), 500Bd would be able to handle 1 new flight per second, which is at least 2 orders of magnitude faster than aircraft arrive & depart, so historical information can easily be interleaved between new flight data for initialisation of freshly powered-up screens. Thus already powered screens would update within 2 seconds of new flight information becoming available, and a freshly powered-up screen would be fully populated with a list of flights in under 2 minutes.
Compared to the rest of the hardware costs, a simple CPU and a few KB of RAM would be an insignificant cost. Add a radio receiver and a 500Bd radio link could be implemented that needs no cabling to the screens other than plugging into the mains.
-
-
Monday 20th August 2018 18:25 GMT Anonymous Coward
no redundancy in the internet link
Thanks heavens (or whatever deity you prefer) that the people who run the Gatwick airport displays don't actually operate the planes that fly from the place.
Left to them, each airliner would have one socking great engine and the plane would be controlled by a pilot based in the cheapest possible location remotely linked by a single non-resilient radio link. No allowance would be made for holidays, sickness etc.
The profit motive is essential in running a business, but it mustn't be the only motive.
-
Monday 20th August 2018 19:11 GMT Anonymous Coward
Re. Die Hard VI: Departed
Yeah, that would be Bruce Willis.
Also IIRC he took over *after* the bad guys did some damage.
To be honest, whoever is responsible for this complete quadroclusterfsck of Biblical proportions should do the honorable thing and fall on his/her/its sword.
Maybe they can get a job doing failure analysis, plenty of prior experience is needed.
-
Monday 20th August 2018 19:20 GMT Anonymous Coward
Total Design Failure
I once worked for a company that builds the old type of system that LGW replaced with this shiny new Cloud System.
Our systems had dual newtork connections. Physical not virtual. These went right back to the two totally separated DC's. You could take out on DC and everything would fail over.
The FIDS Display had totally separate network cable runs apart from the last 5m or so. The runs went off in different directions through the building wiring. Those sort of decisions need to be made really early in the design of the terminal. LGW-South dates from the late 1950's.
LGW-North is at least 20 years old. Just like LHR-T1 you get to a point when you need to knock it down and start again. LHR-T3 should be next for the demo team. Airport Terminals are really one huge great system. Most systems are connected to others in many ways. Changing one is just the same as putting sticking plaster on a cut.
There is a reason why I avoid LGW if I can and I was brought up in Crawley.
Sorry, this is IMHO clearly a cost cutting move by the people who now run LGW that may well come back and bite them really hard.
-
Monday 20th August 2018 22:18 GMT J.G.Harston
When they said they were using their standby manual whiteboard display boards, I pictured something like this - like the big display board you see in WW2 RAF movies with WAAFs scurrying up and down ladders. Not a poxy conference room flip chart.
-
Monday 20th August 2018 23:31 GMT Anonymous Coward
Where are the Whiteboard AI / Bots going to come from?
https://www.bbc.co.uk/news/business-45240758
https://www.bbc.co.uk/news/technology-45219902
~~~~~~~~~~~
The hype of Cloud / AI / Automation / Algos is amazing.
What happens when it goes titsup, as it will sometimes.
Will the whiteboard human robots all have been laid off?
The Gatwick clusterfck is the real Disruptive-Technology.
-
Tuesday 21st August 2018 07:11 GMT Ben1892
Really no excuse, some of the most prolific and redundant dark fibre in the country runs through Crawley heading for London
Having said that, the redundant bit of the connection is probably in "phase 2" of the project as they were in a hurry to deliver something, anything - I mean what are the chances of someone digging throu......
-
-
Tuesday 21st August 2018 13:10 GMT yamahaga
Re: Basic HA
Having worked for one of those ISP's I can confirm this a very common solution for businesses a lot less reliant on their connection than Gatwick airport. I almost spat my coffee out when I heard that a fibre break caused this problem. Everyone involved in this project should be sacked but will be promoted, given bonuses or suspended on full pay for 6 months...
-
-
Tuesday 21st August 2018 09:49 GMT steviebuk
Ironic
"an innovative, cost effective system that are easily scalable, more flexible and resilient, and require considerably less infrastructure and maintenance"
Although it wasn't the cloud host that was the issue, part of cloud is the link to it that can also be an issue. Greedy companies relying to much on "the cloud" is what annoys me.
I still don't understand how London airport can run it's air traffic control 80 miles from the airport. Yes they claim to have redundancy in place. But the whole idea of it being at the fucking airport is when John McClane is around, if the anything happens you can look out the fucking window with binoculars.
-
Tuesday 21st August 2018 11:48 GMT Anonymous Coward
ONLY 3Mb/s?!
I see a lot of people on here talking about how the data connection requirements are so low that they shouldn't a problem. And yes, a simple bridge to a nearby homeowner's WiFi would suffice. Or 'appropriate' a satellite connection from a parked 'plane. Or any number of other solutions.
But 3 Megabits per second of WAN required just to update a bunch of boards that are all working from the same data source and don't need to update quickly? That's a surprisingly high amount of traffic for something with so little actual data throughput, especially for a small dataset that's being generated and consumed in the same facility.
Is it just horribly inefficient (as well as being unreliable) or is there more to these boards than just displaying flight information?
-
-
Tuesday 21st August 2018 16:09 GMT Joe Gurman
I must be missing something
....not that that would be unusual.
This is still 2018, right? When world + dog carries a vibrating, light-up Internet fondletoy with them at all times? Is there any airline passenger who cannot consult their airline's app for a check on flight status, including gate? Or, if their phone was stolen during security screening, ask another traveler politely if they could use their phone to check?
In the last three months, I've been faced with at least on instance with the larger flat-panel displays now at every gate (in US airports, at least) displaying hilariously incorrect information that conflicts with the information on the display at the gate 10 meters away. The app always had up to date correct info.
-
Tuesday 21st August 2018 16:58 GMT GSTZ
Reliable Infrastructure ...
Back in the 70's, some company invented fault tolerant (=failsafe) computers which by design had no single point of failure. They even extended that fault tolerance into their system software - if one CPU tripped over some sporadic software bug it was immediately halted, and parallel CPU's took over. Applications continued flawlessly without loss of data or any impact to the end users. By the way, that product line still exists and you can buy such fault tolerant computers today.
Back at that time, large airports ran their critical operations locally and did link up their devices to their own local computers, not to someone else's cloud. Nor did they outsource any critical IT task that they could run locally by their own staff. Those were the days of reliability ...
-
Tuesday 21st August 2018 20:02 GMT Luiz Abdala
Ouija Boards
Congratulations to the staff that had a whiteboard and marker pens stashed in the back and could still provide information in a nearly fail-safe, albeit EXTREMELY sub-optimal manner.
Just like Ouija boards* aboard Aircraft Carriers relaying the status of every aircraft on the deck, using literally nuts and bolts.
It is better than having NOTHING to show.
*Not literal Ouija boards, just a synoptic representation of the Aircraft Carrier deck drawn over a table.
-
-
Wednesday 22nd August 2018 09:54 GMT GSTZ
Re: Cloud
Another problem with today's clouds is that they are designed, built and operated for utmost cost optimization, not for high reliability and running critical applications. Such reliable clouds might be possible, but it is not very likely that they could become economically successful as a standalone offering - the beancounters would shy away from the price premium. Another obstacle is that cloud providers cannot know enough about their customer's individual businesses to provide them with the right scope of reliability they need.
So cloud providers do promote various options that *theoretically* allow their customers to achieve the reliability needed for critical applications themselves. In real life that approach doesn't work particularly well, as customers are not deep enough into the complex art of making applications really failsafe. This would also require to have at least some control over the infrastructure of the not-so-reliable clouds availble today. More common than hard failures (like the defective fibre cable in the Gatwick example) are temporary overload situations causing application timeouts thus making services unavailable.
-
-
Friday 24th August 2018 20:31 GMT A_Melbourne
Departure Control
This brings back the memories. I helped install the departure control system at Gatwick for British Caledonian Airways - some 35 years ago. I was working for Raytheon at the time. BCal had lots of pretty girls working the desks. I was on crutches due to a recent car accident.