UK flights disrupted by 'technical issue' with air traffic computer system The UK's National Air Traffic Service (NATS) is spending a bank holiday Monday dealing with an unspecified "technical issue" that has disrupted flights across the country. NATS said in a statement today that at 1210 UK time it "applied traffic flow restrictions to maintain safety." Half an hour later the an update was posted …
I'm not sure why you got downvoted for saying an air gapped system is not fully secure. It seems pretty obvious to me. To be air gapped means it's likely to be patched less often because it's a PITA to do so, so security vulnerabilities will be there for longer. Also you have to allow storage to be plugged into it somehow to allow updates to the main software as well as patches.
I worked at a place that used CDs for this but it was a real waste of plastic so they started to allow USBs. We had specific machines only running virus checks on those CDs and USBs with virus definitions updated daily. Still we got a virus on the secure system somehow*
*it was definitely not a senior manager plugging his phone into a USB to charge it. <\sarcasm>
That all depends on your definition of air gapping. My definition of of air gapping means completely air gapped with no transfer of any data whatsoever (without systems in place to check that data before it gets anywhere near said system and that the data is purely just data) and not just on a network connection. If you are going to be plugging usb drives in and out from one to the other you might as well just not bother but you knew that right because you have a clue and I don't?
Yes. The Grauniad says "The failure of the air traffic control system, run by private company Nats, has been blamed on a single corrupted flight plan entered by an unnamed airline, according to reports."
Whether it was Excel, CSV or a 1960s-era punched card image I cannot guess, but rejecting gobbledygook in the input has been a principle of programming since, oh, 1948.
I've been thinking this for a while, sadly. And in far more critical areas of infrastructure than the ability of middle-class families to fly back from the maldives on the last day of the summer holidays..
When utility companies take on debts to pay dividends to overseas shareholders instead of upgrading critical infrastructure, is that incompetence, or malice?
The electric grid in particular needs not just to be maintained, it needs to triple its capacity if we are to phase out gas and petroleum, but it can barely sustain current demands.
But while it's relatively easy to build new 400kv pylons, it's much harder to upgrade the thousands of "low voltage" networks that are buried under our towns and cities.
And while it's easy to build more wind turbines, it's fundamentally impossible to have enough storage for more than a few hours of low wind. Batteries are becoming more and more expensive as the world demand goes up, meanwhile we are draining rainforest aquifers to mine lithium.. Renewables just can't keep the lights on without dispatchable gas turbines. And many people don't realise that a CCGT takes a while for the combined steam cycle to warm up. So as they ramp up and down with the wind, a lot of energy is wasted. They are initially as inefficient as OCGTs.
We talk about carbon capture yet we are chopping down trees and burning them for power, producing more CO2 and particulate matter per MWh than the dirtiest coal plants ever did! It's madness
So far no evil person has attacked the UK grid, but with it teetering on the edge, it needs to be at the front of minister's minds how to fix it when it does all go dark. If I were in charge, i'd organise a test of that scenario. The privatised utilities can pay for the losses caused if it turns out that they have been lying to us about the resilience of their networks. If they can't or won't pay, they get taken public. But all ministers seem to care about these days is lining their own pockets, before they make for the lifeboats as the ship sinks.
Something has to give. If it doesn't, the only outcome is WWIII.
/offtopicrant
In fairness, most normal people understand this, it's the Woke fascist left who will cancel you for susgesting such things, even though they call for tolerance!
Remember for every MegaWatt generated by renewables, we need a backup generating system ready on standby for when the weather doesn't play ball and can be called up on demand and all those mobile power stations on four wheels will be replaced and will need to draw their power from the grid.
Download the planned national grid planned shutdown rota, check the letter on the top right of your electricty bill, that gives you the region your in.
In theory yes, but sadly in practice, not so much. :(
It's a tiny molecule that leaks easily, it's very light so has low storage density unless you liquefy it, and to liquefy it is much, much harder than for LNG.
Hydrogen is also pretty inefficient to produce, and very expensive - it usually requires Platinum electrodes, which means lots of Platinum per MW - (the efficiency of electrolysis is not much better than 50% and it has an electrochemical potential below 1.5 volts, so you'd need over a thousand cells at a thousand amps each for just 1MW - that's a lot of Platinum, even if the surface thickness is very small - and a lot of copper. Multiply by 1000 again for 1GW. The UK has around 15GW of excess Wind capacity that it plans to store... It's just not feasible at utility scale, sadly)
I have heard of a few researchers using "AI" to design special non-platinum electrocatalyst materials, but as far as I am aware, none are very promising.
What was needed was Nuclear power. I say "was", because we no longer have the time, expertise, or the political appetite to build enough new nuclear.
And don't get me started on Fusion. :( It's another pipe-dream that is great in theory but when you think of the practicalities, it's not very feasible to transfer large amounts of power across a vacuum in the form of fast neutrons which destroy and make radioactive almost all materials known to man.
There's a lot of promise in large-scale batteries that use iron. Useless for cars etc., but potentially fine for bulk uses like storing wind farm generation.
Plus, people could just use less. Crazy idea I know - capitalism is all about growth. Maybe that's the source of the problem right there.
Scottish airline Loganair indicating the issue involved "a network-wide failure of UK air traffic control computer systems."
its always a network, except when its not.
Sounds like an application issue to me, but i happen to know they are replacing their core (check job listings over the last 12 months).
major outages just shouldn't happen in 2023, i'd mandate all on prem systems managed from the UK for critical systems like this.
On prem doesn't prevent peering with multiple ISP's across various parts of the nation, or indeed nationwide private WAN's serving all interested parties/customers just like we used to do it.
It's all well and good to call your industry colleagues incompetent, until you f-ck up and everyone points at you.
Someone made a mistake, likely they made that mistake years ago and didn't anticipate something in the validation checking that years later is now possible. Calling people 'incompetent' doesn't help.
Now they've reloaded the tape into the ZX80 hidden inside the expensive cabinets you can see the planes leaving Heathrow at an impressive clip.
Being the curious sort I did some timing and one leaves on average every 90 seconds, with the shortest interval only 45 seconds. What impressed me other than the speed with which they're decanting their parking lot is the sheer volume. I lost count, but I think there must have been some 30 planes leaving while I was watching, with many more taxiing to join that queue.
Zoom out from the link to get an idea just how much traffic whizzes around in the air, it's impressive. The inbound stack is still empty as most of the inbound traffic is probably still underway.
Same if you drive up the M3 or M4 in the morning - if they are landing in the East-West direction, you can often see 4 or 5 stacked up behind each other with their landing lights on, all coming down one after the other. As soon as the bottom one goes out of view, another appears at the top.
A beer to those who manage and handle these things on a daily basis, and a beer to whoever was "on call" on a holiday monday. Looks like you had a bad day.
Oh, it ain't over yet..
Especially Gatwick is presently still in a real mess, but the chaos was to be expected. Planes are in the wrong place, flight and staffing schedules have to be adjusted, the whole logistics chain behind these flights have to adapt - it's not a trivial exercise to get everything back onto the normal schedule so that striking French air controllers have something to blackmail again..
(yes, I'm a cynic).
Over 40 years ago I lived in Southall, directly in the flightpath. The BT tower (or Post Office Tower as it was then) was on the horizon, and on a good (i.e bad) day we could see at least 5 planes on the approach. Thankfully we only had planes flying over us when it was wet and windy weather.
>When you're in Hounslow you can see them lined up in the air. The endpoint of the stack seems to have enough space to line up 4 planes with the mandated safety gaps.
>Then again, you'd have to go to Hounslow. FlightRadar is easier :).
Its the same for Manchester - with the queue starting over the Woodhead Pass and passing over Stockport on the way into landing - only far more scenic than Hounslow.
Yeah Heathrow has an impressive 90-120 ish second turnaround time.
One of it's biggest concerns with the two runways at Heathrow is that they are running at maximum capacity during peak hours. Something trivial like fog results in an increased time per landing due to visibility concerns.
When there is heavy fog dozens of flights get cancelled
Very strange walking around Hounslow on the flight path. The planes feel like they are landing on your head especially when the shadow zooms across your feet
Incoming planes tend to be prioritised due to trivial matters of remaining fuel and gravity coming to call when this runs out.
90-120s between planes is the maximum throughput possible but it does depend on the size of the plane. Small planes do not follow big ones and there is a much longer gap between a larger plane followed by a smaller one. This means that planes tend to be scheduled as, e.g. small planes then medium planes then large planes as this allows the shortest gap between planes. At some point a smaller plane has to go through and the cycle goes back to the beginning again after a longer gap. This just adds extra fun to the scheduling process...
Unfortunately, running at the maximum throughout already means that there is no spare capacity to catch up, therefore cancellations happen. To complicate matters there are rules to follow regarding the number of planes permitted at certain times of day therefore it's not possible to continue running outside of these times to deal with a backlog.
The landing queue was quite a sight this morning, lights as far out as you could see as soon as the noise restrictions relaxed, even for Heathrow it seemed busy. Fortunately we arrived from the US bang on time courtesy of Delta, the equivalent BA flight was four hours late departing, I assume because of the knock-on from inbound flights being delayed - not that BA can moan too loudly, given their own recent IT track record!
"waiting for a keypress to a modal window which was hiding behind something else"
I got caught out with one of those once. Didn't have a clue why a system had frozen, it seemingly had crashed. I was on the verge of doing something drastic like killing the process before I realised there was a hidden window waiting to be clicked on. Such windows really need to be displayed "topmost".
Such windows really need to be displayed "topmost"
Yes, but it's Windows. Their UI design seems to be mostly based on how different they can make it from the last version so the maximum amount of staff time is lost when upgrading. Hiding a modal window (i.e. a modal window that isn't) fits right into that strategy.
"... before I realised there was a hidden window waiting to be clicked on."
I haven't had too many problems like that, but recently started using some new software which is affected by this issue. Sometimes launching Task Manager, right-clicking the application and selecting Bring to Front helps. But that depends on the phase of the moon.
According to the BBC's reporting of the NATS report:
"... at 08:32 on 28 August, its system received details of a flight which was due to cross UK airspace later that day.
The system detected that two markers along the planned route had the same name - even though they were in different places. As a result, it could not understand the UK portion of the flight plan.
This triggered the system to automatically stop working for safety reasons, so that no incorrect information was passed to Nats' air traffic controllers. The backup system then did the same thing."
WTF! Why couldn't it just have rejected the flight plan instead of shutting the entire system down?
And what a surprise - the "backup" system did exactly the same thing.
So, it only took them 4 hours to get through to their sysadmin on his mobile, who was the only person that knew the correct command line switches to restart the database, as he was having a day off at the Notting Hill Carnival and couldn't hear his phone go off due to standing near a massive sound system and being well mashed....
According to the BBC's reporting of the NATS report on the incident - it did exactly that:
"This triggered the system to automatically stop working for safety reasons, so that no incorrect information was passed to Nats' air traffic controllers. The backup system then did the same thing."
This post has been deleted by its author
That's "The Blockchain" to you, sir.
Yes, I have suffered through a snake-oil presentation of someone trying to sell the virtues of "The Blockchain" including all manner of automatic triggers on activity and other stupid lies. He wasn't happy when I told him in front of everyone else that he evidently had no idea what the hell he was talking about. That made the pitch much more interesting... :)
The problem is the country still believes in Milton Friedmans failed economic theory. As implemented by Thatcher and her successors it has made a bunch of public school boys rich and impoverished the rest of us.
Please can someone pass them another book; J. K. Galbraith and J.M. Keynes had theory’s that delivered.
Whilst you may have a point I don't think adherence to a particular Economic theory is the problem. It's years of Conservative and Labour governments underinvestment and constant looking for short cuts and doing things on the cheap. At the same time we always hear "world class" and "world leading" when in fact it's not even fit for "bottom of the list" 1st world country.
Perhaps it's our system of government itself which has just reached it's logical conclusion and we need to change that to change the outcome....
First past the post is really the underlying cause.
Worst method of choosing a government, as it creates a two-party system and then further polarises everything into "us vs them", as everyone has to vote to against the one they fear the most, because they cannot vote for the one they actually think is best.
Fear then leads to hatred...
Starting to wonder if a damp woman distributing swords would make for a better basis of government.
It was written from the ground up when Swanwick ATC was being developed in the late 1990s. I know, since I was interviewed for a place on the development team at the time.
Somewhat worryingly, a number of supposedly non-critical parts were written in Perl. I don't care how non-critical they were, but I don't consider a dynamically typed language to be a good choice for anything related to air traffic control...
Come on the register!! You are the number 1 IT news website. Where are the hourly updates? Do you have the change number? The resulting incident number! Why don't you have a reporter on the MIM bridge. Very poor from the register.
My money is on an expired certificate or BGP or DNS.
This post has been deleted by its author
Dec 2014: “The problem, involving computer code written a quarter of a century ago, was responsible for widespread disruption at British airports.”
“ES7. The systems at the NATS Swanwick operations centre entered service in 2002 but were in development during the previous decade. Failure occurred on 12 December 2014 because of a latent software fault that was present from the 1990s; this is referred to as the proximate cause of the failure. The fault lay in the software’s performance of a check on the maximum permitted number of Controller and Supervisor roles (known as Atomic Functions).”
“These Atomic Function identifiers are used to index (access) the table of data held in the System Flight Server (SFS) and distribute the correct and relevant data to individual roles. The check should have been whether or not the limit of 193 (the total of civil and military roles) had been reached; instead the check was performed against a civil limit of 151.”
“SFS was designed and programmed to shut down the primary SFS in response to this “exception” ,, the same exception was raised in the software in the secondary SFS, and that too shut down.”
Perhaps they should have put Margaret Hamilton in charge of development
https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_engineer)
Apparently the cause is being put down to a bad flight plan filed by a French airline. Seriously, what?
If true, that implies input validation of flight plans was either non-existent or insufficient, both of which are possibilities. Little 'Bobby' Tables might have struck again.
This incident should add another test-case; and probably increase the scope of fuzz-testing of the inputs; with the possibility of the scope increase being from a low base, conceivably zero (hopefully unlikely).
NN
It's the very first thing I thought when I saw that was the cause. Although they should have rebuilt the system long ago. Another symptom of a never ending push from weak management and non-technical decision makers to concentrate on cost-cutting and glossy additional features rather than streamlining codebase and paying down technical debt.
Bet is was a rouge acute accent or grave accent, once we left the EU, we won't take foreign characters anymore, next upgrade for NATS is to go back to miles and feet!
Hmm. Yes, if the input used ANSI escape sequences to generate coloured characters, that might have left the programmers rosy-cheeked with embarrassment at mishandling the input. Of course, it may have been forced upon them by lack of space for any more input sanitisation.
Some people really liked Tiswas as a Saturday morning kids TV programme. It comes out subconsciously in their writing. You can always tell.
Oddly enough, most of the world (exceptions are North Korea, Mongolia, and the People's Republic of China) does altitudes (and Flight levels) in feet. Russia is in the process of swapping over from metres to feet.
OpsGroup: 2011-11-17: Special Report: Russia transition to ICAO RVSM
OpsGroup: 2017-02-22: Big change: Russia finally moving to QNH
But all is not sweetness and light regarding standardisation of units in aviation:
AeroSavvy: 2014-09-05: Aviation’s Crazy, Mixed Up Units of Measure
NN
PPRuNe: U.K. NATS Systems Failure
Please don't contribute to the thread unless you are a professional pilot with a meaningful information to add. PPRuNe is strongly moderated after publication, and fools are not suffered gladly. The 'Rumour' thread has a little more latitude than some of the other forums ('Jet Blast' even more so). I'm not a professional pilot, and I have witnessed many 'contributions' by wanna-bes, Dunning-Krugerites, and narcissists removed. I just find it interesting and informative to read.
There's an interesting observation in the PPrune thread:
"...since in Europe all flights plans are centralized and go through IFPS , checked and redistributed to the ATC centers by Eurocontrol . So if there was an initial filing error by the airline it should have been spotted at IFPS level. but assuming it was not, then every other ATC center receiving the FPL should have been affected, and if not why only the UK system .
https://www.pprune.org/rumours-news/654461-u-k-nats-systems-failure.html#post11493309
The contributions by CBSITCB in that thread are worth reading. This is a link that should work to the most interesting one: PPRuNe: U.K. NATS Systems Failure, post at 29th Aug 2023, 11:13 by CBSITC
- Each National ATC centre runs their own system: they are not identical. So you can expect the flight-plan-to-actual-route conversion programs to differ.
- The current UK airspace model is based up the US system, but has diverged.
- The functional specification of the US system for converting the filed Flight Plan into a route expressed in the national airspace model. are here: [pdf] FAA: NATIONAL AIRSPACE SYSTEM: En Route: CONFIGURATION MANAGEMENT DOCUMENT: COMPUTER PROGRAM FUNCTIONAL SPECIFICATIONS: ROUTE CONVERSION AND POSTING
I'm not copying and pasting CBSITCB's posting here, but it is well worth reading to get some technical background, and some plausible speculation as to what might have happened. Essentially, the speculation is that the conversion of the Flight Plan into the UK national route generated an untrapped error that caused a system crash. It apparently happened before.
CBSITCB said:
I understand at least one major UK NAS outage in the past was caused by errors in this process. Someone had managed to input an FPL route that passed NAS route validation (described in NAS-MD-311 Message Entry and Checking) but “did not compute” when route conversion was attempted.
FPL = Flight Plan
NAS = National Airspace
Indeed, and you can file a flight plan to many airports that aren't customs airports so don't have IATA codes but do have ICAO ones. You can also fly IFR to private strips that don't have either an IATA or ICAO code. I find it unlikely that an airport name change has anything to do with it.
Got it.
That said the same principle holds. As ICAO codes have been a thing for years I find it difficult to believe that, in the Year of our Lord 2023, flight plans loaded by NATS are subject to the vagaries of encoding converters in the airport name in preference to the international code and one mashed up utf-8 or iso-8859-1 string or e.g. someone calling an airport by the town where it is ("Le Touquet Paris-Plage") instead of its real name is enough to bring the whole house of cards tumbling down.
It would mean that NATS would fail much more often.
There's still time to buy a four-way mains strip from Wilko for less than a tenner before they sadly vacate the high street.
No need for the cleaner to unplug NATS to use his/her vacuum cleaner.
(Said in jest: I had a tour round NATS when it was situated at West Drayton, so I'm well aware how tight their systems are).
.. used to have a flaw such that there was an apparently logical command line that a controller could input that crashed the whole system every time. This was in the early eighties and some generations of software back...
However..
If you watch the GB News interview with the Minister for Avoiding Questions, you'll hear the interviewer start with a question suggesting a very similar scenario. (The interviewer was rather excited that the command message came from France.) Had the interviewer heard something from a genuine source?
Surely it couldn't be that several generations of software have actually failed to remove the vulnerability? The word incompetence might even come to mind. We'll probably never know. Certainly the Minister for Avoiding Questions played a very straight bat and revealed exactly sweet FA.
You really have no idea how big bespoke systems are developed. Ever looked at the UI for Adobe? Still s***t.
The fun starts when supposedly clean sheet systems, developed with no (honest) legacy code at all also display the same bug.
I supposed it's total backwards compatability.
But do you really need that?
.. used to have a flaw such that there was an apparently logical command line that a controller could input that crashed the whole system every time. This was in the early eighties and some generations of software back..
Several decades ago, when I was still a schoolboy, the school computer club had a (single) teletype connected via audio-coupler and modem to the county-council mainframe. A boy a year or so older than me was taken up with trying things out on the system's command line, and one of the arcane functions was SYS(x,y) where x and y were integers. I can't remember what it was meant to do, but that's not relevant - the boy tried setting one of the arguments to be negative. SYS(0,-1) or something.
The connection stopped responding. This was usual, as the line would drop out occasionally, but the modem was still 'up', but we tried re-dialling, and still getting no response. After a short wait 15 minutes to half-an-hour, we got a connection again, and so, of course, he carried on experimenting (xkcd 242 applies). SYS(0,-1). The connection stopped responding. Wait a while, re-dial, system back...and he tried a third time. You might see a pattern. Shortly after a teacher appeared with a message from the council computer centre, kindly, but firmly requesting us to stop crashing the computer.
Surprisingly, we (and the school) didn't lose access. But we were told not to experiment with things we didn't understand. So we played Lunar Lander and Star Trek instead, using up rather a lot of fanfold paper.
Oh...and the point is, in those gentler times, the assumption was that people would try not to break the system, and use it as intended, following the rules, so there was a lot less input validation. Things have changed.
NN
Clearly they never though anybody would be naughty enough to supply an invalid / rogue flight plan. I mean why would somebody want to do that, they* wouldn't be allowed to fly if their flight plan wasn't valid, and by filing a flight plan they clearly want to fly, hence pointless exercise so an impossible situation to arise.
*Along with all other flights in UK air space.
Apparently the Flight Plan has to go through two other systems before it gets to NATS and would have been reprocessed by them as well. So it's M2M data, not user supplied data, however it may have featured something out of bounds for NATS. Just remember that that same flight plan also went to every other system in Europe at the same time and it was only NATS that Flop'ed
There is obviously a lack of adequate data validation here if a single bad bit of data in a flight plan (apparently from Air France, nothing like blaming the auld enemy) can crash the system, and also the backup system, which was obviously running the same code. I see a really bad pattern here. If they get bad data, simply crash, as there is a manual system that can be used instead. Except that in this case its capacity is so very much lower that it causes delays lasting days.
This seems a typical attitude of programmers designing safety-critical systems. Take the crash of flight AF447 in the Atlantic off Brazil a few years ago. The pitot tube froze up, so the autopilot could not measure the air speed and passed control to the redundant autopilot, which found a frozen pitot tube in its turn, and then gave back control to the pilots. Except here the inexperienced pilots had no idea what to do and panicked and flew the plane into the ocean. But there are other ways of estimating air speed even without pitot tube input (GPS + met data on wind speed and direction) which the autopilot could have used. But its lazy programmers simply though tthe best thing if you can't easily cope is to hand back control to the humans.
In neither of these cases did that turn out well.
I have been a programmer of real-time systems. It is not easy when you have inputs that are in conflict with one another. In fact it can be impossible, so the only sensible way out is to pass the error up the chain. In safety-critical systems it is not the average Joe/Jo programmer who should dictate what the 'default' action is on a life or death input conflict. It (the decision on the default), needs to be sent up the chain to reviewed by people prepared to take responsibility, which in the case of my ex-employer, was at the very top. These things, once documented are the Holy Grail, and would be presented at an official inquiry if a serious incident occurred. I've been on 'low-tech' changeovers (notably in my case the Hatton Cross extension of the Piccadilly Line), where some really senior people got their hands dirty with some highly repetitive sequences, rubber-stamping what many others in the heirarchy have confirmed to their own satisfaction many times over, in a dark, noisy relay room for many hours at a stretch.
No, let's not call programmers 'lazy', let's say that they value their backsides.
Back in the day*, 'resilience' (aka DR), meant proof that, in the event of a total layer 0-7 outtage at site A, all services would continue happily to run from somewhere else (e.g. site B) with no apparent impact to users/customers.
We users also happen to own 49% of Nats - bank-holiday Sunday must have looked like a good time for a DR test...
*20-ish years ago by now
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
