Tech troubles create aviation chaos on both sides of the Atlantic Technology problems hit the commercial aviation industry hard over the weekend, leading to hundreds of cancelled flights and myriad delays on both sides of the Atlantic. In Europe, the source of the problem is a company called Collins Aerospace that makes a product called ARINC cMUSE that it describes as a “next-generation …
You make a good point which shows true redundancy can be very difficult and/or expensive to achieve. Imagine you want redundant power feeds from separate sub-stations for your facility, the power company are happy to provide them, but it so happens your facility is the other side of the freeway from the substations and there's only one nearby bridge with a suitable cable culvert crossing that freeway. Both cables end up crossing the same bridge, and then a truck hits the bridge and you have a single point of failure. Most bean counters wouldn't sanction the extra cost to build the infrastructure to fix that, claiming it's an "acceptable risk", which is why things like a single sub-station fire take out Heathrow Airport for 24 hours or more.
"which is why things like a single sub-station fire take out Heathrow Airport for 24 hours or more"
The redundancy issue here is about the way that the airport systems were connected, and managerial decisions by the airport about recovery, not about a single point of failure.
Heathrow isn't supplied by a single substation, there are two others that can supply the entire airport demand, but it was reliant upon the one at North Hyde and had inadequate airport systems failover to the alternative power supplies. Even with lack of failover, the two other substations could have supported recovery at Heathrow far more quickly. It was the airport CEO's deputy Javier Echave who took the decision to shut the airport at 01:15 on March 21, that was less than two hours after the fire started, and also at that decided it would remain closed until 23:59. There certainly would have been disruption caused by the power loss and bringing systems back on line, but the chaos and lengthy closure was entirely of Heathrow's own making.
The problem with Heathrow was that they did have redundant power feeds to the facility, but no backups to allow transitioning seamlessly between power feeds, which meant the loss of a single feed meant they had to bring up all the infrastructure again and check that it was working.
It's a case of penny pinching that likely cost them millions of pounds plus long term reputational damae. Backup solutions of the required scale do exist and are regularly deployed for e.g. large hospitals and critical national infrastructure. A 30MVA diesel generator plus UPS/batteries to fill in during the startup of the generator would have come in around £20m.
Diverse routing should ensure separate feeds to at least 2 separate parts of the facility/building.
Re power over a single bridge, you then ensure you have an upstream backed by a generator with enough fuel for at least a couple of weeks time enough to get extra fuel to last until the bridge & main power feed is restored. For data you’d likely use a microwave link to bolster the fibre running through the bridge.
There is always a way provided you pay for it.
... of your separate, redundant comms suppliers. When the upstream decides to merge its various separate links into a BiggerLink™ for cost savings, they've just created a single point of failure for you, which you probably don't know about until disaster strikes.
The only way I see to solve this problem is to own your entire comms linkages yourselves, with no outsourcing. This will be expensive. But how expensive is your downtime?
It happened to the bank I worked for. When Euronext changed their connectivity, they set up connections to market members using two different telecom providers.
There where two connections to Euronext's datacenters, and two connection to the bank's datacenters - but in the middle (connecting France and the Netherlands to Spain and Portugal) there was just one backbone which, inevitably, happened to fail.
Fortunately (?) all market members lost connectivity, so there was no advantage of any of the local banks over the others - we all had to resort to brokers located outside Iberia.
The other redundancy issue catching people out is using two different suppliers, who then merge into one and bundle everything together… Happened on undersea cables in the past.
Happens a lot on terrestrial systems as well. Often because customers (and sometimes sales & marketing) don't understand what they're buying (or selling). Which could be fair enough given the inner mysteries of telcos infrastructure is only known to the initiated. So as a designer, I used to get many requests for wavelength services. Often from prospects who thought if they bought two unprotected wavelengths from two different suppliers, they'd have supplier diversity.. Which they wouldn't.
Which then included things like demands for detailed route maps, which often weren't shared. Then when they were, or prospects asked for a redesign, we'd discover their 'diverse' supplier was our customer and using our network. And then if customers were renting wavelengths, there was no assurance that we might not regroom or redesign infrastructure and the route would change.. Which was also a consolidation issue, ie consolidation usually meant cost reductions, which means exiting leases on PoPs, regen sites, wayleaves etc. But basically 'supplier diversity' is pretty much a myth.
Or getting real diversity and route seperation can get very expensive very quickly. Which from memory was an issue with providing services into LHR given both the airport's location, and the location of LHRs datacentre within the airport. Which meant big digs, and attempting to get wayleaves from Heathrow Holdings, even when the service was for LHR. Or having to explain that to sales that they couldn't use standard wavelength prices, assuming they could connect to a 'PoP'.. And having to explain to sales that the 'PoP' in question happened to be exclusively for a customer that could probably shoot the salesperson and get away with it. But LHR was one of those frustrating customers where despite trying to help their IT director, his management refused to pay to improve their resilience.. which boiled down to beancounters saying they couldn't afford to build a few hundred meters of duct, and us trying to explain that they couldn't really afford not to do it because a SPOF existed that could leave the airport pretty much dead in the water.
But in reality, if customers really want things like assured seperacy, they need to specify it and get it from a reputable supplier that could actually deliver it.. And pay for the service because it required more work to design. Biggest element often being the service IDs getting tagged in databases to indicate seperation needed to be maintained during regrooming, upgrades, network maintenance etc.
El Reg used to be the best source around for up-to-the-minute tech news. It has lagged behing more often in recent years.
The story of a cyber attack affecting multiple major European airports has been running on non-specialist news media since Saturday. It made it to El REg on Monday morning.
Nope, there was a relatively brief weekend experiment but other than that they publish on weekdays and usually take a little longer than Twitter so they can actually give more detailed information.
If it were stories which fresh off the social media feeds then what would be the point? You would have thought the US and European problems were related but it turns out that wasn't the case.
El Reg hacks have always had the weekend off. I don’t think that allowing them to rest is reasonable either but nothing has changed there. You noticed this because you saw the story covered by non-specialist media and came here to get the actual details from a source you trust. The fact you are complaining tells me El Reg is still needed.
Reg hack Simon here - I co-wrote this piece, edited it, and decided on the timing of publication.
FWIW the timing of this story was entirely down to the fact we don't work weekends. I arrived at my desk on Monday down here in Australia and one of the things in my edit queue was a short version of the story that mentioned only the EU outages. I was already aware of the US situation, and decided that as the EU angle was already widely known, a piece on both incidents would be more valuable.
When we are around and in a position to deliver very timely coverage, we pounce on news. For example, I was on duty down here in Oz when the substation fire took out Heathrow back in March. Our story was up at 04:39 UTC https://www.theregister.com/2025/03/21/heathrow_closure_datacenter_resilience/ - we were among the first to cover it, and almost certainly the first to advise on the datacenter impact.
When CrowdStrike's massive fail hit last year, we had the story at 06:46 UTC.
We also break stories almost every day. Stick around. You'll see lots of stuff here before it appears elsewhere.
Not only that, Simon, but RegTards normally are in a position to provide a much more detailed piece than mainstream rolling news outlets can afford - they don't have the bandwidth to go into detail in a half hour rolling news schedule, whereas El Reg can, and are much valued for it.
[reposted, first attempt got lost]
You don't need Russian saboteurs when there're so many numpties with diggers trying to get the job done briskly so they can go to lunch.
I was in my house a couple of years ago when the internet went down. Almost immediately there was a knock on the open window. "Your internet go down?" - an Irish bloke in hi-vis. "Yea it did" I replied. Already beginning to walk off he commented "Cable snagged the digger" in a way that strongly suggested it was the cable's fault.
fuckwittery and stupidity as well as bloody minded cheapness will win over any damage spies can do. It's layers of cheapness, no joined up thinking and just modern capitalism.
I was working for an unmentioned airport a long time ago when they were building a certain olympic village.
Some moron dug through a telecoms providers cable run.
Now said airport HAD ensured that there were 2 totally different technologies for WAN installed, that each one left in a different direction and went to a different exchange run by the telecoms provider. Annoyingly it was so long ago that there weren't 2 providers. Also it's the UK & OpenReach are still a monopoly & even other providers are forced to run much of their infrastructure across them.
What we didn't know was at the OTHER side of the exchanges, the routes came back together, into the same cable runs and out to the East. Cheapness, crap installations and fuckwittery from the supplier.
DAYS of problems and going back to paper & the poor check in staff having to deal with self important business types who didn't think they had to queue because of the price of their suits.
Layered fuckery...
I was painting the picture, especially for the accent, but of course realised almost immediately after posting someone would invent a problem with that. None of the anecdote was 'necessary'.
To clarify: the Irish bloke was the senior guy on site (I believe it was his non hi-vis wearing father who I met later who ran the business) hence he was the one confirming the damaged cable was in use. The numpty driving the digger was a young English chap. There were also 4 or 5 others standing around doing apparently nothing whatsoever, as is usual for these kind of works.
Yeah, The Observer mentions some folks thinking the timing is suspect with respect to when "Nato jets intercepted three Russian MiG-31 jets in Estonian airspace", and Aviation Source News notes some "speculated a link to Collins’ recent NATO contract for electronic warfare" which RTX showcased on 16 Sept.
But The Obs also notes it may be just an "opportunistic ransomware attack" too ... time will hopefully tell.
You also have the built out from relatively little of Foxconn by people like Apple into a global manufacturing titan that has enabled a multi-trillion €$£¥ high tech supply chain in China and cascaded their expansions into other areas to dominate manufacturing of consumer electronics, Solar and BeV or end product, component and raw material supply chains.
4 decades of state espionage could not have achieved a fraction of that :.
Same with IT/Support off-shoring to Bangalore.
Who built the American massive trade defect over half a century … Corporate America did… with the bemused assistance of China.l (and Mexico, Vietnam etc…).
"It also highlights the critical importance of clearing thousands of state and local permitting obstacles, which will delay modernization efforts by years. Moving from aging, analog systems to more resilient, digital technology, is critical to maintaining the reliability and resiliency of the national airspace system"
Whilst I don't doubt the need for modernisation, it's the "modern digital systems" that are the most easily attacked, as in this case.
Collins Aerospace don't have the excuse of being some fly-crap business, they're part of RTX, owners of Pratt & Whitney, and Raytheon. The Collins Aerospace business unit has 80,000 employees and turnover of around $28 billion, larger even than the turnover of either the P&W and Raytheon divisions. Yet still they've been trashed by Russian hackers - I'm sure Krasnov will have a strong response....
it's the "modern digital systems" that are the most easily attacked, as in this case
That's not quite fair... Yes, the "modern digital systems" are shared by many (often thousands) of customers. So when they fail they affect many, many more people than the old analogue systems which effectively served one customer.
On the other hand, the old, non-shared systems didn't scale. Most of the people affected by the failure of the modern systems wouldn't have been able to have service at all, at any time, however reliable that service was. Modern airports didn't get to handling the volumes of flights, passengers, luggage and cargo that they do today without the scalability provided by those shared systems.
In the 90's I was working on (the network management of) the new wide-area "ring" technologies which BT and their competitors were rapidly deploying. Those technologies (SONET, etc) mostly predated ATM and IP backhaul and really opened up reliable, flexible, high speed wide area networking. They were shared infrastructure, supporting thousands of users at any one time, so were affordable (to the big businesses who were the target at the time). They were well designed and engineered and had redundancy (self-healing rings), all at a reasonable cost - a fraction of a dedicated resource.
However, they were not designed to deal with actual adversaries. None of the systems I worked on were designed with military requirements. The military had their own systems, and no one had thought of criminals scaling protection rackets up to the levels of today's ransomware attacks.
I worked for a competitor on what was at the time the leading example of "next generation" passenger processing systems.
We had a model where for nearly all airports it was deployed on prem, not connected to the internet, and isolated from all other airport systems behind its own dedicated network stack.
We did consider a shared/hosted model for much smaller airports, or airports that only wanted a few common-use desks alongside their dedicated desks - because there was demand for a lower barrier-to-entry solution. We were the first to host the solution remotely with thin clients and working peripherals in the airport.
In our shared model, the server hardware/hypervisor and airline network connectivity were the only things that were shared - the individual airports still had dedicated and logically isolated systems, independent virtual machines, on different network segments, on dedicated Windows domains, unable to talk to each other, but able to talk to the relevant airline network depending on which user was logged in to which virtual instance at any given time.
We approached the design with complete paranoia, and I would never have advocated using that model for any large airports like Heathrow.
We did propose a common solution for Dublin and Cork, hosted out of Dublin. But that was still an isolated on-prem solution for effectively one customer, taking advantage of the airline connectivity and data centre at Dublin to reduce costs at Cork which is obviously a much smaller airport.
One of the biggest problems we started to face was more and more airlines demanding access to random websites on systems that had invariably been designed for access between airports and airline datacentres, usually over high SLA but poor bandwidth links. All of a sudden, with no extra funding for NGFWs you've got demands to open up outbound traffic for swathes of IPs over an airport's internet feed, airlines threatening to pull service from airports if they don't get their own way, and before you know it you've got random gate agents who've figured out how to break out of their airline's walled garden and are browsing Facebook between flights. Nightmare. Glad I'm out of the industry now TBH, though it was fun at times.
We did consider a shared/hosted model for much smaller airports, or airports that only wanted a few common-use desks alongside their dedicated desks - because there was demand for a lower barrier-to-entry solution. We were the first to host the solution remotely with thin clients and working peripherals in the airport.
I'm guessing that behind this story will be a 'Cloud' fsk'up. Mainly because the nature of the beast suggests a shared/hosted model. No idea if that'd be a Collins private 'cloud', or they increased the risks by loading it into AWS. And I'm also now a bit curious how locked down any shared terminals might be, especially if airlines 'need' to be able to plug in their own QR or bar code scanners & how that might be abused to inject something nasty. I've often been sitting at gates where there were no agents and terminals left unattended.
The airport/common use provider typically provide the peripherals, with a mixture of industry-standard and proprietary APIs, Firmware, and translation layers.
Most sensible airports have the PCs or Thin Clients locked up inside a cabinet at the check in desk or gate so that Joe Public can't screw with stuff, or bad guys can't plug in key loggers, or ground handlers can't charge their phones.
Typically there's no need, support, or access for airlines to plug their own peripherals in at common use desks.
If they can seamlessly switch to a manual Plan B, using paper, without inconvenience, then they are a well run company.
If not, they can stick their apologies where the sun doesn't shine, get their wallets out and compensate every passenger inconvenienced. Per hour delay, with lots for a cancellation.
Do your ****ing jobs properly or pay up. Doing it properly means having a working Plan B, because tech is not resilient and will go TU. That is where we should set the bar of our expectations, across the board.
The "compensation" you are talking about is paid in advance in the form of the lack of another zero on the end of your fare.
There's so much regulatory stuff to jump through, and so many critical calculations, not to mention stuff like routing baggage on connecting flights and so on, that if you wanted a robust and resilient non-automated backup plan that would be so slick that it wouldn't inconvenience qnyone, they'd have to charge so much they'd go out of business.
It's an uncomfortable truth but many businesses are only viable by strategically cutting a few corners here and there. I'd rather airlines and airports cut a few corners on the resilience of their passenger processing systems if it means they don't start trying to shave costs off safety-critical systems.
> If they can seamlessly switch to a manual Plan B, using paper, without inconvenience, then they are a well run company.
Didn’t look very seamless at Berlin last Saturday. Flight I was on to LHR was four hours late and arrived without luggage.
The problem is that there is still 30 year old code out there that simply isn't up to modern exploits. Unless companies take security seriously (a firewall simply isn't enough) then eventually somebody will exploit it. Ultimately, that is "just" a money thing, balanced against board bonuses and shareholder dividends, so we know where that will go!
The real problem is that the oldest code is run by the biggest companies, and the biggest of all are governments, who are shockingly bad at security (with a few notable exceptions) It's just luck that somebody has not ransomed a missile system yet :(
