Router cockup grounds United Airlines flights WORLDWIDE

A single dodgy router managed to crash the entire United Airlines computer network, leading to the cancellation of flights across the globe. See that long line, that's @united in Boise. see any other lines at other airlines? Nope, don't fly #UnitedAirlines pic.twitter.com/PnWRcGtPJM — Jeralyn Novak (@AgCommunicator) July 8, …

[NP-Hardass]

Redundancy?

'Nuff said.

[asdf]

Re: Redundancy?

Redundancy (without the payment) of some of the IT management sounds like the first step actually.

[Mark 85]

Re: Redundancy?

The staff might be at fault, but the blame probably goes up the food chain. Management has the responsibility to ensure that backups systems are there even if it eats into profits.

[Anonymous Coward]

Re: Redundancy?

Based on inside knowledge, the issue wasn't a SPOF router, but a configuration change to a redundant pair. Also, note that the equipment in question was part of the holdover from the Continental takeover, so it was managed in-house rather than by one of their suppliers.

[Derpity]

Sensational

A couple of the more "creative" US news sites are linking this with the problems at the NYSE and Wall Street Journal. Obviously the only answer is they all shared systems and terrorists are attacking America! The source country is about to get a home delivery of freedom.

[asdf]

Re: Sensational

The only real news story that matters right now (other than waiting for those lovely Pluto pics) is largely being glossed over. China's stock market is losing more money some days than Greece's entire economy for a year. Their volatility curve looks like bitcoins and that ain't good.

[Spaceman Spiff]

Re: Sensational

I am wondering which Wall Street banks are behind this... Heaven help them if the Chinese government finds out!

[I ain't Spartacus]

Re: Sensational

The Chinese stock market is down to China not having read their history of the Wall Street Crash in 1929. Apparently 90% of their stocks are owned by individual investors - and loads of them have borrowed money to buy the stocks, which is why the Shanghai index has doubled in price over the last 18 months. So of course when the price drops, you have to sell before the value of the shares you own goes below the value of the debt you took out to pay for them. When half the other people on the market are in the same boat, things go wrong very quickly.

Margin trading is scary. Don't do it kids!

One big difference is that China's government can do whatever the hell they like. So they've suspended trading in half the market, they can make the banks write-off the loans, or just abolish the stock market.

The Chinese banking sector is in some ways quite well regulated. They have reasonably high interest rates and very high capital requirements. A lot of their bad loans are also to local government, so the national government will probably have to bail them out. On the other hand the shadow banking industry is huge, and barely regulated. Possibly they were too cautious with the banks, so people went elsewhere? And that mess will be painful to clean up.

[ma1010]

Days of glory past

United used to be a great airline which I enjoyed traveling on - about 30 years ago. Nowadays, not so much. If you want to see more of the horror stories, go to untied.com (no spelling error).

[Spaceman Spiff]

Re: Days of glory past

Yeah. Agree. I liked to fly on Untied Airlines until sometime about 15 years ago. Now, I much prefer Southwest.

[Anonymous Coward]

No surprise here

I work in the industry (hence AC), and have seen a number of routers borked by power loss because the morons in charge of them don't always remember to save running config to startup...!

Additionally, at many airports, many airlines only run in a single circuit - they just don't want to pay the fees for redundant circuits...

Granted this sounds like a much bigger SNAFU, but I'd be surprised if anyone loses their job unless they've been truly negligent. Chances are a techie suggested a resilient solution, and a beancounter vetoed it.

[Anonymous Coward]

Re: No surprise here

It's probably nothing as simple as a basic lack of redundancy. In the words of the immortal devops borat

"To make error is human. To propagate error to all server in automatic way is #devops."

[Will Godfrey]

So..

A router closed all routes.

[Captain DaFt]

Re: So..

In flight announcement:

"Ladies and Gentlemen; We apologise, but due to technical problems, this flight has been rerouted to... /dev/null? WTF?"

[Spaceman Spiff]

All senior networking staff at United (or Untied Airlines as I like to call it) should be shown the door. Using advanced tools (that I developed) at Nokia, we were able to predict when networking or other server gear was about to fail - it is called predictive analytics. We captured 10 billion data points per day from 5000 servers world-wide, stuffed that data into a Hadoop cluster in the Amazon cloud, and then would look for out-of-band behavior, such as failing routers, switches, etc. When Microsoft took our division over (and fired/laid-off about 13000 of us) I was about to implement a series of Kalman filters to refine this process so we could be more precise in our predictions.

Since United doesn't have to support 100-200 million active users, this should NOT be difficult for them! Shame!

[Sgt_Oddball]

Now now...

How many ADSL filters can you find on short notice?

Or if you've unplugged the wrong network cable in a rats nest comms cabinet?

Either that or a guitar finally had its revenge....

[Ian Tunnacliffe]

Anyone for some facts?

It was not the "entire computer system" that was out. It was the Weight and Balance application that in US carriers is typically part of the Operations/Dispatch suite. This is different to airlines in other parts of the world that typically do W&B in the Departure Control System (DCS) that also handles seat allocation, passenger check-in, baggage acceptance, boarding control etc etc.

Weight and Balance is a critical application that produces the load sheet. Valid load sheets are a legal (as well as common sense) requirement and without one a flight may not operate. It is possible to calculate load sheets manually but this has not been done as a routine process for many years and probably the skills required to do it are in very short supply. Hence the FAA ground stop.

United's major IT systems are outsourced to H-P Enterprise Services. This is a legacy from when EDS (acquired by H-P) ran IT for Continental Airlines (which acquired United along with its name five years ago). I suspect that there may be some tense meetings between United and H-P execs in the days to come.

[I ain't Spartacus]

Can't they just do it like Das Boot? When you want to take off you make all the fattest passengers run to the back of the aircraft, then when you get to crusing altitude you make them lie in a big heap in the aisle over the wings.

Obviously it's bad for passenger morale when the pilot comes on the PA and screams, "ALLLLAAAARRRRMMMMMM!!!!!!!!". But it's a small price to pay for getting off on time...