
Redundancy?
'Nuff said.
A single dodgy router managed to crash the entire United Airlines computer network, leading to the cancellation of flights across the globe. See that long line, that's @united in Boise. see any other lines at other airlines? Nope, don't fly #UnitedAirlines pic.twitter.com/PnWRcGtPJM — Jeralyn Novak (@AgCommunicator) July 8, …
The only real news story that matters right now (other than waiting for those lovely Pluto pics) is largely being glossed over. China's stock market is losing more money some days than Greece's entire economy for a year. Their volatility curve looks like bitcoins and that ain't good.
The Chinese stock market is down to China not having read their history of the Wall Street Crash in 1929. Apparently 90% of their stocks are owned by individual investors - and loads of them have borrowed money to buy the stocks, which is why the Shanghai index has doubled in price over the last 18 months. So of course when the price drops, you have to sell before the value of the shares you own goes below the value of the debt you took out to pay for them. When half the other people on the market are in the same boat, things go wrong very quickly.
Margin trading is scary. Don't do it kids!
One big difference is that China's government can do whatever the hell they like. So they've suspended trading in half the market, they can make the banks write-off the loans, or just abolish the stock market.
The Chinese banking sector is in some ways quite well regulated. They have reasonably high interest rates and very high capital requirements. A lot of their bad loans are also to local government, so the national government will probably have to bail them out. On the other hand the shadow banking industry is huge, and barely regulated. Possibly they were too cautious with the banks, so people went elsewhere? And that mess will be painful to clean up.
I work in the industry (hence AC), and have seen a number of routers borked by power loss because the morons in charge of them don't always remember to save running config to startup...!
Additionally, at many airports, many airlines only run in a single circuit - they just don't want to pay the fees for redundant circuits...
Granted this sounds like a much bigger SNAFU, but I'd be surprised if anyone loses their job unless they've been truly negligent. Chances are a techie suggested a resilient solution, and a beancounter vetoed it.
All senior networking staff at United (or Untied Airlines as I like to call it) should be shown the door. Using advanced tools (that I developed) at Nokia, we were able to predict when networking or other server gear was about to fail - it is called predictive analytics. We captured 10 billion data points per day from 5000 servers world-wide, stuffed that data into a Hadoop cluster in the Amazon cloud, and then would look for out-of-band behavior, such as failing routers, switches, etc. When Microsoft took our division over (and fired/laid-off about 13000 of us) I was about to implement a series of Kalman filters to refine this process so we could be more precise in our predictions.
Since United doesn't have to support 100-200 million active users, this should NOT be difficult for them! Shame!
Anyone for some facts?
It was not the "entire computer system" that was out. It was the Weight and Balance application that in US carriers is typically part of the Operations/Dispatch suite. This is different to airlines in other parts of the world that typically do W&B in the Departure Control System (DCS) that also handles seat allocation, passenger check-in, baggage acceptance, boarding control etc etc.
Weight and Balance is a critical application that produces the load sheet. Valid load sheets are a legal (as well as common sense) requirement and without one a flight may not operate. It is possible to calculate load sheets manually but this has not been done as a routine process for many years and probably the skills required to do it are in very short supply. Hence the FAA ground stop.
United's major IT systems are outsourced to H-P Enterprise Services. This is a legacy from when EDS (acquired by H-P) ran IT for Continental Airlines (which acquired United along with its name five years ago). I suspect that there may be some tense meetings between United and H-P execs in the days to come.
Can't they just do it like Das Boot? When you want to take off you make all the fattest passengers run to the back of the aircraft, then when you get to crusing altitude you make them lie in a big heap in the aisle over the wings.
Obviously it's bad for passenger morale when the pilot comes on the PA and screams, "ALLLLAAAARRRRMMMMMM!!!!!!!!". But it's a small price to pay for getting off on time...