Fondleslab deaths grounded ALL of American Airlines' 737s Dozens of American Airlines flights were delayed this morning when pilots' iPads abruptly crashed, leaving the entire AA fleet without access to vital flight plans and, resultantly, grounded. American Airlines uses specific software on its pilots' iPads to distribute flight plans and relevant information to the crew. The …
"Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated."
Hmm, that would more than double the cost of getting the software written and of maintaining the two systems, and you'd still have got half your flights grounded today. Unless you're suggesting that every pilot should have permanent acces to two tablets, rather than deploying a mix, in which case even more cost. Hard to justify for a non-safety-critical system. Maybe better to have a web-based fall-back, so in a pinch pilots could use their own smartphones or tablets to sign in and grab the data?
They may not be "safety critical" but they sure are business-critical as shown today.
Also I doubt the cost of having software for two OS is anywhere near double to cost of one, but we will have to wait and see if it was an OS problem killing the connections or an app problem. Either way, it is a timely reminder of just how much companies depend on IT systems working.
The person I replied to said specifically "[...] software written by two independent companies", so if it's bespoke software, you'd pay twice for that; bespoke software is typically more a development cost than anything else. Granted, if it's off-the-shelf, then that might not be the case.
Nonsense. All the work before commissioning a vendor to write the software is portable to the second vendor.
Also, the first is likely to have asked most the questions you didn't think about, making it easier for the second.
If you have two suppliers work simultaneously you may get questions from both that make both products better.
As for the comment above about buying twice as much hardware, the article mentions the pilot and copilot's iPads both going blank. So you already have two devices, this is just about enduring there's redundancy beyond the physical device.
The software is already written: Delta Airlines gives its pilots Surface tablets running the same software from Jeppesen. Airlines would one hardware supplier because they can get better discounts by buying 10,000+ tablets from one maker than by splitting it into 4000 and 7000...
Replacing the bulky 35lbs flight bags also allowed airlines to save fuel.
I don't see how 35lbs results in a significant fuel savings. Rather, I think a smart pilot would want to keep the 35lbs paper flight manuals on-board anyway as a backup. As a passenger, that's O.K by me.
>>I don't see how 35lbs results in a significant fuel savings. Rather, I think a smart pilot would want to keep the 35lbs paper flight manuals on-board anyway as a backup. As a passenger, that's O.K by me.
You don't see it because you're looking at the problem the wrong way. You're looking at 35 lbs. as a fraction of the plane's total weight, in which case it's a small fraction of 1%, which seems fairly insignificant.
Instead, imagine how much energy it must take to lift a 35 lb dumbbell several miles into the air and then move it hundreds of miles per hour to a destination that's hundreds of miles away. That's a non-trivial amount of energy/fuel. Multiply that by hundreds or thousands of flights per day and you're quickly talking about a huge amount of fuel and money.
http://www.nytimes.com/2008/06/11/business/11air.html
Consider that reducing drink tray weight by 17 lbs. is saving American 1.9M gallons of fuel annually. So cutting the flight paperwork by 34 lbs. is probably saving hundreds of thousands of gallons.
Hmm, that would more than double the cost of getting the software written and of maintaining the two systems
The cost of a single outage is in the millions. I doubt that the cost of maintaining different makes and versions for the Captain and co-pilot is anywhere near that.
Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated.
Smart. As I recall, the U.S. Space Shuttle has/had three independent & redundant onboard computers with three different hardware configurations and software written with three different compiler manufacturers.
No, he's suggesting that, instead of Pilot + Co pilot having 1x Tablet-X each, the Pilot has 1x tablet-X and Co pilot has 1x Tablet-Y running the same software but on different platforms for system resilience.
Yes there's an initial cost of porting your app to a different platform but given the business impact of not getting your birds in the air at all for want of a second device type is huge.
In that case scenario all the flights would have taken off as each of the 737s would have at least one functioning flight plan tablet.
Proper resilience and business continuity is only expensive until you don't have it.
"Replacing the bulky 35lbs flight bags also allowed airlines to save fuel."
That made me laugh. What's that, 16kg? I can't see that saving a lot of fuel on a 747... or many other planes.
Besides which any savings were more than wiped out by today's problems.
Seems they'd have been better off sticking with the "old" solution which worked.
Seems they'd have been better off sticking with the "old" solution which worked
Problem with the old solution is cross referencing every minor update and error addendum every time you looked at the book, that and making sure 50000 pilots placed worldwide got the bit of paper with the "don't push the red button mid flight" update.
if Boeing issued an addendum for all aircraft that's a few million flight crew that needed be aware of it.
Since most of these birds have glass cockpits is there a reason why the data couldn't be uploaded directly to the planes? Perhaps something complicated like a USB stick would work.
In order to include additionnal functionalities to a glass cockpit, you have to ensure that this software won't have an adverse effect on the overall system.
This mean developping the software following the good ol' DO178b (or the c if you're up to date), and would probably increase the costs of the developpement by a 4-5 factor, if not more...
From what I recall, the pilot and co-pilot are not allowed to eat the same food as each other, just in case one set of foods a bit off and makes one of them ill, the other pilot who ate different food won't be ill.
So there is history in the airline industry having systems in place in case something happens to one pilot, this should have been thought of the tablets, what happens if both tablets goes down at the same time, what's the worst case scenario, how can we prevent that.
No, NO ONE should be using iPad, Surface Pro, Kindle Fire or Domestic Android tablet for this application.
OS too fragile
HW too fragile.
At the very worse something like a Panasonic ToughPad, with Linux rather than consumer Android or "Windows" and locked down to prevent users adding applications.
Edit:
While QNX was great on embedded systems and disk controllers, I'd not use a Blackberry either.
This happened on take-off and that's a great time to fail. If this happened mid-flight, it's kinda fucking terrifying if you don't have access to an airport chart for the destination.
Let's say there's a comms problem driving the failure, radios and landing instrument tracking - without a paper backup how do you know the runway elevation and length? How about the frequency of the local tower? Loads of good reasons to have this (redundantly) on paper.
"should have used the Surface Pro like most other airlines that use tablets.."
Got a source for that?
When Delta announced they were going with Surface rather than iPad, it was so unusual that there was a joint MS/Delta press release [1], and a flurry of articles e.g. [2] saying that the drivers didn't want Surface, they wanted what they were used to, ie iPads.
Shades of MS desperation and a CEO-level deal, methinks.
[1] http://news.microsoft.com/2013/09/30/delta-to-equip-11000-pilots-with-microsoft-surface-2-electronic-flight-bags/
"Delta to equip 11,000 pilots with Microsoft Surface 2 electronic flight bags
Posted September 30, 2013 By barrettevans
ATLANTA and REDMOND, Wash. — Sept. 30, 2013 — Delta Air Lines (NYSE:DAL) is equipping its 11,000 pilots with electronic flight bags using the Microsoft Surface 2 tablet. Device rollout to pilots flying the Boeing 757 and Boeing 767 fleets will start later this year and all Delta cockpits are projected to be paperless by the end of 2014.
(continues)"
[2] http://appleinsider.com/articles/13/09/30/delta-pilots-fought-against-deal-to-replace-ipad-flight-bags-with-microsoft-surface
"Got a source for that?"
Pretty much every single airline that went for tablets since the FAA certified Windows based solutions last year seems to have gone for a Windows based option:
http://www.lovemysurface.net/surface-pro-3-airlines-equip-their-pilots/
South West Airlines, Lufthansa, Austria for example. Also Air Asia and Delta pilots also use Surface tablets. British Airways City Flyer also went Windows and supplied their pilots with the Panasonic Toughpad
'Latest' versions have bugs; it's not until they've had a damn good kicking in service that they can be deemed stable. If I thought the people flying my plane were using a version that wasn't stable, I wouldn't even get on board.
Nope, the right answer is platform redundancy, as others have noted.
The only realistic subsidiary to Jeppeson is a Lufthansa subsidiary and as far as I'm aware neither support Android or Windows. You're stuck with the iPads. And remember this stuff has to integrate with the airlines back-end flight planning software, so pointless duplication would be a major fundamental cost
"Once is happenstance, twice is coincidence and three times is enemy action".
...So... what is it when the entire fleet experiences the same event?
But really: is it any surprise when Apple's patented Magical Thinking™ security does not work?
Disclaimer: I am assuming that American Airlines have more than 3 airplanes. It seems a fair assumption, but I have no data to back it up.
From other reports, the issue appears to have been a problem with the application (not the hardware or OS) hence multiple affected aircraft.
One thing I am unclear on is the need to return to the terminal for Wifi access - I would hazard a guess that they had backup hardware but needed to re-sync the information but that is speculation rather than inside knowledge.
Hopefully there will be a public report for what went wrong.
This could have been prevented with 80's era tech - a 286 hooked up to a dot matrix printer* and a fax to send out flight plans as required. As in, "put down the shiny toy and look at this white stuff with printing on it. It's called paper. It doesn't require batteries or wifi. It just works™."
* See BOFH 2013, #5. As an added bonus, when they work out who is responsible for this stuff up, they too can be fed to the dot matrix printer.
Could it have been a passenger running a mobile hotspot with No iOS Zone? The symptoms are eerily similar.
Could it have been a passenger running a mobile hotspot with No iOS Zone?
“The pilot came on and said that his first mate’s iPad powered down unexpectedly, and his had too, and that the entire 737 fleet on American had experienced the same behavior,”
That's one hell of a No iOS Zone created if it affected planes all over the US, or there's some magical synchronicity happening between people running that software.
Knowing American Airlines, as I did slightly many years ago, they could well be doing that.
About 11-12 years ago, my work took me to Dallas TX, and my customer had a deal with American Airlines so I had to fly with them. Their in-flght corporate videos - eg "Welcome to American Airlines", "Finding your way around Dallas Airport", etc - commenced with an animated rotating Earth.
But the Earth was rotating in the wrong direction...
But shouldn't the Earth rotate in the opposite direction in the video because you're facing the screen, not looking at the ground, and you're flying in a different direction to how you arrived and because of the timezone differences, the plane, clouds and sunspots and stuff?
- AA Marketing and Media Team.
> why aren't tickets priced based on passenger weight?
They would if they thought they could get away with it - even Easyjet and Ryanair haven't stooped that low (yet) !
It would cause some massive PR problems. People would have to declare their weight when booking (and you know how vain some people are about their weight), and be checked at check-in (when a lot of people would be offended to find they are "overweight", and then there's the admin of collecting all the "excess baggage" fees.
And I suspect that overall, it wouldn't make all that much difference by the time you've averaged out a typical passenger group. Though there have been "issues" in the past with abnormal weight distributions in the passenger set - either a party of children (lighter) or a party of well built adults (eg some sportsmen, heavier).
We still have 2000 years old writings as well as "unauthorized" versions of the Bible. Greatest humanity scientists did all of their work on paper and now we scoff at anything non-touch oriented and my son after five years in school has the same handwriting as a severely brain damaged monkey because no teacher in his school remembers how to hold a pencil.
Enough said. Yeah, I know I'm old and behind the times.
Sweden and Finland are considering removing handwriting from their curricula.
And quite how many documents from "the olden days" survived, as a percentage? It's extraordinarily tiny and usually only the stuff that mentions kings, gods, etc. and was worth carving in stone for those who had the money.
Technology is no different. If you take efforts to preserve it (e.g. UK tax history, criminal records, etc.) then it will be preserved. But the vast, vast, vast majority of things won't have that process applied to them. So only scraps and bits will survive as time goes on.
We now record and generate more information EVERY SECOND than the collective entirety of every work up until the computer age. You can't store it all. You can't preserve it all. You don't WANT to keep it all as you can't even begin to analyse that amount of data sensibly.
However, your child's child likely will have no need of handwriting beyond block-capitals. Even private schools are now beginning to decree "digital pencilcases" in addition to the normal pencilcase, including tablet computers, etc., and it won't be long before they are the norm. The next generation will learn to type their name into an app long before they pick up a pencil.
I think you misunderstand what they plan to do in Finland and Sweden:
They stop teaching a special handwriting script of connected letters and instead tech kids to write ordinary latin script. It doesn't mean that kids cannot write by hand, but that their hand-written text will look more like printed text and less like this or that
But some scientists cannot read text from 200 year ago: Recently I noticed in a scientific paper that a scientist confused f with s (when written as f without the horizontal stroke).
BTW few people can read old bibles: language evolved, writing also [see medieval tricks to write less letters]
The NHS health records debacle. We sent millions of records abroad to be computerised, all we got back was junk because the handwriting couldn't be read or was mis-read (hypo- instead of hyper- makes a big difference in medicine!).
Signatures? Given that most of the signatures I see every day are fake - nothing more than scanned-in JPEGs of handwritten signature, maybe that's a good thing. Does your credit card need a signature any more? Cheques are already dead. A signature is a VERY poor identifier. I guarantee I can copy your signature with a dozen random examples and a few hours of practice.
So maybe it's a good thing that digital signatures, including certificates, are being used for renewing driving licences, submitting tax forms, etc. and have been for years, and that I've even used several contract-signing services online where people "sign" with full verifiable, legal proof of their consent to the agreement (not just a "tick this box to agree").
Handwriting is dying, I tell you. Give it a few years. I can't even remember the last time I used a pen. Sure, I work in IT but I'm a stalwart who only got a proper smartphone a year or so ago. And if I'm living quite happily not using signatures today, you can be sure the rest of the world won't be using them in a few decades. My employer uses smartcards and digitial certificates to authorise payroll payments in the millions. I've signed my contract online. I verified my identity online for CRB purposes. I pay all my bills online. You can authorise Direct Debit without a single signature online. My bank account gave me a smartphone app to replace the secure-pin-pad thing that I've been using for the last ten years.
I honestly can't remember the last time I signed anything binding beyond a "this is our visitor record, name and sign please". And even that, I priced up an iPad one just the other day.
Handwriting, and signatures specifically, are dead.
It's spelled American Airlines. You simply DO NOT use consumer grade gear to run an airplane, even if it is just a glorified manual. They should have been using something like the Blackberry tablets. At least the OS (QNX) is basically unhackable. That doesn't say the software isn't at fault, but even if one application failed, the tablet wouldn't.
you do if the CEOs get iJacked by the sales team *before* having a chance to consult with their tech experts.
My company narrowly avoided spunking £50,000 on a "proof of concept" because the CEO had been shown an "automated" (IoT) dolls house by a supplier. Luckily he had a phone call from my boss while the sales spiel was in progress, and we managed to persuade him that it wasn't good value.
The worrying thing is the only way we could do that, was to suggest that £50,000 was ridiculously cheap for a PoC, so there must be something wrong.
Remember. Dilbert is a documentary.
"You simply DO NOT use consumer grade gear to run an airplane, even if it is just a glorified manual. They should have been using something like the Blackberry tablets"
LOL - those are TOTALLY consumer grade. A business grade tablet would be something like a Microsoft Surface Pro, although other options exist.
"At least the OS (QNX) is basically unhackable."
Riiiiiight - other than the 91 known vulnerabilities in Blackberry Playbook OS to date (Secunia) - of which 22% are unpatched - and the fact it was completely rooted as soon as launched via a flash exploit that simply required visiting a website!
Semantics :
"Emergency Go To Bag" " Let's print them" "Workaround" "It WILL fail, so now what?" "Yes, we have a workaround in place in case this happens"
Welcome to the wonderfull world of Disaster Recovery & Business Continuity.
Also known as "hindsight is priceless, so what about my BCP budget being cut again this year"
http://dilbert.com/strip/2000-08-15
This post has been deleted by its author
Just the NSA testing some new software and there was a minor glitch. Instead of hitting just the target 737 in the AA fleet it hit all of them. The bug has been fixed. Remember the NSA is your friend and is doing this for your safety and not for anything nefarious, remember they are the "good" guys and not some spooky evil government agency bent on world domination.
[/me looks around for a datasheet]
well, this is one for a TDA5144, probably from a consumer hard disk drive's PCB-- but the message is probably copypasta:
"LIFE SUPPORT APPLICATIONS
These products are not designed for use in life support appliances, devices, or systems where malfunction of these products can reasonably be expected to result in personal injury. Philips customers using or selling these products for use in such applications do so at their own risk and agree to fully indemnify Philips for any damages resulting from such improper use or sale."
So how many ICs in an iPad have similar messages in their datasheets? (Hint: probably all of them) FFS, people, use some brain.
--a stupid american, ashamed of his fellow stupid americans
The informations about the flight plan may be mission critical, but they are not safety critical.
In airborne embedded software, safety is paramount and according to the definition of the main aerospace software certification standard (DO178b/ED12b), this should probably be classified as a level D software* - a very low level of criticity, used for example for the cockpit's ground maintenance software. As I have an habit of being overprotective, I would classify it as a level C Software*
Sure, if it happens in flight (which seems not to be the case here), it would be a bother for the crew, and even worse to the ATC** which will have to handle a herd of lost planes looking for directions, but it would not impair the plane ability to fly and safely land (the landing informations are provided by safety-critical application, thanks you)
* For reference, level D is defined as "Minor: Failure conditions which would not significantly reduce aircraft safety, and which would involve crew actions that are well within their capabilities. Minor failure conditions may include, for example, a slight reduction in safety margins or functional capabilities, a slight increase in crew workload, such as, routine flight plan changes, or some inconvenience to occupants."
Level C is defined as "Major: Failure conditions which would reduce the capability of the aircraft or the ability of the crew to cope with adverse operating conditions to the extent that there would be, for example, a significant reduction in safety margins or functional capabilities, a significant increase in crew workload or in conditions impairing crew efficiency, or discomfort to occupants, possibly including injuries." - Might be overkill in such a case.
** ATC : Air Traffic Control
I agree it wasn't a safety critical application, outside the most bizarre worst-case scenario. But I won't hide that I hate iStuff and that to me, doing anything important on an iPad is obscene. But forget it, chip manufacturer liability is overkill. Forget the certifications and testing, just go to YouTube and find videos for "replace iPad screen". Everyone already did the testing, and the dumb thing can barely survive a fall off a coffee table (where it belongs, if anywhere) not like the flight recorder at the opposite end of the ruggedness spectrum. If it really comes down to one or the other all-purpose hardware being used to carry around a clue, I second the ToughBook suggestion.
All of the airplane's flight instruments go through hours and hours of rigorous bench-tests and their software is scrutinized down to the finest detail, as you would expect.
It's a good job the App dev. company isn't writing any flight critical instruments. Did they test it at all?
I think this event does bring into focus the question of when is a technology mature/stable enough for mission critical applications. I remember people criticising NASA for using (supposedly) old processors in the shuttle, but there are a couple of points that I think are relevant from this case.
Firstly, the safety of the flights does not seem to have been compromised as the iPads were only being used to replace the paper versions of the flight documents. As such, the level to which this could be described as "mission critical" is up for discussion. Sure AA has been thrown into a spin, but the planes themselves haven't.
Secondly, how can you develop redundancy into systems which are - essentially - software dependant? The triple redundancy standard (used for much of aerospace) is based on the physical failure rates of independent components. If the issue was hardware, then having three iPads would suffice, but could you (would you?) apply triple redundancy to software? You would have to have three separately written programs in order to avoid the same bug being in all three systems. Is this feasible? And not just on a cost basis, but from a functional point of view could you get three systems to do exactly the same the same thing, but separately developed and separately operating?
"Firstly, the safety of the flights does not seem to have been compromised as the iPads were only being used to replace the paper versions of the flight documents. As such, the level to which this could be described as "mission critical" is up for discussion."
Well they grounded their fleet, and can't or won't fly without it working properly, so I say they see it as being mission critical.
That's the hypothesis some have put forth on another s/w geek board. The flight crew reached the point in their checklist where they were to open a particular file (PDF? Proprietary format?) and they both crashed simultaneously. The solution was to go back to the gate WiFi hotspot and grab a repaired copy.
So we have data required for a flight. And there's no means to checksum it against a vendor's tested copy on download? No signed certificate to make sure Bad People haven't slipped a corrupt copy onto the server? And then a viewer app that crashes the tablet instead of popping up a "bad data" message? If it was up to me, the iPad would keep the last version of map (assuming adequate storage capacity) and allow reverting to the older one.
If this is the same Will then don't take it the wrong way but... either there never was hope, or it's much too late. All Software Sucks. Suddenly I wonder: is there already an established ASS principle? You could teach it right after the KISS principle, if you were feeling skippy.
- A common piece of shared data is misinterpreted by the app causing it to crash - e.g. an unexpected character in a certain field - like getting an '&' symbol where you only expect letters
- A hard coded date/time bug
- External, malicious intent - less likely that simple incompetence in coding and data, given the complexity of creating such a bug and having it spread and hit at the same time
It's unlikely to be the o/s, as my bet is they are all on slightly different versions of Apple o/s and patch levels.
Its unlikely to be a hardware triggered bug such as the malicious WiFi problem (as they all failed at the same time but in multiple locations)
Who is to blame? in the airlines eyes, probably the IT service provider responsible supplying/maintaining the pilots iPads regardless if the problem is with the app or the tablet. Then the service provider will sue one of these suppliers.
">>35 lbs is nothing when you consider the variability of passenger's weight...
Would you want this "nothing" to come out of your pocket?"
It already does come out of other passengers pockets if there are extra massive passengers on the plane paying the same fare as everyone else. Which of course there generally are.
I'm not suggesting that airlines should charge passengers based on their mass. But if they don't, they may find it helpful to explain why saving 35lbs matters so much when it's in the cockpit but not when it's rather more than that in the Self Loading Freight division.
>>It already does come out of other passengers pockets if there are extra massive passengers on the plane paying the same fare as everyone else. Which of course there generally are.
If I already have to pay for the airline to fly fatter people around, why do you think I'd want to ALSO pay for them to fly 35 lbs. of paper around too?
Scraped this off the BBC:
======
"The issue was caused by a duplicate chart for Reagan National Airport in American's chart database," said Mike Pound.
"The app could not reconcile the duplicate, causing it to shut down.
"We were able to remedy the situation quickly, and instruct pilots to uninstall and reinstall the app.
"Until the chart database is updated, AA pilots flying to or from National will use PDF [portable document format] images of the chart, outside of the app."
======
So, sorry to conspiracy theorists, it was just Ronald Reagan reaching out from beyond the grave...
Oh, and if it had happened while in the air, the pdf maps would have sufficed.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
