back to article Fondleslab deaths grounded ALL of American Airlines' 737s

Dozens of American Airlines flights were delayed this morning when pilots' iPads abruptly crashed, leaving the entire AA fleet without access to vital flight plans and, resultantly, grounded. American Airlines uses specific software on its pilots' iPads to distribute flight plans and relevant information to the crew. The …

  1. Neil Barnes Silver badge
    FAIL

    Now an intelligent design

    Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated.

    1. tony72

      Re: Now an intelligent design

      "Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated."

      Hmm, that would more than double the cost of getting the software written and of maintaining the two systems, and you'd still have got half your flights grounded today. Unless you're suggesting that every pilot should have permanent acces to two tablets, rather than deploying a mix, in which case even more cost. Hard to justify for a non-safety-critical system. Maybe better to have a web-based fall-back, so in a pinch pilots could use their own smartphones or tablets to sign in and grab the data?

      1. Paul Crawford Silver badge

        Re: Now an intelligent design

        They may not be "safety critical" but they sure are business-critical as shown today.

        Also I doubt the cost of having software for two OS is anywhere near double to cost of one, but we will have to wait and see if it was an OS problem killing the connections or an app problem. Either way, it is a timely reminder of just how much companies depend on IT systems working.

        1. tony72

          Re: Now an intelligent design

          The person I replied to said specifically "[...] software written by two independent companies", so if it's bespoke software, you'd pay twice for that; bespoke software is typically more a development cost than anything else. Granted, if it's off-the-shelf, then that might not be the case.

          1. R 11

            Re: Now an intelligent design

            Nonsense. All the work before commissioning a vendor to write the software is portable to the second vendor.

            Also, the first is likely to have asked most the questions you didn't think about, making it easier for the second.

            If you have two suppliers work simultaneously you may get questions from both that make both products better.

            As for the comment above about buying twice as much hardware, the article mentions the pilot and copilot's iPads both going blank. So you already have two devices, this is just about enduring there's redundancy beyond the physical device.

        2. AdamT

          Re: Now an intelligent design

          well they are not "safety critical" in the sense that the plane falls out of the sky if the iPads fail but I am curious as to what the procedure is if you lose access to your flight plan (and all the "reference manuals") mid flight ...

      2. Kristian Walsh Silver badge

        Re: Now an intelligent design

        The software is already written: Delta Airlines gives its pilots Surface tablets running the same software from Jeppesen. Airlines would one hardware supplier because they can get better discounts by buying 10,000+ tablets from one maker than by splitting it into 4000 and 7000...

        1. BillG
          Holmes

          Re: Now an intelligent design

          Replacing the bulky 35lbs flight bags also allowed airlines to save fuel.

          I don't see how 35lbs results in a significant fuel savings. Rather, I think a smart pilot would want to keep the 35lbs paper flight manuals on-board anyway as a backup. As a passenger, that's O.K by me.

          1. Anonymous Coward
            Anonymous Coward

            Re: Now an intelligent design

            >>I don't see how 35lbs results in a significant fuel savings. Rather, I think a smart pilot would want to keep the 35lbs paper flight manuals on-board anyway as a backup. As a passenger, that's O.K by me.

            You don't see it because you're looking at the problem the wrong way. You're looking at 35 lbs. as a fraction of the plane's total weight, in which case it's a small fraction of 1%, which seems fairly insignificant.

            Instead, imagine how much energy it must take to lift a 35 lb dumbbell several miles into the air and then move it hundreds of miles per hour to a destination that's hundreds of miles away. That's a non-trivial amount of energy/fuel. Multiply that by hundreds or thousands of flights per day and you're quickly talking about a huge amount of fuel and money.

            http://www.nytimes.com/2008/06/11/business/11air.html

            Consider that reducing drink tray weight by 17 lbs. is saving American 1.9M gallons of fuel annually. So cutting the flight paperwork by 34 lbs. is probably saving hundreds of thousands of gallons.

      3. Voland's right hand Silver badge

        Re: Now an intelligent design

        Hmm, that would more than double the cost of getting the software written and of maintaining the two systems

        The cost of a single outage is in the millions. I doubt that the cost of maintaining different makes and versions for the Captain and co-pilot is anywhere near that.

        1. BillG
          Alert

          Re: Now an intelligent design

          Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated.

          Smart. As I recall, the U.S. Space Shuttle has/had three independent & redundant onboard computers with three different hardware configurations and software written with three different compiler manufacturers.

      4. Sir Sham Cad

        Re: every pilot should have permanent acces to two tablets

        No, he's suggesting that, instead of Pilot + Co pilot having 1x Tablet-X each, the Pilot has 1x tablet-X and Co pilot has 1x Tablet-Y running the same software but on different platforms for system resilience.

        Yes there's an initial cost of porting your app to a different platform but given the business impact of not getting your birds in the air at all for want of a second device type is huge.

        In that case scenario all the flights would have taken off as each of the 737s would have at least one functioning flight plan tablet.

        Proper resilience and business continuity is only expensive until you don't have it.

        1. Neil Barnes Silver badge

          Re: every pilot should have permanent acces to two tablets

          Quite. Two people driving, two different systems.

          1. Anonymous Coward
            Anonymous Coward

            Re: every pilot should have permanent acces to two tablets

            "Replacing the bulky 35lbs flight bags also allowed airlines to save fuel."

            That made me laugh. What's that, 16kg? I can't see that saving a lot of fuel on a 747... or many other planes.

            Besides which any savings were more than wiped out by today's problems.

            Seems they'd have been better off sticking with the "old" solution which worked.

            1. Anonymous Coward
              Anonymous Coward

              Re: every pilot should have permanent acces to two tablets

              Seems they'd have been better off sticking with the "old" solution which worked

              Problem with the old solution is cross referencing every minor update and error addendum every time you looked at the book, that and making sure 50000 pilots placed worldwide got the bit of paper with the "don't push the red button mid flight" update.

              if Boeing issued an addendum for all aircraft that's a few million flight crew that needed be aware of it.

            2. dave 93

              Re: every pilot should have permanent acces to two tablets

              American Airlines spokesperson on the BBC said $1,000,000 saving a year on fuel from the new system. They do have about 1,000 planes in their fleet though.

          2. Eddy Ito
            Facepalm

            Re: every pilot should have permanent acces to two tablets

            Since most of these birds have glass cockpits is there a reason why the data couldn't be uploaded directly to the planes? Perhaps something complicated like a USB stick would work.

            1. Yag

              Re: every pilot should have permanent acces to two tablets

              Since most of these birds have glass cockpits is there a reason why the data couldn't be uploaded directly to the planes? Perhaps something complicated like a USB stick would work.

              In order to include additionnal functionalities to a glass cockpit, you have to ensure that this software won't have an adverse effect on the overall system.

              This mean developping the software following the good ol' DO178b (or the c if you're up to date), and would probably increase the costs of the developpement by a 4-5 factor, if not more...

    2. David Webb

      Re: Now an intelligent design

      From what I recall, the pilot and co-pilot are not allowed to eat the same food as each other, just in case one set of foods a bit off and makes one of them ill, the other pilot who ate different food won't be ill.

      So there is history in the airline industry having systems in place in case something happens to one pilot, this should have been thought of the tablets, what happens if both tablets goes down at the same time, what's the worst case scenario, how can we prevent that.

      1. Tom 38
        Joke

        We had a choice of steak or fish

        From what I recall, the pilot and co-pilot are not allowed to eat the same food as each other, just in case one set of foods a bit off and makes one of them ill, the other pilot who ate different food won't be ill.

        I had the lasagne.

        1. Mark 85

          Re: We had a choice of steak or fish

          Good choice. Don't eat the fish.....

        2. Diamandi Lucas

          Re: We had a choice of steak or fish

          Someone downvoted your post! Surely, they can't be serious?

          1. Tom 38

            Re: We had a choice of steak or fish

            Surely, they can't be serious?

            It's a rorschach test to determine the humourless amongst us. And stop calling me Shirley.

    3. Anonymous Coward
      Anonymous Coward

      Re: Now an intelligent design

      LOL, should have used the Surface Pro like most other airlines that use tablets...Then at least you can control the config totally.

      Failing that, I suspect paper and pencils are still fully operational!

      1. Stoneshop
        Facepalm

        Re: Now an intelligent design

        Given that it affected only part of AA's fleet, it's apparently a problem triggered by the app choking on data specific to that type of plane in combination with the date. So please explain how it could have been caused by the choice of hardware.

      2. Mage Silver badge
        Alert

        Re: Now an intelligent design

        No, NO ONE should be using iPad, Surface Pro, Kindle Fire or Domestic Android tablet for this application.

        OS too fragile

        HW too fragile.

        At the very worse something like a Panasonic ToughPad, with Linux rather than consumer Android or "Windows" and locked down to prevent users adding applications.

        Edit:

        While QNX was great on embedded systems and disk controllers, I'd not use a Blackberry either.

        1. Cliff

          Re: Now an intelligent design

          This happened on take-off and that's a great time to fail. If this happened mid-flight, it's kinda fucking terrifying if you don't have access to an airport chart for the destination.

          Let's say there's a comms problem driving the failure, radios and landing instrument tracking - without a paper backup how do you know the runway elevation and length? How about the frequency of the local tower? Loads of good reasons to have this (redundantly) on paper.

      3. Anonymous Coward
        Anonymous Coward

        Re: Surface Pro

        "should have used the Surface Pro like most other airlines that use tablets.."

        Got a source for that?

        When Delta announced they were going with Surface rather than iPad, it was so unusual that there was a joint MS/Delta press release [1], and a flurry of articles e.g. [2] saying that the drivers didn't want Surface, they wanted what they were used to, ie iPads.

        Shades of MS desperation and a CEO-level deal, methinks.

        [1] http://news.microsoft.com/2013/09/30/delta-to-equip-11000-pilots-with-microsoft-surface-2-electronic-flight-bags/

        "Delta to equip 11,000 pilots with Microsoft Surface 2 electronic flight bags

        Posted September 30, 2013 By barrettevans

        ATLANTA and REDMOND, Wash. — Sept. 30, 2013 — Delta Air Lines (NYSE:DAL) is equipping its 11,000 pilots with electronic flight bags using the Microsoft Surface 2 tablet. Device rollout to pilots flying the Boeing 757 and Boeing 767 fleets will start later this year and all Delta cockpits are projected to be paperless by the end of 2014.

        (continues)"

        [2] http://appleinsider.com/articles/13/09/30/delta-pilots-fought-against-deal-to-replace-ipad-flight-bags-with-microsoft-surface

        1. Anonymous Coward
          Anonymous Coward

          Re: Surface Pro

          "Got a source for that?"

          Pretty much every single airline that went for tablets since the FAA certified Windows based solutions last year seems to have gone for a Windows based option:

          http://www.lovemysurface.net/surface-pro-3-airlines-equip-their-pilots/

          South West Airlines, Lufthansa, Austria for example. Also Air Asia and Delta pilots also use Surface tablets. British Airways City Flyer also went Windows and supplied their pilots with the Panasonic Toughpad

    4. James Haley 2

      Re: Now an intelligent design

      "Would have used pads from two different makers..."

      That would tricky and expensive, as others have noted.

      The simple solution is to publish two versions of the same app, a "latest" version and a "stable" version that's always known to work.

      1. Laura Kerr

        Yer 'avin' a tin, intcha?

        'Latest' versions have bugs; it's not until they've had a damn good kicking in service that they can be deemed stable. If I thought the people flying my plane were using a version that wasn't stable, I wouldn't even get on board.

        Nope, the right answer is platform redundancy, as others have noted.

    5. x 7

      Re: Now an intelligent design

      The only realistic subsidiary to Jeppeson is a Lufthansa subsidiary and as far as I'm aware neither support Android or Windows. You're stuck with the iPads. And remember this stuff has to integrate with the airlines back-end flight planning software, so pointless duplication would be a major fundamental cost

      1. Anonymous Coward
        Anonymous Coward

        Re: Now an intelligent design

        "The only realistic subsidiary to Jeppeson is a Lufthansa subsidiary and as far as I'm aware neither support Android or Windows"

        Jeppeson already supports Windows - and Lufthansa have gone for Windows on the Surface Pro for their pilots...

  2. RIBrsiq
    Trollface

    "Once is happenstance, twice is coincidence and three times is enemy action".

    ...So... what is it when the entire fleet experiences the same event?

    But really: is it any surprise when Apple's patented Magical Thinking™ security does not work?

    Disclaimer: I am assuming that American Airlines have more than 3 airplanes. It seems a fair assumption, but I have no data to back it up.

    1. theblackhand

      Malice?

      From other reports, the issue appears to have been a problem with the application (not the hardware or OS) hence multiple affected aircraft.

      One thing I am unclear on is the need to return to the terminal for Wifi access - I would hazard a guess that they had backup hardware but needed to re-sync the information but that is speculation rather than inside knowledge.

      Hopefully there will be a public report for what went wrong.

      1. silent_count

        Backup hardware

        This could have been prevented with 80's era tech - a 286 hooked up to a dot matrix printer* and a fax to send out flight plans as required. As in, "put down the shiny toy and look at this white stuff with printing on it. It's called paper. It doesn't require batteries or wifi. It just works™."

        * See BOFH 2013, #5. As an added bonus, when they work out who is responsible for this stuff up, they too can be fed to the dot matrix printer.

        1. Preston Munchensonton
          Coat

          Re: Backup hardware

          "As an added bonus, when they work out who is responsible for this stuff up, they too can be fed to the dot matrix printer."

          Now, now. That would be far too messy. This is precisely the purpose for the tape vault.

        2. Mark 85

          Re: Backup hardware

          Until recently, commercial aircraft did have printer on board. Not sure if a computer was involved or some version of fax, though. I suspect it was two way version of fax since the crew could make queries to the company and get back a hard copy printout.

      2. Ralph B

        Re: Malice?

        Could it have been a passenger running a mobile hotspot with No iOS Zone? The symptoms are eerily similar.

        1. Stoneshop
          FAIL

          Re: Malice?

          Could it have been a passenger running a mobile hotspot with No iOS Zone?

          “The pilot came on and said that his first mate’s iPad powered down unexpectedly, and his had too, and that the entire 737 fleet on American had experienced the same behavior,”

          That's one hell of a No iOS Zone created if it affected planes all over the US, or there's some magical synchronicity happening between people running that software.

    2. Mark 85

      You missed "BOFH"... one that was irritated at his boss enough to send the "shutdown iPad" command.

      For the whole fleet of 737's to be shutdown sounds like a single point of failure or malice.

  3. casaloco

    Apple...

    ...what's the betting Apple pulled the app from the iStore as unapproved, and deleted it from the iPads?

    1. Test Man

      Re: Apple...

      We know what you mean but technically speaking it wouldn't have been possible as 1. the apps wouldn't have been delivered from the App Store and 2. they didn't have wifi onboard so any deliberate "pulling" of apps would never have been possible.

      1. The_H

        Re: Apple...

        Maybe it was the "time limited" trial version. Which expired.

        Or maybe the screens went dark in silent tribute to Orville the Duck, who will now never fly?

  4. Vladimir Plouzhnikov

    Stoopid flyboys

    They are holding 'em 'rong! Duh!

    1. smudge
      Facepalm

      Re: Stoopid flyboys

      Knowing American Airlines, as I did slightly many years ago, they could well be doing that.

      About 11-12 years ago, my work took me to Dallas TX, and my customer had a deal with American Airlines so I had to fly with them. Their in-flght corporate videos - eg "Welcome to American Airlines", "Finding your way around Dallas Airport", etc - commenced with an animated rotating Earth.

      But the Earth was rotating in the wrong direction...

      1. Anonymous Coward
        Anonymous Coward

        Re: Stoopid flyboys

        But shouldn't the Earth rotate in the opposite direction in the video because you're facing the screen, not looking at the ground, and you're flying in a different direction to how you arrived and because of the timezone differences, the plane, clouds and sunspots and stuff?

        - AA Marketing and Media Team.

  5. AceRimmer

    It wasn't a fault

    It was the software optimizing itself for maximum fuel savings

  6. graeme leggett Silver badge

    costs

    35 lbs of weight (and hence 'x' amount of fuel) saved every flight against the costs of all aircraft being grounded once - an interesting one for the accountants to evaluate

    1. Anonymous Coward
      Anonymous Coward

      Re: costs per lb

      If saving weight is so critical, why aren't tickets priced based on passenger weight?

      1. Laura Kerr
        Thumb Up

        Re: costs per lb

        Because it would unfairly discriminate against Americans.

      2. SImon Hobson Bronze badge

        Re: costs per lb

        > why aren't tickets priced based on passenger weight?

        They would if they thought they could get away with it - even Easyjet and Ryanair haven't stooped that low (yet) !

        It would cause some massive PR problems. People would have to declare their weight when booking (and you know how vain some people are about their weight), and be checked at check-in (when a lot of people would be offended to find they are "overweight", and then there's the admin of collecting all the "excess baggage" fees.

        And I suspect that overall, it wouldn't make all that much difference by the time you've averaged out a typical passenger group. Though there have been "issues" in the past with abnormal weight distributions in the passenger set - either a party of children (lighter) or a party of well built adults (eg some sportsmen, heavier).

  7. Anonymous Coward
    Anonymous Coward

    Told you so!

    We still have 2000 years old writings as well as "unauthorized" versions of the Bible. Greatest humanity scientists did all of their work on paper and now we scoff at anything non-touch oriented and my son after five years in school has the same handwriting as a severely brain damaged monkey because no teacher in his school remembers how to hold a pencil.

    Enough said. Yeah, I know I'm old and behind the times.

    1. Lee D Silver badge

      Re: Told you so!

      Sweden and Finland are considering removing handwriting from their curricula.

      And quite how many documents from "the olden days" survived, as a percentage? It's extraordinarily tiny and usually only the stuff that mentions kings, gods, etc. and was worth carving in stone for those who had the money.

      Technology is no different. If you take efforts to preserve it (e.g. UK tax history, criminal records, etc.) then it will be preserved. But the vast, vast, vast majority of things won't have that process applied to them. So only scraps and bits will survive as time goes on.

      We now record and generate more information EVERY SECOND than the collective entirety of every work up until the computer age. You can't store it all. You can't preserve it all. You don't WANT to keep it all as you can't even begin to analyse that amount of data sensibly.

      However, your child's child likely will have no need of handwriting beyond block-capitals. Even private schools are now beginning to decree "digital pencilcases" in addition to the normal pencilcase, including tablet computers, etc., and it won't be long before they are the norm. The next generation will learn to type their name into an app long before they pick up a pencil.

      1. Paul Crawford Silver badge

        Re: Told you so!

        "your child's child likely will have no need of handwriting"

        So they can't sign things and thus we all have to be corralled in to a biometric future like cattle, all suitably tagged and compliant.

      2. Andus McCoatover

        "Sweden and Finland are considering removing handwriting from their curricula".

        That's the end of another chance of a Shakesperian masterpiece being written.

        On a typewriter, natch...

      3. Schultz
        FAIL

        "Sweden and Finland are considering removing handwriting from their curricula"

        I think you misunderstand what they plan to do in Finland and Sweden:

        They stop teaching a special handwriting script of connected letters and instead tech kids to write ordinary latin script. It doesn't mean that kids cannot write by hand, but that their hand-written text will look more like printed text and less like this or that

    2. .Cate

      Re: Told you so!

      But some scientists cannot read text from 200 year ago: Recently I noticed in a scientific paper that a scientist confused f with s (when written as f without the horizontal stroke).

      BTW few people can read old bibles: language evolved, writing also [see medieval tricks to write less letters]

      1. Lee D Silver badge

        Re: Told you so!

        The NHS health records debacle. We sent millions of records abroad to be computerised, all we got back was junk because the handwriting couldn't be read or was mis-read (hypo- instead of hyper- makes a big difference in medicine!).

        Signatures? Given that most of the signatures I see every day are fake - nothing more than scanned-in JPEGs of handwritten signature, maybe that's a good thing. Does your credit card need a signature any more? Cheques are already dead. A signature is a VERY poor identifier. I guarantee I can copy your signature with a dozen random examples and a few hours of practice.

        So maybe it's a good thing that digital signatures, including certificates, are being used for renewing driving licences, submitting tax forms, etc. and have been for years, and that I've even used several contract-signing services online where people "sign" with full verifiable, legal proof of their consent to the agreement (not just a "tick this box to agree").

        Handwriting is dying, I tell you. Give it a few years. I can't even remember the last time I used a pen. Sure, I work in IT but I'm a stalwart who only got a proper smartphone a year or so ago. And if I'm living quite happily not using signatures today, you can be sure the rest of the world won't be using them in a few decades. My employer uses smartcards and digitial certificates to authorise payroll payments in the millions. I've signed my contract online. I verified my identity online for CRB purposes. I pay all my bills online. You can authorise Direct Debit without a single signature online. My bank account gave me a smartphone app to replace the secure-pin-pad thing that I've been using for the last ten years.

        I honestly can't remember the last time I signed anything binding beyond a "this is our visitor record, name and sign please". And even that, I priced up an iPad one just the other day.

        Handwriting, and signatures specifically, are dead.

        1. Mark 85

          Re: Told you so!

          While I agree with you, there's more than a few of us who will only give our checkbooks when they're pried out of our cold, dead hands. In the States debit/credit card security, etc. seems to be a joke.

  8. Alistair
    Coat

    I can see the incident call whiteboard

    But someone's taken a bite out of the side with the solution on it.

  9. Geoff May

    I'm curious about somethings

    I would have expected the application to download required data to the machine and, in the event of losing connection, would use that data.

    Anyone able to explain why this wasn't being done?

    1. Preston Munchensonton

      Re: I'm curious about somethings

      Because the iPads powered off by themselves. Hard to function when not powered on.

  10. Anonymous Coward
    Anonymous Coward

    The biggest pain I see..

    .. is that they only have a one year warranty over there.

    Oops.

  11. Anonymous Coward
    Anonymous Coward

    Clearly they had downloaded ..

    .. the no fly list.

    (that's enough - Ed).

  12. Spaceman Spiff

    Can you spell stupid?

    It's spelled American Airlines. You simply DO NOT use consumer grade gear to run an airplane, even if it is just a glorified manual. They should have been using something like the Blackberry tablets. At least the OS (QNX) is basically unhackable. That doesn't say the software isn't at fault, but even if one application failed, the tablet wouldn't.

    1. Anonymous Coward
      Anonymous Coward

      You simply DO NOT use consumer grade gear to run an airplane

      you do if the CEOs get iJacked by the sales team *before* having a chance to consult with their tech experts.

      My company narrowly avoided spunking £50,000 on a "proof of concept" because the CEO had been shown an "automated" (IoT) dolls house by a supplier. Luckily he had a phone call from my boss while the sales spiel was in progress, and we managed to persuade him that it wasn't good value.

      The worrying thing is the only way we could do that, was to suggest that £50,000 was ridiculously cheap for a PoC, so there must be something wrong.

      Remember. Dilbert is a documentary.

    2. Anonymous Coward
      Anonymous Coward

      Re: Can you spell stupid?

      "You simply DO NOT use consumer grade gear to run an airplane, even if it is just a glorified manual. They should have been using something like the Blackberry tablets"

      LOL - those are TOTALLY consumer grade. A business grade tablet would be something like a Microsoft Surface Pro, although other options exist.

      "At least the OS (QNX) is basically unhackable."

      Riiiiiight - other than the 91 known vulnerabilities in Blackberry Playbook OS to date (Secunia) - of which 22% are unpatched - and the fact it was completely rooted as soon as launched via a flash exploit that simply required visiting a website!

    3. Yag

      Re: Can you spell stupid?

      As long as the FAA/EASA/<insert your local authority here> allows it, it's free game.

  13. Anonymous Coward
    Anonymous Coward

    Something's missing...

    The entire fleet of craft were grounded? Surely they weren't all located at the same airport. There's no mention of the location(s) where this happened.

    1. Anonymous Coward
      Anonymous Coward

      Re: Something's missing...

      >>The entire fleet of craft were grounded? Surely they weren't all located at the same airport. There's no mention of the location(s) where this happened.

      Why would you think the bug that caused the problem is related to the location of anything? Weird.

  14. Hellcat

    The real question is if Toni and Bill are related. If not, what were the chances of them both being on AA flights affected by the ipad crash and both heading to Twitter?

    1. Stoneshop
      FAIL

      If, as per the pilot's remark, ALL of AA's 737 fleet suffered the same problem, there's little need for Toni and Bill to have been on the same plane.

  15. Scott Broukell
    Meh

    So what goes in the map box now?

    Oh, of course, it becomes a charging dock for the gadgets I suppose!

    1. Stoneshop
      Pint

      Re: So what goes in the map box now?

      Couple of sandwiches and a banana.

  16. Nifty Silver badge

    no IOS zone?

    This looks spookily as if someone - a rogue passenger? - did this

    http://www.theregister.co.uk/2015/04/22/apple_no_ios_zone_bug/

    and spoofed the plane's WiFi system to ensure the pilots' iPads would bite...

    1. Stoneshop
      FAIL

      Re: no IOS zone?

      That suggestion might make some sense if it had been one plane only that was affected. HINT: it wasn't.

  17. splodge

    Did the pilot have some sort of black ink hipster tattoo anywhere near the slab?

  18. Knewbie
    Mushroom

    That was SUCH a good Idea....

    Semantics :

    "Emergency Go To Bag" " Let's print them" "Workaround" "It WILL fail, so now what?" "Yes, we have a workaround in place in case this happens"

    Welcome to the wonderfull world of Disaster Recovery & Business Continuity.

    Also known as "hindsight is priceless, so what about my BCP budget being cut again this year"

    http://dilbert.com/strip/2000-08-15

  19. This post has been deleted by its author

  20. DXMage
    Trollface

    Test Software Glitch?

    Just the NSA testing some new software and there was a minor glitch. Instead of hitting just the target 737 in the AA fleet it hit all of them. The bug has been fixed. Remember the NSA is your friend and is doing this for your safety and not for anything nefarious, remember they are the "good" guys and not some spooky evil government agency bent on world domination.

  21. Mage Silver badge
    Facepalm

    Perfectly clear who is responsible.

    "specific software on its pilots' iPads"

    Which ever moron picked a domestic consumer product for mission critical application. No, I'd not use a Surface, Kindle Fire or Consumer Android tablet either.

    1. Anonymous Coward
      FAIL

      Re: Perfectly clear who is responsible.

      [/me looks around for a datasheet]

      well, this is one for a TDA5144, probably from a consumer hard disk drive's PCB-- but the message is probably copypasta:

      "LIFE SUPPORT APPLICATIONS

      These products are not designed for use in life support appliances, devices, or systems where malfunction of these products can reasonably be expected to result in personal injury. Philips customers using or selling these products for use in such applications do so at their own risk and agree to fully indemnify Philips for any damages resulting from such improper use or sale."

      So how many ICs in an iPad have similar messages in their datasheets? (Hint: probably all of them) FFS, people, use some brain.

      --a stupid american, ashamed of his fellow stupid americans

    2. Yag

      Re: Perfectly clear who is responsible.

      The informations about the flight plan may be mission critical, but they are not safety critical.

      In airborne embedded software, safety is paramount and according to the definition of the main aerospace software certification standard (DO178b/ED12b), this should probably be classified as a level D software* - a very low level of criticity, used for example for the cockpit's ground maintenance software. As I have an habit of being overprotective, I would classify it as a level C Software*

      Sure, if it happens in flight (which seems not to be the case here), it would be a bother for the crew, and even worse to the ATC** which will have to handle a herd of lost planes looking for directions, but it would not impair the plane ability to fly and safely land (the landing informations are provided by safety-critical application, thanks you)

      * For reference, level D is defined as "Minor: Failure conditions which would not significantly reduce aircraft safety, and which would involve crew actions that are well within their capabilities. Minor failure conditions may include, for example, a slight reduction in safety margins or functional capabilities, a slight increase in crew workload, such as, routine flight plan changes, or some inconvenience to occupants."

      Level C is defined as "Major: Failure conditions which would reduce the capability of the aircraft or the ability of the crew to cope with adverse operating conditions to the extent that there would be, for example, a significant reduction in safety margins or functional capabilities, a significant increase in crew workload or in conditions impairing crew efficiency, or discomfort to occupants, possibly including injuries." - Might be overkill in such a case.

      ** ATC : Air Traffic Control

      1. Anonymous Coward
        Meh

        Re: Perfectly clear who is responsible.

        I agree it wasn't a safety critical application, outside the most bizarre worst-case scenario. But I won't hide that I hate iStuff and that to me, doing anything important on an iPad is obscene. But forget it, chip manufacturer liability is overkill. Forget the certifications and testing, just go to YouTube and find videos for "replace iPad screen". Everyone already did the testing, and the dumb thing can barely survive a fall off a coffee table (where it belongs, if anywhere) not like the flight recorder at the opposite end of the ruggedness spectrum. If it really comes down to one or the other all-purpose hardware being used to carry around a clue, I second the ToughBook suggestion.

  22. Zot

    Airplane testing...

    All of the airplane's flight instruments go through hours and hours of rigorous bench-tests and their software is scrutinized down to the finest detail, as you would expect.

    It's a good job the App dev. company isn't writing any flight critical instruments. Did they test it at all?

  23. 100113.1537

    When is new technology ready for mission critical application?

    I think this event does bring into focus the question of when is a technology mature/stable enough for mission critical applications. I remember people criticising NASA for using (supposedly) old processors in the shuttle, but there are a couple of points that I think are relevant from this case.

    Firstly, the safety of the flights does not seem to have been compromised as the iPads were only being used to replace the paper versions of the flight documents. As such, the level to which this could be described as "mission critical" is up for discussion. Sure AA has been thrown into a spin, but the planes themselves haven't.

    Secondly, how can you develop redundancy into systems which are - essentially - software dependant? The triple redundancy standard (used for much of aerospace) is based on the physical failure rates of independent components. If the issue was hardware, then having three iPads would suffice, but could you (would you?) apply triple redundancy to software? You would have to have three separately written programs in order to avoid the same bug being in all three systems. Is this feasible? And not just on a cost basis, but from a functional point of view could you get three systems to do exactly the same the same thing, but separately developed and separately operating?

    1. Diamandi Lucas

      Re: When is new technology ready for mission critical application?

      "Firstly, the safety of the flights does not seem to have been compromised as the iPads were only being used to replace the paper versions of the flight documents. As such, the level to which this could be described as "mission critical" is up for discussion."

      Well they grounded their fleet, and can't or won't fly without it working properly, so I say they see it as being mission critical.

  24. Paul Hovnanian Silver badge

    Bad Data?

    That's the hypothesis some have put forth on another s/w geek board. The flight crew reached the point in their checklist where they were to open a particular file (PDF? Proprietary format?) and they both crashed simultaneously. The solution was to go back to the gate WiFi hotspot and grab a repaired copy.

    So we have data required for a flight. And there's no means to checksum it against a vendor's tested copy on download? No signed certificate to make sure Bad People haven't slipped a corrupt copy onto the server? And then a viewer app that crashes the tablet instead of popping up a "bad data" message? If it was up to me, the iPad would keep the last version of map (assuming adequate storage capacity) and allow reverting to the older one.

    1. Will Godfrey Silver badge
      Unhappy

      Re: Bad Data?

      If this does turn out to be a data corruption issue then, after all this time, there really is no hope for the software industry.

      1. Anonymous Coward
        Meh

        Re: Bad Data?

        If this is the same Will then don't take it the wrong way but... either there never was hope, or it's much too late. All Software Sucks. Suddenly I wonder: is there already an established ASS principle? You could teach it right after the KISS principle, if you were feeling skippy.

  25. MatsSvensson

    turns out they had all accidentally jammed the orienation-lock slider, when they put it in the holder.

    And that's why they couldn't take off horizontally.

  26. Anonymous Coward
    Anonymous Coward

    A few causes spring to mind

    - A common piece of shared data is misinterpreted by the app causing it to crash - e.g. an unexpected character in a certain field - like getting an '&' symbol where you only expect letters

    - A hard coded date/time bug

    - External, malicious intent - less likely that simple incompetence in coding and data, given the complexity of creating such a bug and having it spread and hit at the same time

    It's unlikely to be the o/s, as my bet is they are all on slightly different versions of Apple o/s and patch levels.

    Its unlikely to be a hardware triggered bug such as the malicious WiFi problem (as they all failed at the same time but in multiple locations)

  27. Wilson! Wilson!

    IT Service provider faces huge fines by airline for tablet outage.

    Who is to blame? in the airlines eyes, probably the IT service provider responsible supplying/maintaining the pilots iPads regardless if the problem is with the app or the tablet. Then the service provider will sue one of these suppliers.

  28. Crazy Operations Guy

    Why not both?

    35 lbs is nothing when you consider the variability of passenger's weight... Even fresh-off-the-line 787s come with two sets of instruments: the glass cockpit systems and a mechanical Altimeter, compass, and air-speed gauge.

    1. Anonymous Coward
      Anonymous Coward

      Re: Why not both?

      >>35 lbs is nothing when you consider the variability of passenger's weight...

      It's hundreds of thousands of gallons of fuel for an airline, annually, and probably close to a million dollars.

      Would you want this "nothing" to come out of your pocket?

      1. Jos V

        Re: Why not both?

        From an appleinsider article:

        " American was able to do away with 24 million pages of documents carried in bags weighing 35 pounds each, projected to save some 400,000 gallons of fuel, or $1.2 million, every year. "

      2. Anonymous Coward
        Anonymous Coward

        Re: Why not both?

        ">>35 lbs is nothing when you consider the variability of passenger's weight...

        Would you want this "nothing" to come out of your pocket?"

        It already does come out of other passengers pockets if there are extra massive passengers on the plane paying the same fare as everyone else. Which of course there generally are.

        I'm not suggesting that airlines should charge passengers based on their mass. But if they don't, they may find it helpful to explain why saving 35lbs matters so much when it's in the cockpit but not when it's rather more than that in the Self Loading Freight division.

        1. Anonymous Coward
          Anonymous Coward

          Re: Why not both?

          >>It already does come out of other passengers pockets if there are extra massive passengers on the plane paying the same fare as everyone else. Which of course there generally are.

          If I already have to pay for the airline to fly fatter people around, why do you think I'd want to ALSO pay for them to fly 35 lbs. of paper around too?

          1. Anonymous Coward
            Anonymous Coward

            Re: Why not both?

            "why do you think I'd want to ALSO pay for them to fly 35 lbs. of paper around too?"

            Perhaps because there is a clearly predictable (and now clearly demonstrated) downside to not having the legacy* EFB on board?

            * Legacy == stuff that works.

  29. Anonymous Coward
    Anonymous Coward

    There is a lesson here

    Have a back-up system so you don't look like a complete fool and lose millions of dollars while alienating your customers.

  30. Andus McCoatover

    Tin-foil-hat mode, but....

    ..have they found and definitely identified the problem?

    Else (crisp metallic rustle) could this have been a 'dummy run' by some nasty Middle Eastern group?

  31. Frank N. Stein

    Perhaps someone should contact the company who provides Enterprise Mobile Device Management for American Airlines (STRATIX) and ask them what they are doing to fix the problem?

  32. Anonymous Coward
    Anonymous Coward

    The reason...

    Scraped this off the BBC:

    ======

    "The issue was caused by a duplicate chart for Reagan National Airport in American's chart database," said Mike Pound.

    "The app could not reconcile the duplicate, causing it to shut down.

    "We were able to remedy the situation quickly, and instruct pilots to uninstall and reinstall the app.

    "Until the chart database is updated, AA pilots flying to or from National will use PDF [portable document format] images of the chart, outside of the app."

    ======

    So, sorry to conspiracy theorists, it was just Ronald Reagan reaching out from beyond the grave...

    Oh, and if it had happened while in the air, the pdf maps would have sufficed.

    1. Will Godfrey Silver badge
      FAIL

      Re: The reason...

      That's it!

      I'm going to take a patent out on the idea of a method of recognising duplicate files, notifying the user, and giving them the choice of which one to to use, based on erm... How about the age of the files?

      I'll call it the Advanced Service Selector.

    2. Anonymous Coward
      Anonymous Coward

      Re: The reason...

      Like I said in my post, input data caused the app to crash, due to poor coding.

  33. Sven Coenye
    Coat

    Couldn't they just fall back to using Apple maps?

    Other than crossing an Interstate on the way to the terminal...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like