back to article Boeing 787s must be turned off and on every 51 days to prevent 'misleading data' being shown to pilots

The US Federal Aviation Administration has ordered Boeing 787 operators to switch their aircraft off and on every 51 days to prevent what it called "several potentially catastrophic failure scenarios" – including the crashing of onboard network switches. The airworthiness directive, due to be enforced from later this month, …

  1. BJC

    Millisecond roll-over?

    So, what is the probability that the timing for these events is stored as milliseconds in a 32 bit structure?

    1. Yet Another Anonymous coward Silver badge

      Re: Millisecond roll-over?

      Exactly and yet internet experts will start blaming Boeing for improper testing when the real problem is the day being too long.

    2. Nigel Sedgwick

      Re: Millisecond roll-over?

      My first thought too, but that rolls over after 49.7 days.

      Still, they could have it wrong again.

      Best regards

      1. Simon Harris

        Re: Millisecond roll-over?

        The ratio difference between 49.7 and 51 days is suspiciously close to 1.024 though.

      2. Anonymous Coward
        Facepalm

        Re: Millisecond roll-over?

        I suspect that it is a millisecond roll over and someone at the FAA picked 51 days instead of 49.7 because they don't understand software any better than Boeing.

        1. Evil Auditor Silver badge

          Re: Millisecond roll-over?

          "...we don't really understand binary and stuff. So, at 49.7 days it may or may not be, but at 51 days it surely is fscked and needs a reboot."

    3. the spectacularly refined chap

      Re: Millisecond roll-over?

      Could well be something like that, the earlier 248 day issue is exactly the same duration that older Unix hands will recognise as the 'lbolt issue': a variable holding the number of clock ticks since boot overflows a signed 32 bit int after 248 days assuming clock ticks are at 100Hz as was usual back then and is still quite common.

      See e.g. here. The issue has been known about and the mitigation well documented for at least 30 years. Makes you wonder about the monkeys they have coding this stuff.

      1. J. Cook Silver badge
        Go

        Re: Millisecond roll-over?

        Makes you wonder about the monkeys they have coding this stuff.

        The kind that work for the lowest bidder, obviously.

        1. A.P. Veening Silver badge

          Re: Millisecond roll-over?

          The kind that work for the lowest bidder, obviously.

          Which just happens to have it outsourced to India.

    4. Someone Else Silver badge

      Re: Millisecond roll-over?

      Windows 95 redux.

      I'm shocked...shocked...to find Windows 95 going on in this airplane!

    5. bombastic bob Silver badge
      Devil

      Re: Millisecond roll-over?

      I've run into that problem (32-bit millisecond timer rollover issues) with microcontrollers, solved by doing the math correctly

      capturing the tick count

      if((uint32_t)(Ticker() - last_time) >= some_interval)

      and

      last_time=Ticker(); // for when it crosses the threshold

      [ alternately last_time += some_interval when you want it to be more accurate ]

      using a rollover time

      if((int32_t)(Ticker() - schedule_time) >= 0)

      and

      schedule_time += schedule_interval (for when it crosses the threshold)

      (this is how Linux kernel does its scheduled events, internally, as I recall, except it compares to jiffies which are 1/100 of a second if I remember correctly)

      (examples in C of course, the programming lingo of choice the gods!)

      do the math like this, should work as long as you use uint32_t data types for the 'Ticker()' function and for the 'scheduld_time'; or 'last_time' vars.

      If you are an IDIOT and don't do unsigned comparisons "similar to what I just demonstrated", you can predict uptime-related problems at about... 49.71 days [assuming milliseconds].

      I think i remember a 'millis()' or similarly named function in VxWorks. It's been over a decade since I've worked with it though. VxWorks itself was pretty robust back then, used in a lot of routers and other devices that "stay on all the time". So its track record is pretty good.

      So the most likely scenario is what you suggested - a millisecond timer rolling over (with a 32-bit var storing info) and causing bogus data to accumulate after 49.71 days, which doesn't (for some reason) TRULY manifest itself until about 51 days...

      Anyway, good catch.

  2. Nunyabiznes

    Windows Server 2000

    IIRC, we had to restart those at a minimum of every thirty something days or they would lock up. Fortunately<sarcasm font>, they tended to fall over quite a bit more frequently than that so we seldom ran into that particular bug.

    WTH is Boeing doing re-using that particular bit of crusty code?

    1. Yet Another Anonymous coward Silver badge

      Re: Windows Server 2000

      It was Windows95/98 and it took years in the wild before anybody noticed for that very reason.

      “ - and this is the rock-solid principle on which the whole of the Corporation's Galaxywide success is founded - their fundamental design flaws are completely hidden by their superficial design flaws.”

      1. Nunyabiznes

        Re: Windows Server 2000

        I could have sworn it was Server 2000 also. There was a lot of shared code there.

        Great quote, btw.

      2. Fred Dibnah

        Re: Windows Server 2000

        And NT4. I worked for a firm that had hundreds of PCs running it, and they were all rebooted every 42 days.

        1. Unoriginal Handle

          Re: Windows Server 2000

          "Reboot every 42 days" - so *that's* the answer!

        2. Tom Paine

          Re: Windows Server 2000

          I got 250+ day uptimes on a personal NT4 box quite a few times. With a 56Kbps dial-up modem, updates didn't seem quite as urgent back then.

        3. Trygve Henriksen

          Re: Windows Server 2000

          How did they get NT4 to run for 42 days before crashing?

          1. bombastic bob Silver badge
            Coat

            Re: Windows Server 2000

            maybe they had it working on the ultimate answer (or the ultimate question)

      3. Roland6 Silver badge

        Re: Windows Server 2000

        >It was Windows95/98

        Funny how my first thoughts were 49 days, that sounds like Windows 95; does that mean the 787's are running Windows for Avionics 95...

      4. chivo243 Silver badge
        Coat

        Re: Windows Server 2000

        Neither saw even close to 51 days of uptime... the "press ctl alt del twice" joke made sure of that!

      5. JassMan

        Re: Windows Server 2000

        The real reason no one noticed was because it was impossible to run any version of windows until XP for 49 days without getting the dreaded BSoD. Reboot and the timer starts again - end of rollover.

    2. chivo243 Silver badge

      Re: Windows Server 2000

      Bad habit passed down from NT4? We had a print server, Zeus, that needed a kick to get the print spooler working, can't verify the days in between. Nobody can print? Time to reboot Zeus...

      1. Version 1.0 Silver badge

        Re: Windows Server 2000

        I run FreeBSD, occasionally I reboot it ever year to two ... just to check the the machines power supply restarts.

        1. Anonymous Coward Silver badge

          Re: Windows Server 2000

          I'm more concerned about the bootloader and RAID controller than the power supply. I have spare power supplies but it's a bit trickier to have a spare bootloader to plug in (especially on a headless server)

        2. IGotOut Silver badge
          Happy

          Re: Windows Server 2000

          I run FreeBSD, occasionally I reboot it ever year to two ... just to check the the machines power supply restarts.

          We had a Nortel Meridian. Had to be rebooted after only 13 years uptime. Granted it did have a lightning strike which melted several of the boards (it was still running).

          Also had to reboot several Audiocodes ISDN to SIP converter after 5 1/2 years.

          You server boys do need to learn about reliability...

          1. Anonymous Coward
            Anonymous Coward

            Re: Windows Server 2000

            Those were the days before Nortel chucked their entire development systems and switched to "industry standard" WNT. (Of course, they had to do a boat-load of work to make WNT run as a subtask but that's another tale of woe.)

            But I am surprised about VxWorks. It is a very robust OS with a high EAL rating. Seeing the real problem -- and I doubt it is time-connected -- would be instructive.

            1. bombastic bob Silver badge
              Devil

              Re: Windows Server 2000

              "Seeing the real problem"

              someone already posted a valid suggestion - millisecond rollover, and an algorithm to test for periodic timing that was poorly written. [during rollover you might end up with a "storm" of data collection for a brief period of time, as one example, or NO DATA COLLECTED AT ALL - even worse]

              Again, my working with microcontrollers has already gotten me to discipline myself with respect to these kinds of maths so that the controller can run for MONTHS unattended, as you would expect it to, and not have a rollover issue after 49.71 days, or anything reasonbly close to that, depending on whether your millisecond timer is actually happening every 1.024 milliseconds...

        3. phuzz Silver badge

          Re: Windows Server 2000

          Uptime is just a measure of how long since you last verified that your machine could boot successfully ;)

          (Don't forget that a reboot doesn't give anything a chance to really stop. Problems are more likely to crop up after a machine has been powered off for more than a few minutes, and parts of it are cooling down).

          1. Fluffy Cactus

            Re: Windows Server 2000

            So, re Windows 95 on a Boeing 787 :

            If you are flying at the standard 30,000 feet, and you need to reboot, how long can the aircraft glide without any power? Because, somehow, if waiting the customary 2 minutes before rebooting, and then adding the time it takes to reboot, will the aircraft have crashed by then?

            As a potential airline customer, these issues are of medium importance to me, as other ways of kicking the bucket could get me first.

            1. jake Silver badge

              Re: Windows Server 2000

              Except it doesn't run Win95, it runs VxWorks.

              Did you bother to RTFA?

  3. Will Godfrey Silver badge
    Facepalm

    Am I surprised?

    Sadly no. I get the horrible feeling we are still just seeing the tip of the iceberg. Is there any part of Boeing that can be regarded as up to standard?

    1. Flocke Kroes Silver badge

      Re: Am I surprised?

      The boondoggle funding department is world class.

    2. Mage Silver badge
      Coat

      Re: Am I surprised?

      So I for one have cancelled all my orders for Boeings. Actually I don't seem to have any customers either.

      1. Fluffy Cactus

        Re: Am I surprised?

        This reminds me to ask another question: We are so used to the great name the Boeing used to have, that no one even wonders anymore if it ever was a good idea to name an aircraft manufacturer

        Boeing!

        I wouldn't do that. I would not name an aircraft company "Kaboom!" or "Oopsidaisy Aircraft".

        Not even the Russians have an airplane manufacturer called "Crashki-Burnski Planes-ky Factory"

        Sorry, Antonov, Tupolev or Sukhoi all do not translate into anything funny at all.

        Then there is Piper - US single engine aircraft - they seemed to have to pay the piper somewhere along the line. The Canadian company "Bombardier" at least gives you the feeling that you are going to be bombing someone else, which is only somewhat reassuring, because pesky SAM's might give you a little bump in mid-air. uuh, stop that!

        "Embraer" does not provide any feeling one way or the other, so is that a good thing? I don't know

        So, may be it's time that Boeing renames itself to "Majestic Aluma-plastic Happy Flying Machines", or maybe "Sitting Vulture Soaring Eagle Planes". Just trying to help them out here.

        Overall, the topic reminds me of Brian Eno's 1970/1980's song entitled, a bit sarcastically:

        "Burning Airlines give you so much more!"

        Here ends the reading!

        1. jake Silver badge

          Re: Am I surprised?

          I'm sure the Sons of Owen fully understand your concern. No doubt they are furiously inventing a time machine so they can go back and rename their family before William is born.

          Or not, as the case probably is.

        2. Marcelo Rodrigues
          Boffin

          Re: Am I surprised?

          ""Embraer" does not provide any feeling one way or the other, so is that a good thing? I don't know"

          Embraer is an acronym:

          EMpresa BRAsileira de AERonautica.

          Something like "Brazilian Company of Aeronautics"

        3. Crypto Monad Silver badge

          Re: Am I surprised?

          > I wouldn't do that. I would not name an aircraft company "Kaboom!" or "Oopsidaisy Aircraft".

          Or a parcel delivery company called "oops", I mean, "ups"?

          I once saw a car from a driving school called Impact School of Motoring. It had a large dent in the back. (I promise it's true, but I wish I had taken a photo)

        4. tfewster
          Facepalm

          Re: Am I surprised?

          @Fluffy Cactus: You're even closer than you realise - Boeing is the Anglicised version of William Boeings German fathers name, Böing

    3. SW10
      Mushroom

      Re: Am I surprised?

      I mean, you don’t get this with Voyager, or even Mars Express. So it’s do-able...

    4. macjules

      Re: Am I surprised?

      My opinion also. "switch their aircraft off and on every 51 days". I would have thought that was reason enough to ground the entire fleet until the problem was clearly identified and a software patch released.

      1. Chris G

        Re: Am I surprised?

        In the article, it mentions that some years back superceded backup flight plans could kick in mid flight and the aircraft would try to change course to the old plan.

        I was wondering if that could have been a part of the problem that led to the Boeing 777 on flight 370 going missing?

        I think Boeing have for years been suffering from 'Too big to fail syndrome' , the US gov' is kind of obliged to bail them out and keep them running so they don't try that hard to produce a good product.

        1. Stoneshop

          Re: Am I surprised?

          I was wondering if that could have been a part of the problem that led to the Boeing 777 on flight 370 going missing?

          I doubt it, because a) while also from Boeing, it was a different model and this FAA notification concerns only the 787, and b) a previous flightpath from Malaysia towards the middle of the Southern Indian Ocean would be quite unlikely.

      2. phuzz Silver badge

        Re: Am I surprised?

        Well I guess that the silver lining in all of this is that most aircraft are getting grounded and probably switched off at the moment. There's literally only ten aircraft operating at Heathrow right now (as per flightradar).

        One of them is a 787 though.

      3. John Brown (no body) Silver badge

        Re: Am I surprised?

        "ground the entire fleet until the problem was clearly identified and a software patch released."

        At this point in time, most of them probably are on the ground and not likely to be going anywhere soon.

      4. MachDiamond Silver badge

        Re: Am I surprised?

        "I would have thought that was reason enough to ground the entire fleet until the problem was clearly identified and a software patch released."

        Most aircraft are power cycled in less than 48 days so the bug took some time to track down. It's often the first thing to try if you are having system issues so it could be any number of things. Granted, aircraft need to be far more robust, but every complex system has some sort of issue.

        I think it might have been Matt Parker that talked about this issue in a presentation on math. Of course, he would have pointed out why it's a good idea to have mathematicians around checking these sorts or things.

  4. Steve K

    If it's Boeing...

    If it's Boeing...wrong data's showing.

    (Yes I know that Airbus have patched somethng similar...)

    1. Hans 1
      Pint

      Re: If it's Boeing...

      Boeing 787s must be turned off [...] to prevent 'misleading data' being shown to pilots

      #TFTFER

  5. Pascal Monett Silver badge
    Trollface

    Turning it off and on

    So, Boeing is using Windows in its planes now ?

    Run for the hills !

    1. Nunyabiznes
      Joke

      Re: Turning it off and on

      Um, that's what they are going to run into. I'd run for the flats!

      1. ClockworkOwl
        Coat

        Re: Turning it off and on

        Um, are Boeing responsible for any mining equipment software???

        NCB donkey jacket>

        1. Anonymous Coward
          Anonymous Coward

          Re: Turning it off and on

          I can turn my Boeing on and off but it still doesn't let me change the homegroup settings, I guess that I'm just running regular the regular Windows, not the premium version.

    2. fidodogbreath
      Coat

      Re: Turning it off and on

      So, Boeing is using Windows in its planes now ?

      Some airlines charge extra for Windows seats.

      Mine's the one in the plastic bin by the X-ray machine.

      1. Hans 1

        Re: Turning it off and on

        Mine's the one in the plastic bin by the X-ray machine.

        I want you to x-plane that one!

    3. Danny Boyd

      Re: Turning it off and on

      According to the article, the OS used is Wind River VxWorks realtime OS. Where did you find any mention of Windows? In other people's comments?

      1. Anonymous Coward
        Facepalm

        Re: Turning it off and on

        Danny, it's called humour.

  6. alain williams Silver badge

    How long does it take to reboot a 787 ?

    I imagine that it is longer than the 30 odd seconds it takes to reboot my Linux box.

    1. Anonymous Coward
      Anonymous Coward

      Re: How long does it take to reboot a 787 ?

      And? Your point is what exactly? Genuinely interested.

      1. alain williams Silver badge

        Re: How long does it take to reboot a 787 ?

        My point ? None at all - just a question.

        Yes: I provide a reference comparison - is that a problem ?

        1. Hans 1
          Boffin

          Re: How long does it take to reboot a 787 ?

          My Devuan system reboots in 12 seconds, from desktop to desktop, no fast boot cheating.

          Not that I need to reboot, ever, even when I switch kernel ... Windows needs to reboot when it wants to update notepad.exe.

      2. Antron Argaiv Silver badge

        Re: How long does it take to reboot a 787 ?

        Its all fun and games until you need to find a 396 cell to replace the one that backs up the NVRAM.

        1. jelabarre59

          Re: How long does it take to reboot a 787 ?

          Could be worse, could be one of those rechargeable Varta barrel batteries permanently soldered to the mainboard (popular in the late 80's and early 90's). They don't explode or die, they just leak and erode the traces on the board.

    2. Nunyabiznes

      Re: How long does it take to reboot a 787 ?

      Hopefully less time than it takes to hit the ground.

    3. Dave 129

      Re: How long does it take to reboot a 787 ?

      The 787 I was on that needed a reboot to try and fix a locked fueling valve needed around 15 minutes to fully power cycle and have systems back up.

      In the end I think they hit the valve with a wrench... you know what they should've done first :D

    4. Kevin McMurtrie Silver badge

      Re: How long does it take to reboot a 787 ?

      You press a toothpick into a hole under the cockpit dash, wait for everything to blink, and it's good to go when all the lights are back on.

    5. MrNigel

      Re: How long does it take to reboot a 787 ?

      When Emirates introduced the first A380's I was on a flight where self-loading cargo was on-board, doors shut then Captain announced "Sorry Ladies and Gentlemen we are going to have to reboot the aircraft". This was after ground power was disconnected so no air-con. I can tell you that it takes a sweaty 18-20 mins before engine start.

    6. DugEBug

      Re: How long does it take to reboot a 787 ?

      First, you have to switch SCE to AUX.

      1. MachDiamond Silver badge

        Re: How long does it take to reboot a 787 ?

        "First, you have to switch SCE to AUX."

        Nope, that's for a Saturn V. The good thing is the engines keep running during the reboot.

    7. bombastic bob Silver badge
      Devil

      Re: How long does it take to reboot a 787 ?

      typically VxWorks will come up really fast.

      a) you compile it for your hardware - so no driver loading and/or hardware detection

      b) it's an RTOS and not a monolithic kernel. Startup and scheduling are different. You could easily optimize restart times [let's say in-flight reboots being made possible].

      c) the processes would all be compiled in, so no program loads either, as far as I can tell. This could be wrong, based on what they might be doing, but I suspect it'll be like it was for wifi routers with VxWorks, which is what I worked on - wireless, networking, WPA, asynchronous packet handling, stuff like that.

      So yeah maybe it boots up in under 5 seconds? Possibly boots up even faster than THAT...

    8. serendipity

      Re: How long does it take to reboot a 787 ?

      In the FAA notice it says the work will take an hour...

  7. Anonymous Coward
    Anonymous Coward

    Ye Gods

  8. Anonymous Coward
    Flame

    In the current world situation I imagine it won't be long before a goodly percentage of 787s are simply powered down somewhere out the way and left until there are lemon-soaked paper napkins again.

    Of course, it might give RR a chance to catch up with engine rebuilds that have left some aircraft on the ground for a fair while in any case.

    If it's Boeing, I'm not going.

    1. Zippy´s Sausage Factory

      "If it's Boeing, I'm not going."

      I think you found my new philosophy of flying...

    2. John Miles

      until there are lemon-soaked paper napkins again

      now if they only ordered a reboot every 42 days

    3. Anonymous Coward
      Anonymous Coward

      "lemon-soaked paper napkins again."

      Ah ah, made my day this one. I may be french but I know my Douglas Addams stuff.

      Great one.

  9. Dwarf

    Designers needed

    Sounds like a number of designers are needed that come from this century and can resolve the endless looking list of shortcuts and issues that seem to be have been designed into this steaming pile of poo.

    Why they are not thinking a bit more long-term in either handling the rollover issue better (more bits to make it a far longer duration), or better still design in the fact that things will roll and expecting that in the platform design and software so that it does work properly. .

    Alternately, work around with a rolling reboots until better software and firmware can be deployed. The whole idea of "turn it off and on again" is so dated now.

    1. Mike 137 Silver badge

      "Sounds like a number of designers are needed ..."

      What, they have designers of any kind in their software teams? I thought it was all about coding these days.

      Agile, as widely implemented, steps directly from concept to implementation, skipping the design stage entirely.

      1. EVP

        Re: "Sounds like a number of designers are needed ..."

        ”Everyone can code”, you know... and design so yesterday.

      2. DiViDeD

        Re: "Sounds like a number of designers are needed ..."

        Ah yes. Agile!

        Thanks to Agile Development, half our internal websites are "experimental" or "beta".

        Basically, that means "broken until the sprint after next. Or maybe the one after that, who knows?"

        1. Down not across
          Mushroom

          Re: "Sounds like a number of designers are needed ..."

          Ah yes. Agile!

          I'm not convinced "Fail fast, fail often" is entirely appropriate for aircraft software.

    2. Wellyboot Silver badge

      Re: Designers needed

      >>>"turn it off and on again" is so dated now.<<< Please inform Microsoft, I think they lost the memo.

      Bring back the 747 I'll pay the extra pennies per customer mile for an aircraft that works as designed.

      1. Anonymous Coward
        Anonymous Coward

        Re: Designers needed

        But can they still get the TTL ICs needed when stuff actually wears out?

  10. Anonymous Coward
    Anonymous Coward

    Simples

    Just switch off the airplane for ten minutes for a proper power drain and power it on again. Best practice is to avoid contact with the ground in the meantime.

    1. Anonymous Coward
      Boffin

      Re: Simples

      Then how do you ensure there's no damage due to ESD? Computer equipment should be properly grounded during a reboot cycle.

  11. Just a geek

    And every 248 days for this bug -> https://www.slashgear.com/faa-boeing-787s-need-to-be-rebooted-every-248-days-uptime-04381899/

    The 787 is mess

    1. Sandtitz Silver badge

      Well, duh, it's mentioned in the article.

  12. HCV

    I remember they were very excited to announce that to mitigate problems like this, Windows now included a feature where you could schedule an automatic reboot.

    Oh goody! You've invented cron!

  13. Mark 85

    Circuit Breakers?

    I would hope that the control system in question has circuit breakers. If so, "popping" the breaker on the ground should reset it.

    1. MachDiamond Silver badge

      Re: Circuit Breakers?

      " If so, "popping" the breaker on the ground should reset it."

      If you have to do a reset in flight, yes, you hope that will do it. If you want to do it right, it should be done on the ground and completely so everything in the sequence comes up the way it was designed.

  14. jake Silver badge

    A point of order seems to need clarifying.

    VxWorks is neither a Windows nor a UNIX. It is also a RTOS. The commentards above who obviously have no experience with this OS and yet are attempting to appear knowledgeable on the subject are painfully obvious ...

    "It is better to remain silent at the risk of being thought a fool, than to talk and remove all doubt of it." —Maurice Switzer, 1907

    1. Hubert Cumberdale

      Re: A point of order seems to need clarifying.

      Maybe you should listen to your own advice.

    2. Frugelhorn

      Re: A point of order seems to need clarifying.

      No. Plenty of experience developing for VxWorks and exactly the same issues apply with int32/time rollover, etc. Moreover, many products are using very old, heavily patched versions of the OS because it's considered too risky/expensive to migrate. Especially in aerospace. And the fact that it's only used in dedicated applications in relatively small volumes, means that many bugs remain undiscovered and unfixed for years. It may be a more deterministic OS, but that only gets you so far.

      1. Yet Another Anonymous coward Silver badge

        Re: A point of order seems to need clarifying.

        Which is why you probably shouldn't fix this 'bug'

        Simply require it to be powered down every 28days as part of the maintenance procedure.

        I'm sure the engines need oil replacing every X 100 hours, nobody is demanding that the plane contains enough oil for a 50years service life.

      2. Anonymous Coward
        Anonymous Coward

        Re: A point of order seems to need clarifying.

        I haven’t got that much experience with the OS. My experience is, though, that the development environment was a steaming pile of bits. If a coder’s attention goes into fighting with tools, quality of code will suffer for sure.

        That was many years ago, maybe it works better now.

        1. jake Silver badge

          Re: A point of order seems to need clarifying.

          To be fair, "many years ago" most so-called development environments were steaming pile of bits.

      3. JassMan

        Re: A point of order seems to need clarifying.

        Just as well its not Windoze otherwise it would bring a whole new meaning to Blue Screen of Death.

        1. A.P. Veening Silver badge

          Re: A point of order seems to need clarifying.

          You mean a Blue Scream of Death?

      4. jake Silver badge

        Re: A point of order seems to need clarifying.

        Did you read mine for content, Frugelhorn? Did it apply to you?

      5. Denarius
        FAIL

        Re: A point of order seems to need clarifying.

        VXWorks ? You mean the OS used in Mars probes and landers that works for years on chips which are radiation hardened variants of PowerPCs originally ? That just works for years a long way from tech support ? Better coders in space work than mere aviation. Seriously, if this is been a known issue for 30+ years, why does not basic code testing get the stufup in basic acceptance testing ? A whole bunch at Boeing need to to sacked and banned from ever going near aircraft, or any other job requiring coding.

    3. Anonymous Coward
      Anonymous Coward

      Re: A point of order seems to need clarifying.

      Of course Jake. Because you have 12 airline pilot friends who flew Boeing 787 Max who all said there was no issue with that plane and Boeing had given plenty of training and it was Pilots form third world countries with insufficient training that caused the issue (after the second crash)

      And yet pretty much every pilot disagreed with that statement, including ones from most of the major US carriers and the unions and the FAA and the rest of the world and Boeing themselves.

      So I would suggest that advice in the flying arena from yourself and you 12 Pilot friends is not worth the screen real-estate it is written on.

      1. jake Silver badge

        Re: A point of order seems to need clarifying.

        It was the 737MAX, there is no 787MAX. Am I supposed to listen to, or reply, to someone who made such a basic error?

        I did not say there was no issue with the plane. I said that properly trained pilots knew of the issue, and the work around. Am I supposed to listen to, or reply to, a coward who makes such egregious logic errors?

        Consider that the day before the Lion Air Flight 610 crash, the exact same plane was kept from crashing by a third, off-duty pilot who happened to be in the cockpit when the exact same problem that brought the plane down the following day occurred. That's right, he stopped the plane from crashing. As could the pilots who were onboard the next day, if they had had the proper training, which clearly existed.

        The fact that this information and training wasn't available to the pilots of Ethiopian Airlines Flight 302 over five months later is criminal, and IMO that airline should be at least partially, if not wholly, responsible. Blaming it all on Boeing is akin to blaming the loss of a team sporting event on a single play by a single player. It says more about the lobbying power of the airlines than it does Boeing's issues (which do exist, and I'm not saying otherwise).

        1. Anonymous Coward
          Anonymous Coward

          Re: A point of order seems to need clarifying.

          Yes of course Jake. That's why the plane has spent over a year being unable to fly. Just because they needed better training.

          Have you not read anything of the scandal and issues that have arisen since the investigation has begun. With pilot instructors saying the system was faulty and could easily lead to a crash. With people who have followed the proper technique with *the real software* in a simulator not being able to control the plane.

          With Boeing having to cancel orders, having to find parking lots to try to find places while they have issue after issue with their software. With all the evidence that has been produced that Boeing purposefully did not produce correct simulator training as they didn't want to have to force retrains on pilots to re-certify (i.e. The proper training didn't exist and Boeing even tried their best to make sure people didn't think they needed proper training).

          And frankly your claim that we shouldn't be blaming it all on 'Boeing' is disingenuous to those that died unnecessarily. This was a problem of Boeing's making and for money and profit they put the lives of passengers at risk to rush a plane out and ensure they didn't lose customers. Engineers at Boeing have testified to that.

          "The fact that this information and training wasn't available to the pilots of Ethiopian Airlines Flight 302 over five months later is criminal". Yes and Boeing execs should go to court for that criminal act. If they had owned up to the issue, grounded the planes, changed the software and then recommended re-certification in a simulator with *real software* then it wouldn't have happened. It is criminal, e can agree and Boeing should be suffering greater consequences.

        2. anothercynic Silver badge

          Re: A point of order seems to need clarifying.

          Actually... It is a collective failure by the FAA and Boeing (and EASA subsequently). The FAA said "ok, we'll believe Boeing when they say 'we don't need to mention MCAS'" and EASA followed their lead. The Brazilian authorities on the other hand didn't and insisted that MCAS be mentioned.

          Ethiopian is a more problematic case than LionAir because it happened shortly after takeoff at a much higher altitude above sea level (2355m). A minute after takeoff MCAS kicked in, which is a damn sight less than the 13 minutes that the LionAir crew had before theirs kicked in. I wouldn't want to have to resolve a software issue at that point (and Chesley Sullenberger also pointed this out when he attempted this in a simulator). Ethiopian policy to make a pilot with 200 flight hours an FO is something else, but arguably that airline is one of the safest in African airspace (along with Kenyan, SAA and Egyptair).

          Boeing screwed the pooch, the aviation industry is broadly in agreement with that view. That this new issue comes to light is not really all that much of a surprise given that Airbus has made the same mistake (see https://www.theregister.co.uk/2019/07/25/a350_power_cycle_software_bug_149_hours/), but then again this is the *second* of such issues with Boeing (see https://www.theregister.co.uk/2015/05/01/787_software_bug_can_shut_down_planes_generators/ for the first). Boeing needs a lot of internal work to resolve the problem of ruling by accountant, whereas it *was* an engineering company first.

          Just saying...

      2. anothercynic Silver badge

        Re: A point of order seems to need clarifying.

        Sorry to point this out but Jake is right. Throwing an 'anonymous' strop doesn't particularly help your cause. And just in case you are wondering about my aviation expertise... you're welcome to look at past posts :-)

        You're *very* welcome.

    4. M. Poolman

      Re: A point of order seems to need clarifying.

      42 days without a reboot results in accumulated stale data causing serious malfunctions in humour detection sensors.

  15. fidodogbreath

    (Fr)agile avionics development

    Performed by the lowest bidder, of course.

    1. A.P. Veening Silver badge

      Re: (Fr)agile avionics development

      And outsourced to India, just as of course.

  16. Mick Russom

    software outsourced to china and india by bean counting SHYSTERS at boeing.

    1. Yet Another Anonymous coward Silver badge

      The avionics for the 787 were built by Rockwell Collins in Cedar Rapids, Iowa

      1. A.P. Veening Silver badge

        The avionics for the 787 were built by Rockwell Collins in Cedar Rapids, Iowa

        It may say so on the box, but what do you want to bet it wasn't outsourced to India? Your life?

        1. jake Silver badge

          Quite frankly, A.P. Veening ...

          I'd much rather bet my life flying in ANY commercial aircraft (yes, including the 737-MAX) than being transported in an over-the-road vehicle driven by yourself, or pretty much any other licensed driver, on public roads. Don't take this personally. It's a math(s) thing.

          Get some perspective, people. Paranoia is all very well and good, but there comes a point where you just look silly.

          1. A.P. Veening Silver badge

            Re: Quite frankly, A.P. Veening ...

            If you are honest enough to include yourself into that "any other licensed driver" category, I can't disagree with you, though I am a bit of a control freak, so I like to do my own steering, whether it be a bike, a motor bike, a boat, a car or a plane. I must confess I am not licensed to do the steering in a plane though, so by necessity I must leave that to others.

            1. jake Silver badge

              Re: Quite frankly, A.P. Veening ...

              Of course I include myself! Furrfu.

              Concur on preferring to be at the controls ... but I can sleep in the Peterbilt when my Wife or Daughter are taking their turn driving cross-country. Not cat-napping, real sleep. It's a trust thing. I also don't mind somebody else taking the controls of the aircraft when we're flying straight & level. Gives me a break. (I admit that I doubt I'd be quite as blasé about this if they weren't dual yoke ... )

          2. anonymous boring coward Silver badge

            Re: Quite frankly, A.P. Veening ...

            Jake, have you done the maths on the MAX?

            Dead people / Total flown people

            will do. I don't have the numbers, but you seem to.

            1. jake Silver badge

              Re: Quite frankly, A.P. Veening ...

              "Dead people / Total flown people"

              No. But then that number by itself is useless. You need miles and/or hours in that figure to take any real meaning from it. I don't have those numbers handy, either.

              However, the MAX flew around half a million flights and only had two fatal crashes. Both of those crashes were avoidable (see my comments elsewhere). In my mind, the MAX could still be flown today, with properly trained pilots at the controls. It's not an inherently unsafe aircraft.

              The court of public opinion says otherwise. ::shrugs::

      2. ricardian

        I was quite alarmed to discover that the guidance system for the nuclear-tipped Thor missiles in the 1960s was designed and built by the AC Spark Plug Company

      3. TheMeerkat

        I remember seen job ads from Boeing for jobs in India mentioning avionics. Next to job ads for Scrum Masters for the same Indian office.

        I obviously could not check.

    2. Updraft102

      Hiring lawyers to do accounting is too expensive, even if they are disreputable.

  17. elgarak1

    The 787 has far more problems. They left metal shavings (from drill holes and tightening bolts) inside where they well. Which may be inside of cable conduits, where they erode away the insulation...

    Normal rules require such debris to me removed, but it was foregone at this one facility for keeping deadlines and cost requirements. There was a quality inspection executive who tried to report it up, and subsequently became whistleblower. He considered it bad enough to advice his family to never fly in one.

    1. Chris Fox
      Happy

      Debris well

      Looks like some of the debris must have found its way into your keyboard.

    2. Screwed

      Which is why it is better for all if people speak up even at the cost of temporarily being thought a fool. Culture which encourages openness is much to be desired.

      "It is better to remain silent at the risk of being thought a fool, than to talk and remove all doubt of it." —Maurice Switzer, 1907

    3. anonymous boring coward Silver badge

      "The 787 has far more problems. They left metal shavings (from drill holes and tightening bolts) inside where they well. "

      Jeeesssusss Chriiiiist! FFS!

      1. Yet Another Anonymous coward Silver badge

        Only in the parts where they moved production to Charleston S.C, the parts made by commie union labor in Washington are fine.

        ... checks... my BMW was made in the former east Germany, not in S. Carolina.... relieved.

        1. Anonymous Coward
          Anonymous Coward

          For their size, MZ (in East Germany) made pretty good motorcycles. Certainly more reliable than the products of Hardly Davidson in the same era (as complained to me by an HD dealer in South London.)

  18. Anonymous Coward
    Anonymous Coward

    I'm surprised it's not done every flight as standard. You'd think that allowing systems to recover from dubious states would be normal practice.

    1. A.P. Veening Silver badge

      That would probably be a bit much for short haul planes with multiple cycles everyday, but for those I would expect that to be a daily (nightly) occurrence.

      1. John Brown (no body) Silver badge

        Do 7878s do short haul?

        Having said that, adding in a 15-20 minute reboot time at each turnaround might not be an option. It depends on whether it can be done while other turnaround tasks are also being done, otherwise I doubt they'd want to do it except as required. Aircraft have to be turned around on time or risk losing their slot.

        1. A.P. Veening Silver badge

          I don't know if 787s do short haul, but I do know there were 747s doing domestic flights in Japan, which are pretty much by any definition short haul flights, so I won't say it is impossible.

    2. ICL1900-G3

      Well, I used to turn mine off after every flight, but it was a Robin DR400, so probably not quite so complex.

  19. -tim
    Flame

    Out of the frying pan into the fire?

    VxWorks doesn't like hard resets.

    I hope there is a nice easy clean reset option.

    1. STOP_FORTH

      Re: Out of the frying pan into the fire?

      VxWorks doesn't like clock rollovers either. Why have they never fixed this or does it only affect older versions?

      (Have personal experience of a horrible 248 day bug, luckily it was only crashing broadcast systems.)

  20. RoHa

    Maybe try turning it on for eight seconds, off for two seconds, on for eight seconds, off for two seconds ...

  21. Updraft102

    "Solving the problem is simple:"

    Simple to describe, surely. Go fix the bug. That's how you solve it. Turning it off and on again is not fixing the problem... it's mitigating it, temporarily. Surely the readership here understands the difference!

    1. Mark192

      Probably all sorts of expensive consequences to actually fixing it with an update while turning it off an on again can be built into the maintenance for free.

      Not the right solution but when shareholders/your bonus package demands ever higher profits you've got to cut corners somewhe... everywhere.

  22. Stuart Halliday
    Facepalm

    Ask and you shall receive

    So, next time I'm on one of these planes, I've to ask when the plane was last turned off?

    FFS.

    1. A.P. Veening Silver badge

      Re: Ask and you shall receive

      More importantly when the plane was last turned on, there may be a prolonged period between turning it off and turning it on again, especially with all those planes being grounded while this Corona crisis is going on.

  23. Arachnoid
    Mushroom

    So.......

    How many would fly on a 747 that knowingly has not been rebooted for 50 days?

    1. Wellyboot Silver badge

      Re: So.......

      Most 747s are from a time when keeping the avionics warm improved reliability and the flight engineer (remember those) knew which breaker to flip if need be.

      The last (747-8) model shares much with the 787 so it may have similar issues.

  24. 2Fat2Bald

    I think it depends on what you mean by "rebooted". Are these systems that stay live in the aircraft when it isn't in use? - if they are then there might be a problem. If they shutdown when the aircraft isn't in use then they're going to get a reboot pretty regularly and I doubt they'd ever get to the end of the period.

    Anyway it isn't hard to fix. All sorts of things need to be checked before/after a flight. So you just add it to that checklist. "Has the aircraft been rebooted in the last 40 days?". But that's so straightforward I reckon they'd have thought of it already.

    Alternatively. Build it into the aircraft's own software to reboot (or refuse to operate without one!) automatically every so often. Again, seems too easy..

    So I reckon there has to be more to this one.

    1. PerlyKing
      Facepalm

      Re: Build it into the aircraft's own software to reboot automatically every so often

      This is bad enough when Windows reboots in the middle of your pr0n Powerpoint session - picture the pilots' faces when the aircraft decides to reboot on final approach!

  25. Anonymous Coward
    Anonymous Coward

    What a Joke

    So much for software testing. sprint-ing to disaster. Why hasn't the FAA shut this down.

    1. Jimmy2Cows Silver badge

      Re: What a Joke

      Regulatory capture.

    2. Anonymous Coward
      Anonymous Coward

      Re: What a Joke

      The FAA was trumped!

  26. Anonymous Coward
    Anonymous Coward

    What a test

    I was told by a test engineer, from a major aircraft manufacturer, that one of their tests was to remove the main command and control network switch in-flight and replace it with a spare. I know that they had performed it many times on the ground, but they must have bigger dangly bits than me, to have done it in-flight!

    [Anon. because, although it was a long time ago, it was my (not ARINC 664) software on that particular switch!]

    1. anonymous boring coward Silver badge

      Re: What a test

      "[Anon. because, although it was a long time ago, it was my (not ARINC 664) software on that particular switch!]"

      So not ARINC 664? You must be that other guy then?

  27. Jim Whitaker

    Date check

    Own up, who else checked the date of this piece?

  28. flayman Bronze badge

    As it stands, I think we can probably power off most of these birds and leave them that way.

  29. TeeCee Gold badge

    Here you go Boeing.

    New corporate anthem for you.

    1. A.P. Veening Silver badge

      Re: Here you go Boeing.

      I thought it was this one.

  30. stevejs

    Old-school habits.

    It wouldn't take much effort to write a post-it note reminder and stick it on the dash reminding all future pilots of the date requiring a reboot.

    Problem Solved!

  31. Anonymous Coward
    Anonymous Coward

    "This is your captain speaking. Nothing to be alarmed about... but you may notice things go a little quiet as I reboot the aircraft"

    1. A.P. Veening Silver badge

      I'd say that would be a bit alarming, I still clearly remember hearing something similar about a train rebooting (multiple times, luckily no more than once in a single journey, so max two times a day when commuting) and that took about 15 minutes every time with the train standing still. Usually it happened about 30 seconds prior to departure, but it also happened between stations and every single time the train came to a full stop (security reasons they claimed) before the train was rebooted.

  32. anonymous boring coward Silver badge

    So you should do it every 10 days or so then, to make sure one or two missed ones doesn't cause the issue.

  33. one crazy media

    Boeng board needs fire all executive staff immediately. Then, they should clean house and replace their software engineering team. They are not qualified to develop a website, much less aeronautical navigation systemss.

  34. Fursty Ferret

    From a cold start it takes about 15 minutes to get a 787 up and running. Most of that time, to be fair, is waiting for the inertial reference system to align. The common computing resource (which runs in software what would traditionally be handled by individual avionics computers) is online within 3 minutes. If the CCR is reset in flight (never needed to do so yet...) it's back up and running within 70 seconds. It is permitted to reset both CCRs (left and right) simultaneously in the event of the loss of all displays.

    The bulletin for this particular problem is quite woolly - I think that when it says "expired" data it means the results of a calculation that didn't complete in the assigned compute cycle (so realistically milliseconds late); not, say, the values from last Tuesday. Merely a layman on RTOS, was never touched on in my Comp Sci degree.

  35. christooo

    Something tells me you have the wrong date on that article!

  36. ecofeco Silver badge

    Are you effing kidding me?!

    Oh FFS.

  37. serendipity

    The next time some article blithely claims that self driving cars will be much safer for us, just remember this article!

    No such thing as a complex system without bugs!

  38. VTAMguy

    Bail out this company for sure

    Yet another design from Boeing which can kill people. We should definitely be bailing them out with taxpayer money. Definitely. They try so hard and they've been so open and transparent about everything.

  39. MrWordNerd

    Common Core System

    I’m no software engineer, but does anyone know whether the 787’s common core system (CCS) programming was done using Common Core Math standards and principles? If it was, that could explain its messed up system. ;)

  40. ITACS LTD

    Gareth...

    writing afterwards is easy, but still required.

    the problem of recoverable software and hardware for safety critical systems were solved in 1978-87 and were subject of my PhD applied for Russian submarines, satellites and later Sukhoy 27, 27I and C (India and China respectively). Later in 94-95 I was preaching the same for British Aerospace - having no aircrafts in design or decision making people in UK in 98-99 I was pursuing the same and further development in Seattle for both: military and commercial aviation dept of Boeing - by invitation from there and full support of US Government.

    Later we did project for EC called ONBASS was done in full swing 2004-2009.

    Russian government is still trying to steal (as far as they can understand) how to make next steps - see patent and patent defence case.

    We have summarised our experience in four books published by Springer.

    Rod Liddle in 2015 did interview for Sun

    Some notes about this saga you can find below. Dan and Andy from WSJ, as well as their predecessor Jeff Cole are fully aware about this all, see attachment in email I sent you...

    Unfortunately - instead of REDOING avionics as it should be people elsewhere talk about functional safety, ( kind of bull shit, safety is only active and preventive, nothing else matter). Also popular nonsense called conditional maintenance - even NASA (like they now how to measure conditions of aircraft, on board system and environment - ( at order of 2 at power 24 states) - nobody knows how to combine and, of cause - verification of a system - for Intel new processor - hardware code if 1.2mln lines verification program exceed 1.5 billion lines - read thousand times more errors in there...

    Well enjoy reading, if you want to implement the solution that is proven - also by patent a the best one for flight mode, tracing and recovering in real time system on board - do ask, but first read our books. The answers are there, excited and proven.

    rgds,

    Igor

    Notes on Active safety of aircrafts and people

    in 1985-89 the first concept of dynamic safety for aviation (CoDySA) was introduced by ATLAB Ltd, Bristol, now IT-ACS LTD, also UK.

    Fields tests of prototype took place and passed STATE TESTS see this:

    https://www.academia.edu/30247663/ITACS_LTD_Devices_and_Results_Chronology

    in 98-99 specially for Boeing, .Lockheed And Northrop results were presented:

    https://www.academia.edu/7119860/The_Concept_of_Dynamic_Safety

    and EXPLAINED for CA and Military sectors of Boeing in Seattle and Orlando.

    Special Talk how to aggregate on-board and Air traffic controllers info was suggested and explained

    https://www.academia.edu/7126686/Safelets_a_Software_Support_for_Dynamic_Safety_System

    a special project ONBASS was funded by EC within FP5 to make it implemented for general and commercial aviation.

    results of this project we presented for Eurocontrol, Airbus, and EASA,

    sent to Boeing and Eurocontrol, as well as German, Swiss, Russian, French governments.

    https://www.academia.edu/40602498/Principle_of_Active_System_Safety_-_Airbus_HQ_2006

    during demonstration of functioning the whole software framework with prototyped devices were presented 18 November 2008 in London.

    there books published explaining all hardware, systems, software and active system control designs:

    https://www.academia.edu/28829113/Resilient_Computer_System_Design

    https://www.academia.edu/39740322/Software_Design_for_Resilient_Computer_Systems_SpringerNature

    Interview for Times, thanks to Rod Liddle and Bianca Britton:

    https://www.academia.edu/18111652/Rod_Got_An_Issue_Where_Am_I_Safe_to_Fly

    and patent was made in UK and stolen by Russian Government:

    https://www.academia.edu/31065112/Patent_on_method_of_active_system_safety

    https://www.academia.edu/31822260/Method_and_apparatus_for_active_system_safety

    https://www.academia.edu/30247663/ITACS_LTD_Devices_and_Results_Chronology

    https://www.academia.edu/36980437/Patent_breach_failed_https_www.ipo.gov.uk_p-challenge-decision-results_o35518.pdf

    defended in Europe

    https://www.academia.edu/38316652/UK_Patent_defended_in_Europe

    and ... Nothing was implemented - instead of two lamps on a panel of 737 indicating loss of oxygen pressure?

    Now world is doing nothing again, and only few super experts and enthusiasts of aviation ( CAPTIO team) are pushing investigation of MH370 getting the grip of what was that and what to do to avoid in the future.

    http://mh370-captio.net

    Codysa etc

    Money on safety - you have it! but you need to learn how to get it:

    https://www.academia.edu/35896810/110218.pdf

    The book Russian Government (Rospatent) tried to endorse and even re-patent:

    https://www.academia.edu/28342759/Active_System_Control

    CAPTIO work the latest video is here:

    https://www.youtube.com/watch?v=Go3K0UUt2Us

    ITACS Follow-up

    https://www.academia.edu/40899039/Mh370_follow-ups_for_RAES_Brussels_event

    and if we do nothing today, tomorrow will be exactly the same as yesterday. Happy flights. And good luck! You need very good one!.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like