Airbus A350 software bug forces airlines to turn planes off and on every 149 hours

Some models of Airbus A350 airliners still need to be hard rebooted after exactly 149 hours, despite warnings from the EU Aviation Safety Agency (EASA) first issued two years ago. In a mandatory airworthiness directive (AD) reissued earlier this week, EASA urged operators to turn their A350s off and on again to prevent " …

[Tromos]

"...need to be hard rebooted after exactly 149 hours"

Not good if the 149 hours is up while on final approach. I'm pretty sure you wouldn't have to wait for the full 149 hours otherwise this would cause MAX sized problems with scheduling.

[big_D]

Re: "...need to be hard rebooted after exactly 149 hours"

You look at the maintenance schedule, calculate in the next flight (plus delay/headwind/circling time) and if it will be near or exceed 149 hours, shut everything down.

[Yet Another Anonymous coward]

Re: "...need to be hard rebooted after exactly 149 hours"

It should reboot before 'Exactly' 149 hours.

I'm not sure I would fix the bug. This is a known feature, the planes all go through a maintenance schedule more often than this. It may be safer to make a reboot part of the maintenance rather than introducing a software change

[ps2os2]

Re: "...need to be hard rebooted after exactly 149 hours"

So, let me get this straight. How many software engineers are going to be needed to accomplish this? 10? 500? This sound like a great job place a software engineer in every first-class cabin and let the soak up the alcohol till they are snookered. So then they can reboot while under the control of alcohol.

[a handle]

Re: "...need to be hard rebooted after exactly 149 hours"

I'm in :-)

[christooo]

Re: "...need to be hard rebooted after exactly 149 hours"

Good thinking Berger!

['s water music]

Re: "...need to be hard rebooted after exactly 149 hours"

Not good if the 149 hours is up while on final approach

yeah, I wouldn't start the reboot below FL300 to be on the safe side

[Mark 85]

Re: "...need to be hard rebooted after exactly 149 hours"

yeah, I wouldn't start the reboot below FL300 to be on the safe side

Since the plane needs a complete reboot/power down, I don't think I'd want to do that while flying at any altitude.

[Anonymous Coward]

Re: "...need to be hard rebooted after exactly 149 hours"

I'm sure the passengers won't mind when everything goes dark and silent, and they hear the engines restarting one by one. They could do it when people are sleeping.

[UncleZoot]

Re: "...need to be hard rebooted after exactly 149 hours"

That would be a big IF the engines do a restart.

I can imagine making an Atlantic crossing in a storm, run out of time, everything goes black and the flight crew fighting to reboot the computers and get engines restarted in the rain while traveling @ 425kts.

Once the engines shutdown, all heat for the control surfaces no longer exists so the leading edge of the wing, horizontal and vertical stabilizers begin to freeze. This would not be a good day.

[spold]

Re: "...need to be hard rebooted after exactly 149 hours"

Particularly since it involves going outside and using a paperclip to push the little button in the small hole in the nose.

[John Smith 19]

Like the Multiplexer DeMultiplexer boxes on the space shuttle.

The collect multiple data sources (and probably provide any driving signals for things like strain gauges) and format the data.

However the MDM's didn't run programs (although I think the later model running on the ISS do)

As embedded flight avionics that suggests they should be under the full DO178b style development process. Nothing less.

Incidentally the first time I heard one of these timer overflow bugs was related to Patriot missile batteries in the 1990 Gulf War for extended periods of time (I heard about it much later)

So it's not exactly an unknown failure mode.

[tip pc]

Re: "...need to be hard rebooted after exactly 149 hours"

Just reload it every 120 hours, leaves plenty of margin for delays etc to be well inside the 149 hours.

[Anonymous Coward]

Re: "...need to be hard rebooted after exactly 149 hours"

"*exactly* 149 hours" !

[Anonymous Coward]

Cue the usual Scotty and the USS excelsior comparison about over complexity.

[TRT]

Blame the gremlins. Just look out of the cabin window!

[Anonymous Coward]

"Cue the usual Scotty and the USS excelsior comparison about over complexity."

Sometimes wannanted IMO. Does the steering control for the nose gear *really* need computer control? Whats wrong with analogue wiring to a steeting motor and a couple of max lock cut out sensors? Seriously.

[Alan Brown]

"Whats wrong with analogue wiring to a steering motor and a couple of max lock cut out sensors?"

That depends if it can detect and warn that the nosewheel isn't straight before it touches down and if the sensitivity can be adjusted with speed. You do NOT want a sneeze on roll (landing or takeoff) putting you in the bushes or snapping the tyres off.

More importantly a separate control path to the nosewheel would mean yet _another_ set of controls in the cockpit and pilots on the ground suffer from operational overload half the time anyway. That's why those checklists are so critical.

[Neil Barnes]

Turn it off and turn it on again...

Do they have to close all the windows as well?

[simonlb]

Re: Turn it off and turn it on again...

I hope not, that would be a real pane.

[BebopWeBop]

Re: Turn it off and turn it on again...

And might be tricky in most real planes?

[Test Man]

So do all the pilots have to read the counter as part of their pre-flight checks in order to make sure that the plane switched-on time doesn't exceed 149 hours? And what happens if the plane is delayed literally when it's on the runway - do the pilot have to recalculate while waiting and if it's likely to exceed while they are in operational command do they have to turn the aircraft around, park again and then go through the switch-it-off-and-on-again procedure?

[b0llchit]

Automatic switch procedure

The system will switch itself off if they forget to check or run into a too long delay. The only problem may be the fireball that causes the switching off. There are no known sparks involved in the switching off procedure and the switching on procedure expires when switching off is done by the system itself. Obviously, YMMV is appropriate. Please ensure to cash-in those miles before the switching off occurs automatically.

[Dan 55]

Presumably they include extra time for how long the next flight could last in their calculations, just as they do with fuel.

[big_D]

One would hope so...

[Mark 85]

Presumably they include extra time for how long the next flight could last in their calculations, just as they do with fuel.

Presumably they would. But then there's Murphy who doesn't play by the rules.

[Bob Magoo]

I'm reliable informed that Murphy failed his plane driving test.

[Anonymous Coward]

Is the counter

Even accessible to pilots, or is it just some hidden value no one can see so the maintenance guys will need to put a stick in the windshield like the ones Jiffy Lube puts in the upper left of your car's that tells you what mileage you should change the oil at next.

[katrinab]

Well it mustn't exceed 149 hours before it lands at its destination, so if delays are going to make a difference you should reboot anyway. You don't want it to fall out of the sky while circling round London waiting for a landing slot at Heathrow.

[Tim Bates]

Can't exceed 149 hours before it is *parked*. They still need some of those systems to taxi, and airports handling larger aircraft are usually too busy to just leave it sitting on a runway exit for 30 minutes to get started again.

[Cuddles]

Why is there a choice?

"The remedy for the A350-941 problem is straightforward according to the AD: install Airbus software updates for a permanent cure, or switch the aeroplane off and on again."

As far as I can tell from this, the issue has been fixed - a patch is available and all airlines need to do is install it. So why is the "turn it off an on again" thing even being mentioned? Surely with a potentially safety critical problem like this, it should be a simple case of grounding all aircraft until the patch has been applied.

[Jens Goerke]

Re: Why is there a choice?

No time for maintenance, we're losing money while the bird is on the ground.

[Steve Davies 3]

Re: Why is there a choice?

Memo... Insurer to Airline

Install the patch or you are NOT insured.

Yours

Moneybags Insurance GMBH

Seriously, Aircraft need regular and importantly periodic inspections That should be more than enough time to install the update.

[JimBob01]

Re: Why is there a choice?

hmm… imagine you are waiting to board your plane when an announcement is made

“There will be a short delay to boarding as the technician carries out some maintenance. We appreciate your understanding and hope to continue boarding as quickly as possible”

Wait 20 minutes…

“I have an update on the delay to boarding. It seems the technician has bricked the plane!”

I would guess that the patching only becomes mandatory at the next planned service of the plane so that the process can be properly planned. Up until that point, cold rebooting every 100 hours should be sufficient.

[Sgt_Oddball]

Re: Why is there a choice?

Having been on a Fokker suffering software issues that required a call being passed to 2nd line phone support to figure it out (it was turned off and back on again but not everything came back up...) I can tell you it wasn't 20 minutes later that they'd bricked it and an hour later they finally got it to start up right.

Escape because they wouldn't let us get off..

[Doctor Syntax]

Re: Why is there a choice?

I've been stuck waiting for a plane where they found the fuel connection wouldn't cut off so they couldn't remove the hose. Very difficult to take off with the tanker still attached.

[Pascal Monett]

Very difficult ? Nah. Very eventful, though.

[PC Paul]

Re: Why is there a choice?

What makes you think the doors would open?

[Anonymous Coward]

Re: Why is there a choice?

Bloody Fokkers

[bpfh]

Re: Why is there a choice?

So they bricked the little Fokker?

[Stoneshop]

Re: Why is there a choice?

“There will be a short delay to boarding as the technician carries out some maintenance. We appreciate your understanding and hope to continue boarding as quickly as possible”

Wait 20 minutes…

"The technician has found he doesn't have the right adapter cable. One will be sent from Toulouse at the earliest opportunity. We're sorry for the further delay."

[I ain't Spartacus]

Re: Why is there a choice?

"The technician has found he doesn't have the right adapter cable. One will be sent from Toulouse at the earliest opportunity. We're sorry for the further delay."

The technician has now begun the process of installating from floppies, 432 of them, but is unable to find the spacebar...

[OssianScotland]

Lemon Scented Paper Napkins

No further comment needed

Icon - there's a frood who knows where his towel is

[keith_w]

Re: Why is there a choice?

"The technician has now begun the process of installating from floppies, 432 of them, but is unable to find the spacebar..."

I suppose that is better than not being able to find the "any" key.

[Anonymous Coward]

Re: Why is there a choice?

> The technician has now begun the process of installating from floppies, 432 of them, but is unable to find the spacebar...

The spacebar was removed at the same time they removed the cattle-class legroom...

[taxythingy]

Re: Why is there a choice?

It was the heaviest key, so made sense to the bean counters.

[Flywheel]

Re: Why is there a choice?

"Bad Sector Error Disk 65 - Abort/Cancel?"

[elkster88]

Re: Why is there a choice?

Believe it or not, one of the ARINC 429 dataloader boxes in my lab loads flight software for some of the avionics that my company supplies to Airbus and Boeing using... wait for it...

3-1/2" 1.44 MB floppies.

At least it uses "modern" floppies, not the 5-1/4" or 8" floppies used by some of the older (but still operational) equipment that's sitting on the next shelf.

[katrinab]

Re: Why is there a choice?

Does anyone still make them?

I searched on Viking Direct [UK branch of Office Depot] and it says it has never heard of such a thing. They used to sell them back in the day, but presumably all the people around then have died off or retired.

[RRJ]

Re: Why is there a choice?

Thought that was the "ANY KEY"

[Anonymous Coward]

Re: Why is there a choice?

Hey! Where's the ANY key ?

[Adrian Harvey]

Re: Why is there a choice?

> Hey! Where's the ANY key ?

It's the big blank one with no letters on it.

[Remy Redert]

Re: Why is there a choice?

The technician has just found that floppy 412 is unreadable. Unfortunately that means the software will have to be completely removed and reinstalled from the back up. We've been assured that this shouldn't take more than 3 working days.

[werdsmith]

Re: Why is there a choice?

My experience was a 787 Dreamliner at Heathrow (AA) where the captain announced that some systems would not come on line and so a complete power off and on was required. We had to all disembark back to departure lounge because the captain wasn’t happy for us to be on board an aircraft with no power.

[AndrueC]

Re: Why is there a choice?

I was on a flight out of Birmingham a couple of years ago that got held on the apron. We sat there for over an hour while engineers came and went. Eventually they closed the door and restarted the taxi. The captain announced that although they couldn't fix the fault the plane was cleared to take off.

So..worth waiting an hour to try and fix but not bad enough to stop us flying.

Hmmm.

[Anonymous Coward]

Re: Why is there a choice?

>the captain wasn’t happy for us to be on board an aircraft with no power

I wouldn't be happy to be on an aircraft with no power either. They tend to fall out of the sky that way.

[Spanners]

Re: Why is there a choice?

I wouldn't be happy to be on an aircraft with no power either. They tend to fall out of the sky that way.

They should look back in history. I specifically remember that Chipmunks don't!

[Stoneshop]

Re: Why is there a choice?

I wouldn't be happy to be on an aircraft with no power either. They tend to fall out of the sky that way.

a) For them to fall out of the sky they have to be in it. Sitting on the apron or on a taxiway there's not really a big opportunity for that.

b) Sully, the Gimli Glider crew and the BA flight 9 crew, among others, may feel the urge to disagree.

[Down not across]

Re: Why is there a choice?

hmm… imagine you are waiting to board your plane when an announcement is made

“There will be a short delay to boarding as the technician carries out some maintenance. We appreciate your understanding and hope to continue boarding as quickly as possible”

Wait 20 minutes…

Display shows:

Update is 100% complete. Please don't turn off your airplane

[some spinner on the display rotating]

2 hours later

The display is still showing same message.

[Ripper38]

Re: Why is there a choice?

Be funny if the reboot sequence showed up on say 253 multimedia screens... Be even funnier if there was a technical support engineer onboard...

[A.P. Veening]

Re: Why is there a choice?

Up until that point, cold rebooting every 100 hours should be sufficient.

I would make that something like 120 or even 144 hours (five or six whole days), four days and four hours seems inconvenient.

[SkippyBing]

Re: Why is there a choice?

Generally depending on the classification of the problem the authorities grant some leeway in applying the AD. E.g. within next 28 days which gives the airlines some flex to incorporate it into the next scheduled down time. In this case they probably believe that the interim measure is safe enough that it doesn't require the AD to be applied immediately.

[Electronics'R'Us]

Re: Why is there a choice?

The update (patch) would not be applied by either the airline or Airbus; the kit itself will have been designed and certified by a 3rd party avionics house who will also have done the low level software (which would implement the actual communication links) which is what seems to be the issue here.

To update, the equipment would need to be replaced and the older units sent back to the manufacturer for software load and testing.

It takes time to replace units so this would be done at the next scheduled maintenance point.

[veti]

Re: Why is there a choice?

You know how much disruption it causes, to hundreds of thousands of people, when a whole fleet of planes is grounded? Even briefly (and we don't know how brief it would be)?

That's an order that only goes out when they find something really dangerous. This hazard is easy to manage, once you know about it. Indeed, if it's been in service for two years without anyone noticing, that suggests it's pretty easy to manage even if you don't know about it

[greatpix]

Re: Why is there a choice?

If the hard reset works that should be preferable to a software fix. Why? Because, as anyone involved in the development cycle knows, fix one thing and that patch is liable to break something else that only shows up once the user has it hands on. It is well understood that no matter how many QA tests are run, in the field the user will ALWAYS execute some action that the testers, in their wildest imagination, would never consider happening.

[mj.jam]

What is overflowing?

Any ideas about what is overflowing? 149 hours of seconds doesn't seem to be that obvious a limit, but I guess they probably have rounded down a little to stop planes falling out of the sky.

I've seen issues similar to the Boeing one turn up in less critical places. Found my customers since in internal testing no system was left up for long enough.

[Dave Pickles]

Re: What is overflowing?

149 hours is near-enough 2^29 milliseconds.

[mj.jam]

Re: What is overflowing?

That makes it sound like they were trying to allocate individual header bits to different fields. So 28 bits only would give them 74 hours, but that wasn't enough, and 30 bits gave 300 which would never be needed, so they choose 29 bits.

I guess they then didn't write any test cases for overflow. I can imagine the problem is that they haven't wrapped the comparison operation correctly. So the newest data ends up looking very old.

[Anonymous Coward]

Re: What is overflowing?

Test case ? We're doing agile software development here.

[Baldrickk]

Re: What is overflowing?

Agile =\= no tests

[eldakka]

Re: What is overflowing?

In theory, correct.

But in practice, which is the only thing that matters?

[Baldrickk]

Re: What is overflowing?

That your code works, which you prove with the tests.

Still trying to get TDD to become the norm here, but at least we do write the tests. It helps that we need the test evidence to pass our milestones and actually release anything.

[Stoneshop]

We're doing agile software development here.

Which is not that different from "It compiles, ship it", only with shorter release intervals

[hellwig]

Re: What is overflowing?

ARINC 429 messages are 32-bits. 8-bits for the label, 2 bits for the SSM (signed status matrix), 2 bits for the SDI (source/destination indication), 1 bit for parity, That leaves 19 bits for data which is 536,288 (2^19).

There are 536,400 seconds in 149 hours. So they're sending time since power-up as a 19-bit value in seconds, and it overflows just before 149 hours has hit.

[mj.jam]

Re: What is overflowing?

Sounds plausible, but 2^19 is 524288, so is under 146 hours.

The 2^29 works, so AFDX must have changed that.

[John Smith 19]

2^19 is 524288

Which is 146 hours

So something's a bit screwy here.

[Boy Quiet]

Re: What is overflowing?

Many years ago I was a software programmer and the ICL 1904 I was using at a client site stopped working. We were much closer to the hardware in those days.

The engineers ran a test and I poked my nose in asking about the results. I deduced addition was faulty, and persuaded them to do an addition (on the switches !! ) they did and told me I was wrong.

8 hours later they replaced the addition unit and all worked. when I asked what they had added it was 1 & 1 Had I specified FFFF and 1 it would have shown the problem - carry in bit 8 was faulty!

Still reminds me to specify exactly what I want when testing.

[Robert Carnegie]

Re: What is overflowing?

The story says "exactly" 149 hours. If I'm following, it also says that if you've installed the software patch to fix the issue... well done but you still have to do the reboots every 149 hours.

[caffeine addict]

Re: What is overflowing?

What am I missing? Why do the systems need to know the milliseconds since it was started, rather than the milliseconds past an arbitrary time? Something that could reset every time the umbilical was unplugged, 00:00 UTC went by, or the pilot got up to shag a steward(ess)?

The black box needs precise timings. The internal indicators just need to be reply in a timely fashion. No?

[DugEBug]

Re: What is overflowing?

It's what happens when you flush an overloaded toilet, but that's not important right now.

RIP Leslie Nielsen

[Lazlo Woodbine]

Re: What is overflowing?

Windows 98 used to crash after 49.7 days due to a timing chip error

[simonlb]

Re: What is overflowing?

I never trusted Windows 98 enough to leave my PC on overnight, let alone 49 and a bit days!

[fishman]

Re: What is overflowing?

So Airbus is almost 3X better than Windows 98.

[Anonymous Coward]

Re: What is overflowing?

You are mixing days and hours there. Please don't write any software for aircraft. Thank you.

[eldakka]

Re: What is overflowing?

...or interplanetary probes.

[Pascal Monett]

Re: Windows 98 ?

I have never seen a Windows 98 that managed to last a week, let alone 49 days.

XP SP3 was much better behaved, but it still had trouble getting through a month in a single stretch.

[Chemist]

Re: What is overflowing?

"Windows 98 used to crash after 49.7 days due to a timing chip error"

Time must have run very differently for you

[KarMann]

Re: What is overflowing?

It comes to almost exactly 2^29 milliseconds, give or take 8 minutes, so I'd think that's what the limit comes to. Why not 2^31 or 2^32, I've no idea. Not enough extra bits for a time zone or anything like that. Could be used for something else mysterious to me.

[Sandtitz]

Re: What is overflowing?

2^31 would be ~600 days. Is it 100% certain that no plane ever is continuously on for two almost 2 years?

I sure hope the fix wasn't raising the time to 2^30 msec... :-D

[Anonymous Coward]

I wouldn't assume it is something neat like milliseconds

It could be counter that counts cycles on some CPU or bus somewhere to generate a unique 'event' timestamp, and if it happens to be clocked at 333.625 MHz then it would overflow a 32 bit value in exactly 149 hours (though that "exactly" is probably rounded down from 149.something)

[Simon Harris]

Re: I wouldn't assume it is something neat like milliseconds

Not sure where you get 333MHz from, I get 8kHz (+ a few Hz) to roll over 32 bits in 149 hours.

[The Man Who Fell To Earth]

Are the software updates free or does Airbus screw you to fix their screw up?

"The remedy for the A350-941 problem is straightforward according to the AD: install Airbus software updates for a permanent cure, or switch the aeroplane off and on again."

The remedy for the A350-941 problem is straightforward according to anyone with morals & a brain (& a healthy fear of litigation): install free Airbus software updates for a permanent cure.

FIFY

Let's hope the patch doesn't just automatically turn the plane off & on at 148 hours.

[Boris the Cockroach]

lucky the patch

is'nt by microsoft

"Caution: update needed, if you choose to delay the update it will automatically install and reboot your flight controls in 7 days , no further warning will be given"

[SkippyBing]

Missile Bug

Reminds me of one of my favourite buffer over run stories. A missile was being developed, possibly AMRAAM I can't remember off-hand, and they had a problem with over runs. So in a move of genius they installed twice as much memory as would be needed in the longest possible flight, solving the problem.

Some years later they produced an improved range variant of the missile, predictably they forgot why they'd installed so much memory in the first place...

[Anonymous Coward]

6 days ! WTF ?

If I'm reading this correctly, 6 days with no power down = a severely crippled A350.

It went to service in 2015, and still in 2019, there are planes with this flaw ??

It shouldn't even have passed QA, in this state ! I remember EMC was not shipping Symmetrix (significantly cheaper than an airliner) without 3 weeks running flawlesslly, one in a cold room (0 degree C), one in a 20 C room and one in a 40 C room ! Do I understand correctly airplanes, those days, don't even come close to the level of QA from EMC, 2 decades ago ?

I'm not sure I want to board any airliners anymore ...

[Phil O'Sophical]

Re: 6 days ! WTF ?

remember EMC was not shipping Symmetrix (significantly cheaper than an airliner) without 3 weeks running flawlessly,

I'd expect disk drives to run flawlessly for years without a reboot, so three weeks testing isn't much. It wouldn't have found this EMC problem https://www.theregister.co.uk/2014/01/15/vnx2_reboot_issue/ for which the solution was, yes you guessed:

1. Reboot SPA

2. Wait 30 min

3. Reboot SPB

before 80 days had passed.

How many airliners never actually get powered down completely sometime in every 6 days?

[Anonymous Coward]

Re: 6 days ! WTF ?

The VNX is not Symmetrix, it is their midrange product. The VNX is to the Symmetrix what an ATR regional jet is to the A350.

[Phil O'Sophical]

Re: 6 days ! WTF ?

The VNX is not Symmetrix, it is their midrange product. The VNX is to the Symmetrix what an ATR regional jet is to the A350.

I know, I have a lot of experience with Symmetrix boxes and they are great kit, but it doesn't change the principle. Three weeks untroubled testing for something that is expected to run continuously for years, whether VNX or DMX, doesn't really have any relation to the kind of testing an ATR or an A350 would have.

[Jason Hindle]

Presumably, the better established airlines

Are applying the work-around while some other airlines tests the patch for them?

[Red Ted]

"CPIOM is effectively a mini computer"

I think it's probably a fully fledged computer, unless you are implying it's a 1970's style mini-computer with core-store and Winchester disks?

[PTW]

Re: "CPIOM is effectively a mini computer"

I read it as a 70s style, albeit a fraction smaller. Does this mean I'm officially old?

[Doctor Syntax]

Re: "CPIOM is effectively a mini computer"

"Does this mean I'm officially old?"

Welcome to the club.

[Anonymous Coward]

Re: "CPIOM is effectively a mini computer"

The huge Winchester disks serve a dual purpose as a gyro to help stabilize flight.

[Simon Harris]

Re: "CPIOM is effectively a mini computer"

Maybe the Airbus hardware division is still working its way through a pile of Intersil 6100s (PDP8 on a chip) they bought in the 1970s.

[Anonymous Coward]

Re: "CPIOM is effectively a mini computer"

"a pile of Intersil 6100s (PDP8 on a chip) they bought in the 1970s."

Back in the day, sensible system builders used to like to be able to source their chips from more than one chip shop.

For the 6100 family, there was (as Simon mentioned already), the Intersil version.

And here we are, some years later, and Simon's post doesn't mention that the 2nd source for this particular chip was a company called Harris..

https://www.hb9aik.ch/computer/6120history.htm (and elsewhere).

Small world, innit :)

[Alistair]

Good Morning, Welcome aboard, may I see your Boarding Pass?

Not till you tell me when this plane was last rebooted you can't!

[chairman_of_the_bored]

Helpful Technical Support

"Have you installed the latest patches?"

[elgarak1]

Why is this a 'news' story?

1) As mentioned, 149 hours is more than 6 days. Since there are no possible flights that long (even with all the possible delays factored in), it should be easy to work in a turn off/on cycle, albeit with some more ground time and financial loss.

2) As mentioned, the problem has been identified and fixed, but we are currently in the period where operators get some leeway to work the update into their maintenance schedules.

In short, there's nothing to see here, move along, folks. The only reason it gets newsworthy is to say "Boeing's baaaaaad, but look here, Airbus isn't that much better, either!", even though the criticalness of Boeing's fault massively overshadows the mentioned fault of Airbus.

[defiler]

Since there are no possible flights that long (even with all the possible delays factored in)

Tell that to Zarniwoop, waiting for lemon-soaked paper napkins...

[Pascal Monett]

Re: "albeit with some more ground time and financial loss"

Financial loss ? Well you're obviously not in charge of an airline, that's for sure.

Airlines are already running close to red, they really can't afford to just go around losing more money.

Honestly, given how difficult it apparently is to operate an airline, I'm surprised they don't just give up and quit. There must be more money in it than I think.

[Mark 85]

Re: "albeit with some more ground time and financial loss"

There must be more money in it than I think.

And the board looks at their bonuses and nods approvingly.

[Anonymous Coward]

The fact Boeing's fault is much worse doesn't make this fault a non issue as you claim. Whataboutism isn't a defense when it comes to aircraft.

[eldakka]

Why is this a 'news' story?

Because of the Boeing incidents, issues with aircraft are high up in current public perception, and stories about that that 12 months ago wouldn't have rated even a footnote are of interest to the general public.

It's like when there's an explosion at a factory. Generally people don't give 2 figs about how their toothbrush is made. But when some disaster, either just spectacular or that results in major tragedy, strikes, people become interested - curious - about how their toothbrush is made, what trials and tribulations surrounded the invention of it, and so on.

Therefore what happens around aircraft manufacturing, science, maintenance, piloting stories, and so on, become of interest.

Any moron knows this.

[diodesign]

Re: Why is this a 'news' story?

Also we've written about aviation software faults for years (see Register passim) because readers love hearing about engineering problems - and we're OK with this. Bugs and weird shit fascinate us.

C.

[Annihilator]

Yep, to be fair, known bugs with workarounds are sometimes better than introducing a fix and all the risk that comes with it.

Besides, not knowing enough about the usage of aircraft, but I'm pretty sure they don't leave them on when not in use, and aren't flying non-stop - even the ones RyanAir hammer the hell out of.

[Proud Father]

Common Remote Data Concentrator (CRDC)

Sounds similar to the CANBUS system used in modern cars ;)

[TRT]

Re: Common Remote Data Concentrator (CRDC)

One wonders if the plane has a buzzer that sounds when you open the pilot's door without removing the ignition key?

[Ryan 7]

Re: Common Remote Data Concentrator (CRDC)

Not sure, let me cheeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

[Anonymous Coward]

Ah the joys of agile development

I see it's now taking a hold in the aviation industry as well.

[Anonymous Coward]

Re: Ah the joys of agile development

Add to this the perspective of a new generation of coders raised with the belief that pushing a princess on the screen and programming CPIOM of a modern airplane are pretty much the same stuff. After all coding is so easy anyone can and should do it.

[SVV]

Could be worse.....

Imagine you're halfway across the Atlantic and the in flight movie cuts out to be replaced by "Windows 10 update installing......".

[Doctor Syntax]

Re: Could be worse.....

Could be worse. Windows 10 updating cuts out to show an in flight movie.

[Fading]

Re: Could be worse.....

Reminds me of a flight I took back from Baltimore to LHR. The cabin plunged into darkness then the low level lights came back on. A brief moment later the Captain informed us that there was a problem with the cabin electrics so there would be no hot food or inflight entertainment. The options were turn around and land in New York or "just go for it"......

TBH I was glad the wine was plentiful on that flight across the atlantic...

[TRT]

Re: Could be worse.....

Could be worse. BSOD.

[Anonymous Coward]

Re: Could be worse.....

Even worse:

Red box with writing saying Guru Meditation.

[Anonymous Coward]

Re: Could be worse.....

Not the inflight movie. Just imagine pilots see this on their displays instead of altitude, speed, heading and other. Now that's the real fun.

[veti]

Re: Could be worse.....

Happens all the time. The entertainment system is not considered critical, so frankly I'm surprised when it lasts a whole flight.

[Sgt_Oddball]

Re: Could be worse.....

The amount of times I got to see tux on various KLM flights (and one Delta flight) boggles the mind.

That said I've been know to have the gift of crashing computers off any kind.

Probably best I stay out of the cockpit.

[Anonymous Coward]

Echoes of the Patriot missiles in Gulf War 1.0 ..

Didn't they stop working after 100 hours ...

[jigr1969]

Re: Echoes of the Patriot missiles in Gulf War 1.0 ..

Nope, the patriot missile system had a software error which meant that for every hour it is left running, it would become less and less accurate. Hence why the SCUD hit the America base despite the sending up of patriot missiles. It hadn't been rebooted for far too long and the error meant that the missiles completely missed the incoming SCUD (cannot remember if they detonated far too short or too long).

[Anonymous Coward]

Re: Echoes of the Patriot missiles in Gulf War 1.0 ..

Hard to see how such an error occurs:

aim_vector = aim_vector + hours_since_last_reboot

Sounds like code written by a defense contractor wanting to insure they could require 24x7 onsite staff to help maintain it. Oh wait...

[Richard 12]

Re: Echoes of the Patriot missiles in Gulf War 1.0 ..

Inertial measurement is hard.

You tell it exactly where it is to start with, and then it tries to keep track.

Over time, noise adds up, rounding errors compound and/or the Earth rotates and moves around the Sun.

Mishandle any of those, and you get significant drift over time. Enough to trigger an emergency escape system at a launch hold...

[rszasz]

Re: Echoes of the Patriot missiles in Gulf War 1.0 ..

To keep tracking an object with radar, you set a "gate" and try and acquire the target again in the next period within the updated gate. The underlying problem was that different bits of the software calculated the time slightly differently leading to the gate being about .3 of a second out of sync after 100 hours. (counting upward with fixed point or floating point numbers is not as simple as people expect)

[Paul Hovnanian]

Boeing 787

... was once every 248 days.

[Anonymous Coward]

Re: Boeing 787

So it was 40x better!

[kain preacher]

So you are telling me this airplane has less uptime then a windows 95 machine ?

[Anonymous Coward]

You had a Windows 95 PC that lasted for 149 hours? They could hardly last that long if you let them sit, let alone if you used them for anything!

[kain preacher]

Ok window 98

[Anonymous Coward]

OK Windows 98 could get 149 hours of uptime, if you ran it for 48 hours before it crashed and then left it on the blue screen the rest of the time.

[kain preacher]

I've gotten 5 days of uptime on windows 98 se before

[Blackjack]

Guys, cost cutting is killing people!

Why haven't they patched? Because it costs money and time.

And the regulators are wusses and wimps, why aren't they raining fines over these people who cares more about saving money that if their saving money kills people?

Maybe they are cousins of certain guy in the US?

[Anonymous South African Coward]

Just bloody fantastic.

What's next? A kick in the crotch every 5 min to ensure reliable operation?

Furrfu.

[dmacleo]

actually easy to implement although should not be have to put in that position.

a/c can time out on certain items while in flight (meaning next flight puts it over hours, good example is batteries timing out) so upon landing a/c is grounded and no tdmi/mel issued until addressed.

so flight and mtx planning would stop a/c at a mtx base before that time came close and reboot it.

meanwhile engineers should be scrambling to fix this or be getting fired.

[bilston]

Apple Air

If this was an Apple aircraft it would be called an optional reboot feature reminder, and require a trip to the Apple store.

[Anonymous Coward]

Apple Air

If this was an Apple aircraft it would be called an optional reboot feature reminder,

[Halfmad]

Re: Apple Air

If it was an Apple aircraft it'd be taxiing around the walled garden.

[TRT]

Re: Restart Before You Depart

Strategy?

[89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921]

...they should divulge the make, model, software version and inhume issues of any plane before we book flights

[guyr]

Turn off and back on - patented process

I think Microsoft patented the process of turning the machine off and back on again to fix any random problem. Some serious fees are going to be paid for this.

[Criggie]

Serious question - how often would an airplane get completely powered down before this issue was detected?

Are plane engineers like sysadmins who have uptime wanks about the longest uptime they've had, and then explain why it ended ?

[Anonymous Coward]

Fly-not-by-wire

I prefer to fly long haul in a 747 precisely because the pilot's yoke is connected to the control surfaces by cables not of the electrical or fibre varieties.

That and the fact that you can roll a 747 over at 40,000', go into a high speed dive for 30,000', pull nearly 5g in the recovery and land without further incident bent wings, missing tail surfaces and permanently lowered main undercarriage notwithstanding. (https://en.wikipedia.org/wiki/China_Airlines_Flight_006)

[Nathan 13]

Re: Fly-not-by-wire

The people on that plane were 1 in 100 lucky. It really should have ended up in the sea, a miracle actually!!!

[Securitymoose]

I wonder if the software development was outsourced...

...to Musoketeba perhaps. (see 'Into The Fourth Universe' - https://www.smashwords.com/books/view/929784)

[sYncRo]

I believe they are changing model numbers so customer's are unaware that they are on a known affected plane.

[Spanners]

Just shut it down when you park it.

No aircraft journey is going to last 6 days and 5 hours. That is between 3 and 4 times round the world.

If the time starts counting when the crew are starting up for their next flight, they will have plenty of time even with the best of British delays - bureaucracy management anti-union actions, or just the wrong type of rain. Then they can wait for a couple of days at the terminal, another one at the threshold before taking off. I don't know the maximum length of time one of these things can stay up but they can't be in-flight refueled so they can reboot when they get back down again.

[toffer99]

I'm going on a walking holiday.

So many faults on planes. Boeing has another problem: "Pilots reveal safety fears over Boeing’s fleet of Dreamliners. Company admits that fire extinguisher switch has failed a ‘small number’ of times" https://www.theguardian.com/business/2019/jun/15/boeing-dreamliner-b787-safety-fears.

[2Fat2Bald]

149 is around 6 days. So I don't know how serious this actually is as it seems pretty unlikely an aircraft would remain sat there for 6 days, powered on and not being used. Unless the aircraft is put in some kind of "sleep" mode, or something like that and the time is still counting up.

Even so. Doesn't sound a hard one to fix with a routine that refuses the start the engines if the aircraft is on the ground and has over 120 hours since the last reboot.

[Sameer]

First question of any support call, Did you try turning it off and back on again?

Ha, Does that work on final approach?

[Anonymous Coward]

Similar to the Ariane failure?

Math error....result $500 million loss. Don't you just love how SIMPLE software failures can be....

*

http://www-users.math.umn.edu/~arnold/disasters/ariane.html

[Bob.]

We hardware engineers are always forgotten. Less money, less kudos and stature but we don't feck around.

Working on Dealing Room Systems (with our custom designed PCBs), many years ago, we had one board that would intermittently and infrequently crash.

Share prices/Currency/Commodity info would freeze on one of the Dealer's screens.

This peeved them somewhat ($ millions trades at stake).

Generally a dealer would have 4 screens and some hundreds of dealers per room. This board was used on each screen.

So 1200 boards per Room, for a 300 Dealer Room.

Thee was a Hard Reset switch on the board, but it required the sysadmin or on-site engineer to wander into the Machine Room (after an irate call from the Dealer) and find Cabinet x, Rack y and Board slot z. Off/On, Fixed. But that took too long.

Our software engineers spent weeks trying to track down the problem and gave up.

When they came to us, we found quickly and easily that we could put a simple hardware Watchdog Timer on the board.

If it wasn't reset every 5 seconds, the board was rebooted.

It worked well and no further complaints.

Obviously for planes, the logic might be a bit more complicated.

If not reset for 100'something hours and stationary on the ground then reboot.

[John Smith 19]

The choice is a bit more complex than it looks.

Option 1. Well understood (but annoying) procedure that must be run on a regular basis

Option 2. Single uploading of software patch.

But.

Does the hardware architecture support uploading and verification (packet corruption being sent through network to end box)?

If not it's a box removal exercise or a direct connection to a box deep in the bowels of the aircraft

How well has the patch been tested?

Has it added some new failure mode?

IOW from the airlines PoV the risk assessment is not quite as straightforward as it seems.

Of course if we assume that all software patches are perfect and have no unintended side effects then the course of action is obvious.

Anyone here who's written software believe that assumption?

[kraftdinner]

Built by M$

Now that's a 3 finger salute!