Software bug in Bombardier airliner made planes turn the wrong way A very specific software bug made airliners turn the wrong way if their pilots adjusted a pre-set altitude limit. The bug, discovered on Bombardier CRJ-200 aircraft fitted with Rockwell Collins Aerospace-made flight management systems (FMSes), led to airliners trying to follow certain missed approaches turning right instead of …
It's easy to override, obvious to the pilots and whilst confusing not immediately dangerous (unless the pilots fail to take any action and the air space is very crowded with another aircraft loitering around where the plane would turn into).
On top of that as soon as it was reported Bombardier investigated rather than ignore the issue and write it off as pilot error.
Pretty much the polar opposite of what Boeing did...
Boeing should have sensed both AOA devices before triggering the elevator trim. However, pilots are supposed to know how to deal with runaway trim. They had controls on the yoke and center console that would have overridden the computer. I won't speculate on why they didn't follow their training.
Is this purely an autopilot error? In general these are not are high risk, as the pilots are suppose to check up on it's behaviour. A bit like if your cruise control goes too fast/slow, but still allows you to override/turn it off at any point. A problem, but not as bad as breaks failing.
"[...] but three lefts do."
My father taught me that rule while on country walks in England - which generally has a dense matrix of interconnecting roads. On my first overseas posting to South Africa I discovered that was no longer true once you left the urban area. All you could do was turn round and go back to where you had taken the wrong turning.
One dark night I crested a hill on a motorway and sensed - rather than saw - something across my two lanes. It was a massive articulated lorry and trailer - sideways on with no visible lights. It was crossing the central reservation in order to get back to a missed exit junction.
Transiting through Germany I discovered that the autobahns generally didn't have roundabouts on junctions. You had to go a long way on a dual carriageway to the nearest settlement before you could turn round.
Not sure if you're joking or not.
There is at least one real precedent for this, e.g. United Airlines UA232, McDonnel Douglas DC 10-10, uncontained engine failure in the tail mounted engine (caused by a latent manufacturing defect) led to the loss of all three hydraulics systems on the aircraft.
https://sma.nasa.gov/docs/default-source/safety-messages/safetymessage-2008-08-01-unitedairlinesflight232crash.pdf?sfvrsn=fba91ef8_4
"using the throttles, they could control one axis of motion at a time. They were able to raise or lower the nose by speeding up or slowing down the engines, and they could make unstable but manageable turns to the right by slowing one engine down and speeding up the other. However, they could only turn right, meaning that to turn 60 degrees to the left, they had to turn 300 degrees to the right. "
Of 296 passengers and crew on board, there were 112 fatalities.
https://en.wikipedia.org/wiki/United_Airlines_Flight_232
I sure remember United Airlines Flight 232.
Our local airport at Clear Lake/ Mason City was was just east of the first turn they made. I think it was considered but is a bit small so it was probably good they went to Sioux City. Landing like they did could have had them cartwheeling on to the Interstate highway just west of the our airport.
The extra flying time let them burn off more fuel and practice their new flying technique.
They are the type of pilots you need when the going gets tough.
The undoubted Dog's Bollocks of commercial flying.
None of the crews they put in a simulator afterwards, given the same situation, got anywhere near landing the thing so 184 survivors is up there with loaves/fishes and water walking as miracles go.
They actually got it on the floor intact, but couldn't keep it on the runway as they had no rudder control. All the damage was done when it veered off and rolled.
Personally I reckon gear up into the adjacent cornfield was a better bet, but....(!)
Also remember the guy on the throttles was a spare pilot travelling as a passenger. Who sacrificed a relatively safe seat in the cabin for kneeling on the floor with no seat, let alone a seatbelt. In the cockpit where the occupants are a highest risk in a crash.
He flew the plane on the throttles, while the original pilots wrestled the controls for what little they could get out of them.
Didn’t they land at something stupid like 300 knots, because they had no working flaps and slats?
There was also a cargo flight that copped a missile over Baghdad that had to do something similar and doing lots of right turns.
"That pilot developed a whole new way to fly an aircraft [...]"
Quote from WW2 bomber archive for Ronald James Auckland MBE DFC.
The rudder control was severed, the elevator trimming tabs were shot away, the bomb doors could not be closed, the fuselage was pierced in many places and worst of all the bomb-aimer had been killed and the engineer badly wounded. Although the aircraft became extremely difficult to control and the general direction was difficult to maintain he managed to get aircraft home.
@AC: I had a look at the archive expecting the plane to be a Wellington, but was wrong - it was a Lancaster!
I loved the bit preceding that incident - "On 19 May, during a raid on the Tours marshalling yards, on final approach and waiting for the bomb-aimer to order ‘Bomb doors open’ his aircraft was hit by another Lancaster flying in almost the opposite direction. Another foot or so and there would have been one almighty bang. He managed to maintain control of the aircraft, continue the run and drop their bombs. Quiet an achievement given that he was now flying on only two engines, the port outer propeller was bent backwards and 12 feet of the port wing was missing. The Perspex top of the cabin just above the pilots head was also smashed which reduced the cockpit temperature dramatically. It was a long, slow, cold and dangerous return for the crew especially as German night fighters were on the look out for damaged aircraft such as his. He landed safely at RAF Tangmere in Sussex although 61 Sqn base was in Lincolnshire." Seriously good flying!
Although he'd lost the rudder, he still had the ailerons.
Many aircraft were that badly damaged in WW2, but few of them landed since the crews had parachutes and would quite happily use them rather than kill themselves landing a wreck. Wrecked airliners without parachutes however...
> There is at least one real precedent for this
Back in the days of rotary engines it was very common to make turns in the direction favoured by the engine torque rather than spend half a day trying to coax the plane to turn in the opposite direction.
As for the principles underlying the technique that you describe, it is taught to students on their very first lesson (or second? I forget). It is called the secondary effect of controls.
The interaction of the various ideally completely separate functions suggests insufficient process segregation. This in turn suggests that the software as a whole has "evolved" rather than being designed as an integrated deliverable. Most software today "evolves", so we need a radical rethink of our approach to application design.
Design? Design? Not only design.. but also let us ensure that true Quality Control happens... with 2 teams.. one with source looking for flaws... one slinging software into situations that designers knew would NEVER happen.
We are unfortunately in an era where we have foundations/classes/ frameworks that everyone uses and few have an idea of what is behind the curtain. Worse... the idea that hardware and software are two differing disciplines is simply not true. Both must be understood to take advantage of the system.
Not thinking through the software makes bad hardware (intel CPU engineers can see this with the pelthora of security violation bugs that have 'appeared' from people doing things that they 'would' not be doing in real life...)
Not understanding the hardware makes bad software.... as Boeing and others have seen.
As part of a code review I saw a very elegant piece of recursive code. Brilliant... short... but in real world extreme cases it would kill the stack.. The young fresh out of school kid..... had no clue.. as memory was something the system worried about... The rewrite was longer, yet saved tons of stack space and time... Kid finally understood.. but... he was a 'software guy' not hardware as he told me at the start... :(
It just turned it the wrong way so instead of a left turn it went right, or instead of a right turn it went left.
Not such a problem with low-use single-runway airfields in the middle of flat areas, but potentially embarrassing if there are hills, mountains, tall structures, (air) traffic or additional runways nearby...
This post has been deleted by its author
Nah, Gib isn't 'fun' the way you mean it - you get such a great view wherever you are sitting in the plane, (though better on the right hand side.) Sure, it isn't anything like as straight as most airports, but the only significant danger is over-running.
Vigie Airport (name now changed to protect the guilty) in St. Lucia is sufficiently notorious that no-one flies jets into it anymore. The first time BWIA (officially British West Indian Airways, mostly called Britain’s Worst Investment Aboard, But Will It Arrive, or Better Wait In Airport) flew a jet into Vigie the aircraft made a low approach over Castries, St. Lucia’s capital and home to about half of the island’s 120,000 or so population, and blew out multiple windows, including those in the French Embassy. If you land short you land in the sea. If you land long you end in the harbour. If you drift to one side you land on a beach, a very popular beach. If you drift to the other side you land on the terminal building. And there’s just one runway. BWIA put 727s and Lockheed Tristars into Vigie for reasons which must have made sense to them. Air Jamaica put various Airbuses into Vigie back when Butch Stewart owned Air Jamaica; Butch also owned two hotels on St. Lucia and considered Air Jamaica to be a way to deliver victims, that is, guests, to them. BWIA and Air Jamaica have merged and no longer fly jets into Vigie. Now only turboprops fly commercial into Vigie, flown by the crazy men who fly into other small islands in the Windward/Leewards and US and British Virgins. The old airport on Monserrat is actually worse than Vigie, the runway ends at the edge of a cliff, and the runway in St. Croix (I think) runs into the base of a hill, there was at least one accident were a 727 ran long and went into a gas station at the foot of the hill, with interesting results.
The newly constructed airport on St Helena is apparently nearly impossible to use due to treacherous wind shear.
More worryingly it goes left instead of right when left is the quicker way of getting to the desired heading. The only reason I can think of for the procedure being designed to go the long way to a desired heading would be to avoid something you'd hit going the other way*.
Although the missed approach plate used to illustrate the article seems to happily take the aircraft over higher ground than the alternative so $deity knows what's happening there.
*And by hit I imagine it'd actually be a case of miss by less than the minimum separation allowed.
This post has been deleted by its author
I remember landing at Windhoek in Namibia. The town is in an old volcanic crater - complete with high-walled surrounding landscape. The internal town airport's runway could therefore not be extended for modern jets. Fortunately the surroundings outside the crater walls are a dead flat plateau - stretching to the horizon and beyond. So they built a new airport with a single long runway on the plateau - some 25km away from the town.
The runway is aligned such that it points in the direction of the town. On approach the 737 had to fly almost to the town's towering crater outer wall - then flip 180 degrees to get the approach to the runway for landing. In the vastness of the otherwise empty plateau - an interesting piece of design for unknown reasons.
reminds me of a bug in the radar of those Australian-built high speed catamaran ferries....................apparently if you go over 20 knots the radar reports distances at half their real value. Bug has been there since the first Seacats were built and I understand has never been resolved
Not sure why you were down-voted.
The aircraft in question was allegedly the F16, but this was back in the day when they did 'proper' testing and the bug was found before it went into production. Pretty sure though that it never happened in-flight. Can't say for sure it was purely a simulator bug as opposed to flight control.
I suspect that it would have taken many lines of code to have included such a function as a deliberate feature. Instead of all that effort, all the manufacturer now needs to do is document it and ask the relevant authorities to update their missed approach procedures so that turns are made in the opposite direction under certain conditions.
It would be very interesting and useful to sort out how that bug managed to get through the testing unseen. Bugs are useful, they tell you a lot about how well the code has been written and tested and what else might be hiding.
I would guess that the programmer writing the code did not have any flight training to realize how important air temperature is when you are flying and suddenly want to change things.
Probably the testers never thought to test flying a simulated missed approach at a runway where the charts tell you to make a 270 degree turn to the left in order to fly right.
At least, that's how I understood it, that fiddling with certain parameters will result in the plane turning 90 degrees to the right instead of the long turn 270 degrees left, but in both cases will end up flying in the correct direction.
This is an autopilot program for a plane. It does not need to detect obstacles, trees or other vehicles. It does not need to worry about weather conditions or if there is snow or ice on the ground (the pilots do, not the autopilot). It does not have speed limits, or wrong ways, and doesn't need to care about street signs. It just needs to manage the ailerons and the engine power.
These programs have been worked on for decades, and we can still find (small) bugs.
And you're telling me that we'll have autonomous cars by the end of the decade ?
https://en.wikipedia.org/wiki/List_of_automated_train_systems
It does have 3 axis of freedom, plus 1/2 dozen different ways to steer it in each axis. Now, an elevator (lift for those types); 1 path; 2 directions, 1 speed; 1 door. Yet, it is 2020 and I sill watch elevators go bonkers, not line up at floors, twitch until their watchdogs reset them.... You can always spot the embedded programmer -- that is the one that will not use their hand to stop an elevator door.
And actually, there were a fair number of plane crashes in 1902, and even earlier. I'd say that about all powered flights that year ended thusly; the ones that didn't are probably not worthy of the designation 'flight' by any measure. Gustave Whitehead's may have been an exception, though.
Many people have commented on the plane (or similar situations with other planes).
I think the article is about the FAA.
As in, how totally screwed up they have been for years.
Refusing to permit a software fix for a software bug. And the vendors are presumably ready and willing to do the recertification.
Can't fly anymore due to corona virus. Now I have another reason not to.
"Refusing to permit a software fix for a software bug. And the vendors are presumably ready and willing to do the recertification."
According to the Directive, the manufacturers suggest that pilots just avoid doing things that trigger the bug. The FAA says that's not good enough, and the function should be not be used until the bug is fixed. Further,
WR Ryan stated that this matter is not serious enough to warrant an AD. The commenter also stated that this issue is being exaggerated, as Collins will eventually fix the problem. The FAA infers the commenter wants the FAA to withdraw the NPRM. The FAA does not agree.
Nope, the function should not be used until the bug is fixed.
That's what the FAA (and ESA) are saying, and I agree with them.
Pilots can start using it again once the bug is fixed and properly verified - this should take quite a long time, as this kind of testing should be as exhaustive as possible.
For example, simulated missed approaches on every known airport and airfield capable of landing the aircraft, covering the entire expected temperature range of each place.
As I've told another dev at work - there's only a few billion floats. Test them all!
<rant warning> This makes me so mad. It reduces my opinion of the people who design planes. To assume the pilots are too stupid to fly, automating ever more and at the same time reducing training so that the pilots really are struggling in some cases, then putting their faith into software based on the readings of non-redundant sensors. It creates situations not just like the 737 MAX but also the French airlines crash into the Atlantic based on a single iced piezoelectric sensor - the Air France pilots - who were not trained to safely identify which systems aren't working - assumed that if they stalled the plane the software would save them! Incidentally those sensors have since been re-designed twice to improve the built-in heater; actually I would like to see 2 more of each so that the software has a quorum if one of them fails. Add to that is the news that all 4 redundant sets of avionics in Aibus A380s were placed in the same cupboard, instead of separated 2 at each end of the plane and in one airline's case right under the 1st class showers which once leaked and took out 3 of the 4 avionics systems nearly crashing the aircraft. I am not particularly clever but I think even I could do a better job of this. Is it politics? Accountants? Is it the costs of extra sensors? Is it aeronautical engineers treating IT design as having lesser importance? Were the IT architects the stupid ones? I'm so annoyed </rant>
Yes it's madness, but you are massively oversimplifying the AF447 picture.
In general when people die in comercial airline incidents, it's a result of a sequence of events, and fixing any one of them properly (rather than crossing fingers and trusting the worst won't happen) would have avoided the casualties and fatalities. (Look up "swiss cheese theory" amongst others).
AF447, for example: (most of this oversimplified story is from memory)
AF447 aircraft was routed *through* rather than *round* bad weather. Going round uses more fuel but going through increases various risks including the risk of sensor issues, although arguably is accepted practice. This particular aircraft was known to have a Pitot tube config which was subject to icing issues, an Airworthiness directive had been issued but not yet implemented on this particular aircraft, the change to the sensors was waiting for ... I don't know what, do you?. Did the flight crew know of the airspeed sensor issues on that particular aircraft prior to setting off? I can't remember whether they did or didn't.
There were multiple pilots on the aircraft at the time of the incident, two in the cockpit and one, the most experienced one, "resting" elsewhere in the aircraft. When things started going wrong, there was a delay before the experienced pilot returned to the cockpit. The situation would probably have been recoverable if he'd returned promptly.
And so on.
"the Air France pilots - who were not trained to safely identify which systems aren't working "
Commercial pilots generally aren't trained to diagnoise this kind of failure. We are all supposed to have faith in the ability of the regulatory authorities to routinely make things as safe as reasonably practical. AF447 is one of many incidents that show that faith is not sufficient, actual timely relevant appropriate action is needed.
I don't fly much these days (even before CoVid). I fear for the future of the UK regulatory authorities, they seem to have been struggling for the last few years with inadequate understanding of the way safety-critical software systems fail (hint: it's different from the largely well-understood and well-documented ways that pure hardware fails, though the introduction of complex "programmable" hardware such as FPGAs is blurring the boundaries somewhat).
Stay safe out there.
My first thought on reading this: nah, the headline is wrong.
Then I go and read the airworthiness directive, helpfully linked in the article: oops! the headline is right.
The rest of the article is nicely done, clearly the author has put some effort into getting things right, which is a refreshing change from most aviation-related articles in the non-specialised press.
Just one minor nitpick, as a former commercial pilot (technically still qualified and medically fit but no longer active):
> Missed approaches are used when pilots aren't confident that they're going to land safely.
I realise this is an intentional simplification but it is not a matter of confidence. More often than not you're completely confident of being able to land but if the conditions dictate otherwise you are going up again regardless. For instance if the weather is below minima, the approach is not stable, or quite simply you did not get landing clearance on time because someone was hogging the radio.
From a cognitive point of view the missed approach is your plan A, landing is plan B.
Also, the article seems to confuse temperature compensation with temperature correction. Those are two different things and it's the latter that seems to be concerned here. It applies in a specific type of instrument approaches (Baro-VNAV) and the consequence of the AD is to make those approaches unusable by aircraft with affected FMS on days colder than the minimum promulgated temperature. The reason why it ends up affecting the turn direction (obviously, other than the there being a mistake in the code) is because the length of the final approach segment (the bit where you aim for the runway, ending at the missed approach point) is indirectly dependent on temperature on those types of approach.
Someone above commented that this is not a serious mistake because you should be able to catch it. That is not so. In my qualified opinion (and the regulators agree) this is a major problem for at least two reasons: when you are in the soup it may not be immediately obvious which way the aircraft is turning (and there is such a thing as confirmation bias) and even if/when you realise that it's gone for a wander, you are still faced with the dreaded "what on Earth is it doing?" question, which is not a good position to be in at 150+ knots, low energy, close to the ground and with a few tons of aircraft and cargo behind you. In a non-radar environment and with the "wrong" sort of environment and circumstances even if you perform at your best you may not be able to recover the situation.
Quite tired so apologies if not as clear as I would have liked.
“From a cognitive point of view the missed approach is your plan A, landing is plan B.“
I see what you are saying. That’s a very insightful comment and the approach (pardon the pun) behind it is obviously applicable to things outside of the world of avionics, too. You gave me something to think of, thank you!
All software has mistakes, so should never be trusted. If you really need software to do something safely, then have 3 separate algorithms on 3 separate cpus all try to solve the same problem, and only allow the software to take control if 2 of the algorithms agree. It is still risky, but then you can still also have a human over ride.
This was being done on most avionics systems I worked on 40 years ago.
Build in test would check the hardware to make sure the readings were sensible, and software would choose / average readings from the devices before acting on it and displaying it to the pilot / navigator etc.
The event of Fly by Wire meant that even more vigorous checking was done.
I did work on the autopilot software for a British fighter jet once.
One of the stories we heard was on one of the early versions of the software the 2 digital compass reading were averaged to produce a more accurate heading, unless one was obviously wrong in which case it was dropped.
But the variable held the data as -1 for magnetic south (180 deg), 0 for magnetic north, and 1 bit less that +1 for 179.999 deg.
So if one compass was reading 180 deg, and the other 179.999 deg instead of averaging almost 180 deg it computed a 0.
Good job that one was spotted before the aircraft got airborne.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
