Airbus confirms software brought down A400M transport plane Airbus has confirmed the crash that stalled its A400M program was caused by engine control software. However, according to Handelsblatt, the problem wasn't that the software is buggy. Rather, someone in the final assembly process installed the software incorrectly. Marwan Lahoud, Airbus' chief strategy officer, told the …
I would have thought that somewhere in millions of quid spent on this a check on having the correct software properly installed would have been built in on boot?
Or maybe I'm just old fashioned in my attitudes?
But a sad loss of life, RIP.
>>Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one errors in the engine serial numbers come to mind.
What do checksums have to do with anything? Checksums are to address data corruption, not software configuration.
the problem wasn't that the software is buggy. Rather, someone in the final assembly process installed the software incorrectly.
Having worked for a defense contractor, let me translate: The software was buggy.
You never, ever admit fault to your customer, ever, as this can and will affect your ability to get new contracts and also scuttle the contractor's stock price. If pressed you only admit to human error (which is impossible, see below).
To make this more clear, the final assembly process is the most solid part of the process. The final code is complied, after which it is run through an automated code check which can take hours. In final assembly, or rather when the code is loaded into each computer, a series of diagnostic tests and simulations are run to verify both HW and SW operation. These are composed of test vectors simulating actual operating conditions. These are all Go/NoGo tests as simple as a green light means pass and a red light means fail.
Upon power-up (here, before each flight) all systems run some built-in self-test (BIST) diagnostics. They are not just standalone tests, they depend upon inputs from other systems on the plane that share the same parameters. A failure locks the system and prevents operation. See how this can't be an isolated failure?
For a failure such as this I have seen two reasons for failure: either the software was buggy, or somehow the system was tricked (hacked or accident) into running the factory test simulation code. In one case (not a plane) the operator accidentally pressed a secret key combination (holding down 3 keys simultaneously, released, then pressing one key within three seconds) that forced the system into diagnostic mode, with tragic consequences.
Maybe things are different with non-U.S. contractors, as the above process is extremely expensive.
Software is common across different engines. The most likely "incorrectness" of the install is installing the wrong parameter tables. From there on the plane (which is 100% fly by wire) is asking the engine to deliver take off thrust (as was in that case) and it delivers a paltry fraction of it. With the expected results.
This still does not excuse Airbus from not having checksums (or other means) of verifying the correct install stored in the main avionics and not interrogating the engine that it has the correct ones.
Instead of calling for take off thrust the system simply needs to monitor the position of the throttles. If it only knows the relationship of those to the position of the throttles in the fuel system then there is no confusion in defining what "take off thrust" means in aircraft to aircraft.
Anyway, it's all water under the bridge now, RIP.
I doubt that the error was in the sensing of the flight control position (i.e. that takeoff thrust was being demanded). It was more likely using the wrong engine performance or sensor parameters which resulted in incorrect engine control settings being used to try to achieve the commanded power.
As an example, if you were to set the parameter on your programmable home thermostat so that its temperature input is interpreted as degrees Celsius, but your digital room thermometer is sending degrees Fahrenheit, your house is going to be a lot colder than you are commanding it to be.
That is the distinction between an EEC and a FADEC. The FADEC does not allow any overriding input from the operator (Pilot). The EEC is a subcomponent of a FADEC System and for many reasons a FADEC has become the choice moving forward. It is no longer possible for a human to calculate and control an engine according to all the variable parameters within the environment that modern jet engines operate.
"A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one."
- Edward Norton, Fight Club
I am guessing the engine control software is generic, why write a very complex piece of software more than once? Then its just configured to the engine/airframe...
I can see how poor configuration could cause a stall on takeoff...
Really with QA on something like an aircraft the rule should be no one person or even team (except pilots I suppose) could cause catastrophic failure even if they tried intentionally. If its possible as is the case here obviously a whole lot of people in management need to find a new line of work. People will make mistakes especially groups of them (none of us is as dumb as all of us) but if the process doesn't account for it the process is garbage.
"I would have thought that somewhere in millions of quid spent on this a check on having the correct software properly installed would have been built in on boot?"
The test systems for aerospace hardware often contend with different software versions in the gear they're testing. I'm on an aerospace company's program for a targeting system and we have different firmware versions for different customers and legacy versions of the system, which means the test rigs are necessarily flexible - they don't scream if you've got (for sake of argument) v1.01 firmware installed instead of the latest v1.03. And it's real easy to let your eyes glaze over when the boot-up data scrolls past and you've got the chance to spot "v1.01" instead of "v1.03." The real meat of the test data is whether the system works and all the flagged functions say, "pass," "working," and so on.
The A400M's engines go through a lot of tests. I can guess at some of the objective evidence that would be required to make Airbus and Europrop happy. At a minimum:
1) The ECU was tested and certified by its maker before it was sold to Europrop (admittedly probably without the software, which would be Europrop's or Airbus's responsibility to install);
2) Europrop tested the all-up engines on a test cradle before it sold the engines to Airbus;
3) Airbus would've powered up all 4 engines on the A400M long before the plane was allowed to take its first, fatal test flight since big aircraft don't get near a runway until their many systems are tested;
4) On that fatal flight, the engines powered up and made all the blinken lights in the cockpit glow happy colors without spitzensparken
The software that crashed the plane did go through check after check, and it worked. Up until the most important check.
Airbus has noted problems with the A400M program's organization: it treats production, development, and retrofitting as separate programs, rather than an integrated single program. That's an environment where version control problems are going to proliferate. I could very easily imagine the poor bastard who "incorrectly configured" the software was just using an out-of-date manufacturing process plan that said something like, "get the USB stick with software version 1.01" because no one had flowed down process changes to the MPP hardcopy at his workstation to say, "get the v1.03 stick." Or something like that.
Evidence, transparency, proper legal procedure.
As opposed to the more traditional response to a military aircraft accident. If pilot==dead then {pilot error.}
Because suggesting that the aircraft was at fault might impact the manufacturer's profitablilty which would put the defence of the realm (and jobs in a marginal constituency) at risk.
Because suggesting that the aircraft was at fault might impact the manufacturer's profitablilty which would put the defence of the realm (and jobs in a marginal constituency) at risk.
I hate to bring this message but some Yuropean countries have the hots for their "friend's" F-35, so your "risk" is already in the intestinal zone.
Evidence, transparency, proper legal procedure.
You know, nowadays you can actually *copy* black box contents while maintaining evidence, transparency and proper legal procedure and getting the engineers to work.
[Our world] No
[Their world] Yes
Discuss, using only one side of the paper.
Let's correct that for you:
[Our world] Yes
[Their world] Yes
There's likely to be a court case over this accident. The investigation in Spain is being conducted by their judicial authorities, as is normal in their (and many other country's) Inquisitorial/Roman system of justice.
Regardless of the judicial arrangements having the evidence, especially if it is damning, made publically available seriously prejudices any court case in any jurisdiction. How could a trial be 'fair' afterwards? That would deny the victims and their families their day in court. Thus anyone pressing for public disclosure of the data (evidence) is being stupid and supremely selfish towards the victims.
I bet you take photos of other people's car crashes, you jerk.
However...
The fact that Airbus are confirming the cause of the crash indicates that they have no intention of contesting whatever legal case is coming up. They're going to take what's coming on the chin, which is the proper thing for them to do. That is to the benefit of the individuals who carried out the installation (who are no doubt completely devastated by the crash): such an admission means that the responsibility is now aimed higher up the managerial chain.
"The fact that Airbus are confirming the cause of the crash [...] means that the responsibility is now aimed higher up the managerial chain."
From dealings with another large company, I suspect that any blame will go up the chain until it hits a committee or two, and then dilute into a fog of collective responsibility.
Full marks though for not just blaming the pilots or some lowly technician.
But, not releasing the black box data to the aviation accident investigation authority responsible is in contravention of international treaty (Annex 13 of the International Treaty on Civil Aviation, I think) for aviation accident investigation. Aviation accident investigation is purely to prevent reoccurence and not to accord blame. There is a slight question over whether this was a civil or military accident, however as it was being flown on a test by an Airbus crew I'd think it was civil at this stage as it hadn't been handed over to the Spanish Air Force.
Of course it's an interesting move by Airbus to confirm the cause of the crash without the data from the black boxes as that may reveal other information.
You're right, but some countries consider airplane crashes a "criminal investigation", especially if there are victims.
In Italy it happened in the Linate crash too - it was managed by Italian "Carabinieri" and magistrates without a clue about properly managing a crash site, debris where removed immediately and piled up at the helipad, while access to the site by the investigators sent by the airplane company and its national agency was forbidden.
When an airplane flown with Alitalia colors (but rented from a Romanian company with its crew) went off the runaway in Rome some time ago (without victims, luckily), the Alitalia logos where removed immediately with white paint, clearly "contaminating" the crash site evidences.
Usually data are shared with the investigators of all interested parties, and of course the manufacturer is the one that is best equipped to analyze some data like engines performance, and fixing the problem is far more important than hiding it, actually hiding issues and having more crashes it's not what an airplane manufacture aims for, it's already difficult to sell planes with a good reputation, selling models known to crash is pretty impossible.
Just some states have outdated rules and are obsessed with police/magistrate control on everything, just because of fear.
Not block. But the Spanish authorities might be having trouble reading it, resulting in delays.
This was a military transport, ordered by Turkey. Odds are that CVR and FDR data is encrypted. So extra steps may need to be taken with Turkey's cooperation to get a 'plaintext' copy.
When I read this:
"Handelsblatt beats Google's translation with the sentence “Die Software für die Steuerung der Motoren sei bei der Endmontage falsch aufgespielt worden” "
Well ... I had to see what Google Translate offers up:
"The software for controlling the motors had been partly filled with wrong during the final assembly"
"Partly filled with wrong" - my new catch-phrase for every time somebody cocks something up!
I have worked on many multi national projects. To cater for customers who want to interface with equipment in their own language, which might be Mandarin Chinese, Sanskrit, Standard Arabic, Bulgarian or what ever, software tends to be hugely parameter driven, to an unsafe extent. Also, to allow for minor differences in customer requirements, parameters are adjusted,
This gets unmanageable really quickly, for example, just 20 Yes/No options have over a million different combinations.
One application I know of has over 24,000 lines of parameters, most with multiple values. This is inherently unsafe, there are not enough hours in the foreseeable future to test all the possible combinations, prior to release of the product.
I have seen people involved in major acquisitions demand "modifications", which, inter alia, complicate the parameter file(s), merely to justify their own role as part of the acquisition team.
Language issues are best dealt with by Locale files - and those locale files are best translated by someone bilingual, then vetted by at least 3 other people (I've just finished making ~1500 corrections to a UK-english locale which was originally translated from french by a guy who learned english at high school...)
It's the fiddling with other parameters which gets really messy, real fast - and leads to Chinook crashes.
As to "inherently unsafe", that depends on whether parameters do need testing in combination. Some will, some won't. No testing strategy, not even for safety critical software like medical equipment, requires testing of every possible combination of individual states. If you think that's a problem, you don't understand ALARP.
Good architecture should minimise linkages so that you *can* trace what can affect what. If you don't have good enough engineers to design a robust architecture, then you have bigger problems than just the testing.
Fortunately, testing of avionics and nuclear plants does require the testing of all possible combinations - things like full MC/DC, etc.
However, even though that test all the paths through the software it doesn't cover all possible combinations of paths through the software...
Supposed to be used before live testing, but often used to fly a profile based on known weather/software/etc or even replay black box data to see what the pilots would have seen.
It's creepy flying a profile (more like riding along) where you know people died and without the benefit of hindsight usually impossible to avoid making the same mistakes that caused it.
Obviously don't know their process, but I've been involved in a project where special test modes are used to test the hardware in production.
These don't perform a functional tests of the software (as that should be done as part of the development process), but do allow all the mechanical / electrical / electronic systems to be tested. Testing like this means that aspects can be tested in isolation, making it much easier to find any issues (e.g. by not having to have all the features required for a closed-loop control system to be working when testing).
The final step on the production line was to load the configuration data for the particular customer. This step was then verified as part of the final QA before shipping (e.g. verify that the configuration data checksum matched the checksum on record for the customer).
One well known vendor of safety critical stuff I'm familiar with is keen on model based systems engineering and similar entirely sensible tools.
Unfortunately their idea of testing is to test solely against the model, on somebody's desktop/datacentre PC, not test against whatever eventual implementation of the model ends up being programmed on the aircraft subsystems. It's so much more cost effective that way, obviously, and what could possibly go wrong between the model and the stuff on the aircraft.
Expect more of this A400M kind of thing in the years to come, and not just on military trials, unless attitudes and processes return to being driven by engineers, testers, etc, (as they used to be in the early days of safety critical software) rather than being dictated by cost-driven MBA types who think Integrity is a piece of productivity software (and who ruthlessly silence anyone who dares question the new improved approach).
A recurring theme, where modern flight control systems control mechanical devices totally with software and electronics rather than using them to assist or resist mechanical actions. One suspects even if the pilots realised the plane was behaving unexpectedly, they would either have to trust the electronics to recover or probably not be able to anything about it in time anyway.
Fly-By-Wire has proven itself to be far safer than the old mechanical systems; check the statistics for modern FBW airliners:
http://www.fearofflying.com/resources/safest-airliners-and-airline-safety.shtml
In this case the software problem was in the FADEC, which replaces the role of the "flight engineer" (planes haven't had flight engineers for about thirty years); FADEC is like the engine management software in your car (unless you have a car that is pre-1990s that is) so it's virtually impossible to replace with mechanical systems and still have the same efficiency and reliability.
Going back to the pre-FBW systems is likely to introduce far more safety problems than you remove so whilst we know that improvements can be made, it won't be by abandoning modern avionics.
Perhaps you are right about the safety of FBW. The site clearly indicated the clear advantage Boeing has over Airbus in safety issues.
As a layman, it is pretty hard to imagine somebody manages to create a situation in which software starts shutting down the engines in mid take off.
Somebody would think that when steel wires and hydraulic pipes for management of the essential controls are replaced with copper wires, there should be some brain behind it.
"when steel wires and hydraulic pipes for management of the essential controls are replaced with copper wires, there should be some brain behind it."
There has been brain behind it in the past.
Safety critical stuff (software in particular) *was* rocket science when it first routinely arrived on airliners almost thirty years ago, when engineers ruled the roost. And it stayed that way for a long time, and rightly so.
Thirty or so mostly accident free years later, the new dynamic young people in charge of the business seem to be starting to get complacent. Nothing's gone wrong [the missing word here is 'yet'], but we need new methods to keep our costs down and improve performance all round. Methods, tools, processes, are being introduced from other parts of the software world. methods that lose much of what has kept safety critical systems as reliable as they have been.
New people in the regulatory authorities think that if there's a defined process and the paperwork follows the process, everything will be OK. The auditors are no longer subjec t matter experts, they're paperwork experts. Well guys, there's a reason engineering-based things like code reviews and extensive (not exhaustive, but extensive) testing of a representative setup have been considered vital in the past. They may or may not discover errors, but they should help *prevent* them, **IF the right people do them for the right reasons** (which does not mean because they're on the project ticklist).
This stuff can be fixed. It needs to be fixed before too long.Fixing it will involve managers listening to engineers, designers, testers, etc. Otherwise at some time in the next few years, there'll be an inquiry and lessons will be learned and all the usual guff. The lessons will likely already have been known, but will have been ruled as outdated by Bright Young Things on their way up the 'career advancement by cost reduction and schedule compression' pole.
"Fixing it will involve managers listening to engineers, designers, testers, etc"
The jobsworthian "tickbox" mentality goes _well_ beyond aviation and engineering. People do stuff by rote without understanding why they're doing it and that in many fields the tickboxes are merely a guide to ensure the job's being done right, not an end in themselves.
Then they get promoted to management and the tickboxes become more important than the end target.
I seem to remember that the UK Chinooks had problems with their engine software, (termed FADEC Full Authoity Digital Engine Control I think) Basically a fly by wire throttle pedal which took loads of other parameters into account when setting the engine power.
A possible contribution to the '94 crash in scotland where a load of anti terrorist experts died en route from Northern Ireland. Initially blamed on the pilots..
I think this was also a case of 'not invented here', where the MOD decided we couldn't use the off the peg Boeing system and had it rewritten. Jobs for the boys I suspect..
I saw in a different forum what the installation problem was.
Its a problem with the motors. Depending on where they're attached to the wing they have to behave differently. You have to first install the software and then add the settings. If you mix them up its very difficult to detect it on the ground. Once in the air its difficult not to notice.
"Die Software für die Steuerung der Motoren sei bei der Endmontage falsch aufgespielt Worden"
Translates as "The software for controlling the motors be incorrectly been played during the final Assembly" using the Bing engine.
No fluent German/English people here willing to weigh in?
Will have to wait until the BBC report it - they have a ridiculous number of multi-lingual people working for them.
"Seems like it should be made very difficult to configure this software incorrectly"
How? There are a load of parameters that the software & hardware has no way of checking, and must thus rely on the installer inputting the correct data. As a trivial example, if one of the parameters dictates the position of the engine on the aircraft, the electronics/software has no way to check that the wires physically go to the left wing rather than the right wing. The only checks that can be made is that all the parameters are consistent with each other - and if they were parameters that were designed to work with a slightly different engine type or configuration, they will be self-consistent and pass all the "sanity checks".
"The only checks that can be made is that all the parameters are consistent with each other "
Confident? Final answer?
Here's a thought for you. How hard would it be to have the parameters (and maybe nothing but the parameters) programmed into something that's associated with the *engine* and follows the engine (does not follow the controller).
An engine-attached widget with a standard interface that can be used by the engine control to read the parameters. A widget which can be (and should be) programmed and verified independently of the engine control itself.
Hint: it's not very hard. Ask people in the industry about what a Data Entry Plug is, and where they're used, and how long they've been using them that way. Or read about Data Entry Plugs (aka rating plugs) on the Interweb.
Please don't try to defend the indefensible by making things up. It's not helpful.
"How hard would it be to have the parameters (and maybe nothing but the parameters) programmed into something that's associated with the *engine* and follows the engine "
Not very (and this is already done), but as some have already pointed out the desired characteristics of the engine vary depending on which side of the aircraft it's on, whether it's inboard or outboard and whether it's driving a reversing reduction gear or not.
Ideally the engine would recognise which position it was in and pick up the right settings automatically, but people insist on individually manually programming things "because it's more reliable" and that opens up the possibility of mistakes.
Let's not go into Murphy's Law (If an aircraft part can be installed 2 ways and one way is the wrong way, someone will do it) - which amongst other things is exactly the reason that the Genesis probe crashed into the Utah desert (someone installed a gravity sensor upside down) and that there was much excitement about Spirit and Opportunity camera images until it was discovered the calibration files had been swapped over...
It's better to design it right and make _sure_ it can't be installed the wrong way up. This applies as much to software as hardware, because the wetware is fundamentally unreliable.
"better to design it right and make _sure_ it can't be installed the wrong way up."
Very wise words (and the rest). The once-much-worshipped Japanese even had a name [1] for that concept.
An example: if an item needs fixing to another item, and the relative position matters, make the design such that (1) the necessary orientation is obvious, and (2, just in case obvious isn't enough) make the wrong orientations impossible.
E.g. If the orientation of a circular or square part matters, and you're attaching it with screws or similar, have the holes at irregular positions so that the correct orentation is the only one which is possible. On many occasions I've seen stuff like this follow the "do everything neat and regular" principle which frequently introduces opportunities for confusion (4 possible orientations, only one of which is correct) or worse.
It's not really rocket science is it. Just 'common sense, properly applied'.
[1] poka-yoke, from the (in)famous Toyota Production System and thus from all the finest LEAN toolkits. But LEAN (and its abuse by those who just tick boxes without understanding why the concepts matter) is another story for another day.
>>How? There are a load of parameters that the software & hardware has no way of checking, and must thus rely on the installer inputting the correct data.
I hope you're not in charge of designing anything important.
Let's say you need to configure some software using the user's birth year. According to you, it sounds like the only way you can do that is ask the user for his birth year and if he's wrong, everything is eff'ed and there's no way you could have prevented the problem.
But what if you ask for birth year AND age and compare the two, as a sanity check? That should sort out of a lot of problems. And if you wanted to triple check, ask for the user's social security number and arrange some way to confirm the year with the government. etc.
I'm sure similar double/triple/quadruple checking of configuration parameters is possible when installing software for a plane.
"Instead of calling for take off thrust the system simply needs to monitor the position of the throttles. If it only knows the relationship of those to the position of the throttles in the fuel system then there is no confusion in defining what "take off thrust" means in aircraft to aircraft."
Yes there is confusion, "take of thrust" might be like 30% throttle on one engine, and 75% thrust on another engine (I doubt it's as low as 30%, but for sake of argument). These vendors tend to order an airframe and then order some engines to be put on seperately... why, I don't know, but that's what they do, so it's not like there'd be an "A400M calibration". This kind of thing does actually apply on cars too -- some car with 1.8L engine, 2.2L engine, 3.0L engine option, they will usually all 3 use the exact same computer, with a calibration ROM (in the past a physical ROM chip, now flashed in parameters) saying "this is a 4-cylinder, here is the fuel curve, here's the ignition timing curve, here's an "fast idle" versus temperature curve", and so on. Although I've never heard of GM f'ing this up by putting the wrong calibration in.
Besides confusion over "how much throttle" with electronic throttle control, there'd also be confusion over fueling (I'm sure underfueling or overfueling a jet engine jacks up your power just like it would on a car engine), the different engines may even be expected to run at different RPM to produce some amount of thrust.
German here. Proper translation: "During final assembly the engine control software had been installed incorrectly."
It doesn't mention parameters and lacks any specifics about how they managed to install the software "incorrectly". In a German military forum (augengeradeaus.net) there has been some speculation by current and former military personnel that either the incorrect revision of software had been installed, or that (indeed) some parameters were used that were incompatible with whatever software they had been using during the maiden flight. Or a mix of both.
However: Airbus is awfully silent about what did lead to the inability to retract the flaps after take-off, which the doomed crew had reported by radio.
A "software installation problem" during final assembly is awfully convenient for Airbus, too. With that they say: "The software is not the problem. The installation was. (i.e.: human error) All we need is better Q/A.". And also: "The engines themselves aren't the problem."
But fact of the matter is: Airbus has (since long) lost the ability to integrate turboprop engines into airframes. Or to design airframes around turboprops. For the A400M they had to start on a clean sheet for that. It would not be surprising if the engines of the A400M (and their integration into the system as a whole) contain some unaddressed issues that haven't yet been fully ironed out.
The German ministry of defence also doubted the statement of Airbus and said that if the software was glitching, the engines should have continued to operate on their last power setting instead of shutting down.
Bottom line: Airbus is in all out damage control mode and it can be speculated that the entire (sad) truth hasn't come out yet.
On the other side of the pond, a security researcher is in BEEG trouble for either patching into a commercial airliner's engine controller system from his seat back and making the aircraft yaw, or he just made up that story and tweeted it, and got kicked off the flight and then corralled by the FBI with some major league criminal charges. http://www.theregister.co.uk/2015/05/19/airplane_hacking_panic_why_its_a_surely_a_storm_in_a_teacup/
There is substantial dust-up as to whether it could have happened, even if he didn't actually do it on a real flight.
And the use of the CRM 114 discriminator absolutely prevents any errors and ensures General Ripper's commands are adhered to. Does the ARINC 629 provide the same protection.
That's amazing. I wonder where Google gets the 'filled up' from though - I don't know of any uses of 'spielen' meaning 'to fill'.
'aufspielen' is the proper German vocab for 'to install' in the context of software. It could include both installation and configuration. It has nice retro connotations of installing software by sticking a reel of on a magnetic tape drive and threading the header around the rollers.
"there are two ways of constructing a software design: One way is to make it so simple that there are *obviously* no deficiencies and the other way is to make it so complicated that there are no *obvious* deficiencies"
The FADEC systems I saw in the 1980s and early 90s were in category 1: as simple as possible (e.g. no loops, no compilers, etc).
Times have apparently changed. Engineering best practice has apparently been replaced by cost effective, productive, etc. For now.
https://www.cs.fsu.edu/~engelen/courses/COP4610/hoare.pdf
Longer version (same concept, different presenter):
http://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf (55 slides)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
