Bad software crashed Boeings. Now it appears the company lacked a singular software supremo

Boeing has created a new role for a vice president of software engineering and filled it with a veteran of similar gigs at SpaceX, Tesla, and Google. The Register reports the new job because Boeing’s appointment announcement points out that it previously lacked such a role – and also because the company has admitted that …

[John Smith 19]

Pity they didn't think it that important earlier.

But it's not just the software.

Driving the software with non redundant sensors (which Boeing has done twice that I'm aware of so far) is not software but is certainly doesn't make that task easier.

Let's see if they can turn improve Boeing's delivery as well as its image.

[Version 1.0]

Re: Pity they didn't think it that important earlier.

I'm not going to downvote you but as a programmer or software application management boss you have to understand the environment that you are working in ... well, that's the way it used to be but the world has changed. These days it's not a pity that they didn't think it that important earlier because the standard is to get the app written and released - and "fix" the bugs with an update later.

"Only one sensor? No problem, I'll create a virtual one ...."

[Paul Hovnanian]

Re: Pity they didn't think it that important earlier.

"But it's not just the software."

This, exactly.

Having worked there, including during a period when their software really hit the fan, so to speak, they suffer from placing it on a pedestal. To be handled only by the High Priests of the mainframe, sequestered in the monastery of the data center. The MCAS failure resulted in software that (from what I've read since I wasn't there during its inception) worked properly per specification. But the system design (including sensor hardware, cockpit warnings and flight crew training policies) all failed miserably.

You can't just cobble together an ill-conceived system and then hope to 'fix it in software'.

[Anonymous Coward]

Re: Pity they didn't think it that important earlier.

they suffer from placing it on a pedestal. To be handled only by the High Priests of the mainframe

That's not exclusive to them and it isn't new either. The exact same situation existed, for instance, at PSION in London in the 80s, but there it had admittedly at least less lethal consequences.

There are sometimes good reasons for silos, usually security, confidentiality or specific approval/audit/certification processes, but you need someone with sufficient authority* occasionally sanity checking if silo building isn't actually losing value rather than adding.

I agree with you that in Boeing's case, the error started with basic fundamentals. If you change an airframe so much that its basic characteristics dramatically change and you then decide to kludge your way past that, you better make damn sure that your kludge is at least as good as you can possible make it and watch it like a hawk. Which it wasn't, and wasn't. Cue planes nose diving, followed by reputation, shares and sales.

* _{Because otherwise exactly zip will happen otherwise}

[niio]

Re: Pity they didn't think it that important earlier.

The specification must have been pure junk. The purpose of having two sensors is to increase reliability. If one fails you use the other, if you can tell which is correct, so you reduce failure rate by half.

The way this was implemented did the opposite. Since the software alternated between senors for sequential flights but only used one, they doubled the risk of failure since there were two parts to fail and both had to be working in order for two flights to complete successfully.

Even an idiot coder or specifier should have caught this.

[Robert 22]

Re: Pity they didn't think it that important earlier.

It also has the potential to cause confusion and complicate troubleshooting - quirky behavior on one flight might not show up on the next flight.

[sanmigueelbeer]

Re: Pity they didn't think it that important earlier.

Driving the software with non redundant sensors (which Boeing has done twice that I'm aware of so far) is not software but is certainly doesn't make that task easier

Just want to remind people that the MCAS is just one part of the equation.

The whole thing all started with a faulty reading from one of two the Angle of Attack (AoA) sensors. Take note: One-in-two.

One sensor is giving porkies. Who to believe? The left one or the right one? In both cases, the MCAS randomly accepted the input from the faulty AoA sensor.

Airbus has THREE (3) AoA sensors. Boeing put two in the MAX to bring the cost down.

In the meantime, EASA and FAA has quietly cleared the MAX to fly.

[anothercynic]

Re: Pity they didn't think it that important earlier.

Sorry to have to pooh-pooh you, but it's not correct to say that 'Boeing put two AoA sensors in the MAX'. It implies that older versions didn't have two, but the 737 has always *had* two. But Boeing decided that to get around the problem of not having a 'quorum' of AoA sensors, they'd switch sides for MCAS on each flight.

They found that *adding* a third would add to the cost and also add to the complexity, and it would move away from the 'this is the same aircraft, just new engines and pretty new winglets, nothing else to see here, move along' mantra that allowed them to get away with grandfathering in a *lot* of old crud (which they wouldn't have gotten away with if the FAA had designated the MAX as a new plane). Ditto for the EASA requirement for a synthetic AoA (via the synthetic airspeed calculations that the 787 had). And EASA is still insisting on this change eventually with the MAX.

And also, no, FAA and EASA have *not* 'quietly' cleared the MAX to fly. There was no 'quietly' and there is no clearance yet. The FAA's chief pilot flew the plane and was positive, and the EASA chief said that their primary concerns have been appropriately addressed (and the findings are being analysed by the JOEB in Gatwick). But they have *not* cleared the jet to fly commercially yet.

[sanmigueelbeer]

Re: Pity they didn't think it that important earlier.

It implies that older versions didn't have two, but the 737 has always *had* two. But Boeing decided that to get around the problem of not having a 'quorum' of AoA sensors, they'd switch sides for MCAS on each flight.

You are correct but MCAS only appeared in the MAX, yes?

What I am saying is the MCAS focused entirely on a faulty AoA sensor and started a nose down. MCAS did not consider "verifying" at the speed or the altitude when doing the nose down. It just kept doing it until impact to the ground.

Another thing, of all the MAX customers, only Southwest Airlines purchased the option to enable (in software) the AoA Disagree Alert. If I remembered correctly, the AoS Disagree Alert costs about US$40k and it shows up as a "AOA Disagree" in the PFD. That is all it does. For US$40k.

After the crash, Boeing made the AoA Disagree Alert a "standard" ... even though the AoA Disagree Alert did/does not really work.

If I remembered correctly, the new software means if one of the AoA should provide faulty readings, it will only inform the cockpit and not wrestle control.

[Evil Auditor]

Re: Pity they didn't think it that important earlier.

That is all it does. For US$40k

It does sound a lot for what it does. But its value is much higher: saving a single life would already outweight the costs of one AoA Disagree Alert.

[anonymous boring coward]

Re: Pity they didn't think it that important earlier.

So you think it's ok to blackmail the customer with a chance that all will die, for a feature that cost the manufacturer nothing to switch on? Is that getting maximum security out of the aircraft?

I think it's utter madness, personally.

"So you want to know if your sensors are acting up? Well, pay $40k then..."

Why not make fuel readings cost options too? "You need higher precision on the last 10% of fuel? OK, pay $100k." "It's just a software toggle, but we like fleecing our customers."

[Anonymous Coward]

Re: Pity they didn't think it that important earlier.

It's a systems engineering problem. The software performed to specification. I suspect there's a PowerPoint presenter who drove the specification development. Focusing on the software is focusing on the wrong part for MCAS.

[Anonymous Coward]

Re: Pity they didn't think it that important earlier.

Focusing on *any* single aspect of this picture is asking for trouble. Here's something I posted here on 3 Aug 2019...

The MCAS kit as originally specified was allegedly intended to have a limited-authority (maybe 25% of jackscrew travel, or something like that??) one-shot effect on a flight control surface. Perhaps in those circumstances it *might* just about have been acceptable to not have much resilience designed in (but the system might also have not had the authority to achieve the intended effect either).

As time went by, the fundamental MCAS design got transmuted into "keep retrying till the aircraft/system is back in control. No limits." So 25% authority on a one off basis, to full authority, whatever it takes, and nobody considered it might call for improvements in sysem resilience and recovery mechanisms?

Presumably MCAS variations got a "delta" design review rather than a "start from a blank sheet of paper" review, just like the 737 in general hasn't had a proper design review for decades.

(This from one of the earlier well-informed blogs on the subject - aircurrent maybe?)

....

See also

https://www.seattletimes.com/seattle-news/times-watchdog/the-inside-story-of-mcas-how-boeings-737-max-system-gained-power-and-lost-safeguards/

"Two people involved in the initial design plans for MCAS said the goal was to limit the system’s effect, giving it as little authority as possible. That 0.6-degree limit was embedded in the company’s system safety review for the FAA."

...

"It also calculated what would happen on a normal flight if somehow the system kept running for three seconds at its standard rate of 0.27 degrees per second, producing 0.81 degrees of movement, thus exceeding the supposed maximum authority.

Why three seconds? That’s the period of time that FAA guidance says it should take a pilot to recognize what’s happening and begin to counter it.

Boeing assessed both of these failure modes as “major.” Finally, the analysis looked at the inadvertent operation of MCAS during a wind-up turn, which was assessed as “hazardous,” defined in a cold actuarial analysis as an event causing serious or fatal injuries to a small number of people, but short of losing the plane (that’s called “catastrophic”)."

Worth a look.

[Anonymous Coward]

I’m sure Agile, DevOps and the cloud will fix all this...

For certain values of “fix”.

[CrackedNoggin]

Re: Pity they didn't think it that important earlier.

Exactly. The question is really, will "Greg Hyslop, Boeing chief engineer and senior vice president of engineering" insist on the necessary hardware and pilot training.

Boeing hasn’t launched a new commercial aircraft since the Dreamliner in 2004. It delivered the first of those planes in 2011. In the short run R&D suppresses quarterly results, and quarterly results are what earn bonuses for the Chicago office.

[Black Betty]

Re: Pity they didn't think it that important earlier.

Part of Boeing's problem is their continuing reliance on "wetware" in an increasingly automated environment. Too many of their systems still rely a human in the loop to look at an instrument and think "Yeah that's not right." when it misbehaves.

All well and good provided the human sees the problem soon enough, and intervenes quickly enough, but absolutely deadly when the computer gets the "bit between its teeth". Thus there were several near misses with Boeing's flare retard landing system and MCAS before disaster inevitably struck.

[Cliffwilliams44]

Re: Pity they didn't think it that important earlier.

I am sorry, as a pilot I am not at all comfortable with software wrestling control of an aircraft away fro a human pilot. Software should alert and/or assist but not take control and fight the pilots decisions. It is the pilot that makes the decision that "yeah, that's not right".

[Anonymous Coward]

OK so he's a veep of software development. Will he be allowed to actually do anything or is he there to be fired next time the beancounters screw up.

Still ain't going if it's a Boeing.

[First Light]

His Twitter says "Actor, Engineer, Scientist."

And the website he links to there - jdh.org - ain't working.

Hmmmm.

[Anonymous Coward]

.. so bye bye engineering credentials :).

An auspicious start.

[e_is_real_i_isnt]

The bean counters did not screw up. It was systems engineering architects that are responsible for this.

[Will Godfrey]

Hari Seldon

Revealed that a long, complicated statement of intent from the empire to the foundation essentially said nothing at all.

[John Brown (no body)]

Re: Hari Seldon

That has struck in my mind since I first read it many, many years ago. As a young teen at the time I'd probably say that was and is the core basis of my entire critical thinking process whenever I hear statements or announcements like this, especially if a politician is doing it :-)

[simonlb]

Re: Hari Seldon

Pretty sure it was Salvor Hardin who said it.

[fajensen]

Re: Hari Seldon

That is pretty much the point of "formal japanese":

A language where one can deliver a coherent, gramatically correct speech, that says absolutely nothing at all, offending no-one because it is expected that nothing is going to be said and the intended audience appreciates the effort of the performance rather than the content of it.

[Electronics'R'Us]

Safety Engineering

Boeing needs a complete safety team. That means you need not only software people, but safety engineers; there is such a field and it is common in avionics companies that take safety seriously.

Then systems engineers, ITAs (Independent Technical Authorities) who can be Boeing employees but have a veto over something going out of the door (cannot be overridden - the ITA is the ultimate signatory) and various design authorities. The list is quite long for a proper safety organisation.

Unless and until the Boeing attitude changes (so they become engineering focused, not MBA focused) this is simply putting lipstick on a pig.

[Pascal Monett]

Agreed.

Once upon a time, Boeing was all about safety. Boeing was the definition of safety. Boeing didn't need a software supremo because everyone had safety in mind.

That lasted until the CEO was no longer an engineer.

Like NASA, the beancounters took over and now here we are. No moon base, and no more redundancy in Boeing planes.

Get the beancounters out of decision roles, stat !

[oiseau]

Get the beancounters out of decision roles, stat !

+1

Been saying it for the longest time.*

O.

*Almost cost me my job once, the wig in charge was an acccountant.

[Version 1.0]

You got a downvote? We have accountants reading El Reg? Well I guess that's a good thing.

[martinusher]

The most irritating thing I've found about beancounters is that they always assume that we engineers are just nerds who are only interested in our toys and aren't aware of real world things like costs, manufacturability and reliability. You get ignored or patronized and quite often you end up wasting effort working around their edicts. To an engineer they're just ill-informed and inexpert, people who are so convinced of their primacy that they don't take input unless its what they want to hear.

The tendency to outsource stems from their primacy. Their detachment leads to thinking of the core business as overhead (even though they're often taking the lion's share of renumeration) so they're prey to any slick salesman that comes across with a 'we can do it cheaper' line. (Because they can -- few beancounters know what 'loss leaders' are).

[fajensen]

... always assume that we engineers are just nerds who are only interested in our toys and aren't aware of real world things like costs, manufacturability and reliability.

To be honest, In My Experience, from vascillating between being an engineer and a project manager, engineers (and scientists) are never going to be finished with something on their own volition.

As long as there are some tweak, adjustment, improvement, et cetera, to be made, most engineers will put that at "80% done" and they just won't stop tinkering.

One needs a Lead Engineer, Product Manager or Project Manager with authority to have to go at the requirements, "Does it meet x, y, z, q ... ... well enough?" and yank it from their hands when it does.

Case in point - the "Leave it to the boffins" Corona Response -> Chaos and Dithering. Politicians are needed to take important decisions on an ill-informed basis*, because the experts will *never* do it as long as the possible solution space has not been exchausted yet!

*) And to take the flack for it too. Sadly in the US, UK and SE we have politicians who can't take the flack so they won't make any decisions, letting the experts tinker and fiddle forever as they are wont to do.

[Yet Another Anonymous coward]

Re: Safety Engineering

Various Boeing have crashed because of metal fatigue. They didn't create a minion of metal fatigue.

If software is on the plane it's the vp of engineering's problem. This is just a way of saying - our engineering isn't crap, it was just a software problem and we're fixing that with a new job title

[Cliffwilliams44]

Re: Safety Engineering

Metal fatigue is a problem that should be identified during maintenance. I this not a design/engineering issue. If you built aircraft that never experienced metal fatigue they would be so heavy they would never get off the ground. This is a regular occurrence in military aircraft, which are pushed to their limits, it is checked for after every sortie. I assume it is checked for in commercial aircraft, just not as regularly.

[Grease Monkey]

Yes the software was crap, but that wasn't the root of the issue.

The root of the issue was they they fitted the 737 airframe with new engines the cowls of which were basically too large and really mandated a complete redesign of the airframe. However they didn't do that, they moved the engines forward so that they could be moved upwards for the cowls to clear. This lead to the centre of thrust of the engines being in the wrong place. This in turn meant that the plane could (weasel words time) exhibit undesirable flight characteristics. They decided to try to fix these flight characteristics with a cludge in software.

Having a software supremo would have been unlikely to have prevented this situation.

What Boeing were trying to do here was basically fit new more efficient engines to an airframe that wasn't suited to them. They wanted to do this not only to save money in development but also to save their customers money in aircrew training. A whole new plane would not only have cost more to develop and have taken longer to get to market, but it would have meant customers needing to get their aircrews certified on a new plane. However if they could just sell it as a 737 there would be no need for customers to retrain their aircrews which would make the plane more attractive to existing 737 users.

Basically it was Boeing gaming the system to make money.

What the worlds regulators need to do is send a clear message to Boeing and the FAA by refusing to certify the MAX under whatever name Boeing try to relaunch it as a new iteration of the 737 and define it as a new craft requiring re-certification for aircrews.

[Dinanziame]

Hear ye, hear ye. It should be made clear what are the penalties for trying to bypass security with kludges.

[Claptrap314]

What is the penalty for the negligent homicide of almost 500 souls?

[Anonymous Coward]

A huge bonus to the excecutives for all of the stress they suffereed, then a sacrificial lamb will get a golden parachute into a bigger, cushier job providing more destructive potential, like Dido Harding.

[Pascal Monett]

Boeing gamed the system because it could

It had people at the FAA to muddy the waters and smooth things over, and it used that advantage to keep the FAA from taking a good look at what was going on.

Yes, Boeing cheated, but IMO the FAA has a large share of responsibility in this matter and nobody is talking about that.

If the FAA had done its job properly, it would never have accepted to just wave the MAX through and would have rightly decided that pilots needed recertification.

The FAA does not exist to save customers money, it exists to save people's lives, and it utterly failed in this instance.

[Grease Monkey]

Re: Boeing gamed the system because it could

You are correct.

The FAA allowed Boeing to carry out a lot of it's own certification. The FAA's argument for which is that proper certification would take too long. Which shows that the body in charge of certification doesn't really believe that certification is necessary. Kind of like the DVLA taking your word for it that you can drive and handing you a licence.

Which is why I said that the world's regulators need to send a clear message to Boeing and the FAA.

[John Robson]

Re: Boeing gamed the system because it could

"Kind of like the DVLA taking your word for it that you can drive and handing you a licence."

Which is pretty much what they do. Ok, there is a test when you first start (normally in your late teens), but for the next seventy years you just declare that you can drive.

[Anonymous Coward]

Re: Boeing gamed the system because it could

The FAA is a US publically funded organisation. As such, it is part of the "big government" which US voters dislike. Allowing private companies to succeed or fail in a free market is "the US way".

For US readers: how much are you willing for your taxes to rise to pay for more FAA?

The FAA was set up to fail by underfunding. Nobody is talking about that either.

[Claptrap314]

Re: Boeing gamed the system because it could

So...what's the ROI on the money that we've put into the FAA, especially in this particular instance.

It takes a special kind of ******* to say, "We're not doing our job properly--obviously, we should have more!"

[Anonymous Coward]

Re: Boeing gamed the system because it could

What about the kind of asshole who says "We are supposed to regulate big business, but every time we try to, they ring up their tame Congressmen and Senators who shit all over us until we back down. We are trying to do our job properly - but Government tells us what to do on one hand, and interferes to keep us from doing it on the other."?

The history of the Thalidomide Scandal in the US may provide a parallel. The FDA came under pressure from the manufacturer to approve it, based on the manufacturer's data from trials. The physician responsible for reviewing the drug, Frances Kelsey, refused to approve it, asking for additional data about side-effects. The manufacturer was evasive (and may also have attempted to pressure the FDA to approve anyway). Kelsey was adamant, and shortly afterwards reports of birth defects stopped the drug from being used.

The difference between this and Boeing is that Boeing is American, and has key lobbyists in Government. Mere government employees don't stand a chance.

[Anonymous Coward]

Re: Boeing gamed the system because it could

And another thing. You just had an election. How many of the Congressmembers and Senators who just got re-elected have lobbied for Boeing against the FAA in the past? How about the ones who lost?

Boeing gamed the system because they could. And your elected representatives will continue to help them because there are no consequences to it. Occasionally, this state of affairs will kill people.

[Cliffwilliams44]

Re: Boeing gamed the system because it could

Typical European misconception!

We have no issues with agencies like the FAA, OSHA, etc. These agencies keep people alive in industries that are inherently dangerous! Aviation, construction, etc. What we have a problem with is run-a-muck agencies like the EPA, Labor, etc that are always used as a political weapon by a particular Party in this country.

[Caver_Dave]

Re: Boeing gamed the system because it could

There is a matrix that defines just how much attention the FAA takes and how much they leave to the company DERs. If you are very experienced, have well defined processes and a good track record, then the FAA monitoring may be as little as reading two documents, the PSAC and the SAS. (For newbies it could be in the 20's of documents, through out the project.)

I think that Boeing will be moving to another part of the matrix after this, as internal processes have been borked by some method to allow this bad solution through, and so they are just not as trustworthy.

[hoola]

Re: Boeing gamed the system because it could

But they were strenuously trying to bury the problems by blaming the airlines, training, pilots, anything but themselves. If these crashes had happened in Europe or the US then the authorities would have been screaming much louder and much sooner. One would think that the view taken was that lives of non-Americans are worth less.........

This has uncovered all sorts of endemic cultural issues at Boeing and the FAA but I rather suspect that after some furious activity claiming that all is squeaky-clean, both parties will be back to the old ways again. It needs someone like the European regulator to stick two fingers up and require the Max and any subsequent revisions Boeing makes to any models be completely re-certified as new aircraft.

Both the FAA and Boeing and proved they cannot be trusted therefore can you actually trust any of the certifications?

[Anonymous Coward]

Airlines

asked for it. There is a primary cause for the development of that abberation. Airbus considerations on the side of Boink helped also.

[oiseau]

What the worlds regulators need to do is ...

What the worlds regulators should have done is ...

There you go.

Q: Is it possible to imagine that the FAA and the rest of the world's regulating bodies did not know or at least suspect what was going on by just looking at the damn bird?

A: Nah!

It was Boeing, so they let it pass.

O.

[Cynic_999]

"

The root of the issue was they they fitted the 737 airframe with new engines the cowls of which were basically too large and really mandated a complete redesign of the airframe.

"

Not sure whether a " complete redesign of the airframe" would have been necessary. Could probably have managed with a redesign of the undercarriage (& retraction mechanism) to give it more ground clearance.

[EnviableOne]

still would have needed a new type approval, and full pilot re-certification, which is what they were trying to avoid.

[Cynic_999]

ISTM that the type approval would be no more than needed for the engine re-positioning. Not sure whether pilots would have to be re-certified for a slightly longer undercarriage - but it would not require major retraining if all flight characteristics remained the same. In fact it would make a tail-strike less likely and so give greater latitude during takeoff rotation and landing flare.

[Morten Bjoernsvik]

Blame Airbus

>still would have needed a new type approval, and full pilot re-certification, which is what they were trying >to avoid.

It was the Airbus A321Neo that stole all their orders. The desperately needed an airframe with more efficient engines, more seats and had no time for a full FAA classification and costly pilot certification.

If it had worked, it would have been hailed as a "stroke of genius".

[Anonymous Coward]

You can't just use longer landing gear.

Apparently redesign of the undercarriage wasn't an option (for engineering reasons).

Another clear reason for not raising the height of the fuselage when on the ground is that airports would have needed new sets of steps for the passengers. Which would have cost them money. This money would have been extracted from the airlines, making the 737MAX less attractive as a purchase option. A key market for the MAX is budget airlines, who use steps rather than jetways as it's cheaper.

Remember that the point of these was that they were a drop-in replacement for existing 737s. In Europe the biggest budget airline, Ryanair had planned to mix them in with their existing fleet across all routes, rather than restricting them to certain airports. Making them more expensive to fly would make them less attractive to buy.

[Grease Monkey]

"Not sure whether a " complete redesign of the airframe" would have been necessary. Could probably have managed with a redesign of the undercarriage (& retraction mechanism) to give it more ground clearance."

You'd think that a longer undercart would allow the engines to be mounted in a more sensible location, however said longer undercart would almost certainly have mandated a redesign of the airframe. And we're back where we started.

[Nifty]

Aren't most modern planes only flyable by wire?

[Electronics'R'Us]

Fly by Wire

Aren't most modern planes only flyable by wire?

Many modern aircraft are indeed fly by wire with a reversionary mode (particularly Boeing aircraft that were designed specifically for fly by wire systems).

Fly by wire systems are clearly flight safety critical and therefore get a great deal of scrutiny from a safety perspective at multiple levels (*). Boeing and Airbus (as well as Bombardier and Embrear) do not design or manufacture the fly by wire electronics or actuators; those are specified by the airframer to a suitable subcontractor as these are rather specialised designs.

The B777 (the original) is a fly by wire design and the electronics has a triplex architecture - 3 independent computers (and that means electrically isolated as well so a major defect in any one cannot bring the others down with it) along with 3 independent sets of sensors.

The problem with the 737MAX is that the aircraft has very little automation and MCAS was a bolt-on driven by certain rules (Part 25 stick force, primarily) but it was clearly never given the scrutiny that a fly by wire system should get.

* Boeing as an engineering company does not seem to exist any more, particularly since the McDonnell Douglas 'merger' (more like a reverse takeover) so in a very sad way it is not really surprising to see a total lack of ethics and scrutiny in the search for ever more profit regardless of the safety of the aircraft.

The fly by wire systems I am familiar with (which includes the B777) have very strict failure requirements, and those same failure requirements should have been applied to MCAS. Provided it can still control the flying surface then even a duplex system (single redundancy sensor and computers) cannot be considered safe unless it gives back complete control to the pilot in the event of a disagreement from the sensors and simply stays dormant in that situation.

The pilot who would need to be trained for such an event, clearly.

[G.Y.]

838

If they named it the 838, and certified it as a new plane, it would not have crashed --at lest not the way it did

[Anonymous Coward]

Not so. Airbus would have similar changes to their software to accommodate the same engine size change on the Neo, but Airbus is not subject to the same "stick-force" rule because Airbus has no force feedback to the pilots per the aerodynamic forces on the plane. MCAS is solely for stick-force regulations. The air frame is perfectly well suited to the engines. In a world that can see the GeeBee R1 fly, the 737 MAX is fine.

It was the airlines refusing to pay for training, gaming the system to save themselves money. The airlines had specified "no training" as basic requirement.

The 737 MAX center of thrust is more in line with the fuselage, producing lower pitch up force due to thrust. The engines are also farther forward, which is often an advantage to stability, but they are larger in diameter and interact with the wing a bit more, so they produce an increment of lift at high AoA, which a small increment of trim from MCAS was supposed to handle.

A software supremo means that software developers have someone to back them when the systems engineering is incomplete and a good one would have avoided this situation by determining what the systems architecture was designed to handle.

It's a pressurized tube with wings, a tail, engines, and landing gear. What way could being completely redesigned not add more potential problems?

[Anonymous Coward]

Anyone ever see a rocket crash, or a self driving car?

https://www.cnbc.com/2015/07/20/musk-this-is-what-caused-the-spacex-launch-failure.html

“That’s an unfortunate thing because we could have saved Dragon [supply ship] if we had the right software there,”.

https://qz.com/1701070/spacex-missed-a-satellite-collision-warning/

"missed a potential collision warning because buggy software did not relay an alert"

https://text.npr.org/809207519

"allows its Autopilot system to be used on roadways that the software is not designed to handle, creating safety risks"

[Anonymous Coward]

Dido Harding MkII?

[Paul Herber]

Type approval invalidated.

[Anonymous Coward]

There aren't many people round here who approve of Harding's type in the first place.

[Anonymous Coward]

"...will lead a new, centralized organization of engineers who currently support the development and delivery of software embedded in Boeing's products and services"

That's an interesting move to take. It can end up being a double edged sword. Creating what amounts to an internal outsourcing organisation for your software engineering function, from which your other departments procure engineering services, can add a huge amount of operational friction and organisational politicking. The counter-argument is normally that the best people to tell experts how to best do their jobs are their fellow experts, which is generally fair, but if that new organisation is not sufficiently empowered to actually make meaningful decisions then they end up in an even weaker position - both unable to steer the project (because they have no corporate-level say) and unable to steer their own output (because they're 'just a supplier')

The fact that Jinnah is coming in with a job title of "only" VP indicates to external observers there is little interest in empowering him to Say No To Other Departments, and at best he'll be a peer of the other corporate project/department leaders. If they were really serious about making a centralised group of empowered, effective and world-class engineers we'd be seeing him rocking an SVP or GM title and heading up a genuine, dedicated subsidiary or global service line with a remit to overhaul organisational processes and culture from the root. Without that it's hard to conclude this is anything other than window dressing.

[Strahd Ivarius]

What do you expect?

In any case the actual work will be outsourced to India, to people who don't know what the system is intended for (rocket, plane, brick, ...) and therefore won't be able to anticipate any glaring mistake in the requirements (like A requires B, B requires C, C requires D, but you can't have A and D at the same time - real story, took 2 weeks to the client to decide what to do).

[Anonymous Coward]

Maybe the FAA is getting a spine......

......because the Boeing 787 is now ALSO getting more attention. The South Carolina plant has been shipping 787 planes out the factory door with factory ladders lying around in various inaccessible parts of the airframe!

*

And the FAA may also be reconsidering all the SELF-CERTIFICATION it as been allowing....where Boeing personnel have been signing off THEIR OWN WORK ON BEHALF OF..........THE FAA!

*

Even so, FAA spine or FAA no spine, I won't be flying in the new 737 Max (or whatever they call it today).....ever.

[khjohansen]

SpaceX / Tesla

Wasn't "trying to make it land on its tail" the problem with the 737MAX ??

[Anonymous Coward]

Re: SpaceX / Tesla

@khjohansen

Yous sick bastard. Hundreds of people died and you try to make a joke out of it?

Fuck off

Cheers… Ishy

[StrangerHereMyself]

Does it really matter?

Does it really matter if the top-level execs are only focused on profitability and their bonuses and not on engineering the best product possible and making profit as a sideshow?

He'll get down-voted every time he tries to improve quality by hiring expensive American software engineers instead of cheap Indian bods. My guess is he'll quit in a couple of years because of all the obstruction.

[A random security guy]

Re: Does it really matter?

I give him a year. That is when some of his bonus comes due, probably.

[A random security guy]

Re: Does it really matter?

Good engineers out of India cost almost the same as the engineers you hire here. They know their value.

[heyrick]

Interesting spin

I perceive the software problem to be a symptom, not the cause.

It's nice they want somebody to oversee software development (I guess tacitly admitting they didn't have anybody before), but really it would be better to cure the disease rather than whacking on a sticky plaster and thinking all is well.

[Dave314159ggggdffsdds]

Re: Interesting spin

The cause was the software issue combined with poorly trained pilots. Neither caused planes to crash on its own. Properly trained and experienced crews didn't panic, and solved the problem without crashing.

Boeing can only fix their software approval process. It appears their software engineers had insufficient clout to get their (necessary) way. Boeing have added a veep to go in to bat for the software side of things.

Ultimately, new plane designs crash, although less frequently than they used to, because we're still discovering new failure modes.

[Electronics'R'Us]

Re: Interesting spin

Ultimately, the root cause of the problem was terrible management overriding engineering which yielded, quite predictably, a terribly engineered aircraft.

A computer that has control of a moveable flying surface without pilot intervention that relies on a single sensor known for its unreliability and that is using that sensor information only and that was not hardened against single event upsets.

Really - what about true air speed, altitude above ground, rate of climb and dive (independent of the AoA sensor) which could all feed into this but even then with everything relying on a single computer the design is provably unsafe.

[Claptrap314]

Re: Interesting spin

Demonstrably, as well...

[UncleZoot]

Re: Interesting spin

At the time of design or entry into service, the angle of attach sensor wasn't a known issue.

What no one has asked or disclosed is who is making the sensors? Name and shame.

When I worked in aircraft parts manufacturing, I/we tracked defects. If I saw more than 2 or 3 parts or operations with the same defect, I was out looking for the root cause.

[elkster88]

Re: Interesting spin

"What no one has asked or disclosed is who is making the [AOA] sensors? Name and shame."

Angle Of Attack sensors are reliable, up to the point that they suffer a bird strike or other FOD. Which is why all A/C have two or three of them. Only using one of them as a basis for actuating MCAS, or not adding a third sensor, is another discussion altogether.

[Mystic Megabyte]

Re: Interesting spin @ElectronicsRus

That's an interesting link, thanks.

Have a look at Spaceweather.com and scroll down to "Cosmic Rays in the Atmosphere".

[First Light]

Re: Interesting spin

So why were the very properly trained American Airlines pilots union so very very pissed off with Boeing when they got a chance to sit down with them? Because they knew it was a BS move.

https://www.theguardian.com/business/2019/may/23/boeing-737-max-crashes-american-airlines-pilots-union-mcas

[A random security guy]

Re: Interesting spin

The US pilots are on record, anonymously, as well as through direct reports, of complaining vigorously about the plane. It took just the right sequence of conditions to cause the crash. The pilots were trained to handle the 737, not an elephant that suddenly starts bucking like a horse.

[Cynic_999]

Re: Interesting spin

The reason that the pilots of some airlines were not trained on that system was because *Boeing did not tell them it existed* If the aircraft does something totally unexpected that you have no idea what is causing it, then no pilot will know what to do.

[Tim99]

And...

How much of that software will be outsourced to an external contractor?

[Caver_Dave]

Re: And...

That in itself is not an issue. You have to have independence in the checks and balances, and outsourcing part of the work is a perfectly acceptable way of achieving it.

e.g. the people that write the specs, are not the people who review them, or who approve them.

the tests specifications are written from the system requirements by a different team that wrote the software specifications. etc.

That outsourcing may be to a separate part of the same company or a separate company, so long as the reviewing and approval is independent and by suitably qualified and experienced people it doesn't really mater who is doing the work, internal or external.

[A random security guy]

Re: And...

They were outsourcing to a company, Cyient, at $7/hr. I have hired engineers in India and never have I met anyone at that low rate of pay.

[Robert Grant]

Is this "centralising quality" (yuck) or actual in-housing engineering with a really strong culture of high performance (yay)?

[Caver_Dave]

Quality should always be independent of development.

VP or SVP should be the first level that both trees report to the same person.

[EnviableOne]

Scapegoat

my first thought, here is someone paid to get canned next time something goes wrong.....

[xyz]

Waft of a Dilbert cartoon here?

No further comment

[tundish]

Re: Waft of a Dilbert cartoon here?

Historians may marvel how Scott Adams' wisdom and insight still failed to penetrate the managerial class despite being delivered in the form of cartoons an eight-year-old could understand.

[Paul Hovnanian]

Bingo!

“Jinnah will be charged with defining and leading Boeing's strategy for software engineering, which includes providing capabilities, technologies, processes and secure and accurate systems to meet the needs of all our customers across the entire product life cycle."

[Steve K]

Re: Bingo!

Sorry Paul, they missed out “Synergy” so not quite a full card there...

[Alan Johnson]

It was not a software problem

Boeing may have poor software quality but the software involved in the Boeing 737 Max crashes did what it was specified to do. The problems were in the overall system design, safety analysis and change control with changes that had a large impact on safety being made without changing the safety analysis.

The failures were not the result of subtle difficult to predict problems but issues that should have been identified and addressed. Given this it would not be suprising if Boeing had issues in their software development process as there seemed to be systemic problems with respect to their safety engineering processes but software failures were not implicated in the crashes.

[Potemkine!]

Re: It was not a software problem

the software involved in the Boeing 737 Max crashes did what it was specified to do

You mean the software was specified to crash the planes if it got irrelevant information?

[Locomotion69]

Re: It was not a software problem

No sir,

The problem was that the software was relying on flawed information from a defective AoA, the pilots not being aware of the presence of this particular software feature in the first place, and therefore not thinking of the option to just switch it off (button was provided, yet not documented anywhere).

[arachnoid2]

the deaths of 346 people who flew on the 737 Max

So when can we expect the murder charge and who caries the can for it?

[six_tymes]

and the big picture is, the more complicated systems rely on software, the more shit will be crap.

[Claptrap314]

Software engineering vs "a software problem"

Software engineers do NOT "write code to spec". Software engineers "engage with the stakeholders" until the spec is not s***. I don't need ANY training in aerospace engineering to understand the problems with code that relies on a single input. If I were writing software for airplanes, I would have more than just a little bit of knowledge of aerospace engineering, and in a hurry.

One of the critical failure points was when whomever was writing the software accepted that s*** spec.

[s. pam]

Still dangerous at any altitude

If it's Boeing I ain't going this will not resolve nor address. Fundamentally the loading area Centre of Gravity of the wings and engines extended forward shift the pivot point of the plane. Software can not fix the laws of aerodynamics Boeing has violated with the redesign.

Boeing tried to ignore:

1) The rules of aerodynamics / gravity

2) Software cannot fix hardware faults

3) Not having an honest process and going after the money

The first point is not resolvable on the 737DEAD^ahem MAX in any form

The second point is a costly immutable fact, ask Airbus

The last point is whomever is foolish enough to take this role is a likely sacrificial lamb

[jtaylor]

Re: Still dangerous at any altitude

Fundamentally the loading area Centre of Gravity of the wings and engines extended forward shift the pivot point of the plane.

Loading area of the wings? You mean wing loading, which is how much pressure per surface area of the wing. As in, 10 tonnes spread over 10 square meters = 1 ton per square meter. Give it 20 square meters and you have half the wing loading. Can you relate this to the topic?

Centre of gravity...of the wings? Do you mean Centre of Lift, as in the balance point of the upward forces in flight? Compared with the Centre of Gravity (mass)? I promise you, the Centre of Lift is quite close to the Centre of Gravity of this plane in cruise, give or take some "trim." As it is with all planes that fly.

I guess you wanted to say something about Centre of Thrust, combined with moment arms, and then bring in auto-trim logic, tail-strike protection, thrust levels, and such. There's a lot of great information available if you want to learn. (FLCH isn't just a dirty word.)

Software can not fix the laws of aerodynamics Boeing has violated with the redesign....Boeing tried to ignore: 1) The rules of aerodynamics / gravity

Can you tell us which "law of aerodynamics" Boeing violated?

2) Software cannot fix hardware faults

It cannot fix faults, but control logic can assist with them. Look at how cars manage fuel mixture and spark timing advance. Or how anti-lock brakes work.

3) Not having an honest process and going after the money....The second point is a costly immutable fact, ask Airbus

Yeah, Boeing really had no shame. How does Airbus come in to this?

[Anonymous Coward]

"veteran of similar gigs at SpaceX, Tesla, and Google."

Oh, and also Aurora. So either this guy has a magic touch, and quality instantly improves, or he's a typical C level ladder climber who hops around every four years?

Establishing a culture of quality or safety takes a long time. Establishing it when there are entrenched bad habits at a massive company like Boeing takes even longer.

The hiring is a good first step, but there's much more work to do.

[A random security guy]

He hasn’t worked at a company like Boeing. His challenges will be different. I doubt heading Boeing software is a step up. His compensation package can’t be even close to a typical startup going IPO.

I think he genuinely liked the challenge. Turning around a large organization requires a different set of skills.

I do really wish him success.

[Big Softie]

...or he's one of those smoke & mirrors guys who only last until the people who work for him twig their new manager is completely inept, wonder how on earth he got hired in the first place, but unfortunately by their own heroic efforts hide the awful reality until he moves on (again)...

[bazza]

Er, Hang On A Mo

I'm not so sure I'd want that post. Guess where the buck will be passed to at any time in the future when software is thought to have played a role in a crash, even if the true cause lies elsewhere...

[David Roberts]

Optional Extra?

Nobody so far seems to have mentioned that the instrument to warn the pilot that there were potential issues with the sensor was an optional extra.

[A random security guy]

How long will he last?

He has experience only with startups and very dynamic companies. I really wish him luck. He is going to be running into entrenched Boeing managers.

He will be running into $7/hr outsourcing to cyient. I have hired engineers out of India and you can’t even get a new college graduate working for those wages.

Why do I wish him luck?

I had to turn around an engineering team of 220 engineers who wrote code that met the requirements but no one could figure out how it worked. When it worked. The politics was tough. Good training for me.

It took me 2 years to get them to write good code. It was my toughest job ever.

Fast forward to a several billion dollar company. Pitched quality battles from 5AM to 8 PM. Multiple entrenched groups, cultures, sensitivities, regions. Just managers doing everything possible not to meet the spirit of software quality as their KPIs were tied to shipping their products on time.

I inherited the mantle from my VP who got fired for doing the right thing. I left roughly after a year. I can claim some success but I was a burnt out shell for a year.

[Potemkine!]

Externalisation?

I know a lot of big companies whose accountants found smart to externalise to the max everything related to IT. The consequence is those companies lost all of the embedded knowledge, and relied on underpaid and overloaded staff of technicians and engineers leased by meat-sellers like Cap Gemini, HCL and al. And now they wonder why IT works badly, with security and privacy breaches, bad software, people angry and/or desperate about IT support...

Is Boeing one of those?

[Anonymous Coward]

It's the culture that matters

Disclaimer: no aerospace experience, but I have sadly worked for a big US corporation.

https://www.nytimes.com/2020/01/10/business/boeing-737-employees-messages.html

Adding more top brass won't fix the problem if the company culture is borked.

[A random security guy]

Re: It's the culture that matters

Uber is a prime example. It took 3 years to fix the problems. Well, sort-of. Change in culture has to come from the top. Will the guy be neutered.

[UncleZoot]

Too bad that they outsourced the software flight control software to a company that didn't speak English as their first language.

Just another seven figure position created to place a bandage on being politically correct.

[Anonymous Coward]

Political correctness is irrelevant. Decisions like that are about money. Nothing else.

[A random security guy]

Language: The Germans, French, and Italians have been working with us for decades. The Germans and Italians since 1945, and the French in fits :) We and the English speak a different language but we have figured out how to communicate sometimes.

This is just hiring incompetent people (will not call them engineers) to write software for $7/hr. There are good engineers in India and here, some even run large tech companies. But you get what you pay for. Currently, a decent DBA costs $120K or so. Fully loaded costs (the company's costs) would be roughly $150K. Yeah, the good ones know their value. I still can't figure out how Cyient gets away with it.

Yeah, the guy now has to figure out if he has any power to make changes. Does he have the power to block a release? Does he have to power to force a group to adopt good software engineering practices? Will he even be invited to participate? Will the entrenched managers work around this guy and ship out software around his back?

[Anonymous Coward]

Anybody else waiting for Trump to tweet that someone with an arabic sounding name like "Hossein" shouldn't be in charge of aircraft software.... because.