back to article You're not Boeing to believe this: Yet another show-stopping software bug found in ill-fated 737 Max airplanes

Boeing today said another software flaw has been spotted in its star-crossed 737 Max. The bug was found during an audit of the passenger jet's on-board technology, held last weekend with America's aviation regulator. These technical reviews are expected to turn up glitches and gremlins for Boeing engineers to fix, so this is …

  1. SuperGeek

    "Our highest priority is ensuring the 737 Max is safe and meets all regulatory requirements if it ever returns to service."

    There, fixed that :)

    1. Anonymous Coward
      Anonymous Coward

      Profits

      Except that the assertion is wholly untrue. Their highest priority is to maximize profits.

      In order to maximize profits, they will attend to quality and safety issues if and when the publicity arising from those threatens profits.

      1. Tom Chiverton 1

        Re: Profits

        Also, as a US public company, their legal responsibility is to maximise shareholder value !

      2. werdsmith Silver badge

        Re: Profits

        It is the safety that will lead to profits, because aircraft that are demonstrated to be unsafe are very unprofitable. Safety is what Boeing should be selling.

      3. bombastic bob Silver badge
        Facepalm

        Re: Profits

        Except that the assertion is wholly untrue. Their highest priority is to maximize profits.

        this is to be expected. keep mind that airlines don't purchase unsafe planes... [which affects profitability]

        In order to maximize profits, they will attend to quality and safety issues if and when the publicity arising from those threatens profits.

        No. See what I said above. it's *impractical* (and unprofitable) to be so heartless. Dead customers aren't repeat customers, after all... (nor are they satisfied).

        1. John Jennings

          Re: Profits

          You seem to confuse boeings customers. Its the Governments and Airlines that buy its stuff, for transport of goods (including human chattel) or to supply weapons.

          Travelers are most definitely not Boeings customers.

          Airlines, when they can go down the route of single supplier to simplify their training and supply lines - to save money - Southwest for example.

          Sure, you could argue that 'people wont fly them' .... They will to save a few dollars or because they cant (or wont) take the time to find alternative routes to X on alternate carriers.

    2. Chris G

      My opinion is that far from being an upgrade that requires some additional training and a bit of software, the fitting of different engines that have significantly changed the flight characteristics to the point that they are dangerous without that software and training, has put the aircraft outside the usually acceptable envelope for a civilian airliner. This is not a modification it is a type change.

      Passenger aircraft should be inherently safe and stable to fly, not optimised for profit and reduced spending to keep up with the competition. In relative terms this has been a cheap fix to avoid falling behind Airbus in sales.

      I can't begin to imagine how much thought is going into arse covering and how many brown envelopes are changing hands.

      1. niio

        Whether or not new engines are a Type change, the problem is that the FAA trusted Boeing to self verify the quality of the change. Boeing took a short cut that literally blew up. They could do the same thing with all the additional requirements for a Type change and the product would be no better off. The extra work unrelated to the engines may pull resources off the more critical things that actually changed.

        I don't think the objective should be to punish Boeing; the stock market, it's customers and all the people who fly are already doing this. The objective should be to fix the problem, defined both as the software flaws that exist and also as Boeing's penchant for cheating. The FAA is now crawling through every line of code for this very reason, and is not going to allow Boeing much self verification going forward. If the review takes another year, so be it.

        1. chivo243 Silver badge
          Facepalm

          FAA trusted Boeing to self verify

          Trusting the butcher to check his own meat... Trusting the pusher to make sure his product is safe...

          1. Anonymous Coward
            Anonymous Coward

            Re: FAA trusted Boeing to self verify

            "Trusting the butcher..." isn't a bad analogy. Late last year the U.S. Government privatised the inspectors in pork abattoirs. Who would you trust to ensure that the carcasses being processed weren't diseased? An independent inspector paid by the Government, or one paid by the abattoir?

            I'm sure we in the UK will be protected by our trade deal from any drop in quality...

            1. Anonymous Coward
              Anonymous Coward

              Re: FAA trusted Boeing to self verify

              See the banks. They had paid for auditors. Turns out the auditors always passed them.

            2. not.known@this.address

              Re: FAA trusted Boeing to self verify

              "I'm sure we in the UK will be protected by our trade deal from any drop in quality..."

              The trade deal shouldn't matter - unless you really think safety is something they should be negotiating separately on, and isn't that what seems to have happened with the MCAS and the whole "failure warning as a chargable optional extra" fiasco?

              Besides, it doesn't matter how well the UK tries to do if the EU is determined to screw us over - as I've pointed out before in other topics on The Register, *both* sides need to negotiate in good faith; if one side says "this is what you're getting" over and over again and refuse to make any changes at all then that's NOT negotiating. And given that the politicians and bureaucrats in the European Parliament cannot let us be seen to succeed in getting away from them lest others try to follow, the chances of them "negotiating" any more fairly this time around don't look too hopeful...

              1. Anonymous Coward
                Anonymous Coward

                Re: FAA trusted Boeing to self verify

                They were talking about the putative trade deal with the US, not the one with the EU.

                1. 's water music
                  Trollface

                  Re: FAA trusted Boeing to self verify

                  ...something about bells...

              2. cynic56

                Re: FAA trusted Boeing to self verify

                I tried a downvote, but that wasn't enough. You're an m*r*n with a distorted view of reality.

    3. spold Silver badge

      No the priority is driven by three sets of objectives.

      1. Corporate - largely based on maximising shareholder value. Board of Directors zone directing management..

      2. Personal - maximising your paycheque and pension. Job satisfaction optional.

      3. Safety.

      Until you align 1 .and 2. with 3. then the first two (one?) dictate your priorities.

      The only levers applicable are the metrics you put in place and incent people on. What you measure and rate people on is what they do (realistically you have to ensure those in 1. and 2. support 3.). The current case demonstrates that a failure to properly incent 3. will lead to a big hit on 1. Ultimately also affecting 2. for some of the management.

      Too complex? Join me at the pub and wait for all of this to blow over - it's safer than getting on a 737 Max (well, probably not since they are all in the airport long-term parking lot).

      1. Lotaresco

        "Corporate - largely based on maximising shareholder value."

        If only that were true. Corporate is largely based on telling enormous lies and rearranging deckchairs on the Titanic in order to massage the bonuses of the senior management team. It does not maximise shareholder value to tie companies down with unsustainable levels of debt. Yet this is a standard tactic for CEOs looking to acquire businesses to give a false impression of growth. See "Fred the Shred" for example.

    4. whitepines

      What happens if it doesn't return to service? Do the airlines that bought it and received it get to sue Boeing to recover at least what they paid?

      Genuinely curious.

      1. spold Silver badge

        They can easily be resold....just needs a bit of marketing and a crayon.

        ....the new 737 CliMAX - the apex of our technology, all for your pleasure.

        Icarus altitude assistance software allows you to reach new heights (<tinyprint footnote> to avoid sudden performance degredation do not fly in near proximity to the sun).

      2. Blank Reg

        "What happens if it doesn't return to service?"

        Well, they could put all that aluminum to good use making beer cans.

        1. Stuart 22

          "Well, they could put all that aluminum to good use making beer cans"

          Aren't they mostly composites these days (aka single use plastics?). But then the fuselage might make a very fine fat straw for a very fat cat.

      3. localzuk Silver badge

        If they don't return to service? Boeing goes bust.

      4. John Sturdy
        Boffin

        Rebuild as a normal 737?

        If the rest of the aircraft hasn't changed significantly, perhaps the engines could be replaced by one of the types they previously used, thus converting them to non-MAX 737s. Probably expensive, but better than having them sit unused or scrapping them.

  2. sanmigueelbeer
    Mushroom

    The audit has to be completed before the FAA can take the updated 737 Max on a test flight, and ultimately certify it to fly again with passengers

    I nominate the entire Boeing board and the upper management echelon of the FAA to be on this so-called "test flight" -- y'know, just to be sure.

    1. Anonymous Coward
      Anonymous Coward

      Add the Boeing shareholders and you’ve got yourself a deal.

    2. Stuart 22

      "I nominate the entire Boeing board and the upper management echelon of the FAA to be on this so-called "test flight""

      I remember possibly an apocryphal story before the certification of the Boeing 747. A problem was sudden flame-outs on the P&W engines which wasn't being addressed. The CEO of Boeing introduced the issue with the CEO of P&W on a test flight while they went through the procedure to induce one - well four on a 747 but luckily they all restarted. It got sorted. Those were the days when the 737 was already in widespread service and in 53 years Boeing had progressed from this: https://en.wikipedia.org/wiki/Boeing_247

      Imagine if they had just strapped a couple of GE Leaps under the wing and a prayer of a 247 Max instead of going to all that trouble and expense of a type change ;-)

      1. A Non e-mouse Silver badge

        I saw that anecdote talked about on a BBC4 documentary about the history of the 747.

  3. nautica Silver badge
    Holmes

    Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

    "‬When someone builds a bridge,‭ ‬he uses engineers who have been certified as knowing what they are doing.‭ ‬Yet when someone builds you a software program,‭ ‬he has no similar certification,‭ ‬even though your safety may be just as dependent upon that software working as it is upon the bridge supporting your weight.‭"‬--David L.‭ ‬Parnas

    "There are no standards for computer programmers and no group to certify them."--David L. Parnas

    David L. Parnas penned these gems WAY before the 737 MAX debacle. He was one of the most vocal--and most effective--critics of Reagan's "Starwars" SDI boondoggle, and is credited with almost single-handedly shutting down the program by arguing, with surgical precision that, "...no, you can NOT solve any and all problems with software..."; certainly not with software which was written the way it was back then.

    Spoiler alert, folks: the writing of software has only gotten worse since then...much worse.

    1. yoganmahew

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      Indeed.

      This:

      " These technical reviews are expected to turn up glitches and gremlins for Boeing engineers to fix, so this is kinda to be expected."

      is just wrong. Very, very, wrong.

      The FAA are not the PO, they're not there to do a demo to. It suggests Boeing don't know how to code safety critical systems.

      1. A.P. Veening Silver badge

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        It suggests demonstrates once again Boeing don't know how to code safety critical systems.

        FTFY.

        1. bazza Silver badge

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          It’s a bit more complex than that I’m afraid. Initially Boeing contracted out the coding job to another company, but effectively they’d already designed the software architecture and functional breakdown, so all the contractors were doing was writing the functions like they’d been told.

          And apparently the contractors coders were wise enough to question the overall architecture and functionality, pushed back questions to Boeing asking “are you sure?”.

          Boeing seems to have said yes, now get on with it...

          What we don’t know now is who has been making the decisions, Boeing or the subcontractor. My bet is that it’s still Boeing. The subcontractor company is well known and respectable, and I wouldn’t mind betting that they’re not far off walking out of the job before this gets ridiculous.

        2. Mike 137 Silver badge

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          "It suggests demonstrates once again Boeing don't know how to code safety critical systems."

          Practically speaking, almost nobody knows how to program (not just "code") safety critical systems, and those few that do are extremely expensive and usually fully booked up.

          1. Anonymous Coward
            Anonymous Coward

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            I work on most DO-178C artefacts. But it's different sections of different projects to maintain the required independence between the defined roles (I'm just completing a Software Verification Results document, parts of which are to prove the correct independence and oversights of a particular project, I was providing technical oversight on another project meting this morning and I will be coding to LLR on another project later.)

            Yes, there are those of us who truly know how to produce safety critical code. My teams are spread thinly and globally and I never expect to physically meet most of them. We are all 'on the rivet' and would love to encourage more people to join the profession. Profession, however, is a key term, chartered, is another. (I am CITP)

            Until Software Engineering requires professional qualifications to practice, then it will never get the same respect as the professions where people have to become chartered to work, and have to complete continuous professional development to maintain that status.

      2. Anonymous Coward
        Anonymous Coward

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        Quote: ..."These technical reviews are expected to turn up glitches and gremlins.."

        *

        Hey...way back in the dark ages of software development, I was taught that "quality" in a sofware product started with a robust DESIGN....long before anyone started to write code. If the DESIGN could not survive a detailed review, then the DESIGN needed some help.

        *

        The article also comments that two computers, formerly working independently, are now "connected together".....and something about this process didn't quite work out. Harking back to a successful safety DESIGN, I seem to recall that the Space Shuttle had a multi-computer control system, where THREE OUT OF FIVE computers needed to AGREE, before a control decision was implemented. Oh...and the five computers involved in the voting were all CODED by independent teams to the same DESIGN.

        *

        Haven't Boeing learned anything about quality and safety?

        *

        But then a wise man once pointed out that another successful company (the family of the Bourbon kings) "had forgotten nothing....and learned nothing".

        1. bazza Silver badge

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          The whole “strap the 2 FCCs together and suddenly there’s a duplex safety critical system” certainly does feel like the bodge of the century.

          It remains to be seen whether this approach will find favour with regulators. Outside the USA - probably not. It’s going to be a tough sell; Boeing’s more modern products have a conventional triplex FCC setup, and they’re going to have to explain why the 737 NAX can be different.

          1. tygrus.au

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            If there's 1 sensor or 1 computer or any other 'single' in the system then you have no redundancy and no guarantee of reliability.

            If you duplicate sensor/computer/network then you add redundancy but how do you determine which is correct and reliable with only 2?

            With triple redundancy you assume only 1 will fail at a time so the other two agree and you assume the third has failed and to be ignored. Hopefully only 1 fails at a time. The other problem with triple redundancy is if they continue flying with 1 broken because they think 2 is enough.

            Can GPS and other sensors be used to model the suspect flight parameters and verify it's suspect value? Sensor says I'm pitching up with a stalling speed but GPS and airspeed (nose) says I'm level flight (also cup of water is level flat in plastic cup) with sufficient speed that stalling is very unlikely. Auto-pilot/MCAS panic button = alert authorities of rouge pilot in the hope that it's a computer fault not pilot trying to crash the plane. Is it better to rely on a person or a computer? If in doubt, let the pilot take control who has responsibility for the plane. Warn and help the pilot but don't let the sensor/computer faults crash planes.

            1. bazza Silver badge

              Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

              Boeing seem to be aiming for a duplex system. This does at least have the ability to recognise something is wrong, even if it can’t work out what’s right...

              The contentious point is that the new MCAS will be hair-triggered to switch itself off in the event of a detectable fault, the problem being that the pilot is then left flying the aircraft’s inherent native aerodynamics. If the aircraft aerodynamics are still legal without MCAS, that’s ok but why bother with MCAS in the first place? Otherwise the aircraft is legally unflyable without MCAS, in which case Boeing’s plan is unacceptable.

              Another issue is that a duplex system has a residual risk of not identifying a fault due to both channels failing in the same way. In the new MCAS, two duff AoA sensors going wrong in the same way (eg put together badly, as was the case with the Lion Air AoA sensor that was sold by a dodgy US referb outfit, now closed down), won’t be detected as being at fault. I think EASA consider that unacceptable all by itself.

              Synthetic airspeed, attitude likely isn’t acceptable for managing the MCAS function, especially if as EASA suspects it is a stall prevention system (ie the native aerodynamics are not legal).

              1. jtaylor

                Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

                "If the aircraft aerodynamics are still legal without MCAS, that’s ok but why bother with MCAS in the first place?"

                Cost. Airlines put a high value on not having to train for the new model. MCAS reduced training costs to just a short differences course that is quick and doesn't require either sim time or check ride.

                Remove MCAS and you have to do more training. Of course, if you have to train to deal with MCAS failures, maybe that bridge has already been crossed.

          2. anothercynic Silver badge

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            "If we were to go to triplex, it would require a new type certificate. We don't want one. It costs money and time. We don't want to spend either."

        2. A Non e-mouse Silver badge

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          Oh...and the five computers involved in the voting were all CODED by independent teams to the same DESIGN

          There was some research done into this type of work. Unfortunately, the researchers found that separate teams still made similar errors.

    2. Anonymous Coward
      Anonymous Coward

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      Actually much of the best software was written back in the 1950s, in the infancy of computing. It was written by engineers, scientists, doctors and other professionals whose standards of work were very high. Those people picked up programming "on the run" and did it very well.

      Since then, successive generations of programmers, software engineers, CS graduates, and now "Web developers", etc. have been writing software with a more and more superficial and partial understanding of the whole computing system and the whole problem to be solved.

      As Kingsley Amis said of British universities in the 1960s, "more means worse".

      1. Trollslayer

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        Software is incredibly complex compared with the 60s.

        Computer languages and frameworks also.

        As to 'superficial', frankly that is insulting skilled professionals.

        1. niio

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          Superficial is a pejorative, implying that coders with partial knowledge of a design could and somehow should comprehend it all. This is false now, and it was false in the 60's, and is false for any undertaking of any complexity. No one's brain is big enough.

          The people responsible for understanding the design are software architects, who do no coding. In complex projects there are teams of architects, and only the senior of these see the design as a whole. They see high level function, so they may not even know all the details of the design let alone the underlying software. Getting coherence all the way from top to bottom is a serious challenge. There is no easy method; review and review.

          The Max problem was not this complicated. Its primary flaw, twin AOA single points of failure, should have been obvious in isolation. Its secondary flaw, repetitive pitch down, was a little more subtle but hardly incomprehensible. Whoever designed the MCAS system was incompetent or intimidated, as was whoever approved the design and whoever coded it.

          Three levels of function should have flagged it for the failure it was to become, which means the rot was structural. If it was intimidation, Boeing needs new management. If it was incompetence, Boeing needs a whole new workforce.

          1. bazza Silver badge

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            Seconded.

            I’d be interested in your views of Boeing’s proposed fix. There’s still going to be an MCAS function, but it will 1) intervene only once per “High AoA event”, 2) limit the cumulative nose down trim, 3) duplex it.

            Fundamentally it will still be a system armed to push the nose down if it thinks it should.

            My views are that “detect high AoA event”, something it needs to do for 1), is probably a function that requires fully triplicate independent implementations. Especially as detecting a high AoA event involves paying attention to AoA transducers, air speed probes, etc. It’s a complex function and, if it goes wrong along the lines of “there’s a lot of AoA events” it will allow the rest of MCAS to repeatedly push the nose down. Just like last time.

            Even 2) is, despite appearances, non trivial. And Boeing’s approach is 3), make it duplex, not triplex!

            Ok, I think that duplex is valid if the system is hair triggered to switch itself off at the first sign of trouble (eg disagreement between AoA vanes, differences in computed results, etc). That is, it would be valid if they were independently written implementations running on different hardware designs. But Boeing aren’t doing that; it’s a single implementation, two instances of it.

            Duplex is also valid only if the system switching itself off leaves the pilot with a flyable aircraft, but what Boeing and the FAA aren’t saying is whether the MAX airframe has near-stall aerodynamic characteristics that are legal without MCAS. No wonder the EASA want to do their own flight test.

      2. Vometia Munro Silver badge

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        I think a lot of the problem lies not with the programmer (though I'm not denying there's a lot of chancers out there... some knowingly, some not) but with the environment. I mean that particular landscape that features a lot of silhouetted pointy hair.

        Casting my mind back to the early '90s, there was a serious problem where training courses were used by managers who wanted to keep their headcount up without having the amount of work to justify it so it gave idle hands something to do. Of course if they were better at managing then they might realise that maybe if they shared the training with the staff who were actually on chargeable account codes they may end up with more staff on chargeable account codes and fewer staff at the extremes of being either burnt out or bored shitless. But that's not how they think.

        The culmination of this was the horror called Rapid Application Development, which meant dragging some unwary, unwitting and unwilling programmer into an ill-defined project involving systems they'd absolutely no experience of, knew nothing about, often never heard of and certainly weren't going to receive any training for. The specification would be verbal, imprecise and would change every time it was discussed typically in that ubiquitous meeting place better known as the corridor, and requests for clarification typically resulted in being signposted to somebody else who predictably knew nothing about it and/or was so pissed off with the whole thing that they didn't want to discuss it either way.

        Oh, and of course there was absolutely no sort of planning, design nor version control of any sort. Better project managers liked version control for the usual reasons but it seemed to be viewed as an obstacle to the rapidity of RAD.

        Even as a fairly green programmer who ended up having to learn a lot of stuff the hard way I realised this absolutely stank to high heaven. To this day I feel pretty awkward about anybody encountering any of the stuff I wrote back then, whether or not it has my name on it. Though to be honest, prior to RAD there were my own inept attempts to find my feet in a team of pretty much one, so they were... well, hopefully not as bad, but still quite random. I'm also reminded of the poor chap who worked on one such project before I did: how I swore about him at the beginning of my tenure, but after not very much time spent working in that environment I quickly realised he'd made the best of a bad job and subsequently and sensibly made his excuses and left.

        The ethos was pretty much "make it work any way you can. I don't care how, just do it quickly", the obvious undertone being "make it look like it works, I'm expecting a botch job that's good enough to get a sale". Not nice. I wonder if whoever was intended to be in receipt of this wondrousness knew what was the deal; or likewise if they even cared. Which seemed to be the problem: a lot of bonuses and promotions for both producer and customer were riding on being seen to have something delivered and nobody really gave a toss about whether it worked or would even be used. Nobody who was making decisions, that is: the people at the ugly end tended to grumble until sooner or later it was time to call it quits.

        1. yoganmahew

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          @Vomet

          "The ethos was pretty much "make it work any way you can. I don't care how, just do it quickly", the obvious undertone being "make it look like it works, I'm expecting a botch job that's good enough to get a sale"."

          They now call it agile development. The way it is implemented, particularly scrum, is exactly this. Sweatshop it out the door, never mind testing it.

          1. werdsmith Silver badge

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            And as for that "RAD".... that has evolved into the "Framework".

          2. Ferry Michael

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            Like many others you are mixing up FRAgile and Agile development. With Agile ONLY tested software is DONE. Agile does not mean "do it quickly." - it means only working on a manageable number of things. and understanding how long things "actually" take to develop.

      3. Stuart Halliday

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        Absolutely. Bring back Lyon's Leo Programmers!

        1. a pressbutton

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          My dad told me a story:

          Leo was used to do all the payslips for Lyons.

          It failed.

          After much grief, just on the deadline it worked and printed a few thousand payslips.

          Someone had loaded the wrong paper - non-perforated A(something or other) pages

          This was where my dad was called in.

          After a bit of thinking they clamped the paper in a block between 2 sheets of ply, sellotaped a blank perforated page on top and sawed down the dotted lines.

          Puts debugging in it's place.

      4. Sam not the Viking Silver badge

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        Richard Feynman warned of the failure to aggressively check safety systems and in his appended report on the Challenger Disaster praised NASA's approach to software faults (one of the few processes he thought was fit-for-purpose). Even then, the managers wanted to reduce software testing because it was time-consuming and expensive.

        His appendix to the report is a good read (and re-read); I frequently recommend it to engineers.

        https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

      5. nautica Silver badge
        Holmes

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        "...They (language designers) also spread the false impression that the important thing was to learn the language‭; ‬in truth,‭ ‬the important thing is‭ ‬to learn how to design and document.

        We are still trying to undo the damage caused by the early treatment of modularity as a language issue and,‭ ‬sadly, we still try to do it by inventing languages and tools.‭" ‬--David L.‭ ‬Parnas

    3. Muscleguy
      Boffin

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      Why are learning to cold and learning to think mutually exclusive? that is a false dichotomy. We should be teaching them to code AND to think. Teaching people to think is hard though.

      During my PhD I was detailed to demonstrate the 3rd year Physiology labs. In the end of year drinks a couple of them came up to me and said that they had realised I had been trying to teach them to think. When one of them came to me with a question I analysed the information needed to answer it and if they had that information. If so I would try and guide them into joining those together to arrive at the answer themselves.

      This is the essence of thinking, seeking, analysing and making connections between aspects of your knowledge instead of keeping it in silos. This has to be an ongoing project, part of why us scientists are lifelong learners. We are teaching ourselves to think better.

      If you want better code you want people who can think instead of just being rote learned and practised. How can software and algorithms improve if nobody can think? Or do you imagine yourself to part of an established priesthood of thinkers and the yoof can’t be trusted with this? I get a whiff of that from your post. Be careful now.

      1. Richard Jones 1

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        @Muscleguy, I am unable to understand the thoughts of your down voter, but they have given no idea of what they were thinking. If someone cannot think enough to understand what they are working on, then the prospect of a solution is reduced possibly to zero. To be able to think a problem through, you need the information pertaining to the problem. The systems analysts should have done the analysis and that they must provide an accurate definition of the problems to be resolved. Safety critical products are in one sense the most challenging, but should all code not be as good as possible? That does take educated, critical thinkers who see and remove as many problems as possible before the systems reach integration and testing. Acceptance testing can never be replaced by best guess previous stages, it is an essential final phase and test plans should ideally combine both critical thinkers and where practical, those with a desire to find faults via intent and those with no clue as to how the final product should work. The latter test the fail-safe, (or not) capabilities of the product.

      2. Electronics'R'Us
        Go

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        I am reminded of a quote from an acquaintance of mine from some years ago:

        Most people consider critical thinking to be hard work and therefore do not engage in it.

        1. Anonymous Coward
          Anonymous Coward

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          "Most people consider critical thinking to be hard work and therefore do not engage in it".

          It's also somewhat risky. Someone might be upset or offended - and that could have consequences.

          1. Martin Gregorie

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            Critical thinking and analytical skills are the last things our self-proclaimed elders and betters want to see the general population understanding and using. They know very well that if this ever comes to pass they, along with the Sirius Cybernetics Sales Engineers, will be first against the wall when the revolution comes.

          2. bazza Silver badge

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            Consequences, such as being fired (if the rumours about what it’s like to work in Boeing are to be believed...).

            Boeing claim to have adopted at least some elements of “lean”, or the Toyota Production System. I suspect they’ve done it largely for the appearance of it, because one of the fundamental points about how Toyota work is that you don’t fire people if they’re pointing out problems. You’re more likely to give them a bonus.

          3. DropBear
            Unhappy

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            These days spotting and pointing out potential (or indeed very much guaranteed) problems is aggressively shunned and called "negativism". Nobody cares who and how will deal with whatever inevitable problems turn up during the execution of a job (or indeed if anyone does at all, instead of just sweeping them under the carpet) - all management (and even your peers) ever care about is that there be no objections raised so the whole thing can be rushed out of the door before it blows up, never mind whether it's ticking or not. "This will need to be solved if that is to ever work" is simply not acceptable attitude - your job is to go out there and shout at the tide without asking questions, and it's your fault if it doesn't stop.

            ...

            There HAS to be a better way, and I want off this accursed universe, over to the one where they found it.

        2. Stuart Halliday

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          Had to take on a 16 year old on IT Work experience once. After a week I asked him what was his thoughts.

          "Having to think".

          Good result I thought.

      3. Vometia Munro Silver badge

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        "When one of them came to me with a question I analysed the information needed to answer it and if they had that information. If so I would try and guide them into joining those together to arrive at the answer themselves."

        I really wish I'd had more tutors like this. Far too many took the approach "I've already explained how it works, just do it" even though I would be trying to explain to them that I lacked sufficient understanding of how and why "it" is supposed to work and was therefore unable to proceed. I don't know if they just impatiently assumed I was trying to get the answer out of them or if they themselves didn't understand the hows and whys.

        Fortunately for me, I encountered some wonderful people who were very patient and clearly found genuine enjoyment in passing on their knowledge.

      4. myhandler

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        Almost all degrees used to do this - including arts and humanities. It's the modern vocational degrees that tend to forget the 'why' question.

    4. dajames

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      Programmers need to be able to code AND to think.

      In the case of the 737 MAX the main problem seems to be that the control of the aircraft could be snatched from the pilots by a software system on the basis of a single (possibly faulty) sensor reading. That's not a software problem, that's a thinking problem.

      That the software systems involved have been shown also to contain defects demonstrates that better coding skills are needed as well ... but first you need to be able to think.

      1. DryBones

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        No, that's an information problem. The necessary bits of information are much more spread out and more numerous, kept in various systems instead of all in one. If you don't know the pieces of information exist or where they're kept, or just how wide you need to fish your net, you are going to miss things.

      2. bazza Silver badge

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        @dajames,

        “ Programmers need to be able to code AND to think.”

        In the case if MCAS the programmers working for the contractor given the job of writing MCAS to Boeing’s design did think. They took one look at it and saw garbage, resulting in the contractor asking Boeing “are you sure about this?”. Reportedly Boeing told them to get on with it.

        That email chain has probably saved the contractor a fortune, and deflected blame away from the programmers and back towards where it belongs.

        The tricky part comes when one wonders if there was a moral responsibility to refuse to do the work, and dib Boeing in to the regulators. I strongly suspect that Boeing gave strong assurances that other safeguards were in place. Which didn’t actually exist. If that’s correct, someone in Boeing needs to be in jail.

    5. martinusher Silver badge

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      >.‭ ‬Yet when someone builds you a software program,‭ ‬he has no similar certification,‭ ‬even though your safety may be just as dependent upon that software working as it is upon the bridge supporting your weight.‭

      This is the fundamental misunderstanding behind writing code (and why its a waste of time to teach it to schoolchildren). Writing code for a system is on the same functional level as physcially assembling that bridge -- you need skills to know how to assemble ironwork, paint it and so on but it has absolutlely nothing to do with the actual design of that bridge. That's an engineering task.

      To use the bridge analogy a bit more one needs to delve a bit further into what we mean by a bridge. At the simplest level a bridge is a plank thrown over a ditch. You don't need a team of structural engineers to figure that one out. You can also build a pretty decent bridge using know-how and on-the-job experience -- people have been doing this for millennia. One of the really important engineering skills is to know when you're crossing the line, when you can build by experience and rule of thumb and when you really need to think things through. Aircraft are a pretty good example of a structure than needs a bit of thought -- you can build a model aircraft out of practically anything using nothing much more than the Mk 1 eyeball but if you're going to put people in it (or fly it near people) then you just can't wing it.

      1. Anonymous Coward
        Anonymous Coward

        Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

        You're describing the difference between software engineering and software programming. Most CS course only teach the latter - the nuts and bolts of programming - since the engineering part is still very, very hard.

        I just reread Brad Cox's book that introduced object oriented programming to a larger audience. He notes how hardware was becoming more sophisticated AND more reliable, whereas programming was still in a pre-industrial phase. Artisans making entire one off systems, rather than specialising in very specific areas. I don't think much has changed since Cox wrote about that - look at how many jobs are advertised as for "full stack" developers...

        1. thosrtanner

          Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

          Re: He notes how hardware was becoming more sophisticated AND more reliable,

          Sadly I think hardware is now becoming less reliable. Faster, but less reliable. That might be because a fair swathe of it is now software, and the boundaries are less clear than they once were.

          1. Anonymous Coward
            Anonymous Coward

            Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

            Nowadays one major reason why hardware is unreliable is that it includes software in the form of firmware.

            https://www.youtube.com/watch?v=fE2KDzZaxvE&feature=youtu.be

            The best possible use of 45 minutes of your valuable time.

    6. keithzg

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      Not that you're wrong, but in my city there's a bridge that only just opened last year after years and years of delays because the piss-poor engineering was such that portions of the bridge were even falling apart while they were building it, so it had to be dramatically re-imagined. Those engineers certified to know what they're doing . . . well, often they nonetheless don't.

    7. bombastic bob Silver badge
      Trollface

      99 little bugs on the wall

      99 little bugs on the wall

      99 little bugs

      you take one down, patch it around

      104 little bugs on the wall

      or, as an associate put it: Kill one bug, grow six more

    8. John Jennings

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      One could argue that Starwars was completely successful.

      An argument can be made that the primary goal was the destruction of the Soviet economy - with the ability to plant US presences in Poland and Turkey.

      Mission successful.

    9. GrahamRJ

      Re: Isn't THIS why we've got to teach 2nd-graders how to "code", rather than how to think?

      I'm afraid you're not even wrong there. Unless your job involves the letters DO178B, MISRA, SIL or ISO-13485, the odds are very high that you don't know enough to have an informed opinion on the subject.

      Software engineering is a degree-level subject on the same level as mechanical engineering. Like mechanical engineering, you can become a chartered software engineer through very similar professional organisations. Those letters I mentioned above are the standards for development of software (amongst other engineering branches) within engineering, together with formal procedures for what level of rigour is required for development depending on how badly things can go wrong.

      Roughly 40 years after Parnas wrote those words, software engineers have worked to ensure we do have those standards in place. Remember that the reason bridge-building has these rules is directly because the Tay Bridge Disaster killed dozens of people, in the early days of civil engineering. With some famous disasters in early software control (Ariane, for example), all this was nailed down.

      What you've spectacularly missed is that this isn't a failure of software, any more than Challenger was a failure of mechanical engineering. Both are failures of *management* ensuring proper engineering practises. Every failure of the 737-MAX should never have happened, because a halfway competent safety assessment would have shown that every aspect of that feature was broken. No training, no redundant inputs, no proper testing - this goes *WAY* beyond software.

      Yes, I'm bloody livid about 737-MAX. I did this for a living for a number of years, on "stuff fails, people die" projects. I know what should be happening with this kind of work, and I know standards have been in place for 20-30 years to ensure these things don't happen, as far as is humanly possible. That kind of thing can only happen when you ignore those standards. How I feel about the engineers involved in this is pretty much how a teacher would feel about a fellow teacher who murdered their pupils. It's a fundamental betrayal of what the profession stands for.

      Does this mean your Flappy Birds app is written to the same standards? Of course not - but also your garden shed probably isn't put together to the same standards as a Galileo satellite. I doubt you'd compare the assembly of your garden shed with the assembly of a satellite. So don't be daft and compare PC and phone apps with safety-related engineering, please.

  4. Notas Badoff

    Those moneychangers...

    Recently read an article positing that it was Boeing's take over merger with their old rival McDonnell Douglas that turned them away from the engineering side. They _were_ an "engineering first" company, but the siren song of the accountants has done the full moon thing on the executive suite.

    Searching I've found Quartz article The 1997 merger that paved the way for the Boeing 737 Max crisis. There are a few other "what went wrong with their culture?" articles if you look around.

    There's a reason we near universally disdain the "alternative routes to riches". "Engineering first" *is* "people first"!

    1. Snake Silver badge

      Re: Those moneychangers...

      Yes, just today I read a similar article:

      https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/

      "What went wrong" is that finance people got a hold of an engineering firm, and pushed money over product - the production of product was seen as only an somewhat 'inconvenient', overly complex stepping stone to what was really important, that being raising "shareholder value" and maximizing stock price.

      The fact is that that very belief is what is destroying Western business: seeking every compromise necessary to add that penny to the bottom line. If that means outsourcing to inexperienced suppliers because they save a lousy £1, or firing your experienced workers because that new bloke who doesn't know what he's doing is cheaper, then all for it. I'm getting my stock bonus when the project's spreadsheets hit finance next month.

      1. Kobus Botes
        Flame

        Re: Those moneychangers...

        @Snake

        ""What went wrong" is that finance people got a hold of..."

        You can enter ANY company/entity there. I do not think there is one example where financial people took over running a company, that ended well for anybody else (part from the financial people). Once an accountant starts calling the shots, the focus moves from the product/service delivered to maximizing shareholder returns. These two goals are diametrically opposed and will almost inevitably lead to the demise of said entity in the medium to long term.

        I once asked an accountant in the company I worked for at the time, why they have such a short term

        outlook. His answer was that they only live from the current financial year to the next. Whatever happens beyond that is unknowable and therefore of no concern. His second reason was that they abhor uncertainty and would rather opt for a certain negative result (even if ruinous), over an uncertain (possibly positive) result.

        My feeling is that, once accountants take over, you should get out as soon as possible and cash in.

        1. NerryTutkins

          Re: Those moneychangers...

          I think more than this, management simply had no interest in the product.

          It's exactly what went wrong with the British car industry. German car companies were run by engineers who cared about cars and who wanted to make them better. UK companies were run by bean counters who knew nothing about cars and made decisions looking at pricelists of components and financial predictions.

          1. deive

            Re: Those moneychangers...

            And also the UK computer industry...

          2. The Real SteveP

            Re: Those moneychangers...

            "It's exactly what went wrong with the British car industry. German car companies were run by engineers who cared about cars and who wanted to make them better. UK companies were run by bean counters who knew nothing about cars and made decisions looking at pricelists of components and financial predictions."

            Oh, right, so how did 'dieselgate' happen then? -And it wasn't just VW involved in that...

            1. Robert Forsyth

              Re: Those moneychangers...

              One the problems with test driven development when the tests do not reflex real use?

              1. Doctor Syntax Silver badge

                Re: Those moneychangers...

                One the problems with test driven development...is that if you're not careful it becomes test determined design.

                Did somebody mention the British education system?

            2. AlbertH

              Re: Those moneychangers...

              "Dieselgate" was an accountant-led cock-up: "Fiddle the figures in any way you can so that we comply with the new rules that the French have imposed on the whole EU".

            3. Anonymous Coward
              Anonymous Coward

              Re: Those moneychangers...

              > Oh, right, so how did 'dieselgate' happen then? -And it wasn't just VW involved in that...

              Yes

              The R129 Merecedes is generally regarded as the last engineering led Merc and by extention German car, and that was launched in 1989.

              That was the last time the management could ask "so when will the new model be ready for launch" and the engineering team were allowed to reply with "oh, when it's ready".

              Since then the bean counters have had an increasing say, usually following the 80/20 rule "we don't care if it's only 80% as good as long as it only costs 20% as much to make/develop"

          3. werdsmith Silver badge

            Re: Those moneychangers...

            In Britain car companies would design a car, like a Rover SD1, a Triumph 2500 or a Jaguar XJ6. Then they would think about how they were going to build lots of them. BL could not build SD1s quickly enough to meet demand, or cheaply enough to maintain demand so they ended up corner cutting with component supplies and quality checks.

            In Germany or Japan they would design a car and its production system at the same time. They could build them quickly enough without stressing the line.

            But now the German car industry has lost its reputation for quality and reliability and is somewhat tainted, not just by VW whilst Japan still produces solid products but the reasons for that are a whole other story.

        2. Doctor Syntax Silver badge

          Re: Those moneychangers...

          "Whatever happens beyond that [curerent financial year] is unknowable"

          The accountants can, if they're in control, hold some sway over whether it's knowable or not. At best that's a cop-out, at worst it's a guilty plea.

        3. Stuart Halliday

          Re: Those moneychangers...

          Hey, right on the ball on why Maplin had to shut down. Accountants bled it dry.

      2. peterfoster

        Re: Those moneychangers...

        I used to work for an established silicon valley software outfit.

        When business was good Sales led the company.

        When Sales revenue was down the Services/Support monkeys would take control on the basis of all the services/support revenue they "commanded". The reality is that Services/support revenue is sticky (think multi-year contracts), and was actually sold by Sales anyway. In many cases the Services team are truly customer-toxic.

        Under Services/support leadership revenue would typically dive as custom-toxic policies kill new sales.

        Finance would then take over. Cuts in all areas follow. Product roadmap screwed.

        When Legal take over you have arrived at the end.

    2. A.P. Veening Silver badge

      Re: Those moneychangers...

      Recently read an article positing that it was Boeing's merger with take over by their old rival McDonnell Douglas that turned them away from the engineering side.

      FTFY, because that is what really happened. And they were smart enough to use Boeing's money for that as well.

      1. Doctor Syntax Silver badge

        Re: Those moneychangers...

        "And they were smart enough to use Boeing's money for that as well."

        Rinse and repeat, this time with Xerox and HP.

    3. bazza Silver badge

      Re: Those moneychangers...

      Engineering first is also money first, providing you let the engineering rule.

      Toyota make a heap of cash, but that is despite their corporate culture having a near disdain of production rates, output quantities, etc. By their measure, they are quite happy with a car plant that produces no profit so long as what it does produce is to the right quality. The belief throughout the entire that quality matters is so important that they’ll reward that ahead of profit. Obviously a lack of profitability is a longer term problem, but they’d sooner die than fire staff simply because of profit; they much prefer to find something else for the plant or staff to do.

      The lesson is that get the product design and quality right and profits will flow. Toyota are very good at understanding how much quality is necessary, and not adding anything else to the cost.

  5. MatthewSt

    If "review process finds bug in code before release" is going to be considered news from now on I think you're going to need a lot more journalists

    1. Dan 55 Silver badge

      It is news if they've done a major change like this to the plane in six months and it doesn't work. It means the culture there hasn't changed, just quickly bodge more cheap code in from HCL and hope for the best.

    2. Dan 55 Silver badge

      And also, what I wanted to say but I missed the edit window was their internal QA didn't catch this but the FAA's audit did. That is why it's news and that is why their internal culture has still not improved since the MAX was grounded.

      1. Anonymous Coward
        Anonymous Coward

        What Next for 737 Max ?

        What next for 737 Max and Boeing ?

        They have about 400 aircraft already with customers - all grounded..

        About the only way to keep them flying would be to fit different more compliant engines

        - which would change the specification and economics..

        The other option is to scrap all the aircraft..

        The headquarters should move back, and many of the managers given the chop.

        They need to re-institute the ‘old Boeing’ culture...

        With the company run by engineers - else it’s going to end up collapsing..

        It would be interesting to see what the final cumulative ‘cost savings’ were - likely negative.

        ALL of the ‘engineering’ taken over by bean counters examples, have resulted in poorer outcomes.

        It’s just not the way to run technological companies, whether they be aerospace, pharmaceuticals, or others.

    3. Chris G

      You may want to consider the context of this particular news.

    4. DavCrav

      If "review process finds bug in code before release"

      You mean after release, and after it caused literal crashes.

    5. Simian Surprise

      It's more like a few years after release, unless somehow this is yet another new piece of 737MAX software.

    6. MatthewSt

      To all those talking about context, and existing code vs new code etc - https://m.facebook.com/1620931826/posts/10219247204563842/

      Text reads:

      Before everyone gets spooled up over the latest MAX software issue, the following (next paragraph) is a brief description of what is happening. It will be entertaining to see how this minor software issue, similar to what happens occasionally with your home computer, gets blown out of proportion. Further, this clearly demonstrates that the rigors of develop/certification testing found an issue before the software went into service, as the process was intended!!!

      The software power-up monitoring function that operates at airplane or system power-up and verifies certain monitors systems are operating correctly was found to have an issue. This particular system ensures that there are no latent faults present in the system or function being monitored. The monitor check is initiated by a software command and if a fault is found it will set the appropriate indication if maintenance action is required.

      Boeing found that one of the monitors wasn’t being initiated correctly by the software at power-up. This issue was found during a technical review that takes place near the end of the formal development process, before the software package is finalized. These reviews are required specifically to identify and address issues like this and then We make the necessary updates. Boeing is working with the FAA on submission of the update change.

  6. Anonymous Coward
    Anonymous Coward

    Captain Obvious

    Boeing had just added code to allow the flight two computers to talk to each other – previously they operated independently.

    Yeah, there was probably a good reason for them to work independently. Much like it was a very bad idea for the MCAS to rely on a single sensor.

    I believe it's called redundancy, something that 100% of the executives at Boeing should now face.

    1. Twanky

      Re: Captain Obvious

      ...two computers to talk to each other – previously they operated independently.

      Erm two computers? So if they give different results which one should you believe? I thought safety critical systems used triple processing and went with the result if any two agree?

      1. steelpillow Silver badge

        Re: Captain Obvious

        That "which one to believe?" issue is one which Boeing are trying to develop some clever software to answer. It is less clear to me how many instances of that clever software will be running...

        The Max architecture has evolved from the 737's decades-old anaolgue-era original, in which dual systems selectable by the pilot are safe enough. But sophisticated digital systems involve a massive complexity which is not understandable in the same way. The competing Airbus A320neo has a clean-sheet triple-redundant digital control system.

        Quite why the airworthiness authorities still accept dual-redundancy in the advanced digital age is also unclear to many observers, and I for one am concerned that Boeing's smoke-and-mirrors approach to certification may simply have moved from flight characteristics to automated fault diagnostics.

      2. Pascal Monett Silver badge

        Triple processing

        That was thrown out the window when the beancounters persuaded the Board that using only two processing units would save 33% on costs. They mentioned that it would be just as accurate and reliable.

        The Board, obviously, only saw how much money they could augment themselves by, so it was approved because there are no lowly engineers on the Board to state the problems with that approach.

        Except that, now that the entire world doesn't want anything to do with the 737 MAX (and the FAA is not going to let them self-certify any more), all of a sudden they have to listen to actual engineers because that plane has to fly, or else Boeing is ruined.

        1. Warm Braw

          Re: Triple processing

          In this case, it saves considerably more than 33% because the flight computers operated more-or-less independently and you didn't need all the additional complexity of connecting them together and deciding on quorum rules.

          However, this really is a side issue. The 737 MAX seems to be a perfectly safe aircraft if you ditch the new software and retrain the pilots. The primary beancounter fault was to try to eliminate the recertification and pilot retraining. Once they'd made that decision, the rest of the debacle followed automatically,

          1. Gideon 1

            Re: Triple processing

            If the 737 Max was safe without MCAS, why did they need to add it?

            1. Warm Braw

              Re: Triple processing

              Because it handles differently to an old 737 and would require pilots to be retrained as a result - making it unattractive to airlines with large existing 737 fleets. MCAS was added to try to "correct" the handling so that it could be flown by a 737 pilot. It's the MCAS that appears to have caused the crashes by overriding the pilots' attempts to save the aircraft - it seems they were doing all the right things, but MCAS kept intervening and overriding their actions.

              It was the decision to use MCAS as, effectively, a cost saving measure, that ultimately resulted in all the other bad decisions about its implementation and the lack of information pilots received about it.

      3. Anonymous Coward
        Anonymous Coward

        Re: Captain Obvious

        2 out of 3 is used in the industry for measuring water level in a boiler.

        1. Robert Sneddon

          Boiler Watch

          My Dad was in the Navy Reserve during WWII, seconded from the coal industry since he was a steam fitter and the RN ships had boilers and steam engines that needed fettling. I read a handbook he used during his service which included the rules about steam engineering on board warships. One thing I recall was there had to be a qualified rating or officer on duty at all times to continuously monitor the water level in each boiler. God himself (the Master and Commander of the ship) could not redeploy the boiler watchman unless they were immediately replaced by another qualified watchman. The Navy had a track record of steam boiler explosions in the past, they didn't want any more if they could help it.

          1. Dazed and Confused

            Re: Boiler Watch

            The instructions for driving a steam engine are pretty much the same. You keep a constant watch on the level in the boiler and there's usually two sighting glasses.

      4. Anonymous Coward
        Anonymous Coward

        Re: Captain Obvious

        "I thought safety critical systems used triple processing and went with the result if any two agree?"

        Well, SF writers back in the 1950s and 1960s (such as Heinlein) understood that very clearly. But then they weren't obsessed with money.

      5. Anonymous Coward
        Anonymous Coward

        Re: Captain Obvious

        If these computers were originally designed to run independently, re-engineering them to run in parallel or synchronously is a massive undertaking, particularly if they don't want to break other certified functions.

        What strikes me is that this is a poor architecture in the first place: If the 737 computers were of a appropriate design (hardware firewall-separated command and monitoring channels), then it would have been relatively trivial to implement dual AoA checking (you'd feed AoA 1 to the command side, AoA 2 to the monitoring side). Vice-versa to the second computer. No need to synchronise.

        Having said that, I was sat on the flight deck of a 787 last week where the system clock was running backwards, so I'm not optimistic.

        1. This post has been deleted by its author

    2. nautica Silver badge
      Boffin

      Re: Captain Obvious

      When I worked for Lockheed designing basic avionics and avionics systems, it was a FACT--not even, ever, questioned as to why--that anything having to do with the safety of the craft would be, at the very minimum, TRIPLY-REDUNDANT, with 2-out-3 voting among the systems happening at all times.

      Forget "...end of discussion..."; there was NEVER a discussion as to changing this FACT, and designing using any other 'algorithm'--unless it was to employ a 3-out-of-5 system implementation.

    3. sal II

      Re: Captain Obvious

      Only it wasn't true redundancy. The secondary/passive one couldn't take over mid flight if the main failed. It required manual switch over.

      In server terms it wasn't a cluster, but a identically configured cold spare sitting powered off at the bottom of the rack, used as a failback while the main is being fixed, or under maintenance etc.

      Their latest fudge I guess is trying to run them in parallel for actual HA. Only every engineer understands that you need a 3rd (or any higher odd number) point for arbitration, they haven't learned even the recent lesson of having 2 pitch sensors speak to the MCAS

  7. Paul Hovnanian Silver badge
    Boffin

    "Boeing had just added code to allow the two flight computers to talk to each other – previously they operated independently." [italics are mine]

    I suspect that this is at the core of the MCAS fix. And if it was the case that the FCs originally didn't have a communications link, this is a MAJOR software change. As anyone who has done asynchronous comms programming knows, getting the handshaking straight between two boxes is a non trivial exercise. Particularly if the underlying O/S and hardware was never originally spec'd to handle the kind of process preemption and interrupt handling needed.

    1. steelpillow Silver badge

      Yep. As the Romans said, "Quis custodiet ipsos custodies?" (Who shall guard the guards?). Seems to me that two identical status-management instances arguing with each other on two identical master control computers may not be a rock-solid fail-safe approach: Airbus plumped for three, with majority-vote arbitration.

      1. Trollslayer

        I know I am being picky but a better translation is "Who watches the watchers?".

        1. steelpillow Silver badge

          The traditional translation during my childhood was always, "Who shall guard the guardians?"

          Mere "watchers" is too passive, like birdwatchers or something, although DC Comics would be justified in translating it as "Who watches the Watchmen?"

          - Sincerely yours, P. "Ed" Ant.

      2. Robert Sneddon

        Three votes isn't enough for some people

        The Space Shuttle had three identical flight control computers -- two of the the three computers could "vote" the other one out of the pool if they agreed. However it was recognised it was possible for two computers to go wrong simultaneously and vote the good one out or for all three computers to disagree with each other so the designers added a fourth separate flight control computer, made by a different company and running completely separate software that could be enabled by the flight crew if there were problems with the primary triplet.

        They then added a fifth flight control computer which was a basic "get the Shuttle safely down on the ground" unit, very limited in capabilities. I don't think it was ever used.

        1. gotes
          Joke

          Re: Three votes isn't enough for some people

          What if the fifth computer goes wrong?? When will people learn?

          1. Robert Sneddon

            Re: Three votes isn't enough for some people

            I have a vague recollection that on early Shuttle flights the fifth flight computer's emergency landing program was loaded from a cassette tape. I always wondered, shades of "Good Omens", if it ever turned into a "Best of Queen" compilation due to cosmic rays or whatever.

      3. EH

        Ozymandias, of course.

    2. Phil O'Sophical Silver badge

      Exactly. They've taken two systems designed to work independently and bolted on code to make them communicate.

      What could possibly go wrong with that?

      1. Bronek Kozicki
        Joke

        ... and bolted on code to make them communicate.

        for one, the bolts might have rusted.

        1. Will Godfrey Silver badge
          Meh

          Hardly surprising. The bean counters got a cheap job lot from China - they were supposed to be galvanized, but were actually just painted with a slightly mottled grey paint.

    3. Electronics'R'Us
      Holmes

      Not just code

      I suspect there will have been quite a bit of surgery in the electronics as well.

      The original kludge system was simplex (a single computer and a single sensor active at any one time); now that these must communicate with each other means that there is new wiring to actually implement a bidirectional channel which will also (of necessity) be galvanically isolated.

      There are a number of reasons for this; the main one is to prevent a fault (a short perhaps) on one side from bringing down the other side and another reason is that the ground potential within each box is highly likely to be different as avionics never connects their internal ground directly to chassis (airframe earth) and the internal power is always isolated from the power source.

      So we will then have (at a minimum) new wiring (and therefore new harnesses - adding wires to a bundle is not simple), new circuitry (to both electrically isolate the channels and to add this functionality as it was not previously implemented or if it was would not have been thoroughly tested as it was beyond the scope of the requirements). All this is quite separate from the necessary software changes to actually implement all this (and that alone will be huge).

      This work would have been carried out (for the boxes at least) at the manufacturer (one of the usual suspects no doubt - Boeing does not make their own avionics) with a new set of cable harnesses; if that harness is not identical to the new ones in the Boeing test rigs and aircraft things are highly likely to break as software needs to be written to take these things into account (and it is possible that this functionality is within an FPGA - that brings up an interesting possibility of 1,000s of manhours to qualify it).

      The aircraft level software is done by Boeing (or its cheap subcontractors) but the software within the box that runs the internal hardware is done by the box manufacturer; the integration rigs are used to verify everything, but a sample of only a few will not necessarily prove everything - for that you need production hardware.

      The communications is quite possibly ARINC 429 (designed specifically for avionics) which is notoriously difficult to get right without significant iterations - once it is right it is incredibly robust.

      Keep in mind that the change to the internal software is about more than just the communications; there is also the decision process (are both boxes agreeing is just one part of that) so this update is enormous.

      1. steelpillow Silver badge

        Re: Not just code

        I am not so sure. The standard system automatically switches from one flight controller to the other for alternate flights. But both talk to the same cockpit controls and instrumentation, so the digits must already cross paths somewhere. New hardware of this order risks a whole raft of new testing and certification headaches. If the existing data bus can be used for the monitoring and adjudication, it will be.

      2. peterfoster

        Re: Not just code

        Each 737 max flight computer contains two CPUs with code written by different teams to reduce possibility of code errors. One CPU is Intel 80286 variant, the other a similar vintage Motorola chip.

        Totally agree with previous comments that re-architecting this stuff from dual standalone to some federated Frankenstein would be tuff (impossible?).

        To make life more interesting for the software team it is widely speculated that the CPUs are already heavily utilised.

      3. Anonymous Coward
        Anonymous Coward

        Re: Not just code

        "So we will then have (at a minimum) new wiring (and therefore new harnesses - adding wires to a bundle is not simple), new circuitry (to both electrically isolate the channels and to add this functionality as it was not previously implemented or if it was would not have been thoroughly tested as it was beyond the scope of the requirements). All this is quite separate from the necessary software changes to actually implement all this (and that alone will be huge)."

        Or you could use an optic fibre link, with no electrical connection between the two.

  8. anthonyhegedus Silver badge

    Oh this makes me feel just great. They need to work on that.

  9. Anonymous Coward
    Anonymous Coward

    Can't wait to see them give up on this model...

    ....and launch a "new" Boeing 737-Super. Completely different from the MAX series. We guarantee you!

    1. A.P. Veening Silver badge

      Re: Can't wait to see them give up on this model...

      Yup, that one will start grounded and never get up in the air.

      1. Anonymous Coward
        Anonymous Coward

        Re: Can't wait to see them give up on this model...

        Which will save a lot of time, money, effort and lives.

    2. Mud5hark

      Re: Can't wait to see them give up on this model...

      Already ahead of you there.

      https://www.bbc.co.uk/news/business-48995509

      1. A.P. Veening Silver badge

        Re: Can't wait to see them give up on this model...

        No problem, I am not going to fly RyanAir anyway (and not because they don't fly from around here).

  10. STOP_FORTH
    Unhappy

    Five stages of company life

    Pretty sure I have posted this before (don't recall where I first read it)

    Run by engineers.

    Run by professional managers.

    Run by accountants.

    Run by lawyers.

    Run by liquidators.

    1. AnoniMouse

      Re: Five stages of company life

      @STOP_FORTH

      You forgot HUMAN REMAINS, who rose to ascendency in the UK Engineering industry on the basis of promises to cut employment costs. That cutting of costs was reflected in their reward models (huge bonuses, more power); but there was no penalty in the reward model for severely debilitating the engineering capability of the organisation; cost cutting was the over-riding consideration. That's the culture that leads to inadequate engineering.

      Add to that the fact that mankind is developing increasingly complex systems, invariably comprising multiple component systems interacting with each other. The complexity of such systems (of systems), often specified, developed and operated by different authorities, has outstripped mankind's capabilities to develop and ASSURE those systems.

  11. Adam Foxton

    An easy solution

    They're already seeing how much this is costing them.

    So the FAA now has to keep turning the financial screws. Increase liability if/when things fail in the real world. Mandatory, in-depth FAA testing on every single aircraft.

    Make screw-ups cost a fortune. Make the accountants hand control back to Engineering, and make it clear that's the aim.

    And then, when Boeing get the MAX in the air, announce that the rest of the industry is going to face the same scrutiny in, say, 12 months with massive fines for undeclared problems.

    Regarding the Software industry, are there any actually respected certifications for safety-critical software engineering?

    1. Trollslayer

      Re: An easy solution

      https://www.york.ac.uk/study/postgraduate-taught/courses/pgdip-safety-critical-systems-engineering/

      1. Pascal Monett Silver badge

        By God that's a good article

        "They always understood that they were an engineering-driven company, not a financially driven company . If they’re no longer honoring that as their central mission, then over time they’ll just become another company."

        Yep, well that train has now arrived at the station.

    2. sanmigueelbeer

      Re: An easy solution

      So the FAA now has to keep turning the financial screws

      I do not believe the FAA is the ones turning the financial screws at Boeing. Remember, if the FAA "had their way", the MAX would still be flying.

      The status as to when the MAX will fly is all up to the aviation regulators from other countries. They are now the ones pushing the FAA's "buttons". And they are the ones who can decide if-and-when the MAX can fly. The FAA is just a "mouthpiece" of this group.

      The other option is to scrap all the aircraft

      The US government will never allow that to happen. MAX will fly.

      What happens if it doesn't return to service? Do the airlines that bought it and received it get to sue Boeing to recover at least what they paid

      This sort of scenario isn't in any binding contract. That's the reason why some carriers have "sued" Boeing because they want Boeing to pay for the daily cost to keep the plane on the ground. Most of the time the carrier and the Boeing will come to an agreement and instead of money being exchanged, an accounting debit/credit is made.

      How does a 737 Max land on a carrier?

      Pick it up piece by piece (from the ocean floor) and assemble it on the deck.

  12. Version 1.0 Silver badge
    Joke

    App update time

    It's just a bug, the plane app will update after takeoff.

    Are they writing the 737 OS in Java?

    Maybe Boeing management will start blaming an external malware attack from another manufacturer?

    Ops sorry, the INT to FLOAT conversion dropped a byte.

  13. Anonymous Coward
    Anonymous Coward

    Write once, crash anywhere.

  14. adam payne

    "Our highest priority is ensuring the 737 Max is safe and meets all regulatory requirements before it returns to service."

    Sadly after 346 people had already died.

    After all the problems and bad publicity is any airline going to order these things? Could the Max become the new DC10?

    1. steelpillow Silver badge

      "After all the problems and bad publicity is any airline going to order these things?"

      Horribly, the answer is "yes". Some airlines have even been ordering more while the ones they already ordered are piled up in Boeing's parking lot. Without the Max, their business plans and pensions are down the tube, they cannot afford Boeing not to deliver on its "real soon now" optimism and so even after all that has happened they still swallow it without question.

      1. Doctor Syntax Silver badge

        But will the actual customers - those for tickets, not planes - buy them?

        1. Anonymous Coward
          Anonymous Coward

          Yes, they still will, because the only metrics which are used to buy flights are the route, the schedule, and the price. You generally don't see what type of plane you'll be flying in until you actual go to board it, much less have any way of specifying any preference of that when you're actually buying the ticket.

          I suppose you could research the carriers and only select the ones which don't own any 737 MAXes at all, but I doubt enough passengers would do that to make any difference to overall sales.

          1. nautica Silver badge
            Boffin

            YOU take an active role in giving Boeing what it deserves!

            "...I suppose you could research the carriers and only select the ones which don't own any 737 MAXes at all, but I doubt enough passengers would do that to make any difference to overall sales."

            It ain't that hard, folks. Simply drop a postcard--do NOT WRITE AN EMAIL--to the airlines which have 737 MAXs on order, and tell them you WON'T fly on that airplane.

            All you have to do is type "airlines ordering the 737 max" into your search engine, and you'll be greeted with two very informative entries:

            "Which Airlines Have The Biggest Boeing 737 MAX Orders?"

            and

            "List of Boeing 737 MAX orders and deliveries" [Wikipedia]

            GO !

            You will--absolutely and unquestionably--have done the world a favor.

  15. Anonymous Coward
    Anonymous Coward

    737 MIN

    I think I might fly in one of these.

  16. Anonymous Coward
    Anonymous Coward

    You mean a company with processes so broken they allowed a subsystem to be installed with authority to trim to the point of overpowering pilots inputs, in response to a single sensor reading unchecked against anything else for sanity? An idea so stupid when you repeat it back that you ask yourself how on earth anyone with a working brain could have allowed that to happen? And that then for commercial reasons insisted that information about that system be hidden from pilots, lied about to the FAA, and fought against providing them any simulator time? Which were probably all broken anyway?

    Yeah I really trust them. I'm sure it was a one off.

  17. TrumpSlurp the Troll
    WTF?

    Still no lessons learned

    Boeing are attempting to integrate two (and only two) critical computers which as far as I can tell were never designed to be fully integrated.

    Is their approach:

    (1) This is big, seriously scary stuff. We need to recheck all of the design from the ground up. Take as long as necessary and let us know when it is done.

    (2) Fix it now! I don't care what you have to do, what corners you cut, but do it quickly!! The company's future {and my job and bonus} depends on you doing this fast!!!

    Answers on a postcard........

    .

    .

    .

    .

    .

    .

    .

    .

    My personal view is that they will try to solve problems caused by a rushed, bodged and concealed set of software using more rushed, bodged and concealed software.

    Because that is the only way they know, now.

  18. TrumpSlurp the Troll
    Trollface

    All the fault of Environmentalists

    Forgot to add.

    The whole 737 Max issue revolves around fuel economy.

    So this is the fault of those nasty Environmentalists and, in line with other policies recently announced for dishwashers and the like, POTUS will support planes which burn more fuel, just like the good old days, Airlines will get a fuel subsidy if they buy and fly a 737 Classic.

  19. Anonymous Coward
    Anonymous Coward

    Just check out any _proper_ safety related PLC. (And I don't mean the kludged hot standby or dual redundant systems) Try Triconex, which is 2oo3 voted inputs, outputs and processing. Has a very impressive safety record, even withstanding deliberate intrusion via a vulnerable DCS system and was certified to continue operating for something like 1500 hours with a single, flagged fault.

  20. Anonymous Coward
    Anonymous Coward

    It’s like Windows...

    Shit upon shit upon shit. When the turdball gets too thick...

  21. 89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921

    Is this poor jet running Windows 10?

  22. FlippingGerman

    Don't trust the buggers

    Or if you must, verify.

    The whole plane should be re-certified, not as a minor upgrade but as a whole new plane, before being allowed to fly. And lots of people should be fired and forbidden from working in aerospace again. Fucking up when it gets someone killed - or in this case lots of people - should not be given second chances.

  23. Aynon Yuser

    These "glitches" sound deliberate. Full stop.

  24. FozzyBear
    Mushroom

    Thought Exercise

    Boeing execs/Managers

    You forced short cuts simply so the plane would be commerically ready based on your unrealistic timeframes. Cut costs so it would fit your budget expectations. All so you could bolster revenue, met your Performance margins and get a nice little bonus for the year.

    It has been almost a year since these aircraft have been grounded. Tell me, if you had taken this last 12 months to do things properly the first time. Proper software checks, pilot training, etc.

    Would these aircraft now be flying about the skies? Rather than being grounded indefinitely

    Would your share price be riding a high with the release of a more efficient, safe aircarft . Rather than having tanked over 25% from last years high point.

    Would your companys reputation be intact, if not even improved. Rather than the joke it is today.

    The "earning" of a bonus last year worth it. Knowing that you are likely going to be sacked over this debacle . If not have your personal reputation destroyed. Oh, and If I had my way facing multiple criminal and civil charges.

    Tell me is not taking that extra 12 months not worth the 300+ lives that were lost due to your fucking greed.

  25. Kev99 Silver badge

    I'm sure this is a silly even ridiculous question, but...

    Why on earth did Boeing decide to install a brand new software kit instead of just using tried and true from their other planes?

    Oh, and as Benny Hill once asked, " Would you really want to fly on an airplane called Boeing?" Think about it.

    1. sanmigueelbeer

      Why on earth did Boeing decide to install a brand new software kit instead of just using tried and true from their other planes?

      Because there was never a "tried and true from their other planes". MCAS was first used on the MAX.

  26. Anonymous Coward
    Anonymous Coward

    Might

    Crash

    Alarmingly

    Swiftly.

  27. Anonymous Coward
    Anonymous Coward

    This late in the cycle, a major system rewrite ?

    "Boeing had just added code to allow the two flight computers to talk to each other – previously they operated more or less independently."

    Are they really doing SW architecture change this late in the life cycle of the MAX ????

    This tells a lot about about the distance between a fully and truly validated plane and the current situation.

    SW people at Boeing won't lack work in the next year !

  28. MJB7
    Boffin

    Technical reviews

    These technical reviews are expected to turn up glitches and gremlins for Boeing engineers to fix, so this is kinda to be expected.

    No, I don't think it is to be expected. The initial architecture review would certainly be expected to turn up glitches, similarly the initial design, the detailed design, the test design, the coding, the units tests and the integration tests. However, by the time that lot has been done, one would expect safety critical code to be fairly near bug free - so a post-hoc technical review would *not* be expected to find any glitches.

  29. Dabooka
    WTF?

    Oh is that all?

    'Just added code to allow the computers talk to each other'.

    It's that easy then. Right, okay.

  30. Luiz Abdala
    Stop

    Computers designed to be independent.

    I hope these were designed like this to ensure redundancy, so that no single spurious signal can disable both at the same time.

    This was a good thing, by any chance, right?

    That's why you have 2 independent brains inside meat bags called pilots, right? Yes, they can talk to each other, but one single spoiled plate of food can't disable both.

    Ah forget it, I suck at analogies.

    But back to my point: I hope those independent computers were built like that to make sure they don't fail for the same reason.

    1. Anonymous Coward
      Anonymous Coward

      Are they designed independently?

      Take two computers with the same design and running the same code. Now feed in the same inputs and watch buggy software doing what it does best - getting things wrong reliably.

      See the recent 737-NG runway 270 bug

      Safe systems have three computers running different code on different hardware

  31. s. pam Silver badge
    Facepalm

    Boeing's new theme song..

    (sing to Frank Sinatra's voice)...

    Come flaw with me, we'll die we'll die away

    If you can use a bug or two, there's a casket in Bombay

    Come flaw with me let's die, die die

    Come flaw with me let's die, and be put away....

  32. Anonymous Coward
    Anonymous Coward

    Sunk costs

    I wonder how much the MAX debacle has cost Boeing in comparison to the clean-sheet design that was originally proposed to replace the 737 NG? By the time the plane gets back into the sky it's possible that an all-new airliner with superior characteristics to the A320-neo would have been approaching completion. And Boeing would have avoided the deserved trashing of its reputation.

    Tragically, much as the 737 MAX deserves to go the way of the Comet 1, Boeing will get away with it. The US won't allow it to go bust because it is the last US commercial airliner builder of any size; and with both Embraear and Bombardier falling into the hands of the big two there are simply no other aerospace companies that can step into the gap.

  33. Kerisun

    Their highest priority is to maximize profits.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like