back to article NASA trusted 'traditional' Boeing to program its Starliner without close supervision... It failed to dock due to bugs

At a press conference on Tuesday, NASA confirmed why Boeing's CST-100 Starliner spaceship failed to hook up with the International Space Station last year. The answer: as expected, buggy code. Crucially, NASA admitted it did not supervise Boeing closely enough during the craft's software development stage because the agency …

  1. Mark 65

    The key statement is at the end - you need multiple suppliers, principally to allow for competitive pricing. Alternatively to allow for double the pork-barrelling.

    1. poohbear

      ...the illusion of competitive pricing.

    2. Anonymous Coward
      Anonymous Coward

      @Mark 65 the reverse

      The issue is that with a single source supplier, that supplier has a monopoly and therefore can control prices regardless of quality.

      Having two vendors allows you to play one vendor off another yields a better price, albeit you have potentially higher maintenance costs because you have to support two different systems.

      That said, the issue is that NASA trusted Boeing to actually hire competent staff. You always run the risk of not controlling the use of subcontractors and the quality of their code.

      1. Anonymous Coward
        Anonymous Coward

        Re: @Mark 65 the reverse

        "That said, the issue is that NASA trusted Boeing to actually hire competent staff. You always run the risk of not controlling the use of subcontractors and the quality of their code."

        There are many coders able to write and maintain good-quality code. There are many managers able to manage people and projects.

        The trick is getting both groups together. Boeing seem to be excelling at getting the other coders and other managers - or maybe the good coders and the other managers - but we should not assume that all Boeing staff are not competent just because a few idiots are signing off on products that are not fit for purpose.

        1. EnviableOne

          Re: @Mark 65 the reverse

          but going off current performance and projected costs of the two projects

          SpaceX have achieved almost everything they set out to

          Boeing have 400 737MAX sitting on the ground due to a software bug and no hardware redundancy....

    3. Anonymous Coward
      Pirate

      Competitive pricing

      The key statement is at the end - you need multiple suppliers, principally to allow for competitive pricing. Alternatively to allow for double the pork-barrelling.

      You also really need to build a proportion of the stuff yourself, so you know how much it really costs. Otherwise you won't spot the cartel or the price gouging. It's not hard to smell a rat if they charge billion quid each when you've just built one for a quarter of that.

      The Royal Navy did precisely that for its ships from the fifteenth century until the 1960s. Whilst the majority of their ships were built by private companies, the navy had its own dockyards where it built a small proportion itself. The last one was a frigate launched at Portsmouth Dockyard in 1967.

      1. Rich 11

        Re: Competitive pricing

        The last one was a frigate launched at Portsmouth Dockyard in 1967.

        That would be one of the last Leander-class frigates built. My uncle was a CPO on the Arethusa and gave me a tour round the ship one Sunday morning, when she was being refitted at Portsmouth in 1976. I'll never forget looking along the length of the cavernous hull, with all the lower decks ripped out, and walkways and strings of lights hanging between the ribs and bracing.

        1. Anonymous Coward
          Pint

          Re: Competitive pricing

          That would be one of the last Leander-class frigates built.

          Yes. It was HMS Andromeda.

    4. fishman

      Or when one system is grounded, you have another way to get in space. Look at what happened after the two shuttle disasters - they were grounded for a couple of years until a "fix" was found.

      1. GrahamRJ

        The redundancy in that case was provided by the Russians and Soyuz.

    5. A random security guy

      They didn’t Really care for competition till SpaceX came along. Now they are all for it.

    6. Trigonoceps occipitalis

      "double the pork-barrelling"

      As Mervyn Griffith-Jones nearly said:

      "Is it the sort of thing you would wish your wife or servants to see?"

  2. aki009

    So what happened?

    Were they designing in Seattle, implementing in India and testing in Zimbabwe? (Not that I fault Zimbabwe's space program for the failures.)

    It seems to me that Boeing shifted its focus from engineering to "share holder value" some time ago, and this is the result (along with what happened to the 737MAX MCAS software). Perhaps a few C-class firings without their pretty golden parachutes is what the company needs now.

    And maybe bring software development back to the USA. The guys who can't make the firmware on a coffee maker work right shouldn't be writing this kind of critical stuff.

    1. chrisw67

      Re: So what happened?

      They should be able to manage two out of three golden parachutes.

    2. DS999 Silver badge

      Re: So what happened?

      It seems to me that Boeing shifted its focus from engineering to "share holder value" some time ago

      That's the fault of the merger with McDonnell Douglas, who pushed out the engineering meritocracy based management at Boeing and replaced them with beancounters.

      1. Chris G

        Re: So what happened?

        It certainly looks to be a near terminal infection of bean counters, corner cutting wherever possible and relying on the too big to fail mantra.

        I bet one of the main reasons they are still innthe running is the size if the Boeing hole that NASA has already thrown money into.

        It says a lot about Boeing that they are unable to even make 'traditional' docking programming work.

        1. Muscleguy

          Re: So what happened?

          <quote>It says a lot about Boeing that they are unable to even make 'traditional' docking programming work.</quote>

          Exactly the ISS is hardly ultra modern tech and the Americans and Soviets managed to create common docking technologies in the ‘70s iirc. Also Soyuz is hardly hi-tech any more just more reliable than the American supposedly hi-tech stuff.

          I can see the Americans going into Space and trying to colonise Mars and the Moon etc then the Chinese and Indians will come along and do it better, cheaper and more reliably and end up having to rescue the Americans when some crucial bit of hi-tech goes awry and cannot be fixed and Boeing and Space X have just gone bust so no resupply.

          Chinese scifi such as Cixin Liu just take Chinese dominance as read with varying degrees of internationalism. The original Enterprise crew should have had more Chinese but back then a Chinese space program would have been a joke but the population would still suggest lots of Chinese and Indians in Space. Indian Engineers were hardly unknown in the ‘60s either.

      2. Flocke Kroes Silver badge

        Re: Bean counters

        Calling Boeing management "bean counters" implies an unproven ability to count. Boeing management do have an extremely valuable skill for their shareholders: They can get laws passed requiring NASA to buy from Boeing. They are able to do this because of Boeing's proven record of late and over budget delivery of defective equipment requiring purchase of yet another law forcing NASA to spend even more tax payers' money on Boeing.

        1. EnviableOne

          Re: Bean counters

          plus the fact boeing has the right sized plants in the right Senator's states....

          oh and the lobbying budget to match

      3. AlanS
        Linux

        Re: So what happened?

        The software company I worked for was owned by McDonnell Douglas 1988-91 and there was no shortage of engineering-based management then, so Boeing's decline likely has other causes.

        My view from England is that the US stock market drive changed: preferring rising share price to dividends. In the short term it's easier to cut costs and increase profit margins, rather than improve the product and increase market share.

        (Icon 'cos SpaceX mostly use Linux.)

        1. goldcd

          This

          Although they seemed to have adopted the much simpler approach, of buying back their own shares.

          I've no idea why this isn't seen as a giant red-flag.

          You, as a company, have say 500 million you're not quite sure what to do with. Traditionally you'd leverage your resources and expertise to invest in new research/manufacturing with a view that you'll get a decent return on the investment, and ultimately an increase in your share price. Current approach is to just buy your own shares - same result to share price, but a lot easier.

          1. lglethal Silver badge
            Boffin

            Re: This

            Not defending it completely, but it can have a benefit, by reducing the amount of dividends (i.e. profits) you have to pay out to shareholders each year. Therefore, increasing your future ability to invest in research R&D etc.

            However, most of the time thats not the reason companies are doing it, I know...

            1. Justthefacts Silver badge

              Re: This

              No, that’s not the reason:

              Share price is two things (simultaneously) - a fraction share of the company value, and a fraction share of the whole company income stream. Share buybacks give investors a bigger share of both, such that the Price/Earnings Ratio remains theoretically unchanged.

              The actual rationale is that if you give dividends, investors have to pay dividend tax. Share buybacks are tax avoidance, pure and simple. If they do neither, the cash sits in the companies bank account, earning less interest than (hopefully) the company normal business - the overall margin of the business drops.

              Another way to look at it: a company making 10 planes a year, at 10% margin, would be looking to expand that to making 20 planes at 10% margin. But if they can’t, they can leverage up so that half their shareholders get the “user experience” of being invested in the 20-plane company, without any transaction friction. The other half shareholders were willing sellers.

        2. iron Silver badge

          Re: So what happened?

          You're about 10 years too early. The damage was done when Boeing management, who used to be McDD management, decided to divorce themselves from engineering and move almost 2k miles away. The Atlantic has a very good writeup on how Boeing got into this state:

          https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/

          1. conel

            Re: So what happened?

            This quote from Boeings CEO on why they moved the headquarters goes a long way to explaining the problems they're having.

            Condit makes no secret of another factor: as CEO, he didn’t want to be bothered with tiresome “how-do-you-design-an-airplane stuff”.

            https://www.forbes.com/sites/stevedenning/2013/01/21/what-went-wrong-at-boeing/

            They've been going wrong for about 20 years now.

            1. Anonymous Coward
              Anonymous Coward

              Re: So what happened?

              I worked with a whole bunch of ex Boeing software people just after the merger. Best software team ever. And I had worked in the Valley for more than 20 years.

              McDD had a toxic shark pool reputation for middle / upper management and the sentiment in Boeing at the time of the merger was that even though McDD was the much smaller company the McDD people would quickly clear out the Boeing people.. Which is what happened. The numbers I heard was when HQ was moved to Chicago all the McDD people from St Louise offered relocation took it whereas only about 20 of the 300 plus in the Seattle HQ took relocation. Hardly surprising. Chicago is a huge step up from St Louis but a huge step down from Seattle.

              Anyway in the next few years basically 70 years of Boeing institutional memory walked out the door. One way or another. Pretty much one whole generations of the most experienced engineers took early retirement or other positions. And the management culture was very deliberately destroyed. The consensus at the time was that it would take a while, a decade or two, but the company would suffer some huge catastrophe that would probably kill the company. Which is what just happened.

              So the there maybe a company called Boeing still building aircraft buts its definitely not the same company as the one that built aircraft until twenty years ago. In the same way that AT&T is not the old At&T. Or Bank Of America. Or even IBM for that matter. Which by this stage is really just PWC. Only the names are the same. The culture and people that created the original great company are long gone.

        3. conel

          Re: So what happened?

          It was the McDonnell Douglas acquisition, the same thing happened when McDonnell purchased Douglas.

          The below paper was written by a Boeing engineer 19 years ago and is eerily prescient about the problems they're having now.

          https://pdfs.semanticscholar.org/e9da/f5cc1c94c6e34e29095ca168e8fa2d750df9.pdf

      4. not.known@this.address

        Re: So what happened?

        "That's the fault of the merger with McDonnell Douglas, who pushed out the engineering meritocracy based management at Boeing [...]"

        That would be the beancounter mentatilty that took the Naval air superiority fighter proposal and turned it into the world-beating Phantom 2, the Air Force's air superiority fighter and turned it into the world-beating F15-E Strike Eagle, developed the MD-500 series helicopters, etc etc etc?

        And exactly how many world-beating designs have come out of Boeing since the 1950s? Ones not relying on technology acquired from other companies, that is.

        1. Snake Silver badge

          @not_known@this_address

          "And exactly how many world-beating designs have come out of Boeing since the 1950s?

          Oh, that would ONLY be the 707, the world's most successful first-generation jet airliner; the 737, the world's most successful airliner period; and then the 747, the world's first, and most successful, jumbo jet.

          You give McDonnell-Douglas FAR too much credit: when you have been given a blank check by the government, one that covers cost overruns, 'unforeseen' expenses AND allowances for product delays, you'll be amazed at what even an incompetent can accomplish.

          As proof, witness the "success" of MD's in-house, self-developed, self funded project, that ended up going over like a lead balloon: the infamous DC-10. So good that the sequel was renamed, to "MD-11", in order to avoid the stink of the original.

          1. anothercynic Silver badge

            Re: @not_known@this_address

            Snake's got a point... and do remember how well McDonnell-Douglas played the PR game with the cargo door and the pylon issues... (hint: not very well).

        2. imanidiot Silver badge

          Re: So what happened?

          F-15? The design closely resembling the Grumman F-14 and based heavily on work done by NASA? Referred to by many as the last good aircraft they designed?

          MD500 helicopter. You mean the Hughes 500, initially developed for the army as the OH-6? Only named the MD500 after Hughes was bought by MD?

          McD-D wasn't always a bad company, but the rot set in somewhere in the 70s. By the time of the "merger" (or reverse hostile take-over) McD-D certainly was a hot mess in many aspects. The consequences were predicted by many (see links provided by others above)

    3. lglethal Silver badge
      Stop

      Re: So what happened?

      "And maybe bring software development back to the USA."

      I think you can stop trying to lay the blame on "dirty foreigners", there tiger. Unless things have changed significantly in recent years, only Americans can work on Starliner (and most NASA funded programs but especially anything to do with Human Spaceflight).

      So this is completely down to American programmers. Or more likely their bosses, refusing to allow additional resources to be wasted on little things like reviews, testing, and hardware/software testing.

      Leave the Xenophobia out of it...

      1. ComputerSays_noAbsolutelyNo Silver badge

        Re: So what happened?

        I would not trust Boing on that, surely they could out-source some components of the software. After all, if you're developing some top-secret military aircraft, an engine is an engine; and the sub-contractor is none the wiser.

        Heck, in the aircraft industry it seems you need certification for every cable, nut and rivet you use. I wonder whether the 9$/hour Indian programmer involved in the MCAS system was FAA or otherwise certified for safety critical stuff?

        https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers

        In theory, it might be only Americans being allowed to work on Starliner; in practice, who knows?

        Outsourcing done right, will return the same result as in-house development. However, what you save in in-house dev-work, you need to invest in a clear definition of the project, since you are by definition way outside the loop. Furthermore, you need to invest much more into quality control and oversight. Now, factor-in the bean-counters and wait for disaster to strike.

        1. lglethal Silver badge
          Stop

          Re: So what happened?

          Well done on spouting complete bollocks and not knowing what you're talking about. But dont let that get in the way of making up conspiracy theories.

          When working for NASA or on NASA funded Human Spaceflight projects you need to be an American. Fullstop. I can point to procurement documents that say that. Try and outsource around that and get caught and you would lose your preferential status, costing Boeing potentially billions. Not too mention the media and political fallout that would accompany it, and frankly not even Boeing management are that stupid.

          As for comparing this with the 737, civilian aircraft have zero nationality requirements. You can outsource you're design and construction work wherever you want and indeed aircraft companies do, because it encourages those nations where the work is being carried out to buy Boeing aircraft and support their own local industry. Airbus does the same thing.

          Even with military aircraft there are usually no nationality requirements. It's entirely based on getting a certain level of security clearance. Of course at the higher levels, you wont get the security clearance without being of the right nationalities, but that only applies to a small percentage of positions in the aircraft design.

          As for your statement that "an engine is engine; the sub-contractor is none the wiser". All i can say is thank god you're not in charge of anything security related. Security through obscurity is what you're advocating and when your opponents (other nations in this case) are actively trying to find out more about your aircraft designs (and military aircraft specs are very high on the list of every intelligence services wishlist!), you really think that outsourcing like that is going to remain a secret? Intelligence agencies would have a field day with such an approach.

          Whilst Boeing are definitely infected with Beancounter disease at the moment - cutting corners, failing to test critical items, and dropping the ball on quality - their are some rules you cant get away with cutting if you want to keep your cash cows. But by all means if you've got evidence that Boeing have been outsourcing their space program by all means - post it here. It would absolutely be newsworthy and El Reg would love to report it. But frankly you're talking bollocks.

          1. arctic_haze

            Re: So what happened?

            Maybe that's the problem. The Boeing cost cutting is incompatible with hiring highly paid US developers so the conditions that only Americans can work on it makes Boeing use incompetent beginners.

        2. Doctor Syntax Silver badge

          Re: So what happened?

          "in the aircraft industry it seems you need certification for every cable, nut and rivet you use."

          In the case of the CMAX Boeing seem to have been allowed to write their own certification.

        3. macjules

          Re: So what happened?

          After all, if you're developing some top-secret military aircraft, an engine is an engine; and the sub-contractor is none the wiser.

          Oh dear. "That engine looks like it could run at Mach 15: are we going to put it in a Project X jet? No, its just an engine - we are fitting it in a 737MAX."

        4. Marcelo Rodrigues
          Headmaster

          Re: So what happened?

          "I would not trust Boing on that, surely they could out-source some components of the software. After all, if you're developing some top-secret military aircraft, an engine is an engine; and the sub-contractor is none the wiser."

          Tell that to the SR-71 and the Mig31. Both used unique developed engines. Or the GAU-8, that used the A-10 (yes, the cannon used the aircraft, not the other way around). By the way, it's engines where unique too: made to run continuously at 100% thrust at sea level, as required to close ground support.

          One could outsource the commodity part of a military project, but even this must be done with extreme care - sometimes the most innocent piece of hardware can give a secret away.

          1. anothercynic Silver badge

            Re: So what happened?

            Just remember... The SR-71 was a Lockheed product... Lockheed being a company that built rock-solid products for decades. The Electra doesn't count.

    4. Doctor Syntax Silver badge

      Re: So what happened?

      "Perhaps a few C-class firings without their pretty golden parachutes is what the company needs now."

      Just require them to be passengers on every test flight whether it be of Starliner, 737 MAX or anything else. That would concentrate their minds.

      1. Michael B.

        Re: So what happened?

        It won't make a difference. Just look at what John Selwyn Gummer did during the BSE crisis in the UK in feeding his daughter a Beef Burger in front of the camera.

        A tragic postscript is that the daughter of a friend of his did die of vCJD.

        1. ChrisC Silver badge

          Re: So what happened?

          As per DS's suggestion - "Just require *them* to be passengers".

          Some execs might be cold hearted ruthless bastards who wouldn't think twice about putting their partners, children etc. at risk if the payoff for a successful test was a bigger bonus cheque at the end of the year, but if you enforce requirement for them to put their *own* lives on the line, rather than allowing them to delegate the responsibility to anyone else (even their closest family members as per JSG), then maybe it would make a difference...

        2. Anonymous Coward
          Anonymous Coward

          Re: So what happened?

          It reminds me of that story at an amusement park, where rollercoasters were experiencing (fortunately non-serious) issues even after passing daily inspections.

          The issues stopped when the boss made the inspectors ride the first ride of the day, after they'd done the inspections!

          I can't find a link. This story may be aprophetical an urban legend.

          1. grumpyoldeyore
            Paris Hilton

            Re: So what happened?

            Well, this is what happened when the theory about sending the boss up first was tried with Portsmouth's troublesome Spinnaker Tower lift :

            http://news.bbc.co.uk/1/hi/england/hampshire/4351452.stm

        3. Stork Silver badge

          Re: So what happened?

          I remember asking how Gummer's career did not end there and then. In other places it would have.

    5. nematoad
      Devil

      Re: So what happened?

      “Perhaps we didn’t have as many people embedded in that process as we should have,”

      Oh they were "embedded" all right. Only they were "embedded" where the sun don't shine.

      Boeing? Another Midas in reverse and NASA keeps throwing money at these bozos?

    6. Anonymous Coward
      Anonymous Coward

      Re: So what happened?

      Perhaps a few C-class firings without their pretty golden parachutes is what the company needs now

      Nah. Mandate that the brass goes up and down with the vehicles they're designing, ditto for their planes. Personal exposure to dramatic and potentially lethal failure tends to wonderfully refocus the mind on delivering quality.

      After all, once you get the quality right, shareholder value pretty much takes are of its own.

      1. Anonymous Coward
        Anonymous Coward

        Re: So what happened?

        I once saw a TV show on US Army paratroopers. Part of the show covered the soldiers who packed the chutes. To ensure high quality work, the following measures were used:

        1) all chutes went into a big pool and were issued randomly (no chance to be extra careful, or "extra careful, wink wink" with a chute for your buddy or your commanding officer)

        2) those packing the chutes were fully-qualified paratroopers, who jumped regularly.

        3) at any time, the supervisor could order you to immediately go on a jump with the chute you just packed.

        I think I've related the story before, but it's a good one.

    7. Bearshark

      Re: So what happened?

      Boeing troubles started immediately after the McDonald Douglas acquisition in the late '90s. As aki009 noted:

      "Boeing shifted its focus from engineering to "share holder value"

      People ask even today, "Who really bought who" during that acquisition. Once the transaction was complete, the MD executives basically took over. MD's culture has always been the "Share Holder" comes first.

      That's 20years Boeing will never get back. And it seems this culture has spread to their industrial military complex businesses (including NASA).

      Such a shame, Boeing used to the one of the worlds premier engineering companies.

    8. A random security guy

      Re: So what happened?

      Rocket technology: even Elon Musk can’t hire foreign nationals. You have to hire US nationals.

      Plus it is the management that needs to enforce quality hiring and practices.

      You can’t have anyone to blame but yourself if you hire bad implementing and don’t run the system well. Money wasn’t the problem.

    9. Mark 85

      Re: So what happened?

      The C-suite suits decided to hire monkeys, pay them peanuts and outsource everything to the lowest bidder. The suits at the top never learn that cutting costs will at some point kill profit and the business.

      1. Terry 6 Silver badge

        Re: So what happened?

        Not, from what I gather the C-Suite suits a sucj

        h. One or two. These are beancounters put into place by the big shareholders, who want short term gains. And since everyone in and around the C level keeps their insanely well paid job and gets paid big bonuses for achieving those short term gains no one is going to challenge the strategy that creates them.

    10. jelabarre59

      Re: So what happened?

      Were they designing in Seattle, implementing in India and testing in Zimbabwe? (Not that I fault Zimbabwe's space program for the failures.)

      If it was engineered in Seattle, they probably hired their engineers in Chaz/Chop... (yeah, I know, the timeline is backwards...)

  3. Magani
    FAIL

    History repeats...

    "NASA admitted it did not supervise Boeing closely enough during the craft's software development stage..."

    See also: The FAA admitted it did not supervise Boeing closely enough during the B737 MAX's development etc, etc.

    See a pattern here?

    1. Caver_Dave Silver badge
      Facepalm

      Re: History repeats...

      I work with DERs (company representatives and FAA).

      The FAA tend to do a good job with what resources they have. Unfortunately they seem to be constantly running with about 1/2 the staff they would need to provide comprehensive oversight. Thus, company DERs are often given far too much rein on the quality side, whilst they still receive the corporate pressure.

      And the US government still has the FAA hands tied when it comes to space qualification.

      1. F111F
        Childcatcher

        Re: FAA Hands Tied w/Space Qualification

        For very good reason. The Space industry is in its infancy, so until and if/when rocket transportation moves mainstream, let the rich folks pay their money/take their chances. They'll build up the knowledge and processes to build/operate/maintain a transportation node that can, hopefully, eventually translate to the public sector.

    2. Wellyboot Silver badge

      Re: History repeats...

      >>>Boeing’s approach to writing and testing software in its Starliner was described as being “more traditional”<<<

      After the 737 fiasco this should have been ringing alarm bells across NASA

      1. Anonymous Coward
        Anonymous Coward

        Re: History repeats...

        Maybe NASA don't read The Register?

        El Reg should be required reading IMHO...

        1. Anonymous Coward
          Anonymous Coward

          Re: History repeats...

          A thumb down?

          At least *somebody* from Boeing reads through the forums then! :)

    3. Gonzo wizard

      Re: History repeats...

      Yep, company with long history of delivering, people grow complacent.

    4. bazza Silver badge

      Re: History repeats...

      To further extend the exploration of history. Anyone remember NASA not doing a full up integration test of Hubble before it was launched? You'd think that anyone with a passing knowledge of Things That Have Gone Wrong With Space Projects would know the mistakes that happened with Hubble and would be resolved not to repeat them.

      I wonder if there's any internal emails from engineers to managers in Boeing saying "I advise that a full integration test is performed"?

      Another thing I suspect is that, technically, as written, there's nothing wrong with Boeing's development processes. I suspect that their management is incapable of adherring to what's actually written down. This would explain why NASA seems to have been unsighted; they'd read the process docs from Boeing, the docs were fine, but the practice on the shop floor differed...

      1. swm

        Re: History repeats...

        "Anyone remember NASA not doing a full up integration test of Hubble before it was launched?"

        I believe that the main mirror of the Hubble telescope was figured with very precise laser interferometric testing. Unfortunately there was a .01" in the dimension of part of the test setup. This was cross checked with a knife edge test that showed the error but, since the laser interferometer was much more precise, it was assumed that the laser interferometer was correct.

        You really can't do a "full up integration test" on this kind of system.

  4. Gonzo wizard
    Holmes

    It feels like Boeing suffered from overconfidence and the desire to grow the share price and therefore cutting corners, while SpaceX have more skin in the game (relatively speaking). I hope Boeing succeed, not just to give SpaceX a run for their money but because they are trying different things.

    I like the idea of returning to land rather than the sea, with Boeing refurbishing the capsule whereas SpaceX will build a new capsule for each manned mission, the 'used' capsules being repurposed for cargo only. Lots to look forward to.

    1. tonybarry

      NASA has decided to allow SpaceX to use refurb Crew Dragon and Falcon 9 for crew launches. There are caveats - the units must not be leading-edge reuse numbered I.e. if the booster to be used is at six relaunches (and no booster has done seven relaunches) then it does not get used.

      https://spacenews.com/nasa-to-allow-reuse-of-crew-dragon-spacecraft-and-boosters/

      Regards.

    2. iron Silver badge

      SpaceX never wanted to land on water, that was a NASA rule for some reason. SpaceX wanted to use thrusters to land just like the Falcon 9 first stage but NASA said no.

      Why Boeing are allowed to do it I have no idea.

      1. F111F
        Boffin

        Powered vs Parachute Landings

        SpaceX planned on extending 4 legs through the heat shield and using the Super Dracos to fly a powered descent, with parachute backup. NASA, understandably, was concerned that poking 4 holes through the heat shield was not a proven procedure, and a powered landing vs parachute was a radical departure from the norm. So, if SpaceX was willing to invest a billion or so on their own dime and possibly delay the Crew Dragon, they could do it. SpaceX wisely backed down and changed over to parachutes and water.

        Boeing all along designed their Starliner capsule to have impact-cushioning bags and not interfere with the heat shield, so since the Russians had done it forever, NASA said it was OK to proceed. SpaceX was too far along with Dragon to redesign to include impact bags, so they stayed with water landings.

    3. renke

      > Boeing suffered from overconfidence

      I'm curious if the Starliner project was (one of) the first projects without close supervision by NASA boffins. I wouldn't surprise me a lot if the NASA staff was always a kind of outsourced QA dept for Boeing's space stuff.

  5. Anonymous Coward
    Anonymous Coward

    "information that could allegedly provide its competitors an unfair advantage"

    Yes, others being able to deliver working spaceships is seen as an unfair advantage by Boeing....

  6. Doctor Syntax Silver badge

    "The full report detailing these changes will not be publicly released, however, as it contains Boeing's proprietary information that could allegedly provide its competitors an unfair advantage"

    Does being able to point and laugh count as an unfair advantage?

    1. Anonymous Coward
      Anonymous Coward

      This was basically what the Civil Service told the PAC when they asked why British government outsourced IT projects ended up over budget failures. "We can't tell you that because it's sensitive commercial information."

      And so it continues till today.

  7. arctic_haze

    Did they use the same competent team which deisigned the 737 MAX software?

    I would like to learn the full list of projects they worked on just to be on the safe side.

  8. Anonymous Coward
    Anonymous Coward

    It's a Boeing

    So I ain't going

    1. stiine Silver badge
      Pint

      Re: It's a Boeing

      That's a good way to date yourself...

  9. Elledan

    Nothing wrong with 'traditional' engineering

    Traditional engineering techniques got humans into space, men on the Moon and allows us to this day to communicate with space probes outside of our solar system (Voyager 1 & 2). Flawed as the STS program was, it was not due to poor engineering. US companies like Boeing didn't roll all over competitors to their airplane business because they were famous for sloppy engineering, instead the US managed to churn out hit after hit with the 7x7 and a number of highly impressive military airplanes, most of which were designed before the advent of the IBM PC.

    Speaking as someone who has been introduced to the miracles of 'modern' engineering (hi Agile/Scrum), I can tell you that it is definitely not the method, but the people. That includes not only the engineers, but also the managers, customers and everybody else in the same chain. Clearly something has gone horribly wrong within Boeing that they have stuff literally falling out of the sky and catching on fire, and I very much doubt it is because they don't hold enough daily stand-up meetings and retrospectives.

  10. Mike 137 Silver badge

    "“Perhaps we didn’t have as many people embedded in that process as we should have,” she said"

    Perhaps you didn't have sufficiently competent people embedded in that process.

    Software development is not yet a mature engineering discipline, but we're now using software for things that make that an imperative.

  11. Terry 6 Silver badge

    A calculation

    Size of reputation = S

    Pressure of time to completion = T

    Need to keep costs low to win a bid = N

    Shareholder pressure to reduce costs across company = P

    Risk of failure = R

    (S*T)+(N*P) = R

    1. vogon00

      Re: A calculation

      Nice little calculation, that.....however, based on my thought/opinion of:

      “The strategy was because we’re buying a service, NASA did not have a requirement to have a systems engineering management plan,"

      Errrrr, if they were at all professional here, NASA should have at least had a requirement to *see* a plan, and preferably review it.

      ....that makes me want to suggest a modification of the formula to (S*T)+(N*(P*P))*K = R.

      IMO the 'P' term is the one that is the prime cause of risk at the moment so I think it should be P-squared,..and it's influenced & modified by K, which is a measure of common sense (Ranging from 1=a healthy engineering based scepticism, to 100="We don't know how, but it'll be OK - it will just happen" optimism.

      Space operations require quality, not cost and speed.

      PS Full disclosure : I doubt I could do better now, as I've just caught myself with my nose inches from the screen when trying to figure out if I'd typed quotes or apostrophes. Damn getting older, and Damn small fonts :-) Looks like I need to change the handle to 'Magoo'...

      1. John Brown (no body) Silver badge

        Re: A calculation

        "Damn getting older, and Damn small fonts :-) Looks like I need to change the handle to 'Magoo'..."

        CTRL-+ is your friend, at least in the browser. Going through the settings for your OS/Desktop and increasing all the font sizes is a little more involved.

        1. stiine Silver badge
          Happy

          Re: A calculation

          Do what I did. Take the 43" monitor from the living room and buy an hdmi cable to connect it to my pc. Now I have a 24" monitor and a 43" monitor. (when by better half is at work, i take the 55" moniitor from the bedroom)

          I can see everything now.

          1. JCitizen
            Happy

            @stiine

            I plugged my PC into a 61" HDTV - NOW I can see everything!

        2. Anomalous Cowturd
          Linux

          Re: A calculation

          > Going through the settings for your OS/Desktop and increasing all the font sizes is a little more involved.

          Windows user, presumably. P-p-p-pick up a penguin.

          1. John Brown (no body) Silver badge
            Devil

            Re: A calculation

            I said OS/Desktop specifically to differentiate between Windows (an OS) and Desktop, which here on FreeBSD in my case is either KDE or XFCE - The only penguins here are running on RasPis for Kodi. All the rest are Daemons, which eat penguins for lunch :-)

  12. Version 1.0 Silver badge

    But "bugs" are normal these days

    Today's software environment (look at Android phones) is that you write code and if it looks like it works you release it. No need to check that the code is completely bug free because you can release an update tomorrow... the "problem" that the management and bean counters see is that verifying that code is bug free costs money and takes too much time. Why bother doing a comprehensive test when you can post an update tomorrow?

    As for Boeing's problems, do you think that anyone running the projects has ever read The Mythical Man Month?

    1. Stork Silver badge

      Re: But "bugs" are normal these days

      It didn't harm Bill Gates, did it?

    2. EnviableOne

      Re: But "bugs" are normal these days

      but this doesnt work in mission critical infrastructure or life-critical systems.

      There needs to be a greater level of scrutiny, as if the thing is running v1.111111 and it is heading towards a crash at mach2, you cant update it to v1.111112

  13. Anonymous Coward
    Stop

    Certification and Funding

    Hardware certification is something NASA and the FAA know how to do but both are constrained by their funding. Software certification is something neither of them really knows how to do and their funding doesn't allow them to build up a competent staff to do it. Plus, both are subject to political pressure and artificial deadlines.

    Boeing, OTOH, has few funding constraints due to their Congressional support won by political donations and lobbyists reminding Congress that they or their subcontractors have a significant presence in almost every Congressional district.

    Boeing could use a small portion of this largess to improve its software development process but it's cheaper for them to chop up the software and farm it out to subcontractors. Ultimately, this approach benefits the shareholders and that is management's top priority.

    Besides, their contracts allow them to increase the price to fix the problems that they should have seen earlier.

    1. Charlie Clark Silver badge

      Re: Certification and Funding

      This isn't really about the quality of the engineers or programmers but how heavily politicised any large project at NASA is. Funding is continually being allocated, reallocated and removed. As a result, companies favour lobbying over product: it will only cost a couple of million to get a law that requires your company gets involved.

      The next question: are the problems so severe that they can't be fixed? Seeing as the failure was not catastrophic that does look to be the case.

      1. Stoneshop
        Flame

        The next question: are the problems so severe that they can't be fixed?

        Even a sufficiently large piece of space hardware dropping through the roof of the Capitol won't fix it as the lobbyists won't be affected and will just resume their work with the next bunch of representatives and senators.

        1. Charlie Clark Silver badge
          Pint

          Re: The next question: are the problems so severe that they can't be fixed?

          Oh, you cynic.

          See at the L-Street bar later?

  14. A random security guy

    Holding people accountable

    Having headed security for a few tech companies, I realized that the biggest obstacle to developing quality code is the management, not the engineers.

    Managers who are not measured on quality but on features and schedules will ignore quality even when they know it is important.

    It is almost like a death wish.

  15. spacecadet66

    “From my perspective, every early space company goes through these anomalies and you learn from it,"

    Such as scrappy startup Boeing--founded 1916 and involved in the US space program since the 60s. I assume there are rocket scientists working at NASA: this guy is not one of them.

  16. Anonymous Coward
    Anonymous Coward

    FIFY

    "The full report detailing these changes will not be publicly released, however, as it contains Boeing's proprietary information that could allegedly provide its competitors an unfair advantage. That's amusing given Boeing is far behind rival SpaceX, and its tech doesn't even work properly. "

    The unfair advantage is in the pointing and laughing part.

    SpaceX has made mistakes, but less so like this. I do hope Boeing do better elsewhere, for the safety of those flying on them.

  17. Stevie

    Bah!

    For shame, Boeing! No Red Team? Even after the Aeroplane Of Death fiasco?

    I expect there will be a sizeable refund of taxpayer monies ... why are you all laughing?

    And WTFingF are "traditional methods"?

    Prediction:

    "Of course, using FLOAT just because we were operating in zero-g was, in hindsight, a mistake."

    1. vogon00

      Re: Bah!

      "Of course, using FLOAT just because we were operating in zero-g was, in hindsight, a mistake."

      Do you know, I wouldn't be at all surprised if that exact idea featured in some manager's thought process somewhere..

  18. Anonymous Coward
    Facepalm

    From the company that brought you the 737 Max software suite.

    Title says it all.

  19. Boris the Cockroach Silver badge

    Sounds typical

    Aerospace crap to me

    When we get new aerospace stuff in, its examined, inspected, process written up and 16 different forms created before we even cut any metal... and then its 100% with the inspection, check methods, dump the programs to the backing store and sign off the dump (means we know who created it all and when.... so no modifying)

    Then send the parts to the customer with all the paperwork and hope it all flies through... heh flies.. get it... ok I'll quit that.

    Take an aerospace part we've been making for years , its all 'meh' it looks ok... its size... send it... and the programs were changed about 3 yrs ago by someone to make it a bit quicker but we cant remember who changed it or what the change actually was....

    Its the same attitude from NASA SpaceX.. new... check everything 14 times and then 15 more for luck... oh its Boeing... they know what they're doing... plus its 5 hrs from DC to Seattle.... it will be ok....

    1. Paul Hovnanian Silver badge

      Re: Sounds typical

      "plus its 5 hrs from DC to Seattle"

      2 hrs from DC to Florida. Please. Seattle had nothing to do with this.

      Maybe Chicago.

  20. Scene it all

    My long experience in software development at big companies leads me to beleive that those "closed source to protect our intellectual property" claims are bogus. The real reason is, "our code is such crap we would be laughed at".

  21. Imhotep

    I'm curious exactly what Boeing's traditional coding methods are as opposed to SpaceX.

    Traditional = buggy?

    Honestly, more detail on what that means would be appreciated. It's what I used to do for a living and I haven't got a clue.

    1. diodesign (Written by Reg staff) Silver badge

      More detail

      Unfortunately, we're going on what the NASA people said on the phone call. If the report emerges, even in a redacted version, we'll go over it.

      I imagine it's something like Boeing using a traditional waterfall development process, and SpaceX using an Agile process. At a guess.

      C.

  22. Chris Evans

    anomalies?

    "...every early space company goes through these anomalies and you learn from it,"

    The trouble is that in rocket science 'anomalies' often equals deaths!

  23. kbutler.toledo

    The article states: "At a press conference on Tuesday, NASA confirmed why Boeing's CST-100 Starliner spaceship failed to hook up with the International Space Station last year. The answer: as expected, buggy code."

    Can anybody confirm that the same people who did the code for the 767MAX are NOT working on the Starliner?

    1. jtaylor

      "Can anybody confirm that the same people who did the code for the 767MAX are NOT working on the Starliner?"

      I confirm that nobody who worked on the Starliner also worked on an airplane called 767MAX.

      Look, these problems aren't the result of some rogue employee at Boeing. They're the result of deeply flawed management. And I'd bet more than lunch money that the problems go all the way to the top.

      Turn it around: a capable manager can deal with an uncooperative employee. At Boeing, they successfully buried warnings from their employees in order to deliver an unsafe product.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like