back to article NASA engineers got their parachute wires crossed for OSIRIS-REx mission

NASA has revealed how a wiring mix-up resulted in a parachute problem on its otherwise successful OSIRIS-REx asteroid sample return mission. The release triggers for the parachutes could have been wired incorrectly, resulting in the signals designed to trigger the drogue parachute – a small parachute used to provide some …

  1. lglethal Silver badge

    So easy to do...

    On my current project, I am the responsible for cabling a large system together in an aircraft. I'm a mechanical aerospace engineer, and cables apparently belong to us, but that's another topic.

    It is super easy if you don't pay attention to have connections not match correctly, due to incorrect or inconsistent naming conventions. What's a transmit signal on one device will be a receive on the other, what can be a primary signal for one device is only a secondary for another. You really need to have this being controlled at a high System level, with every signal being defined by the System, which allocates it to the lower equipment including to the level of defining exactly which pin on which connector gets the specific signal.

    Let the individual boxes in your system control their own naming scheme and you are in for a world of hurt, which sounds a bit like what happened here.

    You can't really blame the technicians who connected Main to Main for the issue here, because it sounds correct. Unless of course, it was clearly stated in their documentation, where each connection should be. Actually, there absolutely should have been a wiring diagram they were working to, which would not allow them to connect things up incorrectly without raising questions - which would/should also be checked by an inspector to see that the wiring matches the drawing - so that's a pretty big failure of management to allow any construction without the relevant documentation to make it failure free or the required level of inspection to find the error before it's integrated into the next assembly. Mistakes happen, even with documentation, but if you have the correct documentation, then hopefully those mistakes are picked up before the next stage of assembly...

    1. My other car WAS an IAV Stryker
      Thumb Up

      Re: So easy to do...

      Everything ^^^ said above, agreed, from an electrical engineer who works more with ground (wheels & tracks) and water stuff (but a certain unmentionable "d"-word -- check my nickname again -- gets lumped with "aerospace" all the time). The job is hard enough when the "wiring label definitions" are "inconsistent" (always!) without something actually being, you know, *incorrect*.

      I've crossed my own wires between various documents/drawings/design artifacts, and the folks who are supposed to be checking me often don't catch the errors, which doubles my own workload. Thankfully, when I've also had to play technician and build some of my own designs, I found out many of the errors before they became (human and/or equipment) safety or performance issues.

    2. Sam not the Viking Silver badge

      Re: So easy to do...

      When I was working on commissioning large plant at a UK Power Stations, all instrument cables were marshalled into terminal rails within junction boxes with the outgoing cables on a parallel but separate terminal rail. All cables were labelled/numbered, not always consistently. To get the sensors powered up and working, a procedure involving 'Temporary Jumpers' was used to connect the two rails. These jumpers had different labels on each end and a table was completed showing the status of each junction box and its cabling. Eventually, when things had been shown to work correctly, 'Permanent Jumpers' were made with a different coloured wire. You could not modify Permanent Jumpers without a Permit-to-Work. It was cumbersome but in my experience, very effective.

      1. david 12 Silver badge

        Re: So easy to do...

        It was cumbersome but in my experience, very effective.

        Hi-reliability devices don't use jumper blocks, because jumper blocks are typical points of failure.

        This system was weight-limited and subject to large vibration and acceleration loads and large temperature variations. The "trial and error" phase was supposed to be completed before it was wired up.

    3. Lord Elpuss Silver badge

      Re: So easy to do...

      In a previous life I worked in a crew setting up electrically triggered fireworks for a local council display. About 100k's worth of fireworks, so not small but luckily not nuclear either.

      The (council-owned, and probably custom-made by the local 6th form college) master firing box had about 100 banana plugs, two buttons and a badly positioned label saying Master Sequence Start, which sounded to us like they started the programmed timing sequence. We figured both buttons needed to be pressed together to act as a safety.

      We found out during the pre-test that the two buttons did very different things. One was actually labelled Master Sequence, and it started the built-in sequence to set the connected fireworks off in the programmed order and timing.

      The second button was labelled Start, and it fired a signal down all wires simultaneously. You used this when you had no directly connected fireworks, but were using multiple daisychained slave boxes each with their own programmed sequence and the fireworks connected to them . The Start signal just meant that they all started their sequences at the same time.

      If we hadn't run the pre-test and had just pressed (what looked to be called) Master Sequence Start, we would have set off all the fireworks simultaneously; which would have been a hell of a bang.

      Lessons were learned on that day.

      1. Killfalcon Silver badge

        Re: So easy to do...

        That happened in Sandiego a few years back, where the whole display fired off over 30 seconds or so.

    4. Killfalcon Silver badge

      Re: So easy to do...

      The "send" vs "receive" issue is familiar to me - I think of it as the "London Road problem". The only thing you know is that this road, eventually, gets to London. You could be in nearly any town in England.

      Say you have to send a file to the Finance team, you call it the Finance File, and send it off to them. Finance end up getting dozens of 'Finance files' from all over the business, and every now and then some of them get confused with others!

      Names are hard.

      1. Andy Non Silver badge

        Re: So easy to do...

        Maybe a solution would be to use a different naming convention for wires, one not related to their function e.g. Elephant, Zebra, Giraffe, Gazelle etc.

        1. Neil Barnes Silver badge

          Re: So easy to do...

          Well, after forty years, who can remember whether Tx and Rx are from the point of view of the UART, or of the connector?

          1. ChrisC Silver badge

            Re: So easy to do...

            Quite. SPI had the right idea with MISO and MOSI, but then you get some manufacturers who seem to think that using these tried and tested bits of terminology are beneath them, and define their SPI pins as something like SDI and SDO instead, reintroducing that same "is that from the perspective of THIS device, or the one it's talking to?" problem you've mentioned.

            Not so bad if t'other device at least continues to use MOSI/MISO and you've remembered which device the M and S refer to, but if BOTH of them use SDI and SDO then it's pretty much a certainty that your rev A PCB will be wired up wrong, no matter how many times you've checked it...

            Been there, done that, etc

  2. Yet Another Anonymous coward Silver badge

    Space cyclists

  3. Andy Non Silver badge

    No physical testing?

    I'd have expected the (mostly) built systems to be tested prior to final assembly i.e. that the correct signals were being generated and going to the correct places prior to attaching to the mechanisms that deploy / cut parachute lines.

  4. Doctor Syntax Silver badge

    This may be a bit self-centred but I'm more worried about how the Apophis encounter affects the Earth, assuming I'm still in residence.

    1. Anonymous Coward
      Anonymous Coward

      Don't worry too much about Apophis

      They've known about it long enough they have a pretty accurate orbit figured out. There are small scale forces that throw things off over long terms so good estimates only go out about 100 years.

      I do think it's getting closer than satellites in the Clark belt so it might be a problem for them if it doesn't come in at a steep enough angle.

  5. Chris Evans

    Start at 0 or 1?

    Does numbering starting from 0 or 1 in a particular system?

    This I suspect is most frequent single cause of wiring errors as well just a few in programming!

    Fortunately on most occasions systems with this error will fail on even limited testing and hopefully without a cloud of smoke.

    1. Yet Another Anonymous coward Silver badge

      Re: Start at 0 or 1?

      We compromised, numbering starts at 0.5

    2. trevorde Silver badge

      Re: Start at 0 or 1?

      Obi-Wan (Off By One) Error

    3. Neil Barnes Silver badge

      Re: Start at 0 or 1?

      It goes blue/blue-white, orange/orange-white, green/green-white, brown/brown-white, slate/slate-white, blue/blue-red, orange/orange-red...

      What could possibly go wrong, unless the wireman isn't aware that they come in pairs?

      1. Kevin McMurtrie Silver badge

        Re: Start at 0 or 1?

        And also the hell of automotive wiring. The primary color indicates the system and stripes indicate the wire number. You stare at a bunch of cyan wires for a while and your relative perception of the stripe colors starts shifting. Yellow as a base color is even worse, and those might be hooked up to explosives.

        1. Spherical Cow Silver badge

          Re: Start at 0 or 1?

          How many automobiles contain explosives??

          1. Anonymous Coward
            Anonymous Coward

            Re: Start at 0 or 1?

            Most cars by now, in the airbags.

  6. Anonymous Coward
    Anonymous Coward

    The article described a hardware issue, but it's related to the two biggest problems in software engineering

    * naming things

    * cache invalidation

    * off-by-one errors

  7. ecofeco Silver badge

    I see this everywhere

    Words are apparently hard even for the original makers of things.

    I am always reminded of an odd report I saw years ago. McDonalds reported a significant increase in profits and they attributed it to one simple thing: they made the text larger on the customer orders and the employees could read them better.

    Clear, concise instructions and labels always seem to be a last thought if thought of at all. Looking at you, every website made in the last 20 years.

    My all-time favorite is of course, black on black controls with literally microscopic labels. Brilliant. Simply, effing brilliant. /s

    1. John Robson Silver badge

      Re: I see this everywhere

      "black on black controls with literally microscopic labels."

      So long as the labels are also in black, and the indicator is also a black light....

      Hot Black Desiato's ship has fantastic user experience...

    2. ChrisC Silver badge

      Re: I see this everywhere

      "Brilliant. Simply, effing brilliant. /s"

      I had much the same conclusion about the choice of colour scheme for the centre console on the OHs old Ford Galaxy - silver-grey buttons with red backlighting. During the daytime, the ambient light illuminating the cabin meant it was trivially easy to make out the symbols because they were nice and dark against the light background, but at night...

      At this point, some readers will be going "yeah, at night, what's the problem?", whilst others will be nodding their heads and going "yeah, they didn't think about colour vision deficient drivers there did they"

      Because, to those readers in the former grouping, I should explain that as someone with red/green colour vision deficiencies, I see red things rather less vibrantly than someone with normal colour vision, which means that what to you would look like a bright red symbol against a dark background, looks to me like a rather dark red symbol against a dark background. Which, when you're scanning the controls trying to find, say, the one for demisting the windscreen, makes the task rather harder than it ought to be...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like