back to article Dodgy procedures doomed Arianespace's Vega before it even left the launchpad

ESA has published its report into the loss of the Vega VV17 mission and said the screwup was due to an "inversion of electrical connections" during integration. Which, frankly, sounds a lot like someone plugged something in the wrong way round. The rocket blasted off on schedule at 01:52 UTC on 17 November from the French …

  1. Anonymous Coward
    Anonymous Coward

    Are they really saying there were no design issues?

    Wouldn't the use of keyed connectors and/or better testing have caught this?

    1. Persona Silver badge

      Re: Are they really saying there were no design issues?

      Only if the wires are put into the connector in the right positions.

      1. Anonymous Coward
        Anonymous Coward

        Re: Are they really saying there were no design issues?

        Yep, but testing of wiring looms before installation isn't rocket science.

      2. DropBear
        Facepalm

        Re: Are they really saying there were no design issues?

        FFS, we're basically an underground East-European sweatshop and we're still not allowed to deliver the cables we make without a full all-pins-to-all-pins connectivity test, for every single cable shipped...

    2. Chris G

      Re: Are they really saying there were no design issues?

      "The integration procedure was "misleading" and the resulting incorrect routing and connection of the control lanes was not picked up in testing."

      That says it is a design issue, in the actual design as well as the testing process.

      1. Anonymous Coward
        Anonymous Coward

        Re: Are they really saying there were no design issues?

        Or have different lengths of the spurs from the loom so the short spur cant reach the connector at the end. But never underestimate the ingenuity of the installer to make it fit!

    3. Dimmer Bronze badge

      Re: Are they really saying there were no design issues?

      Red plug, red socket. Blue plug, blue socket. It is all about decreasing the chance of error. Color coded connectors will not stop mistakes, but it helps find them.

      Color blind? Well....

      The best wiring guy i have is color blind. In terminating a cable, he looks at the twist on the cat6 wires to determine which wire goes where.

      1. StargateSg7

        Re: Are they really saying there were no design issues?

        Our company went with Tip, Ring and Sleeve bayonet (kinda like a ruggedized BNC-based 1/4 inch audio plug connector) on ALL our video and motor control systems on our space-rated gear! Fully grounded with extra rad-hardened braided insulation/grounding and NO POSSIBLE WAY to screw up the connections and it is very difficult to remove needing a LOT of force for best security-of-connection!

        ---

      2. Giles C Silver badge

        Re: Are they really saying there were no design issues?

        Better than me

        A cat 5 cable is blue, red, orange and orange to me.

        Yes I am a network engineer no I don’t try and attempt to wire cat 5 plugs....

    4. vtcodger Silver badge

      Re: Are they really saying there were no design issues?

      "Wouldn't the use of keyed connectors and/or better testing have caught this?"

      Keyed connectors -- Maybe. I was once tangentially involved in an incident investigation of a missile that crashed a few hundred meters downrange during a routine test while tumbling rapidly end over end. The cause was determined to be a manufacturing defect. As it was explained to me, there were steering vanes in the rocket exhaust that were driven by a motor. The motor had tabs to which the control wires were connected. In this case, backwards. Using tabs of incompatible sizes would (probably) prevent that. But if your Mil-Std qualified motor comes as a lot of (most?) motors do with identical connectors for both wires, you're sort of stuck with it.

      Better testing? For sure. At least in that case. The test for the wired motor was that powering it moved the steering vane. Nothing about which direction. The test spec was changed forthwith.

      1. don't you hate it when you lose your account

        Turn left

        Not that left the other left

  2. Tessier-Ashpool

    Note to self

    ...Must remember to design in polarised connectors on billion-dollar missions.

    1. Anonymous Coward
      Anonymous Coward

      Re: Note to self

      The problem is that if you have a hundred connectors, you now need a hundred different connectors, all space-qualified. You could maybe solve this by having one kind of connector which could, after manufacture, be keyed to fit only its partner, but then you're relying on people keying them properly.

      1. Anonymous Coward
        Anonymous Coward

        Re: Note to self

        You don't need to have hundreds of different keyings - just different sets that are within reach in the same area.

        Keying is "tested" when am attempt in made to plug the loom into the test fixture.

    2. Grease Monkey Silver badge

      Re: Note to self

      As has already been stated polarised connectors do not prevent conductors being inserted into the connectors incorrectly. Which is why testing is so important. And that is the biggest failure here. Somebody made an assumption and didn't test.

      Testing at every stage is important. Pin to pin testing of cables is vital, it it was indeed a single cable.

      I can believe what the report says about the integration procedure, but what really seems to be missing here is operation testing of every system before launch. Had that been carried out the cock up would have been picked up. Fixing the issue and re-testing may have taken time and cost money, but which would they rather have, a total loss or maybe going over budget? Unless of course they were insured against the former and not the latter.

  3. chivo243 Silver badge
    Facepalm

    Wrong photo for this story

    I remember when El Reg would have pulled the Airplane still of the guy holding cables and that look on his face...

    I would hate to be the guy who mixed up these connectors. I'm no rocket scientist, but wouldn't it be prudent to make different shaped connectors for stuff like this? Making it impossible to cross lines?

    1. LogicGate Silver badge

      Re: Wrong photo for this story

      Without trying to shift blame here.. There will be a limited amount of connector types available that are certified (in house) for the use in the product. Typical mil-spec round connectors are not keyed by a polarisation pin, but raher by keying in the enclosure. So per shell size and pin count, the designer may be left with only 4 different key options. If these, one may be available on the market, with lead times of 10-20 weeks for the alternative key options.

      Now add to this, a push to use as many as possible identical modules in order to get the part count up (and the number of non-interchangeable parts down), you may end up with Port A of Gizmo X always having the same keyed connector. Now you are left with cable length, cable routing and cable labeling as the only things preventing you from plugging cable A into the correct Port A in the wrong Gizmo A.. And you must have routines in place, that will pick it up if you DID plug it in the wrong way. This may also be solved in software if the signal sent over cable A contains ID information. However, if the signal is a (semi) analogue controlling signal, then this may not be possible.

      Generall speaking Aerospace deals with very expensive hardware. However, the very low production numbers mean that component manufacturers will not bend over backwards to provide a large range of infividualized parts at shortest possible notice even if a single connector may cost $100. It is easier to make money by switching on a machine that makes 100.000 identical pars for a car- or a smartphone-manufacturer.

      As the saying goes, the quickest way to make a small fortune in aviation is by starting with a big one.

      1. Anonymous Coward
        Anonymous Coward

        Re: Wrong photo for this story

        Add in the possibility that you might have to use 2 of the same items - then they'll have the same connector. (Normally sub-D like old serial ports)

        Then you are down to cable lengths.

        Think about what happens in a comms cabinet when someone unplugged a few ethernet cables from a core switch, and puts them back in a different order, thinking that one network cable is the same as any other....

        It really isn't a trivial matter making flight compatible keying to add to this connector type

  4. spold Silver badge

    A matter of deduction....

    When faced with all the evidence, and you have eliminated the most likely causes, one has to conclude that what is left, is actually right....

    1. Anonymous Coward
      Anonymous Coward

      Re: A matter of deduction....

      The day of the failure, the TMs made it obvious.

      By the start of the next day, they had found photos of the assembly showing the incorrect connections.

    2. zuckzuckgo Silver badge

      Re: A matter of deduction....

      Did the contract out the assembly documentation to IKEA?

  5. Mike 137 Silver badge

    Causes

    "One can't help but remember 2013's Russian Proton-M mishap, reportedly the result of an engineer slotting in a critical component the wrong way around."

    The component was an attitude sensor, and in addition to having an arrow on it, it was mechanically keyed to fit the right way round. The original Russian report noted that "considerable force had been used" to insert the component incorrectly - i.e. someone probably hammered it in.

    The cause of the Vega failure seems rather different - an error due to poor communication and the high complexity of the system.

  6. Pascal Monett Silver badge

    "not picked up in testing"

    Then the testing is not sufficient.

    I really can't imagine any scenario where plugging something in the wrong way around won't create merry mayhem with the electrical signals. Electricity is not something that adapts itself to the configuration - either you plug it in right or you're going to get a melted component.

    Seems like Vega needs some physical modification in the connections to ensure that you plug it in right - like computer PSU connectors on motherboards. We've had those since forever, why does rocket science not have them ?

    1. Andy Non Silver badge
      Coat

      Re: "not picked up in testing"

      Yes, its not rocket science is it.

    2. Jimmy2Cows Silver badge
      Boffin

      Re: plug it in right or you're going to get a melted component

      Depends on what is sent over the connection. If it's just a relative voltage that is perfectly acceptable as positive or negative, your component will be fine but it will be outputing inverted results.

      Testing should still have picked this up though. An expected result suddenly inverting should have raised some eyebrows.

      Also there's the possibility a connector is pinned-up incorrectly. So it's possible to plug a connector in with the proper orientation, keyed or unkeyed, but still the connections are wrong.

      1. Anonymous Coward
        Alien

        Re: plug it in right or you're going to get a melted component

        I am reminded of the Hawkwind 'PXR5' album cover. Fairly sure that it either had to be changed or it was covered by sticker when it was sold.

        1. Martin Howe

          Re: plug it in right or you're going to get a melted component

          Yep, just googled that: https://en.wikipedia.org/wiki/PXR5 . Never seen it before. Given that PAT is part of my job these days, I would surely notice it :)

          Link to image in question (don't do it at home folks!):

          https://www.vinyliciously.com/images/uploads/HAWKWIND---PXR5-2015-UK-Grey-2LP-2.jpg

        2. Vulch

          Re: plug it in right or you're going to get a melted component

          Well, it does contain the tracks "Death Trap" and "We like to be frightened"...

    3. John Brown (no body) Silver badge

      Re: "not picked up in testing"

      "I really can't imagine any scenario where plugging something in the wrong way around won't create merry mayhem with the electrical signals."

      USB-C? Or any two pin mains connector. Or any of many others I could think of but will leave as an exercise for the reader.

      1. max allan

        Re: "not picked up in testing"

        USB C and 2 pin mains do not have a "right" and "wrong" way. They are designed to work in either direction.

        So,.can you think of something that has a defined "wrong" and can still be connected that way without failing?

  7. Vulch

    From an interim report a while ago I think it wasn't so much a plug put in upside down as the plug that was supposed to go to unit 1 of something was plugged into identical unit 2 and vice versa. This meant that when the flight computer said "Left hand down a bit" the wrong unit responded and it screaming "No! Your OTHER left!" just made things worse. Splitting the routing of control and sensor feedback would be a sensible solution if it isn't done already, a test of "Left hand down a bit" with the feedback going through a different route would need at least two wiring errors to not give a result of "Left hand moving up a bit".

    1. Nigel Sedgwick

      Colour-Coding and Software Connectivity Checks

      Vulch: "From an interim report a while ago I think it wasn't so much a plug put in upside down as the plug that was supposed to go to unit 1 of something was plugged into identical unit 2 and vice versa."

      That was my interpretation too, from this Register article. Four thoughts.

      1. Use of keyed connectors for different cable function raises the count of component types; this adds complexity which itself adds risk.

      2. With manned flights, increasing the cable types itself adds risk, by making spares holdings significantly greater, or making more difficult, cable substitution arising from (say) physical damage.

      3. Use of coloured cables and colour-coded sockets would provide a soft confidence check, which might be more suitable for mitigating problems identified in 1 and 2 above.

      4. In addition, if the cables carry signals embodying sophisticated (software) protocols, these could be checked for the correct connectivity by software checks post-installation and pre-launch. In addition (for cables with analogue signals), parallel wires in the same cable could carry signals (possibly serial digital) that support such (soft) connectivity checking by modest electronics at each end of each cable. Are these things not done already, in many cases, with avionics cables and subsystem connections?

      Keep safe and best regards

  8. Anonymous Coward
    Anonymous Coward

    just use a connector that only fits one way, it's not rocket science

  9. Sgt_Oddball

    And yet....

    All I can think of is Project Binky and the lengths they took to make sure correct connectors were used at all possible junctions..

    (For the record Project Binky is the mad science project of fitting full running gear off an early '90s rally car, think suspension, 4 wheel drive and 2ltr turbocharged engine....into a '70s classic mini - building it from scratch would probably have taken less time but have been far less fun)

  10. TeeCee Gold badge
    Coat

    "inversion of electrical connections"

    Sounds like they plugged the left in right and the right in wrong.

  11. Caver_Dave Silver badge
    Boffin

    Testing wiring looms

    I was involved in a project in the late 1990's for the automated testing of wiring looms (in this case for lorries).

    The latest FPGA's at the time were just becoming fast enough to be used for Time Domain Reflectometry.

    With a little knowledge of the dynamic signals that were meant to be on the wires, you could use them to perform the tests without injecting your own measurement signals and interfering with the working system. (Static voltages needed test signals adding, but could be a small enough voltage not to cause issues.)

    Open, short, extra impedance, too tight corners in the wiring, ringing, wire lengths between each connector, all could be measured and checked against the expected values.

    We charged them what was a pretty penny for us as a very small company. Apparently, they saved the equivalent in less than 6 months use!

    I moved into software only soon after that, but similar system must still be available?

  12. PassiveSmoking

    "Catastrophic success"

    The same thing happened to a little test rocket called Little Joe II which NASA used to launch a dummy Apollo capsule in order to test out the launch escape system. Due to one of the control gyros being wired up backwards, the rocket's control system exacerbated an uncommanded roll instead of damping it out, until the forces on the test vehicle tore it apart. This resulted in the Launch Escape System getting a far more thorough test than was intended, and has been described by one engineer as a "catastrophic success". Shame that this rocket was launching an actual payload, really

    https://www.youtube.com/watch?v=AqeJzItldSQ

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like