Different day, different country, same idiots. When I was in the Army, a communications van was set on fire when its power wires were hooked up wrong. (Yes, it was all color coded. Couldn't get it wrong, no sir...)
Russia's latest rocket mishap, which saw three of the Glonass satellites needed to upgrade the country's global positioning system destroyed in a fiery inferno, is reported to have been the fault of a technician not knowing which way is up. After sorting through the wreckage, the investigating team has found a flight control …
Why did no one catch the error? Did the test plan check for the error? Did the design allow for testing for the error? Why did the design not prevent the error with, say, an asymmetrical connector?
This is management error, but let's not even say error. This was the result of management's decision to gamble.
From what I saw, rockets are assembled on their sides in stages/levels of assembly. No up or down, unless one is pointing up then, which would be to the side.
Even so, what did the engineering plans show the technician on how to install that sensor? I remember many decades ago seeing power supply after power supply (OK, two before I halted smoke testing) produce a mushroom cloud of smoke.
Turns out that the installation diagrams for the installation techs shorted the output of the rectifiers, sending 100A into a dead short on one pole of the filter capacitors, rather than each lead going to each lead of the filter capacitor bank.
Then, change management wanted to hold a committee meeting about it. That screwed up three days of production.
"Why did no one catch the error? Did the test plan check for the error? Did the design allow for testing for the error? Why did the design not prevent the error with, say, an asymmetrical connector?
This is management error, but let's not even say error. This was the result of management's decision to gamble."
According to the JPL "lessons Learned" database power supply problems (including wrong voltage, wrong current, wrong polarity, wrong power up sequence) are the #1 cause of orbital payloads being either degraded or destroyed.
And this is in an environment where everyone is aware of the consequences of failure (into the $Bn) and many of them can't be fixed.
"Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago."
Management issue to ensure design is up to date with Captain Murphy's law.
It's only annoying that it took so long for undergrads to learn about Murphy's law when considering design.
Especially since Captain Murphy served back in the late 1940's and the actual term has been dated to 1877 at the current earliest found reference from one Alfred Holt.
That said, I'd not be surprised to learn that the reality of it is, such was first uttered in the early days of complex lever/pivot systems or complex pulley systems were first employed.
"Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago."That is fine, but normally wires come in standard sizes, and the article says that a part was wired up with incorrect polarity. If someone swaps two wires in a connector or on a component, then it is the fault of the wireman, and later the inspector, and tester (possibly a design error if the designer has not allowed a mechanism to test for this failure.)
This post has been deleted by its author
"If someone swaps two wires in a connector or on a component"
Unidirectional connectors can reduce human errors, but not eliminate them. Did we really think that Murphy is so easy to defeat?
Next suggestion: Use a machine! Now we're immune against human errors and Murphy's laws. He he.
"Or is Google Translate being overly modest?"
No, that's very mild. No real obscenities in the cited Russian sentence. Just a bit of euphemism.
(Note that the following is an unnecessary excursus on colloquial Russian. There's no need for anyone to read it...)
The Russian language has a very extensive collection of expletives, known as "mat' ". (I'm not going to bother typing in Russian here, so forgive me.) See for example Flegon's "Za Predelami Russkix Slovarei" which, to be sure, includes not only "real" mat' but also euphemistic cases, such as the aforementioned "yaichka" (a diminutive form of the feminine noun "yaitsa" in the nominative singular) which translates, literally, as "little egg" but which here means, well, you know. Dictionaries of "blatnaya muzyka" (thieves' cant) and prison-camp slang are also very heavily larded with the various uses of profanity and obscenity, much of which is due to the way that Russian forms words by the use of prefixes, suffixes, and diminutives.
Whereas, for example, one would take a verb, such as " et' " (to fuck, either literally or figuratively) and by combining it with the prefix "s' " (meaning, among other things, "motion away from") and putting a verb ending followed by a reflexive particle "(vats'ya" ) you would end up with one word, s'ebyvat'sya" which is an exactly equivalent of the the English "fuck off". I.e. Russian forms new words, where English collocates existing words into set expressions. (I'm simplifying here, of course, but it's generally correct.) This method of word formation has enabled the very, very luxurious growth of Russian obscenities into an extremely extensive and expressive lexicon.
(Note that the stray apostrophes here are for two Russian "letters" which do not in themselves signify any sounds, but indicate whether the consonant preceding them is palatized (palatalized) or not. Well, don't ask. Really.)
Hmm, let's consider things a bit.
The US was notorious, to the great mirth of the nations of the world, for having rockets that went in reverse or simply fell over and exploded.
When we finally managed to get them to go up, they frequently disassembled themselves or decided to take a random, if toward the launch/recording bunker is random direction and be destructed (one film that I saw had the range safety officer thanking the officer that destructed the defective missile, said officer's response, I didn't, destruct failed).
Much, much, much later, the Apollo program began. The rockets had major pogo oscillation problems.
By your candle, the rocket would've been retired.
Nope, it was partially redesigned to remove the flex that caused the interruption in fuel/oxidant flow. From that fruit, a bunch of men managed to not walk on the moon, but hop, jump, shuffle and most often, fall on their asses or knees, largely asses.
NASA hit that one from the world for a long time. Pity, it's funny and educational. Low gravity equals grave risk of suit rupture.
Thankfully, the suits were overdesigned to compensate for the unknown.
Which resulted in a significant number of said spills.
Directional arrows alone are hardly ever enough for complex systems. There are too may ways to obscure the markings and there's always the real chance that 'up' doesn't mean up when the system is in the ready state. If a component is directional it is also important and adding physical deterrents to prevent this from happening generally doesn't add too much cost considering most of the parts are custom or semi-custom anyway.
Ideally complex systems meant for production should mostly snap together so there is no room for error and assembly doesn't require specialists with advanced degrees. A day or two of training should have anyone ready to start assembling rockets. We know it can be done, WWII proved that COTS Humans could do this stuff if it was designed well. At the end of the day it is all about cost savings, bureaucracy sucks up so much of a projects funds there isn't enough left to properly engineer the project.
According to Russian press it was not orientation but swapped polarity.
Regarding asymmetrical mountings etc - for whatever reason the Soviet, now Russian, industry never bothered to make the little things foolproof. Maybe it's laziness, maybe arrogance, but most likely the consequence of the "planned" economy - all small things were supposed to be "universal", "standardised" and produced by the millions. If your rocket needs a non-standard connector - you're doing it 'rong...
"...for whatever reason the Soviet, now Russian, industry never bothered to make the little things foolproof."
Their problem was that of budget. A serious budget, with little to no cost overrun tolerance. It wasn't unique to the space program, it was endemic to anything not military during the Cold War, later to everything due to economic depression.
Still, when the full failure analysis is completed, there will be the public response and the quiet response to idiot proof systems, for even the brightest person on the planet can have his or her village idiot moment, something that space programs won't tolerate due to the massive failures involved.
It's not prestige, it's finance. Failures cost, both financially and in client base.
And the Russians are no fools.
And as my father frequently said over the years, "My father may have raised a dummy, but he didn't raise a fool".
EVERY nation knows that as a truism, though the definition of dummy is variable based up speaker.
"Their problem was that of budget. A serious budget, with little to no cost overrun tolerance. It wasn't unique to the space program, it was endemic to anything not military during the Cold War, later to everything due to economic depression."
Not strictly apposite to your comment but the phrase "There's never enough time to do it right but there's always enough time to do it over" did come to mind.
(I guess we'd have to change that to "There's never enough money to do it right but there's always enough to do it over.")
"Ideally complex systems meant for production should..."
The problem is, space flight isn't an ideal situation. Mass is of grave concern. Do the snap in approach, add in hundreds of pounds of additional mass at best, a ton or so (tonne for some out there) at a more realistic approach.
That said, whereinhell was QC who should inspect critical assemblies? Of which, nearly every assembly is critical in a rocket designed to go into orbit or near orbit.
For the price per unit mass, I'd not have one QC inspection, but three, considering the Russian labor rate.
Yeah, but you know what happens when 3 people check something independently?
Person 1: I've looked at it a bit, that'll do, there's 2 other people checking this after me.
Person 2: I'm sure person 1 checked it, and there's always person 3, so I'm knocking off early.
Person 3: Persons 1 & 2 already checked it, so what's the point of doing it again?
Apply game theory: If a QC inspector finds a mistake one of the others didn't he gets a big bonus. If he finds a mistake both the others missed, he gets a MASSIVE bonus.
If a QC inspector misses a mistake found by one other, he pays a penalty. If he misses a mistake found by two others (and hence easier to find), he goes to Siberia.
NOW they will focus on the job...
Although I recall a tale of a company paying testers per bug found and paying coders per bug fixed. Unfortunately the coders and the testers spoke to each other and it all got quite lucrative...
"What troubles me is that for the first 17 seconds the rocket must have thought it was flying DOWN."
What fascinates me is that for the first 17 seconds, the rocket sensed it was flying down, but operated correctly.
That, in and of itself is amazing. It suggests a software solution to the issue.
Some only see a problem, others see the exception and find a solution based upon that exception to capitalize upon.
Again, the Russian sources say that the problem was an "angular velocity" sensor, don't know in which coordinate.
From the footage of the launch, the rocket looks spinning quite fast, so probably that was the roll rate sensor.
That would perfectly explain why the lift off looked normal for a few seconds and then the system went into positive feedback, trying to correct a slight roll but instead speeding it up.
...and if they've got a brain cell, there will be multiple redundancy. So if one sensor says "down" and all the others say "up", the on-board computer marks the anomalous sensor defective and ignores it unless or until it reliably returns to service or the OBC's telecommanded to bring it back into the fold.
Or, as in this case, not.
I write on-board software for space systems, so I do have a little insight into this.
One advantage of hypergolic fuels is that they self ignite as soon as they mix so you don't get much of an explosion. The American Titan II used the same sort of fuel which led to the Gemini launch escape system being ejector seats instead of the solid rockets used on Mercury and Apollo.
Biting the hand that feeds IT © 1998–2022