I can't get the sensor to fit
Hit it until it does.
The Russian space agency, Roscosmos, has completed its investigation into October’s Soyuz mishap in record time, pointing the finger of blame at problems during assembly. Mutterings emitted from the space agency earlier this week suggested that the issue was related to a sensor that detects stage separation of the booster. In …
I'm certainly not going to disagree that by 2018 standards Soyuz QA clearly sucks equine testicles. (In 40 000 miles my car has suffered zero faults, zero advisories, and the Toyota service manager says "we expect that".)
But it is worth remembering that the fire which killed three Apollo astronauts was caused by a dreadful error chain, starting with a contractor stuffing wires through ducts so badly that insulation was stripped in places (IIRC). How did anybody, knowing that people were going to travel on that thing, do that?
I recall being told that one reason so few existing car industry workers were taken on by the Japanese companies in the UK was that years of bad management attitude had meant that they simply didn't have the mindset for defect free assembly.
tl;dr: perhaps the space industry has to go the way the car industry did with the new entrants setting new standards, because once people have been kicked long enough to just get stuff out of the door, untraining them is impossibly long and expensive.
When I worked for Ferranti, we had the usual Military and Space standards.
Amazing how many assemblers would botch their job to get a time bonus.
We as the prefinal testers would regularly see amazing errors that people would do.
The next stage from us was environmental testing. Then it was fitted into the Space craft or Fighter Jet.
I was in disbelief at one PCB that had a dozen 2" long component leads still attached to the board so that you couldn't close the carry case it came in.
The Quality Inspector who was supposed to check assembly before me said, "anyone can make a mistake". Yeah right a little mistake caused by the Assembler and you!
If the British Air force pilots or Ariane knew how many errors were spotted and fixed at the last moment, I doubt the Pilots would switch their planes/Rockets on!
We as the prefinal testers would regularly see amazing errors that people would do.My first new car was a 1969 Pontiac, which I kept for many years until it was destroyed by another car striking it while it was stopped. I was still driving it when the state I lived in instituted emissions tests. It failed because of excess hydrocarbon emissions. Compression test looked okay so let's look at the carburetor. Removed top cap on carburetor, then the dried-out gasket. In the top rim of the main housing was a _serious_ dent, where something pretty heavy had fallen on it, or the carb had been dropped before assembly. That carb never should have found its way into a car. The gasket had sealed the dent for many years, until it dried out. Put a little dab of gasket sealer on the dent and let it cure, then replaced only the gasket, and the car passed its emissions test well below the PPM limits for that year. I forget which company built that carb, but that wasn't the only quality problem with the car. The first month I had it a rear-window regulator let go and the glass disappeared inside the body. Then the latch that held the seat in place front-back let go. Buy American? Call me a traitor, but I don't need the drama.
Space vehicles undergo rather more stress than that carburetor, so even if a problem isn't immediately apparent, poor QA can result in catastrophe later on. As John Glenn famously noted, every part of an American space vehicle was built by the lowest bidder. The problem is, once the thing is assembled, it's impossible to know if something was damaged or mis-assembled. As someone else noted, you have to trust in the quality culture of the organizations responsible for making the beast.
British cars were worse. My grandad once owned an Austin Princess. Upon having owned the car a number of years found himself needing to use the spare tire. Not so unusual....
When said tire was removed he noticed an odd lump. After poking it he found a nearly full pack of cigarettes under the spay paint. Bollocks to picking the things up. Just spray paint over ir
> The Quality Inspector who was supposed to check assembly before me said, "anyone can make a mistake".
Quality Bru^Hitsh workmanship - and why "Made in the UK" became a warning label across the Commonwealth.
In the fields I worked, even in the late 1980s anything which came out of the UK was regarded with deep suspicion - and this was despite Marconi putting out some quite decent kit.
As for cars - I think the lesson might have finally sunk in when GM tried to reintroduce Vauxhall as a brand in New Zealand in the 1990s and only sold about 3 vehicles until they gave up and rebadged them as Opels. (When Japanese cars went to the same import duty as British cars in 1973, British cars went from 40% of the market to 3% in less than 12 months.)
"If the British Air force pilots or Ariane knew how many errors were spotted and fixed at the last moment"
I'd be far more worried about the botches that _didn't get spotted. One of the reasons for the "deep suspicion" mentioned above was the British love of sending pristine demonstrators, but shoddy sales items which broke down a lot and then stiffing the buyer on warranty support. Wining and dining the bosses might win orders for a while, but Ferranti and a few others lost major orders when staff threatened to strike if they had to work on any more substandard kit - it helped that the accountants pointed out that the "more expensive" Japanese or American kit had a total operating cost far lower than the UK version and generally worked first time (and on budget) instead of needing 2 years of tweaking before being ready.
It wasn't just poor workmanship either: Who on earth designs pieces of equipment that are meant to be opened up to be serviced but doesn't have a wiring harness allowing this to happen? You can guarantee that having to desolder 50+ joints to get the device apart and resolder them afterwards is going to shorten its life. It was obvious that in most cases someone had just taken a prototype and not bothered to make it ready for production/service life before firing it down the assembly line and we ended up making our own replacement looms for a lot of kit in these kinds of cases as it was cheaper than the wasted time dealing with the craptacular originals.
"When I was a lad" - people used to lambast us colonials for a "she'll be right" attitude, but it was pretty obvious that it was nothing to the attitudes in British manufacturing (mainly at management level). Britain joining the EU and dumping all its cozy trading arrangements with former colonies was one of the best favours it could have done those colonies in the long term. They've moved on, but it's clear that the Brexit crowd haven't and they're still convinced that Britain rules all, instead of being a minor northern European country with a far greater interest in getting on extremely well with its neighbours than cosying up to the rich kids over the Atlantic who've repeatedly shown that the "special relationship" is only special when it suits them.
(As for Russian stuff: Surprisingly, their high tech works well, their low tech is bullet proof and everything in between tends to be crap)
Yes, I have worked in electronics assembling etc. too, some people don’t care, want to complete things quickly (speed of assembly is paramount to management), don’t give a care about anything.....science and tech drive our civilization, expensive oops(es) get noticed! It is truly amazing that some or other big nuke warhead command and control systems have not actually decided to go BANG by technical accident/and/or unforeseen design problems. and/or human malfunction...?
The last hundred years conditioned the whole country to be a QA nightmare, and it can't easily be conditioned away. I'm writing from a country with only about 40 years of "peace and socialism" under its belt, which was done away with thirty ears ago, and the ripples are still to be seen and dealt with today.
It's all well and good to say that but if you do any job a hundred times it stops being a special job, it just becomes step 7b of job 256. And you've got 4 more jobs to do before the end of the day or the boss is going to get pissy with you. And this bloody bracket, which you've complained about half a dozen times but no one listens, is not going in easily. So you just take your hammer and get it to fit easy as pie. And move on to step 7c.
This is how cock ups like this happen. Because people are more worried about the pressure they'll get from managers for missing a deadline. Their manager is a more immediate problem then some future potential problem for some astronauts they'll never meet.
To avoid this, you really need a culture in the workplace where problems and delays are dealt with properly and no blame is put on the worker when they occur. It sounds very much like Roscosmos does not currently have that culture...
@Big John
You seem to have a naive expectancy that any assembly line worker would really think about the criticality of any component, or take more care because of that.
It might be the case that the sensor was simply dropped whilst being handled, and then put on anyway, because the alternative is for the worker to own up to damaging it, and suffer the consequences - which might be punitive.
I wonder what you would do in the circumstances?
"It might be the case that the sensor was simply dropped whilst being handled, and then put on anyway, because the alternative is for the worker to own up to damaging it, and suffer the consequences - which might be punitive."
Exactly. It's cultural.
In Toyota or Sony, to name just two I know a bit about, the worker would admit to having dropped it, the sensor would be replaced properly, and the quality circle would address itself to ways of preventing sensors from being dropped and damaged.
An example: wall hung toilet bowls. These are prone to damage by being dropped. Unless you buy them from a certain German manufacturer who supplies them with the foam packing formed into an assembly jig which holds them secure and presents them to the wall at the right height.
Then there's the marine engine builder who supplies each of their engines with a beautifully shaped jig which simultaneously compresses the piston rings and slides the piston down the cylinder in a straight line, preventing ring damage. When you see that compared to the crudity of people trying to get ring compressors to line up over a piston, you appreciate real quality engineering.
I've worked for a company where the production director had to be fired to start developing a blame-free culture to get towards zero defect assembly.
>Exactly. It's cultural.
If only the Russians had a quality first mindset like the US then they too could have the F35 program as a shining example. Not a russian troll just disgusted with how greed turns everything to shit in the US and causes us to lose the ability to even engage in space travel. Think positive. Looking forward to the JWST launching in 2025.
"If only the Russians had a quality first mindset like the US"
I am not an American, though I have worked for US companies in several capacities.
Some US companies I've come across are as good as any in the world.
Some are crap. Some are in between.
But the problems of the F-35, as so often with military programs, are political.
My friendly consultant to the US government says that the pork barrel basically prevents things from happening but is essential, because it's the US equivalent of the welfare state - providing jobs in depressed parts of the country, either in the armed forces or in making arms. Every politician wants their share.
So unlike our own dear country where Labour has to support Trident because of where the jobs are.
>When you see that compared to the crudity of people trying to get ring compressors to line up over a piston, you appreciate real quality engineering. The Japanese bikes were never that much better than the British ones, especially when they were both relatively new, but the attention to detail in the assembly is much more obvious with the Japanese one. (They Japanese also seemed to put 'working' as a key criteria for outsourced components such as electrical parts -- they like stuff to be cheap but they regard working as somewhat more important.)
When you compare the service manual for my old Triumph with that for a generic Japanese motorcycle then you immediately understand why the old bike leaks and has had periodic reliability problems (up to and including a broken connecting rod) while the Japanese bike(s) just run. Its not really the line workers that are to blame but the company culture, it sets the tone.
The good news with Soyuz is that the emergency escape system got a good workout and seems to be functioning. Its also impressive that they could find the problem by looking at the pile of bits of crushed and twisted metal (and the telemetry).
"The Japanese bikes were never that much better than the British ones, especially when they were both relatively new, but the attention to detail in the assembly is much more obvious with the Japanese one."
I like to watch the old videos of engine assembly on Youtube, and one thing I've noticed on both a British and a Harley video is the con rods being allowed to knock against the crankcase mouth, something the guy who taught me how to do that stuff (PhD mechanical engineer) would have regarded with utter horror. You do not risk making even the smallest scratch or dent in a stressed part!
Video of assembly of small Honda engine - nothing is ever allowed to bump into anything else. And the workers are wearing white gloves (skin grease can corrode metal).
"It might be the case that the sensor was simply dropped whilst being handled, and then put on anyway, because the alternative is for the worker to own up to damaging it, and suffer the consequences - which might be punitive."
Quite likely. When you look into reports of soviet nuclear accidents it's clear that more than a few fatal doses of radiation poisoning were down to this kind of incident. The country might change name and politics but ingrained cultural training takes much longer to alter.
The underlying problem is a culture that encourages middle management to treat line workers like that and not escalate issues that need fixing. Organisations will get politicised to the point that people start behaving that way if you don't keep an eye on the culture and values. The hard part is to avoid letting carpetbaggers into your management team. Once you get one manager with that mentality the toxicity will spread to everyone that person is in a position to put pressure on. It affects culture - by allowing a culture of undue pressure on workers - and hiring practices - A's hire A's, B's hire C's.
Keeping a culture alive where people are able to work on complex tasks in a zero-defect manner is not a trivial undertaking, and the vast majority of management teams are not capable of doing it. Certainly, British hiring practices for managers favour politically savvy 'people manager' types who aren't inclined to rock the boat, and the trend in management culture seems to be moving more in that direction in pretty much all industrialised countries. It happened at NASA with the O-ring fiasco, it's happened at Toyota (coming out with brakes IIRC) and now it's happened at Roscosmos.
@Big John
"Clearly the Russians have a systematic quality assurance problem, and those ain't cheap"
Before this latest problem, the last time a Soyuz launcher needed to use its escape system was 1983.
<http://www.russianspaceweb.com/soyuz_sas.html>
A 35 year run of manned launch successes implies that the problem which occurred isn't systematic.
Clearly, there is some sort of quality management problem which needs to be dealt with. But given the long-term reliability of the Soyuz system, whatever the problem is, it isn't deeply embedded into the engineering or operational culture so isn't likely to be particularly difficult or expensive to deal with.
I do wonder about this point:
<http://www.russianspaceweb.com/soyuz-fg.html>
"As of September 2017, the two final Soyuz-FG rockets had been scheduled to fly in April and September 2020. However the Roskosmos leadership was pressing the industry to retire the FG variant as early as 2019 to cut costs and avoid problems with Russian security services, which put serious obstacles in obtaining avionics and associated technical assistance from Ukraine's Kommunar plant. As a result, at least one of previously planned launches of Soyuz-FG in 2020 could be performed in 2019, industry sources said."
- one might speculate that - just maybe - management has perhaps been pushing workers to get things done faster than usual, which could have contributed to the problem in this case. I don't suppose we'll find out about that one way or another.
Clearly the Russians have a systematic quality assurance
I would have given the poster of this the benefit of the doubt if not for:
1. It being posted by a known AI-synthesized Trump Clone Bot
2. It having a proper statistical basis in the actual data from the field. Do we like it or not, but based on actual launch statistics the old village bus beats any of the other manned flight programs by a very large margin. Shuttle - 135 launches, 14 dead bodies, zero successful landings under major failure conditions. Soyuz - 139 crewed launches, 4 dead bodies, 3+ successful landings under various failure and/or abort conditions with crew alive in all cases (not something the shuttle can claim).
Actually, if you look at the stats differently, you might say the US and USSR/Russia are much the same. Both had two flights kill all the astronauts on board. And the Shuttle had flights like STS27 where there was major heat shield damage that the vehicle survived. The Shuttle both carried cargo and astronauts into orbit, and was much more complicated than Soyuz, so it might be reasonable to say the US has the better safety culture.
Oooh, my cue to post the L5 ground loop audio from Eileen Collins's launch. Stay with it for the low LOX level premature SSME cut-off half a sec before the scheduled MECO, which IIRC was due to a big hole in a nozzle cooling circuit. (The auto cut-off is there because apparently cryogenic gas turbopumps tend to explode if suddenly fed vacuum when running.)
"Yikes!"
"Concur."
Oy, this makes the hair on the back of my neck stand up every time I watch it!
https://youtu.be/O9WjCyWq-iA
Context https://waynehale.wordpress.com/2014/10/26/sts-93-we-dont-need-any-more-of-those/
And there's a WP article of course.
Clearly the Russians have a systematic quality assurance problem
So systematic that it caused a single failure.
Now if you'd mentioned the problem they had with defective gas generators being installed in RD-021x engines during 2015/16, which ultimately killed off the Proton rocket, you might have had a point. However, that was due to attempted cost-cutting, which is the kind of thing you expect to see in the west...
"Wasn't The Morning Star the Crommulist rag?"
Still is AFAIK. I can only assume the idiots who modded me down never lived through the 70s and have some sentimental view of salt-o-tha-earth union workers fighting against The Man. Truth was british car workers back in the day were short sighted, cretinous, lazy fools who would use strong arm tactics to get what they wanted no matter what the long term cost - which turned out to be their jobs in a lot of cases. A bit like the current RMT tbh.
>Truth was british car workers back in the day were short sighted, cretinous, lazy fools who would use strong arm tactics to get what they wanted no matter what the long term cost
I lived through the 70s, and recognise that propaganda from the right-wing press of the time. The real truth was that British management was crap - as we discovered when foreign management managed to produce decent cars - with British workers.
As Geoffrey Goodman later observed - you could always predict when there was going to be a wildcat strike; when the storage car parks filled up with unsold cars, some issue would be created, to annoy the workers enough to walk out (and allow some cars to be sold, to clear the storage parks).
That hammered a sensor into place because it wouldn't fit - because it had been carefully designed with locating pins stopping it being installed upside down ?
...I much prefer this kind of incident investigation to that other kind that digs itself into a mountain of reports and committees with all flight operations halted for half a decade. Besides, they allegedly do intend to take apart all assembled boosters again to make sure there are no further issues of this kind. It sure beats sitting grounded until you can design, test and certify your new ISO-compliant procedures for establishing management guidelines and decision strategies for drafting quality assurance protocols that will hopefully ensure this exact specific particular fault can never occur again.
DropBear,
That's all very well - until it turns out that you've got larger problems than a quick investigation will turn up, and you end up killing your cosmonauts live on TV.
It's not exactly very reassuring to say, "we broke a vital component of our rocket at some point in manufacturing, but we don't know how or when. Still that's OK - because it passed all the checks - or there weren't any checks - or people aren't doing the checks properly. But seeing as we don't know which, we'll just invent some new procedures and hope they work.
Meantime our current Soyuz in orbit has a hole in it, that was subsequently crudely covered up with epoxy (presumably by the guilty employee). And that also got through inspection, assuming there was any) and we also still don't know how that happened, when that happened, or how it got through checking, or if it was checked.
Oh and some of our senior leadership decided to publicly accuse the personnel in orbit of doing it, without any evidence or even credible possibility. Because that's really reassuring that management culture is all about solving this problem, rather than just rushing back to full operations with the minimum of arse-covering neccessary.
Yes a quick report can be an opportunity to get back to operations with maximum efficiency. But it can also be an opportunity for a whitewash that's eventually going to get people killed.
You can't just blithely say, "we have a manufacturing quality control problem of unknown dimensions, but fuck-it we're just going to fly anyway."
If NASA have got any sense, they will run a mile from the next launch. I hope they don't get pressured into it because they'll be accused of cowardice for not flying on a rocket the Russians will. Unless they're getting more indications that this is being taken more seriously than it looks to be on the surface. I'll be delighted to be wrong - but I'm seriously worried.
NASA being unable to meet any kind of deadlines or cost estimates is why we are depending on our fr-enemies to get into space in the first place. As Neil deGrasse Tyson says our space program looks awesome if you reverse time's arrow. Thanks to shitty decisions made for decades our two choices are now no space flight or trust the Russians. Easy to poo poo their culture but the alternative is a jobs program ran by defense contractors with too many heads in the trough that delivers nothing these days but invoices for the taxpayers.
> Yay for free markets
The American space program is most definitely NOT anything close to a free market. It is closely controlled by Congress and used as a political football.
For example, the current hoorah over launching American spysats on rockets powered by Russian engines. They'd be powered by American engines, except guess what? Congress cancelled the last FIVE of the NASA engine R&D programs. One of which (FASTRAC) was rescued by that bi-polar stoner Elon Musk and resurrected as Merlin
> NASA being unable to meet any kind of deadlines or cost estimates
See above. This is because Congress has cut the COTS crew program funding by 67% and the BEST thing about it is that the senator responsible for the cuts is now bitching it's behind schedule!
And when you have to punt deadlines and jerk contractors around, of course it all gets more expensive... and I end up footing the bill for all this bullshit, as a taxpayer.
Oh yeah, I forgot:
Those Russian engines launching American spysats?
Yeah, the design of those were licensed to us and supposed to be "made in America" except guess what?
Again, Congress denied the promised funding for that to happen and told ULA to go fuck itself.
For the record yes Elon Musk is amazing and yes SpaceX is too but was on a rant and have to admit the guy hasn't been a rock of stability lately. Sarcasm aside he may still well be the US's best hope.
Also I am well aware of our recently history of failed political leadership especially in regards to science but then an article like this shows NASA shares some of the blame as well. ISS was also a mistake IMO.
https://spectrum.ieee.org/riskfactor/aerospace/space-flight/gao-warns-of-deteriorating-costs-and-schedules-in-nasas-major-project-portfolio
Alexey Ovchinin is definitely in trouble for this: he had Nick Hague sitting right next to him throughout the launch yet somehow failed to notice him sabotaging the first stage of the rocket! Most likely a frank yet courteous exchange of words with Putin awaits him.
Just to get this story straight: The fault that caused a soyuz to misbehave and crew to climb to the escape pod has been identified and corrected. No BS with people blaming others or withholding information (Challenger, anybody ?), or escape routes too long and pods too far away for the nauts to gets out.
Whatever you say, the Russians have a track record nowhere near anybody else when it comes to space missions, like, 4 deaths in almost 70 years ? [have not checked] Their kit is rudimentary, but it usually does the job nicely - shit sometimes happens AND gets fixed ...
I do not like the Russian President or his Administration, but Roscosmos definitely know what they are doing!
Just to get this story straight: The fault that caused a soyuz to misbehave and crew to climb to the escape pod has been identified and corrected.
It may have been identified, but it has not been corrected. For a start, it has already happened and is therefore beyond correction. Beyond that, it's not enough to know that something was wrongly fitted; you also have to know why it was wrongly fitted. If it was accidental, can you be sure that further accidents will not occur elsewhere. And if it was sabotage - which seems at least possible - how do you know you have caught all the saboteurs?
Oh, and all that aside ,,, what "escape pod". Soyuz capsules don't have "escape pods".
It may have been identified, but it has not been corrected. For a start, it has already happened and is therefore beyond correction.
I obviously mean the manufacturing fault has been identified and procedures adhjusted, meaning any newly built soyuz will NOT have this manufacturing fault.
Soyuz capsules don't have "escape pods".
True, they landed in the crew capsule. Sorry!
As to Roscosmos security track record, read this:
https://en.wikipedia.org/wiki/List_of_spaceflight-related_accidents_and_incidents
Not saying accidents will not happen, just that their death toll is incredibly low, especially considering all the humans they put into orbit ...
When it comes to rockets (and aircraft), you don't compare fatalities.
You compare near-misses and vehicle losses, because vehicle losses aren't survivable.
Soyuz takes up to three people, while the Shuttle took up to seven. So you'd have to lose more than twice as many Soyuz to kill the same number of astronauts.
So given an equal death rate, you'd choose the Shuttle as you're half as likely to die.
Soyuz currently has very obvious quality problems, as there are now two that were launched in the last year with issues that should never have made it off the production line.
That is indicative of a manufacturing culture where mistakes are covered up, rather than fixed - the workers don't feel like they can say "oops, I broke it" without consequences to their livelihoods, and thus will hide that.
I work with a guy who used to be an aircraft heavy maintenance tech. We were discussing formal tool and component accountability procedures for a high quality line we are working and I asked how rivets and other fasteners in aircraft are accounted for as they are not serialized.
Tech: "They're not"
Me: ?
Tech: "You sweep and vacuum out the rivets and fasteners from wing tanks and whatever the best you can. That's why the fuel system has strainers and filters."
Me: ??
Tech: "Well, yeah, they have their limits. Thats why on your Airbus or Boeing the fuel intakes are not in the lowest part of the tank. Sometimes if the wing is gutted we will pull all the crap out of the sumps. Usually they're there forever"
Me: whimper
Tech: "To really eff things up you need a socket head floating around or similar, that's why we have accountability at that mass level. Every part, every shift"
I do love postmortems on incidents, whether techy or IT, at El Reg. Always guaranteed to bring out the best Harry Enfield style 'You don't waana do it like that from people who do every job perfectly (and if it fails it was someone else's fault). The press conference basically said the sensor got bent. It didn't say it was dropped or hammered in when it didn't fit. No one commentating (probably) has any idea what the sensor looks like and how robust it is. Or what the allowable bend might be. Maybe it was bodged, maybe it was fragile, maybe once installed it's not visible for external inspection - we don't know.
It's fair to wonder why a damageable sensor is a single point mission critical element, but not to extrapolate to systematic organisational failures on that basis. Spacecraft are designed to be light and engineering 'light enough to lift, strong enough to work' is always going to be a fine compromise.