Once considered lost, ESA and NASA's SOHO came back from the brink of death to work even better than it did before Welcome to the final episode in The Register's series on engineering longevity in space. We conclude with the joint ESA and NASA project, more than 24 years into a two-year mission: the Solar and Heliospheric Observatory (SOHO). Launched atop an Atlas II-AS from Cape Canaveral Air Force Station on 2 December 1995 at 08:08 UT, …
Pete 2,
I'd imagine it's cost issues, combined with flight experience. You'll note that later in the article it states that SOHO control is now operated by many fewer people - and I'd imagine that means more automation.
Obviously this is partly a function of cost saving. Can we automate things, to save money? And you might object to that, but government funded science is always going to suffer from that.
But there's also a much better reason for cost savings. Doing more science. It's now common for space missions to over-run their design lifetimes by orders of magnitude. The Opportunity rover on Mars (14 years instead of 3 months), being an awesome case in point. Yes, we know, that spacecraft are over-engineered - and have many backup systems. Plus we're getting better at using software and clever tricks to overcome a situation when even our back-ups have failed - using different systems to overcome that - in SOHO's case using the star trackers as gyros, or Kepler using thrusters to overcome the failure of reaction wheels.
But this has a problem. If you're only budgeted for having x number of active missions to control in 5 years time, and now you've got 2x still working, what are you to do? You either delay new missions, until you've got the staff to operate them, abandon perfectly good spacecraft, or ask for a bigger budget. Which you might not get.
My final point is that we gain experience by flying these missions. Each spacecraft is unique, even when they use some common parts. Along with different mission profiles, that means you get better at controlling them over time, you understand the problems - and have a better idea of what needs to be done. 3 monthly maintenance procedures may not be worth automating on a 12 month mission - where you're only doing it 4 times - but on a mission that now lasts 10 years I'd suggest the chances of someone getting it wrong one of the now 40 times you're doing it manually becomes greater than the risk of problems with automation.
Beers all round! Especially to the guys still getting the 2 Voyagers to do science. Although I think they lose power for the radios this decade.
Well said. I was stuck doing maintenance on a system that required maintenance occasionally. I found it annoying so I developed an automated process to take care of it. I didn’t ask management for permission, and when they found out what I had done, they were oh so happy to take credit for it. I didn’t care. It saved me from annoyance..
It's a tribute to the engineers... these days if something is designed to last two years - like a washing machine - you can bet your ar** it will be f**ked around then - there is money to be made in the replacements. Kudos to all, that from soup to nuts, they made the highest quality something, within whatever budget was negotiated, and that has performed well beyond spec... and to the peeps that have continued to do help it do that.
The entire history of SOHO is a demonstration of what the human race can accomplish when the best minds are determined to solve a problem.
The goof about Gyro A that almost scrapped the mission in the first place ? I would blame the UI for that. The status of all gyros should always have been visible to the operations team. Apparently, they only had the reading without knowing the status. An easy mistake to make when things are going pear-shaped.
Everything else is just the brilliance of engineers. There's a reason they are somewhat apart in the world, and this whole saga is proof of why : they get results when everyone else thinks it's the end of the line.
That is why Humanity needs to get into space. It brings out the best in us.
We'll find a way to fuck that up, but still, we need to go to space.
> they get results when everyone else thinks it's the end of the line.
Second that.
However, I often wonder if engineering mindsets could also reshape other areas, where rationality and the will and capability to make things work (instead of the opposite) often appears to be in short supply.
Politics comes to mind, national and international, where engineers are strongls underrepresented.
Wrangling control somewhat away from manager types, economists and lawyers might prove worthwile also down here ...
Then again, I understand every engineer that fails to find enthusiasm for politics at the current state of affairs ...
However, I often wonder if engineering mindsets could also reshape other areas, where rationality and the will and capability to make things work (instead of the opposite) often appears to be in short supply.
Politics comes to mind, national and international, where engineers are strongls underrepresented.
The problem is that politics involves people, and they are generally not too happy if you try to force on them an upload sequence that makes them operate in a different fashion. Hardware is a lot more biddable than meatware.
> The problem is that politics involves people
Yep, I need to agree :)
> and they are generally not too happy if you try to force on them an upload sequence that makes them operate in a different fashion.
Oh, but that exists. It's called "convincing" people ... And even engineers have to do that a lot.
> Hardware is a lot more biddable than meatware.
I'm not so sure about that point. Looking at the state of politics, conspiracy theories and the like, it seems rather easy to convince at least some people of very weird things.
If I upload code to hardware it at least has to be syntactically and logically correct :)
It's great being able to find wonderfully creative solutions to problems, but it would be far better not to have the problems in the first place. It's a bit like the old adage about never employing someone who says they are good at debugging.
One has to wonder if they outcome would have been the some if it occurred now. There appears to be so much general incompetence in the management chains that the people who really understand what can be done find their voice is lost.
Some really amazing work to recover the mission using good old fashioned thinking and problem solving.
Fleck has spent nearly 28 years with SOHO and became emotional when discussing the eventual deactivation and safe disposal of the spacecraft, which will consist of a final uploaded command sequence.
I wonder if there is enough fuel left in those tanks such that, when the day comes, it can be pushed away from L1 and into the sun. It seems that it would be a fitting burial.
You know, until recently I wouldn't understood what you were talking about: "why not point it at the sun and hit the thrust button?"
Then I started playing Kerbal Space Program. Now I was able to nod sagely at your comment.
ElReg, could we have a Kerbal icon? The ET one doesn't quite work here.
Getting it out of L1 is the problem, that is a relatively stable position for stuff. A serious burn against the orbital direction should result in a slowly decreasing orbit around the sun and finally result in a close enough approach to the sun (unless Venus or Mercury comes in between). It will however take a long time.
Whatever orbit you can get it into will either get close enough to the Sun that atmospheric friction dumps enough energy to put it into the Sun (possibly after many orbits) or, modulo being able to do some slingshot around something else, it will be slowly decaying only in the silly sense that gravitational radiation will, eventually, make the orbit decay, for suitably large values of 'eventually.
Orbits don't just gently decay: they need to dump all that energy and angular momentum somewhere.
As the team have shown how it's possible to bring it 'back from the dead', why not make a planned close-down with the intention of seeing what can be resurrected after say 12 months. Not only would this be instructive about the toughness of dormant technology on long space voyages, but also, if the instruments 'work better than before' then long time base observations can be useful to measure gradual change.
Thoroughly agree!
I note the initial cock-up was to "save time". This has been a common theme in cock-ups I've seen and been involved in :(
Writing a procedure takes a lot of time and testing. If you are going to change it, at least give the changes as much thought (and testing as you can) as the original procedure.
Of course nothing can take away the fantastic skills and thinking of the team to recover the craft. In a world where a lot of people are doing really stupid things, it's great to see what humankind can achieve.
Brilliant article, what a plot!..
Not sure luck had much to do with anything though - its good luck was to have skill, imagination and perseverance (heh) in its ground team.
I would love to have an idea of what resources it had, in terms of CPU, MHz, RAM, FLASH, etc? - just to get some sort of calibration.
Thanks el Reg for yet another very interesting article.
We are lucky that ESA is not part of the EU, otherwise the monkeys in No 10 would like to see our contributions scrapped there too. We've already lost so much by their stupidity, and this piece shows if we ever pull out of ESA that would be a big loss.
Kudos to all the minds involved. We're living in an age where we are learning exponentially more about our solar system (and beyond) than we've ever learned before.
It's the muppets on the ground we need to be wary of. :-(
Penguin -- because I'm sure all those ESA engineers just used their Windows skills -- not!
I think they were on Dec/Vax VMS on the ground for SOHO. They later moved to Solaris. And as I was leaving ESA about 10 years ago, starting to get some penguins.
I imagine space craft themselves were and always will be very specialised systems.
Although there were always some windows machines around with stuff like Excel for doing non critical stuff.
If you get a chance to work there, do. It is quite unlike anywhere else I've worked. (Darmstadt in Germany is where the operations centre is, so I guess working there will become more challenging when we leave the EU...)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
