back to article Once considered lost, ESA and NASA's SOHO came back from the brink of death to work even better than it did before

Welcome to the final episode in The Register's series on engineering longevity in space. We conclude with the joint ESA and NASA project, more than 24 years into a two-year mission: the Solar and Heliospheric Observatory (SOHO). Launched atop an Atlas II-AS from Cape Canaveral Air Force Station on 2 December 1995 at 08:08 UT, …

  1. Pete 2

    If it ain't broke

    the team were using an update command sequence ...

    It got worse. After the momentum management manoeuvre, gyro B was erroneously left in its high-gain setting (thanks to a change in command procedures)

    1. I ain't Spartacus Gold badge
      Pint

      Re: If it ain't broke

      Pete 2,

      I'd imagine it's cost issues, combined with flight experience. You'll note that later in the article it states that SOHO control is now operated by many fewer people - and I'd imagine that means more automation.

      Obviously this is partly a function of cost saving. Can we automate things, to save money? And you might object to that, but government funded science is always going to suffer from that.

      But there's also a much better reason for cost savings. Doing more science. It's now common for space missions to over-run their design lifetimes by orders of magnitude. The Opportunity rover on Mars (14 years instead of 3 months), being an awesome case in point. Yes, we know, that spacecraft are over-engineered - and have many backup systems. Plus we're getting better at using software and clever tricks to overcome a situation when even our back-ups have failed - using different systems to overcome that - in SOHO's case using the star trackers as gyros, or Kepler using thrusters to overcome the failure of reaction wheels.

      But this has a problem. If you're only budgeted for having x number of active missions to control in 5 years time, and now you've got 2x still working, what are you to do? You either delay new missions, until you've got the staff to operate them, abandon perfectly good spacecraft, or ask for a bigger budget. Which you might not get.

      My final point is that we gain experience by flying these missions. Each spacecraft is unique, even when they use some common parts. Along with different mission profiles, that means you get better at controlling them over time, you understand the problems - and have a better idea of what needs to be done. 3 monthly maintenance procedures may not be worth automating on a 12 month mission - where you're only doing it 4 times - but on a mission that now lasts 10 years I'd suggest the chances of someone getting it wrong one of the now 40 times you're doing it manually becomes greater than the risk of problems with automation.

      Beers all round! Especially to the guys still getting the 2 Voyagers to do science. Although I think they lose power for the radios this decade.

    2. bombastic bob Silver badge
      Devil

      Re: If it ain't broke

      the team were using an update[d] command sequence ...

      updates. highly overrated! [even for meat-space procedures]

    3. spold Silver badge

      Re: If it ain't broke

      It's a tribute to the engineers... these days if something is designed to last two years - like a washing machine - you can bet your ar** it will be f**ked around then - there is money to be made in the replacements. Kudos to all, that from soup to nuts, they made the highest quality something, within whatever budget was negotiated, and that has performed well beyond spec... and to the peeps that have continued to do help it do that.

  2. Pascal Monett Silver badge
    Thumb Up

    A fitting tribute to intelligence and sheer dogged determination

    The entire history of SOHO is a demonstration of what the human race can accomplish when the best minds are determined to solve a problem.

    The goof about Gyro A that almost scrapped the mission in the first place ? I would blame the UI for that. The status of all gyros should always have been visible to the operations team. Apparently, they only had the reading without knowing the status. An easy mistake to make when things are going pear-shaped.

    Everything else is just the brilliance of engineers. There's a reason they are somewhat apart in the world, and this whole saga is proof of why : they get results when everyone else thinks it's the end of the line.

    That is why Humanity needs to get into space. It brings out the best in us.

    We'll find a way to fuck that up, but still, we need to go to space.

    1. EricM

      Re: A fitting tribute to intelligence and sheer dogged determination

      > they get results when everyone else thinks it's the end of the line.

      Second that.

      However, I often wonder if engineering mindsets could also reshape other areas, where rationality and the will and capability to make things work (instead of the opposite) often appears to be in short supply.

      Politics comes to mind, national and international, where engineers are strongls underrepresented.

      Wrangling control somewhat away from manager types, economists and lawyers might prove worthwile also down here ...

      Then again, I understand every engineer that fails to find enthusiasm for politics at the current state of affairs ...

      1. Arthur the cat Silver badge

        Re: A fitting tribute to intelligence and sheer dogged determination

        However, I often wonder if engineering mindsets could also reshape other areas, where rationality and the will and capability to make things work (instead of the opposite) often appears to be in short supply.

        Politics comes to mind, national and international, where engineers are strongls underrepresented.

        The problem is that politics involves people, and they are generally not too happy if you try to force on them an upload sequence that makes them operate in a different fashion. Hardware is a lot more biddable than meatware.

        1. EricM

          Re: A fitting tribute to intelligence and sheer dogged determination

          > The problem is that politics involves people

          Yep, I need to agree :)

          > and they are generally not too happy if you try to force on them an upload sequence that makes them operate in a different fashion.

          Oh, but that exists. It's called "convincing" people ... And even engineers have to do that a lot.

          > Hardware is a lot more biddable than meatware.

          I'm not so sure about that point. Looking at the state of politics, conspiracy theories and the like, it seems rather easy to convince at least some people of very weird things.

          If I upload code to hardware it at least has to be syntactically and logically correct :)

          1. I ain't Spartacus Gold badge

            Re: A fitting tribute to intelligence and sheer dogged determination

            Both hardware and meatware problems can be easily fixed - so long as you have big enough hammers.

            The only reason that hardware is easier to deal with is that it hasn't yet worked out how to get its own hammer.

          2. Pascal Monett Silver badge
            Trollface

            Re: syntactically and logically correct

            True, but that still doesn't mean it's going to do what you expect it to.

            1. Doctor Syntax Silver badge

              Re: syntactically and logically correct

              Still less what you intend it to do.

        2. Missing Semicolon

          Re: A fitting tribute to intelligence and sheer dogged determination

          Don't the Politburo in China all have STEM degrees?

          1. tfb Silver badge

            Re: A fitting tribute to intelligence and sheer dogged determination

            More interestingly perhaps, what is Angela Merkel's background, and how is Germany doing with CV19? Hmm.

    2. Thoguht Silver badge

      Re: A fitting tribute to intelligence and sheer dogged determination

      It's great being able to find wonderfully creative solutions to problems, but it would be far better not to have the problems in the first place. It's a bit like the old adage about never employing someone who says they are good at debugging.

      1. hoola Bronze badge

        Re: A fitting tribute to intelligence and sheer dogged determination

        One has to wonder if they outcome would have been the some if it occurred now. There appears to be so much general incompetence in the management chains that the people who really understand what can be done find their voice is lost.

        Some really amazing work to recover the mission using good old fashioned thinking and problem solving.

    3. bpfh Silver badge
      Joke

      Re: A fitting tribute to intelligence and sheer dogged determination

      That's quantum mechanics for you. You can know position or speed, but not both at the same time

    4. oldfartuk

      Re: A fitting tribute to intelligence and sheer dogged determination

      This is why you dont let programmers design UI's. They assume too much of the user.

  3. Loyal Commenter Silver badge

    Fleck has spent nearly 28 years with SOHO and became emotional when discussing the eventual deactivation and safe disposal of the spacecraft, which will consist of a final uploaded command sequence.

    I wonder if there is enough fuel left in those tanks such that, when the day comes, it can be pushed away from L1 and into the sun. It seems that it would be a fitting burial.

    1. Spherical Cow Bronze badge

      Nice idea but the ∆v required would be enormous. Not happening.

      1. Michael Hoffmann
        Alien

        All I know about space flight comes from KSP

        You know, until recently I wouldn't understood what you were talking about: "why not point it at the sun and hit the thrust button?"

        Then I started playing Kerbal Space Program. Now I was able to nod sagely at your comment.

        ElReg, could we have a Kerbal icon? The ET one doesn't quite work here.

    2. avo

      There is still over 100kg fuel left. Fuel use per year is less than 1kg. So getting it out of L1 is no problem.

      But not into the sun.

      1. A.P. Veening Silver badge

        Getting it out of L1 is the problem, that is a relatively stable position for stuff. A serious burn against the orbital direction should result in a slowly decreasing orbit around the sun and finally result in a close enough approach to the sun (unless Venus or Mercury comes in between). It will however take a long time.

        1. tfb Silver badge
          Boffin

          Whatever orbit you can get it into will either get close enough to the Sun that atmospheric friction dumps enough energy to put it into the Sun (possibly after many orbits) or, modulo being able to do some slingshot around something else, it will be slowly decaying only in the silly sense that gravitational radiation will, eventually, make the orbit decay, for suitably large values of 'eventually.

          Orbits don't just gently decay: they need to dump all that energy and angular momentum somewhere.

  4. Peter Prof Fox

    Do another deep-freeze experiment first

    As the team have shown how it's possible to bring it 'back from the dead', why not make a planned close-down with the intention of seeing what can be resurrected after say 12 months. Not only would this be instructive about the toughness of dormant technology on long space voyages, but also, if the instruments 'work better than before' then long time base observations can be useful to measure gradual change.

    1. stiine Silver badge

      Re: Do another deep-freeze experiment first

      Or we might be able to increase the performance of instruments by powering the spacecraft off for 1 orbit?

  5. Doctor Syntax Silver badge

    "the final episode in The Register's series on engineering longevity in space."

    When does series 2 start? You know you're going to need it.

    1. I ain't Spartacus Gold badge
      Happy

      The second series will be when Voyager returns, in its new repaired/upgraded form.

      The interviews with the engineers in question might be rather harder to do though.

      1. Doctor Syntax Silver badge

        Or https://xkcd.com/1504/

      2. Martin J Hooper

        We;ve already seen that - See Star Trek the Motion Picture :)

  6. Will Godfrey Silver badge
    Thumb Up

    Never give up

    That's the lesson here, and it applies to all forms of survival.

  7. HildyJ Silver badge
    Pint

    Systems who need people

    This is the difference between artificial intelligence and real human intelligence. When systems encounter an unexpected situation they get stuck. It takes boffins to figure out why and unstick them. Beers all around (and continued funding).

  8. bazza Silver badge
    Pint

    Great Article, Thanks

    That was a fabulous article, a thoroughly good read. Many thanks :)

    1. Archivist

      Re: Great Article, Thanks

      Thoroughly agree!

      I note the initial cock-up was to "save time". This has been a common theme in cock-ups I've seen and been involved in :(

      Writing a procedure takes a lot of time and testing. If you are going to change it, at least give the changes as much thought (and testing as you can) as the original procedure.

      Of course nothing can take away the fantastic skills and thinking of the team to recover the craft. In a world where a lot of people are doing really stupid things, it's great to see what humankind can achieve.

      1. oldfartuk

        Re: Great Article, Thanks

        Its absolutely true in all computing. The more time you spend making the algorithms and UI utterly foolproof, the more it pays off in less support calls from the great unwashed. Its also why you dont let programmers design UI's unsupervised.

    2. Falmari

      Re: Great Article, Thanks

      Agreed best part of the day was reading that article

  9. bonkers
    Pint

    Luck?

    Brilliant article, what a plot!..

    Not sure luck had much to do with anything though - its good luck was to have skill, imagination and perseverance (heh) in its ground team.

    I would love to have an idea of what resources it had, in terms of CPU, MHz, RAM, FLASH, etc? - just to get some sort of calibration.

  10. Uncle Ron

    Thanks

    All I can say is Thanks So Much for this great article. A very nice read. Well Done.

  11. RegGuy1 Silver badge
    Linux

    These articles are great!

    Thanks el Reg for yet another very interesting article.

    We are lucky that ESA is not part of the EU, otherwise the monkeys in No 10 would like to see our contributions scrapped there too. We've already lost so much by their stupidity, and this piece shows if we ever pull out of ESA that would be a big loss.

    Kudos to all the minds involved. We're living in an age where we are learning exponentially more about our solar system (and beyond) than we've ever learned before.

    It's the muppets on the ground we need to be wary of. :-(

    Penguin -- because I'm sure all those ESA engineers just used their Windows skills -- not!

  12. max allan

    Back then...

    I think they were on Dec/Vax VMS on the ground for SOHO. They later moved to Solaris. And as I was leaving ESA about 10 years ago, starting to get some penguins.

    I imagine space craft themselves were and always will be very specialised systems.

    Although there were always some windows machines around with stuff like Excel for doing non critical stuff.

    If you get a chance to work there, do. It is quite unlike anywhere else I've worked. (Darmstadt in Germany is where the operations centre is, so I guess working there will become more challenging when we leave the EU...)

    1. Snapper

      Re: Back then...

      "like Excel for doing non critical stuff"

      I don't like it doing non-critical stuff!

    2. Joe Gurman

      Re: Back then...

      hp/UX.

  13. nojava

    Keep it for education

    Rather than decommission it should be turned or to some educational society. National Geographic or such. Even Amateur Astronomers.

    1. Joe Gurman

      Re: Keep it for education

      Know any amateurs who have US$20M or $25M a year for the Deep Space Network time? That's how much NASA Is chipping in every year to keep the mission running.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020