back to article Revealed: How NASA saved the Kepler space telescope from suicide

Waking up to a phone call in the wee small hours of the morning is almost never good. It’s usually a wrong number, a drunk ex wanting to talk, or the news that someone has died. When NASA's Kepler space telescope mission manager Charlie Sobeck was woken by ringing at 0125 on April 8, it was close to the latter. The Kepler …

  1. zaax

    which is why we need space elevator

    1. Mage Silver badge

      Engineers and Scientists

      No, it's why we need good education and people doing science and maths at school.

      The jury is out on space elevators.

      Some sort of robotic repair spacecraft able to stay in a high orbit and be re-supplied from a craft docked at the ISS would be possible today.

      While I like the romance of manned (or womaned) space exploration, till we figure Starships (if ever), robotic / remote ships and probes are better. We do need orbital repair bots and orbital "sweeper" bots. Still, amazing how cheap India did a Mars Mission.

      [Even if the material issues of the cable of a space Elevator can be solved, the central platform has to be built in orbit. Then you have maintenance. The building of it needs a horrendous number of launches. Mars and the Moon are better candidates for a space Elevator as Kevlar could be used, but some other launch mechanism likely cheaper on Moon (see The Moon is a Harsh Mistress) and maybe on Mars too. A space elevator is a nice idea in a story. The reality is problematic ]

      1. I ain't Spartacus Gold badge

        Re: Engineers and Scientists

        Mage,

        I'm not sure I buy the idea of the robotic repair spacecraft. If satellites had been designed from the start for in-orbit maintenance - then a robot could be built to do the job. But given even the low level of bodging that was required on the Hubble repair mission - I very much doubt it would be possible.

        It seems to me that space tourism is going to be too expensive for decades given current technology. Isn't a Falcon 9's fuel cost something like $300,000 to orbit? So with 8 people in a Dragon capsule and totally re-usable rockets, that's still $50k minimum - and that would only be if you could do it in bulk to get the costs down, which you can't do at that price. Whereas some kind of space plane can operate with much less fuel - but it's a question as to whether we currently have the materials to build one. I'd say a 2 stage to orbit, big fat carrier jet to get to 40,000 feet and a smaller space plane looks to be the best bet - but maybe Reaction Engines will prove me wrong.

        Anyway the most likely commercial application I can see for manned spaceflight is going to be satellite repair. A team of say 4 guys, with a Bigelow inflatable space habitat, launching presumably with Dragon capsules (assuming they're going to be cheaper than Soyuz in a few years). The Dragon has multiple engine firing capability and solar panels, and it's your in-orbit taxi too. So if each mission lasts a couple of months and requires one unmanned resupply with food, fuel and parts, that's say $100m in launch costs. If you can repair or refuel 4 satellites in that time, which doesn't seem unreasonable - you ought to be able to charge $60m to give an extra ten years life to a $500m satellite.

        It's a big investment, but I'm sure the numbers work out. Particularly so if you can start building new satellites to be able to take in-orbit refuelling - and maybe even processor upgrades, new reaction wheels and such.

        Then the question comes, how much do you have to compromise satellite design for what can go up on one lauch, and self deploy? Is there a demand for a small amount of in-orbit assembly (once you've got a small workshop up there)? So you could have a bigger unit, that maybe takes two launches and a small amount of assembly. The Ikea flat-pack satellite...

        1. Mage Silver badge

          Re:If satellites had been designed from the start for in-orbit maintenance

          Yes. They should be. The Robotic / remote repair needs craft designed for it. But it is a live project as the the rubbish collection.

          However working in spacesuit is REALLY hard. A remote or robot could actually be very capable.

    2. Floydian Slip
      Boffin

      Need a bl**dy long bit of wire

      According to the story, the satellite is some 70m from earth, a space elevator would't be much help

      1. I ain't Spartacus Gold badge
        Happy

        Re: Need a bl**dy long bit of wire

        According to the story, the satellite is some 70m from earth

        If it's only 70 metres from earth, you could presumably fix it with a normal elevator...

      2. Allan George Dyer
        Boffin

        Re: Need a bl**dy long bit of wire

        @Floydian Slip - 70 metres isn't long for a cable, even ethernet can reach 100m.

        Oh, sorry, you meant 70 million miles... well, the space elevator only gets you to geostationary orbit, 22236 miles, but that's the hard part, anyway.

        1. oldcoder

          Re: Need a bl**dy long bit of wire

          No, getting to geostationary orbit is only about 2/3 of the length of the elevator - the other 1/3 is beyond that and attached to a counterweight to the elevator closer than geostationary orbit.

          And since anything beyond is going faster than orbital speed, it allows launches to anything outside without any additional expenditures.

    3. Joe Gurman

      Er, no.... but it's a good idea anyway

      Kepler is in a heliocentric orbit, so not much direct help for it, but if a space elevator were feasible, it would be very useful for servicing things in geosync orbit of which there are lots) or "tipping off" payloads with boost motors to head elsewhere.

    4. mi1400

      This is all just because they had faith in DevOps. They kept clinging to good deeds and values it preaches and hence the divine light of DevOps led them to their success in this world and the next!.. Oh fix me too.. yeah yeah just right there! ... fix me harder!!! ... and the next time phone rang at 0125 the panning out camera saw an AVN awards winner porn flick being rehearsed!

  2. david willis

    Just Like Home?.

    "Kepler has discovered hundreds, if not thousands of planets, and maybe someday we'll find one that's just like home."

    Hopefully thats a home that is less violent, less polluted and doesn't have another Donald Trump?

    1. David Knapman

      Re: Just Like Home?.

      But we *need* to find the planet of the Trumps so that we can return the one that landed here.

      1. Kharkov
        Joke

        Re: Just Like Home?.

        Now, now. One, just one, is funny. Open the door to a planet of them and, well...

        Besides, we may find out how he does that amazing thing with his hair, it looks so natural!

      2. I ain't Spartacus Gold badge
        Happy

        Re: Just Like Home?.

        Obviously, if we did discover the planet of the Trumps (a gas giant with methane atmosphere presumably?) it would be the incentive that instantly gave us world government and a space navy, plus a mission for it. I know it's not quite the way Gene Roddenberry imagined, with the Federation's first fleet on a ten year mission to seek out and destroy all weird-hair based lifeforms.

      3. TitterYeNot
        Coat

        Re: Just Like Home?.

        "But we *need* to find the planet of the Trumps so that we can return the one that landed here."

        Unfortunately, one of Kepler's lesser known early discoveries was that of a Dyson sphere being constructed around the Trump planet's solar system (home star - Hysteria Minor, located near the Running Chicken Nebula, IC 2944), so that may no longer be possible.

        Whether the sphere was constructed to keep illegal aliens out or the Trump population in is unknown, though there has been speculation that it was because of allegedly illegal genetic recombination experiments involving a proto-ape species and the common Tribble...

        1. Anonymous Coward
          Anonymous Coward

          Re: Just Like Home?.

          Whether the sphere was constructed to keep illegal aliens out or the Trump population in is unknown, though there has been speculation that it was because of allegedly illegal genetic recombination experiments involving a proto-ape species and the common Tribble...

          There's also the possibility that they built it to stop the one we have here from ever returning :)

          1. Francis Boyle

            We are all Charlton now

            I fear that one day (probably some time in November) we'll wake up to discover that The Planet of the Trumps was Earth all along.

      4. Michael Thibault
        Mushroom

        Re: Just Like Home?.

        >But we *need* to find the planet of the Trumps so that we can nuke it from orbit. It's the only way to be sure.

      5. JLV
        Alien

        Re: Just Like Home?.

        >return the one that landed here.

        Sorry Earthlings, you have him now, you get to keep him.

        mwahahaha

    2. Anonymous Coward
      Joke

      Re: Just Like Home?.

      Because you plan to migrate to that planet illegally?

  3. Pascal Monett Silver badge
    Thumb Up

    "balanced against the force of charged particles streaming from the Sun"

    Awesome. There is simply no other word to qualify finding a solution like that.

    This is what happens when Science and Engineering meet at the highest levels. Pure genius and exemplary DIY are the result. And they're probably not even going to write a paper on that. It's just business as usual at NASA.

    Thumbs up Kepler team. You deserve it.

    1. Anonymous Coward
      Anonymous Coward

      Re: "balanced against the force of charged particles streaming from the Sun"

      Previously, they could send a Shuttle to replace wheels... (I know Kepler is too far away for the Shuttle - one able to reach the L points would have been great...).

  4. Ru'

    So (if I've got this straight) they switched it off and on again? Granted, that's tricky enough to do at distance, but...

    1. Phil O'Sophical Silver badge

      The really tricky bit was hitting F8 at just the right moment to get it into safe mode...

  5. 2460 Something
    Thumb Up

    Wow, those are some people who really deserve the accolade of 'smarty pants'. Well done that team. A raised pint in your general direction!

  6. Efros
    Pint

    The bonus

    from this sort of mission is all the tricks and dodges tried and used that can be incorporated into future missions to lengthen them or make them more reliable. This is not the first time that bodges and solutions to in flight problems have significantly added to the value of the mission but also to future missions capabilities. You can't beat working under pressure to get the results you need, rarely pleasant at the time but incredibly satisfying and exhilarating when it comes together. Pint for those involved, or several!

    1. Alan Brown Silver badge

      Re: The bonus

      "from this sort of mission is all the tricks and dodges tried and used that can be incorporated into future missions to lengthen them or make them more reliable. "

      Better quality reaction wheels should be extremely high on that agenda.

      1. Anonymous Coward
        Anonymous Coward

        Re: The bonus

        "Better quality reaction wheels should be extremely high on that agenda."

        If your car already ran for seven years continuously without any servicing whatsoever, I think that a better quality engine might not be high on the agenda.

      2. Anonymous Coward
        Anonymous Coward

        Re: The bonus

        "Better quality reaction wheels should be extremely high on that agenda."

        Doubtful, they were designed (as a system of wheels) to last 3.5 years. The redundant wheel was invoked after 3 years, and the whole system failed after 4 years (the bodges have followed).

        "Better quality" generally = "more expensive". Why would they spend more money than was needed to meet the requirement?

  7. allthecoolshortnamesweretaken

    "Waking up to a phone call in the wee small hours of the morning are never good. It’s usually a wrong number, a drunk ex wanting to talk, or the news that someone has died."

    Well, I once got a phone call at 3am from a drunken ex (not mine) that wanted to talk about the recent death of his first(ex) wife. Took a few minutes to convince him that he got the wrong number and I wasn't his third (ex) wife. Having got that of my chest I'd like to say:

    If this world we live in was even close to how it should be*, teams like that and the work they do would be celebrated in the way we celebrate guys kicking footballs or drive a fast car round and round.

    * According to me, of course. YMMV.

  8. kmac499

    Self organising team

    "I was impressed with everyone’s ability to help when they could, and to stay out of the way when they couldn’t."

    What a superb compliment to an obviously very skilled group.

    1. Martin

      Re: Self organising team

      Indeed.

      In my whole career, I have never once met a manager who has learned the trick of the second part.

      1. Annihilator
        Coat

        Re: Self organising team

        "In my whole career, I have never once met a manager who has learned the trick of the second part."

        Well, no, if you had a manager who stayed out of the way when they were unable to help, you'd pretty much never meet them.

        Unless you checked under a table leg to see if they were stopping it from being wobbly, I can't think where else they would be...

  9. Anonymous Coward
    Alien

    Soft errors in electronic memory

    "We don’t yet know what spawned the problem, and we may never know, but the first effects that we’ve found were a sudden series of alarms that caused the onboard fault protection to react"

    How about a cosmic ray hitting a memory cell and flipping it. ref.

    1. Anonymous Coward
      Anonymous Coward

      Re: Soft errors in electronic memory

      Agreed. With so many high energy particles, one of them is bound to pass thro CPU / memory at some point and flip some bits.

      1. lglethal Silver badge

        Re: Soft errors in electronic memory

        Its certainly the most likely cause, but its next to impossible to prove with any level of certainty. Hence why they call this a transient event.

        You don't claim something you cant prove without a very high level of certainty in the space biz. Its far too costly when you find out you were wrong later! ;)

      2. PaulFrederick

        Re: Soft errors in electronic memory

        we use radiation hardened electronics where that cannot happen. The pitch size of the components themselves is simply too big. It is like trying to deface a billboard with a shotgun. Yeah sure you'll put some little holes in it, but you're not going to obscure the message. This ain't our first rodeo bucko.

    2. Alan Brown Silver badge

      Re: Soft errors in electronic memory

      "How about a cosmic ray hitting a memory cell and flipping it."

      ECC (actually ECC on steroids), redundant control computers. It'd take more than one bit-flip to scramble things.

    3. Anonymous Coward
      Anonymous Coward

      Re: Soft errors in electronic memory

      I'm assuming they have ECC and one bit flip doesn't matter. They may be happening all the time. It's the event that manages to either (a) flip a whole lot of bits or (b) crowbar a power circuit that you have to worry about.

    4. Seajay#

      Re: Soft errors in electronic memory

      I've heard a few people say "No. My code can't be wrong, I've checked. The only way I can explain the crash you saw is cosmic rays." Invariably they're wrong and it's their bug. This system is exposed to more cosmic rays but it is also better protected against them.

      On the principle of "if you hear hoofbeats, think horses not zebras" I'm going to say that if there was a software glitch, it was probably a software bug.

      1. Anonymous Coward
        Anonymous Coward

        Re: Soft errors in electronic memory -cosmic rays

        Cosmic ray background radiation at sea level a little under 0.3mSv/year

        Radiation at Earth orbit outside the influence of the Earth around 400mSv/year

        It isn't a small difference, it is a factor of more than 1000 times. What's more, there are some really big particles out there that never reach the bottom of Earth's atmosphere, particles with multi-Joule energies.

        Your observation may well be correct as regards terrestrial computing, but conditions where Kepler orbits are completely different. If you were in Mara Masai rather than, say, Hampshire, and heard a lot of hoofbeats, would you still assume horses rather than zebras?

  10. Tezfair
    Happy

    awesome

    Need to make a movie with keanu reeves as Charlie

    1. Swarthy
      Meh

      Re: awesome

      Woah.

  11. Old Handle

    Well done team! But it seems like these emergency software modes on spacecraft have a record of doing more harm than good. Maybe I just don't remember it, but have you ever read a story where it says "foocraft suffered a bar failure, but fortunately it when into emergency mode enabling us to resume contact and continue its mission"? As far as I can recall it's always "foocraft is stuck in barmode for some reason and we're trying to get it running again."

    1. Ken Hagan Gold badge

      Without an emergency mode, the story would be "Foocraft suffered a bar failure and its game over. Thanks for playing.". Don't blame the emrgency mode for the situation that it lets you recover from.

      1. Annihilator

        Without an emergency mode, the story would be "Foocraft suffered a bar failure and its game over. Thanks for playing.".

        More accurately it would be "Foocraft has gone silent - we haven't heard from it since last Tuesday. Missing, presumed dead"

    2. Seajay#

      There are cases where emergency modes saved the day. Spirit went in to fault mode after landing. Worked out very well in the end.

      http://www.spacedaily.com/news/mars-mers-04zh.html

    3. I ain't Spartacus Gold badge

      Old Handle,

      They often go into safe mode (or one of the several safe modes), which is fine and dandy. And usually easy to recover from, though it takes ages - as NASA tend to do a lot of thinking before deciding to make changes.

      In this case I guess they'll have had to react faster than they'd have liked, because they were losing so much fuel. But they obviously had some sort of major error - either bug, cosmic ray or possibly an interaction of both.

      What if the main processors had been affected long enough that they couldn't reboot? Then communication would have been lost, the solar panels lock on the Sun would have gone, so no power, and therefore no way to recover the craft.

      By having this backup system take over you're taking out a bit of insurance that might save your bacon at some point. And has at several points in the past. Like all insurance it has a cost, in this case wasted time on the DSN disrupting other projects, and some wasted fuel.

      In a lot of cases you won't need such an expensive fail-safe, but where you're getting power from solar panels that require alignment you probably do.

  12. Version 1.0 Silver badge

    Great Story!

    That's a great story, well written and with a good ending - there are so many lessons to be learned when you operate in that sort of environment, unable to touch and see the problem, just relying on a data stream or someone's description of events - or their interpretation of events.

    BLOODY WELL DONE EVERYONE!

  13. Oengus
    Pint

    Going "Above and Beyond"

    I think the entire team deserves the rest of the afternoon off and a beer (or two) for their effort.

  14. Florida1920
    Pint

    People who can do this stuff

    are the gods of our times. It's too bad so much of humanity is unable to appreciate what gifts they bring to us.

    1. Anonymous Coward
      Anonymous Coward

      Re: People who can do this stuff

      "are the gods of our times. It's too bad so much of humanity is unable to appreciate what gifts they bring to us."

      Look at some of the religions we've managed to invent and you understand that being compared to gods is not a compliment.

      The human ideas of gods are usually either sociopathic power maniacs, indulgent narcissists or murderous thugs. What this tells you about the qualities the average person respects in someone else is left as an exercise for the reader. But don't forget there are still people in the East End who kind-of-worship the Krays.

  15. Dr Patrick J R Harkin

    So the fix was -

    Turn it off and reboot in Safe Mode?

  16. Comedy of Errors

    So... you just turned it off and on again?

    "After rebooting the main computers the readings from the thrusters, wheels and instruments showed all systems were normal... it’s an unknown problem that sorted itself out"

    So... you just turned it off and on again and that mysteriously fixed the problem? Even rocket scientists do it.

  17. anonymous boring coward Silver badge

    "The incident has now been classed as a "transitory event" – meaning it’s an unknown problem that sorted itself out."

    How lame is that?

    Don't they have logs to analyse?

    Normally you would want to learn from an event like this.

    One obvious lesson is to not have dramatic "panic modes" that get invoked too quickly.

    Guess NASA is just fed up with Kepler now, after all hacking to keep it going..

    1. PaulFrederick

      They're running Systemd so the logs are encoded in binary. No one has a clue how to decypher them now either.

      1. anonymous boring coward Silver badge

        Perhaps they use Windows 10 Embedded Everywhere Edition? (Auto "upgraded" from W7)

        A scheduled maybe-update at 3am disrupted normal operation...

  18. alferdpacker

    Aliens!

    Obviously somebody doesn't want us to find these exo planets.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like