back to article Hubble memory errors persist despite NASA booting long-idle backup payload computer

NASA has fired up the Hubble Space Telescope's backup payload computer to find that the spacecraft still has problems. The primary payload computer halted on 13 June, ceasing collection of science data and tripping the veteran observatory into Safe Mode. Troubleshooting has continued ever since. Last week NASA engineers …

  1. Bartholomew
    Pint

    I wonder if ...

    If NASA put all the people who worked on this 30+ years ago, and who are still alive, in multiple virtual rooms and had them bounce around a few ideas. Would it be cheaper and faster than having people today look at the problem. Would 30+ year old knowledge (some of which would no doubt be obsolete, upgraded by standard scientific rigor) converge on the exact cause of the problem and maybe suggestions for possible solutions (if any exist) faster ?

    1. Bartholomew
      Happy

      Re: I wonder if ...

      Maybe I should explain a bit more, since everyone seams to think that I'm totally wrong.

      Hubble was supposed to be launched in 1986, but the space shuttle Challenger explosion in January 1986 delayed the launch by nearly four years. So in reality most of the core backbone systems of the Hubble would have been lock in stone at least a couple of years before then. So you could be talking about technology from the early 1980's forming the backbone system of Hubble. The stuff that is not easily swapped out the core functionality in the heart of the chassis is hard to change without a major redesign. And that is rarely done once the money has been allocated. And changing it after launch on a system that was never designed for upgrades would be next to impossible.

      1. Anonymous Coward
        Anonymous Coward

        Re: I wonder if ...

        No because there have been 30 years of patches, fixes and hacks put in place to work-around faults that have arisen and also to improve results from science instruments as better ways to get more out of them have been developed. The "old guys" would have to spend ages getting up to speed with the current state of affairs versus the way they left it. On top of that - assuming they are still in the industry - they will have moved on to more modern electronics so may not actually be as well versed as the current team are anyway.

        1. Electronics'R'Us
          Holmes

          Re: I wonder if ...

          On top of that - assuming they are still in the industry - they will have moved on to more modern electronics so may not actually be as well versed as the current team are anyway.

          You seem to be assuming that moving on to more modern electronics means they don't remember the older stuff, but that is not the way it works (for me, at least).

          I have been moving to 'more modern' electronics for decades but I can still remember the details of the (much) older stuff I worked on 50 years ago; the modern stuff builds upon the knowledge of the older stuff. There are newer methods of achieving the same results (software defined radio comes to mind) but doing a radio with purely analogue electronics is still perfectly achievable although there may be limitations (in both instances here).

          I did my first multi-gigabit switch over 20 years ago; the underlying rules for that and more modern parts are the same but there are details that are not that important at lower speeds that can be major issues as the speeds become faster. These types of issues are not confined to modern processors - as an example, modern switching power supplies are a very different beast today although the functionality (from a top level perspective) is the same.

          Layout has certainly changed over the years; what was appropriate in the 80s (with 70s and 80s era parts) is not appropriate for newer parts but that is because the newer parts have to be dealt with in context of their I/O speeds, not because the underlying physics has changed.

          I currently work with very modern microcontrollers but I also work with a decades old radar system and I am quite comfortable with both.

          It is perfectly possible that the original designers (and those who developed the various hacks over the years) have retired or even passed on but that is why documenting the design thoroughly (very likely more in depth for something designed in the 80s than present) is incredibly important.

      2. Field Marshal Von Krakenfart
        Boffin

        Re: I wonder if ...

        Hubble has the NASA Standard Spacecraft Computer-1 (NSSC-1) computer which was developed at the Goddard Space Flight Center (GSFC) in 1974. A co-processor for the DF-224 was added during Servicing Mission 1 in 1993, which consisted of two 80386 processor and 80387 math co-processors.(wikipedia).

    2. arachnoid2

      Re: I wonder if ...

      Whats needed is Clint Eastwood, Tommy Lee Jones, Donald Sutherland and James Garner to go up there and deactivate the nukes.

    3. Annihilator

      Re: I wonder if ...

      Not really, for at least two reasons;

      1) Hubble is a bit of a Trigger's Broom these days. Fairly sure that the entire telescope has been replaced over the various servicing missions with the exception of the case and the primary mirror.

      2) Their remotely trying to fix it and following a diagnostic process and flowchart. Jumping about at random with guesses (which is what it would be) is a risky game. The methodical approach they're following is the correct one. You seem to be making the assumption that minimising recovery time is the priority here, it's not.

  2. Kevin Fairhurst

    Have they tried

    Turning it off, and on again?

    1. bazza Silver badge

      Re: Have they tried

      Always worth asking, but I think that in this case they have tried that.

    2. John Robson Silver badge

      Re: Have they tried

      It's something you avoid doing in space - if it doesn't boot then you're completely out of options.

      Of course the advantage of redundant equipment is that they can boot up the second compiter whilst still talking to the first.

    3. bombastic bob Silver badge
      Coat

      Re: Have they tried

      yeah those "El Cheapo" power supplies never last very long, do they?

      Maybe they could blow it all out with some canned air, and put a desktop fan on it to help keep it cool...

    4. NoneSuch Silver badge
      Coat

      Re: Have they tried

      They have, but the notification that they will be turning it off and on again between 1 and 5PM keeps getting returned in the Post.

  3. Christopher Reeve's Horse
    Coat

    Perhaps...

    the telescope has taken one PEEK too many?

    Maybe it just needs a POKE?

    etc...

    1. Chris G

      Re: Perhaps...

      Precisely!

      A well considered application of percussive maintenance is probably all that is stopping it from it's celestial peeping.

    2. Anonymous South African Coward Bronze badge

      Re: Perhaps...

      Maybe a RANDOMIZE USR 1331 command for some funky stuff?

      BTW try it out on a Speccy and see what happens :)

      1. Bartholomew

        Re: Perhaps...

        I have never used a speccy in my life, but willing to try:

        Went to https://jsspeccy.zxdemo.org/ selected 48k spectrum and "48k BASIC".

        Then I needed to look up the needed keys at https://en.wikipedia.org/wiki/Sinclair_BASIC because nothing is ever simple.

        So "RANDOMIZE USR 1331" you say:

        T (for RANDOMIZE)

        shift+AltGR (To change the default cursor from a L to E - EXTENDED MODE)

        L (for USR while in EXTENDED MODE)

        1331 (1331)

        Then I hit the Enter key and wondered was it worth it ?

        1. PTW
          Pint

          Re: Down voted...

          'cos you're a miserable curmudgeon! You do realise that almost anyone that knew that command is an old fart like me, and it took them back to the early days of home computing. That's what got most of us here [both el reg, and in IT as a profession]. And it made me smile, and think of simpler, happier times.

          Anonymous South African Coward have a beer on me, I went from a ZX81 to a C64, but plenty of friends had Speccies. There was also a weird POKE command on a 32k BBC B that would make it 16k and it'd persist across reboots. It needed another POKE to fix it, thankfully the Comp Sci Master saw the funny side in the end.

  4. Pete 2 Silver badge

    Forgot to renew the support agreement?

    > The primary payload computer halted on 13 June

    Because the warranty ran out on 12 June.

    1. Robert Moore
      Coat

      Re: Forgot to renew the support agreement?

      >> The primary payload computer halted on 13 June

      > Because the warranty ran out on 12 June.

      They really should have paid for the extended warranty.

  5. Disgusted Of Tunbridge Wells Silver badge
    Alien

    Somebody didn't like being looked at.

    1. Korev Silver badge
      Coat

      Yeah, it’s in a lot of Hubble…

  6. Anonymous Coward
    Anonymous Coward

    Jeff, while you're taking Blue Origin out for its run, could you just take a little detour via Hubble... oh, and can you remember to get a pint of milk on the way back

    1. The commentard formerly known as Mister_C Silver badge
      Joke

      Way back?

      What is this "way back" you're asking Mr B to perform?

    2. imanidiot Silver badge

      They'll be about 460 km altitude short, and a whole lot of delta-V.

    3. Chris 239
      Joke

      Now there's a conundrum.

      Is getting Hubble back online actually worth having Bezos get back from his little hop? Probably not!

  7. x 7

    Faulty windows?

    Or faulty mirrors\?

    1. The commentard formerly known as Mister_C Silver badge
      Facepalm

      Re: Faulty windows?

      still too soon?

  8. TVC

    Give it a thump? Somehow.

    Any opportunity for a bit of percussive maintenance?

  9. Neil Barnes Silver badge
    Coat

    I hope they didn't install bubble memory back in the day...

    Because, of course,

    Hubble bubble, toil and trouble...

  10. Winkypop Silver badge
    Pirate

    Expired McAfee?

    Literally

  11. Intractable Potsherd

    Pessimism time

    I hope we aren't going to lose Arecibo and Hubble in the same 12 months. Come on NASA, work your magic!

  12. Charles Smith

    Holey smoke

    I think there's hole in my puter, Dear Lisa Dear Lisa. There's a hole in my puter .....

    1. arachnoid2

      Them damn aliens throwing rocks at our space debris

      A micro meteor insertion into the power supply maybe

    2. Anonymous Coward
      Anonymous Coward

      Re: Holey smoke

      With what shall I fix it, Dear Lisa, Dear Lisa - with what shall I fix it?

      With your back-up puter, Dear Charles, Dear Charles - with your back-up puter.

      ...There's a hole....

    3. BugabooSue
      Happy

      Re: Holey smoke

      On a total tangent, you just triggered a happy memory...

      When working for the Ministry of Silly Walks (back in the 80's) as an "Electronics Bod" I secured a Mac IIcx for doing 'lab work' <cough> so we could capture waveforms using devices via an IEEE-488 interface card and play them back using an arbitrary waveform generator.

      The IIcx had expansion slots and used a 'Real-sized' colour monitor as standard unlike the little one on SE30, et al.

      However, every time I wanted to do something I would find there was no driver available, or more likely -

      "This program is not supported on a Lisa..."

      Swapped to PCs for lab work after that, but I was left with the IIcx as a fantastic machine for CAD, and writing reports of the HUGE 24" screen!

      Nobody else wanted it because the screen itself took up 70% of a desk and the computer about 20% of the rest.

      Fantastic in the winter, but in the summer, you could fry an egg on the screen!! :D

      I miss my "Lisa."

  13. WargRider

    WargRider

    Send a couple of actual humans up with some spares and poke it with a stick alternatively we could have jeff bezos try to get it to work for peanuts after all this is what amazon workers are paid.

  14. sp1nyn0rman

    I think someone uploaded some new OS software that demanded TPM2.0 compatibility...

  15. Archivist

    When I learned fault finding

    I was told: always check the power supply first.

    1. Anonymous Coward
      Anonymous Coward

      Re: When I learned fault finding

      We had a very intermittent disk search problem that was causing subtle corruptions in a database. Disk controller power supplies were checked every week and were "within tolerance". Turned out that one particular type of logic board needed the voltage to be "spot on".

      1. arachnoid2

        Re: When I learned fault finding

        She canna take any more Captain.

  16. Locomotion69
    Holmes

    "it is highly unlikely that all individual hardware elements have a problem"

    Last time I heard that statement while troubleshooting a problem it made me go right there to look for it.

    And for good reason...... Statistics are just "numbers".

    Needless to say it was indeed a bunch of malfunctioning units which was supposed to be statistically "impossible".

    1. Anonymous Coward
      Anonymous Coward

      Re: "it is highly unlikely that all individual hardware elements have a problem"

      Occam's Razor is a good general rule - but an intractable problem is sometimes due to several concurrent root causes - each producing apparently the same overall symptoms.

      You get suspicious when you are in a situation of "chasing your tail". Especially if you fixed an obvious problem - that was actually counteracting another problem elsewhere. Two wrongs can make a right.

      1. Hazmoid

        Re: "it is highly unlikely that all individual hardware elements have a problem"

        reminds me a recent issue with a windows 10 machine failing to recognise the domain network ity was connected to. Tried various fixes and even used a USB ethernet adapter to rule out the card. Turns out that the common factor was a shitty 5 port switch with 2 cables in it that was no longer needed. As soon as the switch was removed and the machine directly connected to the wall port, the situation resolved itself. Only took me 3 hours of numerous reboots and google searching, before I found the network switch hidden at the back of the desktop machine :)

  17. jylee

    If the computer cannot be fixed, can the hubble keep it's orbit ?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like