back to article Hubble, Hubble, toil and trouble: NASA pores over moth-eaten manuals ahead of switch to backup hardware

NASA has completed a formal review of what engineers will have to do to switch the Hubble Space Telescope to its backup hardware. The formal review comes more than a month after a problem with a payload computer aboard the veteran observatory sent the telescope into safe mode and engineers scurrying for contingency plans. …

  1. Hero Protagonist
    Holmes

    Step 1

    “Turn it off and back on”

    (Presumably they’ve already tried that…on to step 2 then)

    1. katrinab Silver badge
      Black Helicopters

      Re: Step 1

      I would be extremely nervous about rebooting a computer remotely when there is no chance of a visit to the computer to turn it back on if the reboot command fails.

      1. Little Mouse Silver badge
        Coat

        Re: Step 1

        Only $134 for a single ILO licence. I bet those boffins are kicking themselves now...

        1. David 132 Silver badge
          Happy

          Re: Step 1

          They should just ask the REvil or Cosy Bear gangs for help. Or have NASA found the only computer in the solar system of which Russia/China haven't got clandestine control?

      2. FuzzyTheBear
        Mushroom

        Re: Step 1

        Nervous ? I am not .. ill push the button myself if that's all that's needed . Hell , it's fried as it is .. What could possibly go wrong ? 8)

      3. Diez66

        Re: Step 1

        Every time I bog up a remote computer, they are in the same building so "not in a far away galaxy".

        I am so chuffed I do not make the software for these beasts.

        Also, I am very aware that no one in their mind would consider me for the job.

        YAY!

    2. bernmeister

      Re: Step 1

      I am sure they would not do that. The standard IT support advice does not work well in orbit. Its OK on the bench and might work but a lot of settings might need to be reset and loss of the spacecraft might occur. Notice how many mights were in that sentence? There are probably a few more unknowns that are unknown.

  2. Hubert Cumberdale Silver badge

    This has the potential to be all too familiar, only turned up to 11 – I'm not surprised they're carefully checking all possible scenarios. I've more than once found myself pressing Enter in an SSH session to a router only to realise my mistake a fraction of a second later as the connection is "unexpectedly closed" (yes, it is accepted that I'm an idiot). Difficult enough when it's three hundred miles away, even more so if the direction of those miles is up.

    1. Paul Crawford Silver badge

      Most of us have done it at some point, perhaps a new firewall rule that blocked SSH, or a reboot command that failed to complete as a remote machine hung on failing hardware and crap firmware that never times-out, etc, etc.

      That amp it up to 11 !

      1. Giles C Silver badge

        As I have said to people I work with, find an engineer who has never done any of the above and we have found either a liar or they have only worked in the field for a couple of weeks.

        All of us have done this and the honest ones admit to doing it and will explain how they recovered from the problem.

        1. A Non e-mouse Silver badge

          I met an instructor who said they liked students who made lots of mistakes as the students learned a heck of a lot more from undoing the mistakes than those that got things right all the time.

          It sounds corny, by mistakes are great learning experiences. Just don't make them a habit ;)

          1. andypbw

            Experience and Luck

            You start with an empty bag of experience, and a full bag of luck.

            The trick is, to make sure you fill the former before you empty the latter.

            1. Mike 137 Silver badge

              Re: Experience and Luck

              There's also a need for intuition - which is not guesswork, but in the words of the late Robt. Ornstein "arrival at a correct answer without recourse to inference". It's based on accumulated experience but experience alone is only a part of it.

              For example, in the days of dumb terminals, I was thrown a dead one to fix. A huge board covered with TTL chips, no manual and no circuit diagram. It suddenly occurred to me that TTL is tough stuff, so I raided the digital chip drawers in the electronics shop and, one at a time, I sat a replacement chip on each of the onboard devices and turned on. Half a dozen chip tests down the line the terminal sprang to life. So I turned off, replaced the said chip and everything was fine. I haven't a clue why that approach occurred to me - it's distinctly oddball, but it succeeded. That's intuition, and without it the terminal would have been scrap.

          2. Julian 8

            Had this during training once. All the machines had been left from the previous class and only a few bits were "reset". During the training I had something fail and would consistently fail. Most of the class left and there was the trainer, myself and one other person left. It took us around 2 hours to find and fix the problem, but it did reoccur a few times over the next few years with that app.

            Without that course I would never have known

            The trainer said he preferred to leave the machines this way instead of reimaging as a freshly imaged machine would not have these problems.....

          3. Arthur the cat Silver badge

            It sounds corny, by mistakes are great learning experiences.

            A quote from James Davis Nicoll(*) that turned up on tor.com a few days ago:

            Nothing teaches one not to try to stamp out burning thermite quite like real-life experience.

            (*) If you've never heard of him, the man's a first class accident magnet with the resulting collection of tales to match.

            1. Gene Cash Silver badge
              Pint

              Holy crap. I looked at his wikipedia entry, and he's the originator of 5 different pieces of wisdom I've picked up independently over my lifetime. Definitely a bloke I'd buy a few beers at the pub.

            2. Red Ted Silver badge
              Flame

              Putting out fires

              The first step is always to understand is what it is that is on fire, before you try to put it out!

              See icon —>>>

          4. David 132 Silver badge

            Or phrased another way:

            "How do you avoid mistakes? Experience..

            How do you get experience? Through making mistakes.

            1. Anonymous Custard Silver badge
              Headmaster

              The often quoted mantra here:

              Knowledge is what you learn to do.

              Wisdom is what you learn to do again.

              Experience is what you learn never to do again.

              1. David 132 Silver badge

                I like that. A related variant I heard years ago:

                Knowledge is knowing that the tomato is a fruit.

                Wisdom is knowing not to put it in a fruit salad.”

                1. Anonymous Coward
                  Anonymous Coward

                  "Wisdom is knowing not to put it in a fruit salad."

                  That's not wisdom - that's bias. A tomato has the texture and sweetness that fits in the range of fruits one eats like that. It is not even at an extreme of that range.

          5. Down not across Silver badge

            It sounds corny, by mistakes are great learning experiences. Just don't make them a habit ;)

            However it is true. And making it a habit is not a problem either, as long as you learn from your mistakes and don't make same mistakes repeatedly.

            1. Anonymous Coward
              Anonymous Coward

              Or learn from other peoples' mistakes.....

          6. Trigonoceps occipitalis Silver badge

            “Wisdom comes from experience. Experience is often a result of lack of wisdom.”

            Terry Pratchett

        2. Ex-Code Monkey

          The value of mistakes you've made and won't make again is underappreciated.

          1. Anonymous Custard Silver badge
            Headmaster

            There's only one thing more valuable than learning from your mistakes, and that's learning from other people's mistakes before your end up making them yourself.

        3. Anonymous Coward
          Anonymous Coward

          Ka-ching! thank you :-)

          I'll take your bet - and your money. My father has never screwed up a firewall, rebooted the wrong router/server/disk array, or had remote hardware or software go into an unrecoverable loop.

          But then he's a licenced aircraft engineer, not just an electron-botherer. ;-)

          1. Trixr

            Re: Ka-ching! thank you :-)

            And I'm sorry, but I do not believe that he NEVER made a mistake on any production IT gear.

            Yes, as time goes on, you certainly learn not to make stupid mistakes, and in highly-regulated, relatively static environments, it's much less likely you'll make a mistake, especially one that has a noticeable effect.

            I'll even accept no mistakes that cause an outage - certainly possible in said highly-regulated and highly-redundant environments. I personally never had any downtime in Exchange while running it over 15 years, as one complex system I dealt with.

            But never, ever any mistakes, from the beginning of his career? I'll have to take that with a big grain of salt, I'm afraid.

          2. Anonymous Coward
            Anonymous Coward

            Re: Ka-ching! thank you :-)

            neither has mine. To be fair though, he was a psychiatric nurse

    2. Vometia has insomnia. Again.

      Guilty as charged. Did that from home to a router at work in London and found myself having to catch the Stansted Express early on Sunday morning. I figured it would at least be quiet but it was packed with sweaty journalists. I didn't find out until later that day that my router woes had been somewhat eclipsed by the news that Diana had died.

      So I remembered that much, but did I learn anything? *cough*

      1. Hubert Cumberdale Silver badge

        Maybe you learned to breathalyse your chauffeur before getting in the car.*

        *Too soon?

        1. Yet Another Anonymous coward Silver badge

          And if you are going to get into a limo with a drunk driver and recreate Ronin - wear a seat belt

  3. HammerOn1024

    Sounds Like...

    Another repair mission is needed. I'd thinking a Dragon on Falcon Heavy could easily lift the necessary hardware and personnel to Hubble. The issue is reentry: Can Dragon take the higher reentry speeds?

    If it can, then bring along new stabilization gyro's and a shortened Canada Arm to grab the satellite. Break out the shuttle space suites for on-orbit repairs while your at it.

    1. aregross

      Re: Sounds Like...

      I would be thinking first of a small drone-like unit that can be sent up to visually inspect dear Hubble for F.O.D. (Foreign Object Damage)

      1. iron Silver badge

        Re: Sounds Like...

        X-37

        Small, drone like, reusable and probably in space on a secret mission right now.

        https://en.wikipedia.org/wiki/Boeing_X-37

    2. Little Mouse Silver badge
      Mushroom

      Re: Sounds Like...

      A repair mission would be far preferable to the "NASA has a plan to send up a de-orbiting module that attaches to Hubble and drives it into orbital decay mode to allow it to be brought down into the ocean or on unpopulated land" mission that El-Reg reported a few years ago...

      1. Yet Another Anonymous coward Silver badge

        Re: Sounds Like...

        NASA fitted an attachment point at the last Shuttle servicing mission - but there is no plan to use it.

        The current plan is to let it reenter and not worry because it can't hit the continental USA (except for bits of Florida that nobody cares about)

        1. Jim Mitchell Silver badge

          Re: Sounds Like...

          There are definitely bits of Florida I would prefer to be impacted over others. Can no one think of the Manatees!?

          1. David 132 Silver badge
            Coat

            Re: Sounds Like...

            I can visualize it now...

            Burning space telescope crashes down onto a large sea cow, news reporter loses his cool and is caught on audio weeping "Oh, the huge manatee!!!"

            1. John Brown (no body) Silver badge
              Coffee/keyboard

              Re: Sounds Like...

              Bastard! ------------>

            2. Trixr

              Re: Sounds Like...

              Honestly, chapeau. That's the best ever El Reg riposte and I have seen a few...

            3. dc_m

              Re: Sounds Like...

              You sir, are a genius.

        2. Anonymous Coward
          Anonymous Coward

          Re: Sounds Like...

          >> The current plan is to let it reenter and not worry because it can't hit the continental USA (except for bits of Florida that nobody cares about)

          Is Mar-a-Largo within reach?

          1. Yet Another Anonymous coward Silver badge

            Re: Sounds Like...

            Mar a Lago = 26.67N

            Hubble orbit inclination = 28.5 deg

            So yes, Cthullu willing, it could hit Florida man

    3. Anonymous Coward Silver badge
      Big Brother

      Re: Sounds Like...

      If only it were that simple

    4. RobThBay

      Re: Sounds Like...

      Where's Elon's space Roadster these days?

    5. iron Silver badge

      Re: Sounds Like...

      Dragon isn't a pickup truck, you can't just add an arm and we're good to go. It would need a complete redesign. Dragon lacks a proper airlock to support spacewalks for a start.

      1. Gene Cash Silver badge

        Re: Sounds Like...

        Sure, but in a situation like this, they can probably depressurize the cabin a couple times, especially if they bring spare cylinders. Gemini did it.

        1. LDS Silver badge

          Re: Sounds Like...

          Could an astronaut wearing a EVA suit exit and enter the Dragon? Is there space to dress and undress the suit?

          Gemini was designed to let astronauts get out with its two large "doors", and for astronauts wearing the suits.

          Moreover a complete platform to catch the Hubble and keep it while astronauts can work around it with al the required tools and replacement parts available would have to be designed and build. And would need to fit an available rocket - people tend to forget how large the cargo bay of the Shuttle was.

          There is a lot of space capabilities that have been lost with the Shuttle retirement.

          1. Yet Another Anonymous coward Silver badge

            Re: Sounds Like...

            >Could an astronaut wearing a EVA suit exit and enter the Dragon? Is there space to dress and undress the suit?

            Couldn't we just send Geordies astronauts ?

            1. Paul Cooper

              Re: Sounds Like...

              >Couldn't we just send Geordies astronauts ?

              Only female ones!

    6. awavey

      Re: Sounds Like...

      Incredibly risky mission for the astronauts even with Shuttle, its multiple times riskier now, it's something Starship is probably more capable of than Dragon eventually,but I think youd struggle to even get through the hatch of Dragon in a shuttle style space suit to begin with. Let alone "capture" the telescope in a safe manner that astronauts on EVAs could work on it.

      Look it's done 30+ years of service, it was only designed to do 15, if you compare it to ground based telescopes the costs/science actually Hubble is a poor choice now, but it's pretty pictures capture the public imagination so it gets cited way more often as delivering that special science dividend,when we are learning more about the universe from stuff sitting on tops of mountains.

      Let them try to fix it if they cant just accept it's done its job and move on, the James Webb telescope though not a direct replacement by any means launches in October assuming ESA can resolve the cargo fairings issue.

      1. Jon 37 Silver badge

        Re: Sounds Like...

        Since they already have a working and debugged design that they have already manufactured, launched and operated once, can't they just make a new Hubble and launch it?

        Note that I don't mean an "improved" design, I mean a straight copy but with the mirror shaped correctly this time. We know that an "improved" design is expensive and time consuming. We know that Hubble, as-is, is doing good science, but will fail soon.

        I'm wondering how much a new Hubble would cost compared to the price of a servicing mission.

        1. Down not across Silver badge

          Re: Sounds Like...

          Well, they don't necessarily need to start from scratch...

          NRO donated couple of Kennens to NASA

          https://spaceflightnow.com/2016/02/18/nasa-moves-forward-with-mission-using-spy-satellite-telescope/

          https://www.planetary.org/articles/nasa-gets-two-hand-me-down

          Plenty more articles on that around.

        2. Cuddles Silver badge

          Re: Sounds Like...

          "Since they already have a working and debugged design that they have already manufactured, launched and operated once, can't they just make a new Hubble and launch it?"

          Who is "they"? It was designed and built 30 years ago. Many of the companies involved in manufacturing will no longer exist, many of the people who actually did the design and build have likely retired, the plans on record are likely far from complete and in any case many things have changed since it was first built and there's virtually no chance all the designs for each bit are actually in the same place. There will be materials no longer used, components no longer available, processes now all completely different, and many, many other issues. Yes, in theory everything required should be properly documented and neatly filed away. In practice, that is never actually the case.

          Once things get past a certain age, especially with a one-off project that used bespoke parts, it almost always ends up much easier to build something new from scratch rather than try to reproduce the same thing again. It's easier to design something that does the job you want that it is to design something identical to an existing thing.

      2. Yet Another Anonymous coward Silver badge

        Re: Sounds Like...

        >can't they just make a new Hubble and launch it?

        Hubble was super-compromised by it's need to work with Shuttle, and be built from a KH-11 spy satelite. It's not what you would build today.

        Arguably neither is JWST, if you assume launches are now relatively cheap and regular you might go for a production line of 2-3 year launch cycle with new instrument technology on a common bus.

        For all it's whizzy space technology - the best thing Hubble did in terms of research/$ was the whole Space Telescope Science Institute, the IRAF software infrastructure, the data archive and the funding for post-docs.

        It led to a much more efficient processing and publishing than typical astronomy of the time - although ground based has learned a lot from it.

    7. John Brown (no body) Silver badge

      Re: Sounds Like...

      Does Dragon have an airlock? I thought all it had was a "half-airlock" docking port.

  4. x 7

    Would......

    closing and reopening all the windows work?

    1. arachnoid2 Bronze badge

      Re: Would......

      It might have got stuck in a loop updating to windows 11.

  5. Doctor Syntax Silver badge

    It's getting difficult to keep typing with my fingers crossed. let's hope it works OK. If it does then it's time to reflect on how much more life it might have and prioritise observations.

  6. Scott Broukell
    Alien

    If it all fails to come back up then perhaps they could turn it into one of those IOT doorbell systems for alien visitors to use? Just a thought.

    1. Red Ted Silver badge
      Go

      Alien visitor: “Oh,I wonder what this button does?” *clang* “owww, it just hit me” as the primary cover opens past their head!

      1. David 132 Silver badge

        "Ow. This makes me very angry, very angry indeed. Where's my Illudium P-36 Explosive Space Modulator?"

        1. js.lanshark

          How many of you just read that in the voice of Marvin the Martian?

          1. Cynical Pie

            Where's the KAboom?

            1. arachnoid2 Bronze badge

              KAboom

              Theres Klingons on the starboard bow, starboard bow, starboard bow.........

              1. David 132 Silver badge

                Re: KAboom

                “…Scrape them off, Jim!”

  7. picturethis
    Holmes

    Experience is what you get....

    When you didn't get what you wanted...

  8. Ol' Bit Slinger

    "Is it plugged in?"

    1. arachnoid2 Bronze badge

      Has someone set the transformer to the incorrect voltage?

  9. MrBanana Silver badge

    Repair slot available next week

    Too late to ask Beardy Branson, but they could give Bezos The Bald a space hammer, some gaffer tape, and ask him to "go outside". It would do us all a favour.

  10. rmstock

    Build a new one on the ground

    Even better, build several. For sure build a replica and build one which has the latest tech inside. There's several body shop's who do things like `Restomod' for late 60's and early 70's muscle cars. The same mentality should be applied at NASA.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021