back to article A borked bit of code sent the Hubble Space Telescope into safe mode, revealing a bunch of other glitches

The Hubble Space Telescope resumed science operations this morning after a software error knocked the veteran spacecraft offline. In what sounds for all the world like an on-orbit Blue Screen of Death, a software update uploaded to the spacecraft attempted to write to a location in computer memory to which it didn't have …

  1. Anonymous Coward
    Anonymous Coward

    Test before deployment ?

    Or they moved to "Move fast and break things" DevOps paradigm and other stuff like that.

    1. Martin Summers

      Re: Test before deployment ?

      It sounds more like they've just hired Boeing to do the dev now.

    2. eamonn_gaffey

      Re: Test before deployment ?

      That, plus it sounds like they may not have a test rig that reflects production on the Hubble....or if they do, adequate testing was not performed.

      Still, sounds like they had sufficient contingency/resilience to enable a fix, which is impressive given the nature of the live environment.

    3. bombastic bob Silver badge
      Devil

      Re: Test before deployment ?

      yeah what I was thinking - they need to have a 'cloned' Hubble (simulator at least) upon which they stage EVERY update. And if it did not catch THIS problem, it needs to be fixed...

      1. Wzrd1 Silver badge

        Re: Test before deployment ?

        Or tested it in a virtuous virtual environment, which lacked the virtue of being faithful to the physical environment.

        Such as a failed patched environment that doesn't reflect what's actually in use. Muddle the testbed, back it out poorly, everybug passes after...

  2. karlkarl Silver badge

    Hubble Space Telescope as a Service means that it always getting better!

    We listen to our user feedback and ensure that any crucial features are monetised appropriately.

    1. yetanotheraoc Silver badge

      Boilerplate

      A limited number of Hubble Space Telescopes have been affected by this issue.

  3. quxinot

    Be nice, guys. For some perspective:

    Hubble was launched in 1990.

    Windows 3.1 was launched in 1992, and is thus more modern.

    Which would you rather use?

    1. G Watty What?
      Coat

      It depends...

      Can Hubble run Minesweeper?

      1. cookieMonster Silver badge
        Joke

        Re: It depends...

        Can it print??

        1. TaabuTheCat

          Re: It depends...

          Can it print?

          It could until the last patch Tuesday.

          1. zuckzuckgo

            Re: It depends...

            >It could until the last patch Tuesday.

            For the Hubble that would be eye-patch Tuesday.

      2. Anonymous Coward
        Holmes

        Re: It depends...

        It's last CPU upgrade was to an Intel 486 so if you can program it, it should run.

        1. Wzrd1 Silver badge

          Re: It depends...

          Precisely the nature of the failure kill chain.

        2. 89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921

          Re: It depends...

          Actually, it was a 486 Overdrive, some might remember what those are. Still are as well as were, obviously, because it's still up there. So there's at least one still working. Probably.

    2. Bruce Ordway

      For some perspective

      In the late 80's I worked briefly at Perkin-Elmer in Minneapolis.

      A long commute for me but, it was one of the "cooler" places where I spent any time. They were building a lot of "neat stuff" (I thought), including some of the Hubble components.

      Last I heard Perkin-Elmer has become "a shadow of it's former self"

      Too bad if that's true.

      1. Anonymous Coward
        Anonymous Coward

        Re: For some perspective

        Now PerkinElmer, that can't be good

    3. I miss PL/1

      Not even windows

      I think the Hubble uses a Commodore 64.

  4. Sparkus

    alternate servicing..

    Could a pair of Orions, or Dragons reach the HST?

    1. Anonymous Coward
      Anonymous Coward

      Re: alternate servicing..

      Sorry, "might one day" doesn't mean "can now".

    2. Anonymous Coward
      Anonymous Coward

      Re: alternate servicing..

      Maybe, but without all the hardware needed for servicing, and the EVA capabilities support... unless you bring them there with another vehicle, that, without re-entry capabilities, means everything would be destroyed after the mission.

      1. Sparkus

        Re: alternate servicing..

        Might be a good what-if exercise for NASA or SpaceX to work on; what would it take in terms of a combined Crew Dragon & Cargo Dragon mission to perform a possible next round of service on Hubble?

        What kind of 3 or 4 port mission support docking module could be developed, quickly, to ease the task?

        1. hoola Silver badge

          Re: alternate servicing..

          That would require some joined up thinking that appears to be way beyond most of the people they have around today.

          Bluntly, there is no servicing capability and no sign of anything being put together that could remotely do a tenth of what the Shuttle could. All the focus has been on getting bigger lumps of stuff into space, not how we might usefully service and maintain them.

          There is some work by ESA to refuel satellites with a standard interface but that is nothing compared to the handling capacity needed to actually maintain assets.

          The costs and potential for things not working on James Webb are just mind-blowing. Yes, it is a sophisticated package but the knock-on of this extravagance is already being felt and will continue to do so for years. I would also be very surprised if it's functional life is anything approaching that of Hubble.

          1. Anonymous Coward
            Anonymous Coward

            "The costs and potential for things not working on James Webb"

            Moreover being at L2 it's beyond the reach of anything operative today - and beyond the Shuttle range too, of course.

            Instead of designing propaganda missions I would have invested in broadening space operations beyond low orbit. It would also build the basis for longer range missions to the Moon and Mars. But that plan is not useful for political propaganda.

        2. Anonymous Coward
          Anonymous Coward

          Re: alternate servicing..

          The Shuttle provided the capability for several EVAs - which means being able to store the EVA suits, and let astronauts exit and reenter several times. AFAIK the Dragon does not allow that.

          Then the cargo bay of the Shuttle had the Canadian Arm to capture and keep the Hubble safe and steady while maintenance activities were performed and let easy access to all the replacement parts needed. Getting them out a Cargo Dragon while wearing EVA suits I don't believe is easy or even possible. Nor I believe the Cargo can be fitted with external devices for maintenance work and keep its re-entry capabilities. A separate platform with such devices would not be able to re-enter intact as well, and would be an expensive non re-usable one.

          I don't remember if the Shuttle also powered the Hubble when it was not able to do it itself, to keep instruments safe. The Shuttle also could restore an higher orbit.

          The Shuttle was an outstanding orbital workshop, and should have been used and kept operative for that task alone - the idea of using it as a launcher was stupid. Its real limitation was it could only reach low orbits.

          Today whatever is not attached to the ISS can't be repaired in orbit.

          1. Byham

            Re: alternate servicing..and the bean counters

            You are expecting politicized bean counters to think constructively like engineers. That has never been known.

            What does it cost? When do you want to spend it? Is it in the existing budgets? You have now exceeded the most constructive bean counter responses. Even "there is an exposure to even more costs if you do not spend this" is outside their cognitive capabilities.

            Service life and service life costs - unrepaired and repaired should be part of the costings of every large item put into space. That also requires the design to allow in orbit repair and someone thinking ahead to what would need to be repaired then ensuring the designers considered that in the system designs. This would raise the ire of the bean counters on the large item projects and the arguments would be rehearsed again.

          2. yetanotheraoc Silver badge

            Re: alternate servicing..

            "A separate platform with such devices would not be able to re-enter intact as well, and would be an expensive non re-usable one."

            ?? Just leave it in orbit, obviously. Non re-enterable != non re-usable. Only people need to re-enter, plus whatever ship is needed to move people.

            1. Anonymous Coward
              Anonymous Coward

              "Just leave it in orbit"

              It would need maintenance as well, be able to move to the required device to be maintained, able to receive the spare parts and refueled. Not impossible, true, but probably even more complex than something that can return to earth.

              Instead of a Cargo Dragon, probably something alike the X-37 would be more practical.

    3. Alan Brown Silver badge

      Re: alternate servicing..

      an ion tug definitely could - and has been postulated as a way of extending lifespan as well as deorbiting it when the sad day comes (it's too big to allow to come down randomly)

      1. herman Silver badge

        Re: alternate servicing..

        Well, if Boeing is involved, then bringing it down is not a problem.

      2. Anonymous Coward
        Anonymous Coward

        Re: alternate servicing..

        A tug can only resolve the decaying orbit issue and re-entry after being EoL'd... won't be useful to replace gyros, solar panels, instruments...

  5. Anonymous Coward
    Anonymous Coward

    revealing a bunch of other glitches

    After quite a long day, and perhaps because rather more used to "space-probe success" news of late, I read that as "revealing a bunch of other galaxies".

    .

    Well, I thought, that *was* a fortuitous occurrence!

    .

    ... but sadly not. But at least the old thing is still hanging in there...

  6. deive

    Time for NASA to switch to rust?

  7. Fruit and Nutcase Silver badge
    Alert

    toil and trouble

    A blast of sunlight could do all manner of damage to the sensitive instruments inside the observatory.

    Double, doubleHubble, Hubble toil and trouble;

    Fire burn and caldron bubble.

    1. itzman
      Headmaster

      Re: toil and trouble

      ...cauldron...

      1. Fruit and Nutcase Silver badge

        Re: toil and trouble

        ah, yes.

    2. Kane

      Re: toil and trouble

      The night was as black as the inside of a cat. It was the kind of night, you could believe, on which the gods moved men as though they were pawns on the chessboard of fate. In the middle of the elemental storm a fire gleamed among the dripping furze bushes like the madness in a weasel’s eye. It illuminated three hunched figures. As the cauldron bubbled an eldritch voice shrieked: ‘When shall we three meet again?’

      There was a pause.

      Finally another voice said in far more ordinary tones: ‘Well I can do next Tuesday.’

      - Wyrd Sisters, Terry Pratchett

      GNU

      Pterry icon, El Reg?

  8. AndrueC Silver badge
    Joke

    Bloody Windows updates.

    1. David 132 Silver badge

      In this case it’s more like, “bloody Door updates”.

  9. ThatOne Silver badge
    Unhappy

    Hubble is fragile

    This is one of the reasons we can't just discard the thousands of terrestrial telescopes just to accommodate Starlink, as some suggested here.

    On earth a stuck barn door or some broken support modules would just require the janitor to grab a screwdriver and a ladder; In space it requires using some extremely expensive and currently unavailable transport, so you risk all the time to lose the totality of the telescope (and investment) just because of some malfunctioning .50 cents part you could had fixed down here with duct tape while waiting for a spare.

    Space is big, and unlike SciFi movies tend to suggest, it's more than a "whoosh" and a "zoom" away.

    1. Ken Moorhouse Silver badge

      Re: Space is big

      Deep, actually.

      https://www.youtube.com/watch?v=cULwlnEok1c

      (Memories of one of the best concerts I ever went to)

      1. tiggity Silver badge

        Re: Space is big

        Lost count how many times I have seen them (or offshoots) - seemed for years that every festival I went to Hawkwind were on the bill somewhere (in addition to going to "normal" gigs where they were the main or support act)

    2. Anonymous Coward
      Anonymous Coward

      Re: Hubble is fragile

      On a much less exotic note, that is similar to an argument we used to put to suppliers in the early days of North Sea oil development. A compressor (say) in a normal factory could be fixed by a service engineer turning up in a white van; even local maintenance folk could fix things with spares arriving by van in a few hours. Not so easy on a platform 100 miles out in the N.Sea. Small parts could be sent out on the next helicopter (most likely the next days crew-change flight) but larger parts would need a separate chopper (expensive - though often a lot less than the cost of not repairing asap) or by boat (which would often be weekly). The lost production cost of breakdowns can be eye-watering!

      Setting aside the spares issue, a service tech can't get on a chopper without appropriate certification (including medicals and survival training).

      As I said, not as exotic as HST but the comment took me back.

      As an aside (and off topic) I recall a dispute I had with a supplier of downhole pumps, regarding the limited testing given before delivery. The supplier happily replied that they would replace any faulty pump at no charge. I asked if that meant at their cost; we agreed to split the risk and forego back-charging lost production - but the cost of removing and reinstalling one of their pumps started at $2m. The supplier rep stuck to his guns. So did we, and switched supplier (to one that built a dedicated test rig) for future pumps. As the first supplier's pumps failed (the operator had already installed quite a number and the failure rate had triggered my involvement), they were replaced with the competitors...

  10. Anonymous Coward
    Anonymous Coward

    Hubble saw aliens. NASA shut it off.

    We all know that's what really happened.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hubble saw aliens. NASA shut it off.

      For all you think that's what really happened. For all you think, anyway.

  11. sbt
    Facepalm

    I'm surprised they tried the backup motor

    I mean, what if they'd closed the door and it wouldn't re-open. I hate those kinds of repair attempts, you try fix a minor problem that isn't really causing any harm and the whole thing falls apart in your hands.

    1. Anonymous Coward
      Anonymous Coward

      Re: I'm surprised they tried the backup motor

      The first Law of Support in any field. "If it ain't broke - don't fix it".

      Someone once discovered - in passing - an obvious error in a payroll package. They fixed it - and payroll failed. There were actually two errors - whose effects cancelled each other out.

      1. Wzrd1 Silver badge

        Re: I'm surprised they tried the backup motor

        If it's broke, we can fix it.

        If it ain't broke, we can fix that too.

      2. AndrueC Silver badge
        Facepalm

        Re: I'm surprised they tried the backup motor

        I spent a fair chunk of last Thursday fixing a collection of unit tests that had a similar problem. I removed a redundant class and suddenly found myself facing over a dozen test failures relating to code that I hadn't directly touched.

        I was in a bad mood by the time I was done.

      3. Grumpy Fellow
        Go

        Re: I'm surprised they tried the backup motor

        Yes, I've found out the hard way that Mom was wrong. Actually, any even number of wrongs do make a right.

  12. Version 1.0 Silver badge
    Pint

    Cheers to NASA

    Working in an environment that can have problems like this one, better testing might have helped but it's no guarantee. I think that NASA did very well to actually find the issue and fix it. I've worked with NASA engineers in the past and it's always been fun, they see complex problems and fix them even if it takes the rest of us days to even figure out what might be the problem.

    Cheers and Beers to NASA for keeping Hubble running!

    1. Ken Moorhouse Silver badge

      Re: fix them even if it takes the rest of us days to even figure out what might be the problem

      It's a pity that so many of us are constrained by the "time is money" mantra.

      1. ThatOne Silver badge

        Re: fix them even if it takes the rest of us days to even figure out what might be the problem

        While I agree that, unlike everywhere else, NASA decisions aren't taken by accounting department, given the price of Hubble (or other space probes) it is justified to have any amount of engineers spend the time necessary to find a good solution (as opposed to a quick/cheap one).

    2. Anonymous Coward
      Anonymous Coward

      Re: Cheers to NASA

      In any endeavour it is always wise to build in some fuzzy flexibility*** to allow some chance of fixing an unexpected problem. In the days when Crosby "Quality" courses were mandatory - that was usually called "over engineering" by our PHBs.

      We called it "open-ended design" - which often became "here's one I made earlier" solutions for problems or enhancements.

      ***"In Search of Excellence" author Tom Peters described a successful collaboration between multiple suppliers - that had deliberately fuzzy interface definitions between their modules.

  13. PaulR79
    Joke

    Typical NASA cheaping out

    I bet they didn't pay for the launch day "door closing" DLC and just hoped it was included in the version they bought. SMH my H.

  14. Adrian 4

    And the others ?

    Since Hubble is widely believed to be a prototype for a line of inward-pointing spy telescopes, are all those worn out too ?

    1. Norman Nescio

      Re: And the others ?

      Since Hubble is widely believed to be a prototype for a line of inward-pointing spy telescopes, are all those worn out too ?

      Actually, it is the other way round. NASA chose to use a 2.4 metre mirror in Hubble instead of the originally suggested 3 metre mirror as "changing to a 2.4-meter mirror would lessen fabrication costs by using manufacturing technologies developed for military spy satellites."

      Reference: NASA: The Power to Explore: Chapter XII - The Hubble Space Telescope: page 483

      The 2.4 metre mirrors were initially used in the Keyhole KH-11 Kennen series of surveillance satellites. - these pre-date Hubble.

      "A perfect 2.4 m mirror observing in the visual (i.e. at a wavelength of 500 nm) has a diffraction limited resolution of around 0.05 arcsec, which from an orbital altitude of 250 km corresponds to a ground sample distance of 0.06 m (6 cm, 2.4 inches). Operational resolution should be worse due to effects of the atmospheric turbulence."

      NN

      1. Alan Edwards

        Re: And the others ?

        Hubble can't focus that close though. It can't focus on the Moon let alone Earth

    2. Paul Hovnanian Silver badge
      Big Brother

      Re: And the others ?

      The Keyhole satellite series predates Hubble by a decade or two. (Maybe more. But nobody's talking.) And they are regularly replaced, thanks to the bottomless funding pit of the NRO and other intelligence services.

      If NASA was funded to the same degree, the JWST would not only have launched by now, we'd be on JWST number 5.

      1. davidp231

        Re: And the others ?

        "If NASA was funded to the same degree, the JWST would not only have launched by now, we'd be on JWST number 5."

        And we'd probably have V'ger knocking on our door in a couple of centuries.

    3. CAPS LOCK

      Re: And the others ?

      Didn't the NRO give NASA a couple of spares?

      [Edit] Aha, yes, in 2011. They're using one to creta a new space telescaope - https://en.wikipedia.org/wiki/Nancy_Grace_Roman_Space_Telescope

  15. jobst

    Normally SysAdmins can walk to the machine

    .... but that hardly possible here, is it?

  16. Alan Edwards

    Hard reset

    That's a long trip to plug in a USB keyboard to Ctrl-Alt-Del it.

  17. Stevie

    Bah!

    The history of e Hubble Space Telescope is one of management control of budget resulting in spectacular near disaster and bolt-on remedies that ate all the cost savings of not doing the ground-based testing the science required.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like