Hubble Space Telescope may now depend on a computer that hasn't booted since 2009

The Hubble Space Telescope may need to boot up a backup computer that's been dormant since 2009 to carry on operations. Science work by the orbiting rig came to a halt on June 13 after the computer tasked with controlling the instruments stopped responding to the main computer, and its sensors were put into safe mode as a …

[Anonymous Coward]

Here's hoping

To clarify the article, the computer was replaced in 2009 but it was replaced with an identical NASA Standard Spacecraft Computer-1 (NSSC-1) system using the 1980s design and parts.

Between this and the InSight power problems NASA hasn't been hasn't been having the best week

[diodesign]

Er, yes, mate?

"The computer was replaced in 2009," and hasn't been turned on since it left the lab.

C.

[Occasional Comentard]

Re: Er, yes, mate?

I think the point being made was that the computer, although being replaced in 2009 as stated in the article, was based on a 1980s design and had not been updated to a 2009 design.

[Joe W]

Re: Er, yes, mate?

From the article

"We note that the replacement unit contains the same 1980s-era parts as the original computer unit."

[HammerOn1024]

Re: Er, yes, mate?

Just remember folks, these are space rated computer systems. While the designs are from the 80's a lot of the chips were 70's vintage designs.

But, they were designed to take a LOT of radiation and just keep chugging along. Old does not equate to infirmed nor non-functional.

I'd bet it boots and works just fine for another 20 years.

Also, since these were made before the lead solder hysteria, tin whiskers are not even relevant.

[David Austin]

Re: Er, yes, mate?

Reminder that in the 90's NASA ran an active scavenger hunt on eBay to grab the out of production chips and boards it needed to keep the space shuttles running, like the intel 8086;

https://www.nytimes.com/2002/05/12/us/for-parts-nasa-boldly-goes-on-ebay.html

Not an expert, but I've been told that due to circuit track width, Pentium I era kit is better for space travel without needing excessive radiation protection; modern processors are so small a stay beam of space radiation can completely severe the track.

Get the right kit for the job, not the newest.

[RM Myers]

"Old does not equate to infirmed nor non-functional."

Thanks for that comment. The way things have gone this week, I was beginning to think maybe the opposite was true.

Really, who thought it was a good idea to use micro-mini screws to secure M.2 drives in a computer. Darn whippersnappers should be taken out back to the woodshed and have their behind tanned!

[herberts ghost]

Radiation effects systems that are off.

One problem is that the components of the backup computer have been exposed to roughly the same radiation levels as the primary computer. Damage can occur when the system is powered off (as people who depend upon Flash drives for long term backup will likely find out.

[wolfetone]

Re: Er, yes, mate?

I think too much is made of it being a 1980's design.

All that would happen, if it went up today, it'd be a computer designed in this century but made of components not used in mainstream IT for 20 years.

[NoneSuch]

Re: Er, yes, mate?

Voyager 1 & 2 are stull chugging along and they were based on the best 1970's tech.

I have faith in the older tech. No Facebook / Google data slurping to slow it down, no NSA CPU microcode to crash.

[John Brown (no body)]

Re: Er, yes, mate?

""The computer was replaced in 2009," and hasn't been turned on since it left the lab.

Apart from the in-situ diags they just might possibly have run when it was installed in Hubble. It'd be a bit silly to spend millions getting it there and hours of spacewalk time just to plug it in and leave it there without even checking that it powers up.

[chuBb.]

Even in space without a means to reach it, its infinitely more repairable than a smartphone

Hope this isnt the end for hubble, it deserves a less ignominious demise than digital dementia

[Anonymous Coward]

To be fair, your smartphone could have an entirely redundant motherboard, battery and memory module in it if there was a market for it... But that would probably be rather extreme.

[ThatOne]

> Even in space without a means to reach it, its infinitely more repairable than a smartphone

Obviously, Hubble wasn't built to be replaced a year later by a newer, shinier model...

[iron]

If like Hubble your phone had four backup memory modules and an entire backup main board, SoC, etc it would cost twice the price. Would you have still bought it on the off chance it has a problem or would you have bought the much cheaper version that wasn't full of redundant backups?

[fidodogbreath]

its infinitely more repairable than a smartphone

Smartphones also contain a computer and a camera, so therefore they are directly comparable to space telescopes?

[Daveytay]

The mirror replacement spacewalk was streamed.

I watched it for hours drinking beer! It was amazing, especially when there were so many little screws to undo. I was so amazed by the skill of the astronauts.

[batfink]

Re: The mirror replacement spacewalk was streamed.

Also amazed that there were so many little screws to undo, which meant that the designers hadn't thought about the circumstances under which it might need to be repaired?

[[e-mail]]

Re: The mirror replacement spacewalk was streamed.

Maybe they hoped it might be superseded by a bigger, more capable model before that became an issue?

FIJAGH.

[Anonymous Coward]

Re: The mirror replacement spacewalk was streamed.

Not sure which spacewalk you're referring to, but they've never replaced the mirror. They installed the corrective lens thing, and then gradually replaced all the instruments to cope with the mirror flaws - the same duff mirror is still in Hubble.

[Bryan Hall]

Re: The mirror replacement spacewalk was streamed.

Apparently they didn't consult Monroe on the design...

[jtaylor]

Re: The mirror replacement spacewalk was streamed.

Those spacewalks were amazing!

They carefully prepare for those missions before launch. The astronauts repeatedly practice the mission on a replica of the satellite while wearing space suits, maybe underwater or in a 5-degrees-of-freedom chair. They work very hard to give the missions the highest chance of success.

I'd love to see more ambitious space missions, and of course to watch more Extra-Vehiciular-Activities (spacewalks). Hopefully soon....

[aaaaaaaaaaaaaaaa]

CP/M, Now where did I hear that before??? Hope they have extra floppies on board...

[Totally not a Cylon]

8 inch or 5 1/4 inch?

[Brewster's Angle Grinder]

The most relevant point is the computer has only been sat in space for one decade, not three. And as it's not been turned on, there's no chance of damaging current surges or voltage spikes; it's only at risk of cosmic rays having carved a few bonus tracks. But being 1980s design (aka built like a brick shit house), the parts are going to be pretty robust against that.

X-fingers

[BobC]

Cosmic rays can leave "punch-through" conduction channels.

Even in unpowered equipment, cosmic rays can cause significant damage. Fortunately, much of this damage (ionization tracks) self-heal, but the remaining damage (crystal disorder, dopant displacement) will be permanent.

Extremely careful power-on sequences can assess this damage without risking further damage from localized over-current. Briefly energize the first supply, measure it's current ramp, turn it off. If good, repeat adding the second supply, and so on until the system is fully powered.

In general, keeping redundant spares in the off state is vastly preferred to running them all the time with voting logic. Only the few most critical subsystems need that level of robustness.

[thondwe]

Shuttle!

Shuttle being missed - any plans for an orbit only spacecraft so ISS astronauts can "fly" to Hubble etc to fix/upgrade them?

[Anonymous Coward]

Re: Shuttle!

No, but the last servicing trip installed a Soft Capture Mechanism (NASA-speak for "handle") to allow Hubble to be grabbed by a remote craft with the possibility of returning it to earth for display - at some point they'll want to de-orbit it either intact or safely, as it's expected that much of the main mirror will survive re-entry..

[Binraider]

Re: Shuttle!

The only vehicle that isn't stuck in a museum with the potential to do such a thing would be a reconfigured upper-stage Starship. X37 isn't big enough.

In many respects recovering it would be extravagant. Hubble's contributions perhaps deserve a better fate than just burning up. I'd pay to visit it in a museum if it were brought back...

[Unoriginal Handle]

Re: Shuttle!

Is that because it'll reflect the heat of re-entry?

Mine's the one with the book about reentry physics with a missing page

[Anonymous Coward]

Re: Shuttle!

No, NASA just plans to de-orbit it safely, making it hit the sea in some remote area. There was something like plan to bring it back with a Shuttle - but now there's nothing that could bring back to Earth such a large thing intact.

[Anonymous Coward]

Re: Shuttle!

Don't think they've decided that yet. They would still have to reach it in some capacity in order to de-orbit it safely, it has no meaningful method of propulsion to de-orbit by itself - that's why the installed the handle. A future craft will need to do it, but nothing exists yet to do it either manned or otherwise.

[Timbo]

Re: Shuttle!

No need for a shuttle - Just send a SpaceX Crew Dragon up, with the Project Daedalus crew: Clint Eastwood, Tommy Lee Jones, Donald Sutherland and James Garner - the Space Cowboys.

If they can handle an old Russki "communications satellite", they can easily deal with Hubble. :-)

[Unicornpiss]

I have confidence..

This is NASA, after all. With the exception of some hiccups (*cough*... Mars probe with wrong unit conversion programming, myopic lens on Hubble, and sadly Challenger, Columbia), they have a great success rate and a reputation for pulling off hardware and software miracles. (Voyager, Spirit, Cassini, etc., not to mention the whole Apollo program)

The computers may be based on 1980s technology, but that technology was hardened and pushed to its limits. It will still be capable of running Pac-Man for decades after our current smart phones and laptops have been shredded or are rotting in landfills. (though even then there may still be some Amigas, Apple IIs, and the PC that controls our building automation still functioning out there)

[Andy The Hat]

A technician sits back in his chair ...

takes a deep breath, leans forward and nervously initiates the Power Initialisation Process (presses the red button) on the 1970's Hubble re-initialisation management computer.

Whirr churn chunder chunder zzzz zzzz chunder chunder ...

>PLEASE INSERT DISK 2

This is only the second time the pimple-faced technician has handled a floppy disk. With some trepidation disk 2 is inserted correctly and the locking flap closed.

Whirr churn chirrup chunder zzzz chunder zzz chunder ...

>SENDING INITIALISATION SEQUENCE

Wheee brrr barrrrrrrrrrp ...

<time passes> ...

A green terminal screen prompt suddenly appears!

> "I'm sorry Dave, I can't allow you to do that ..."

The technician begins to fill in a requisition for new pants and asks if the cleaners can take a look at his chair ...

A 75 year old semi-retired engineer lurking at the back of the room coughs violently and dabs his eyes with a handkerchief ...

?

"My mate Malcolm set that prompt, he thought it was a great idea!"

Angry that he'd been suckered by a jape he hadn't thought of, the technician stabs at the return key so hard he gets muscle spasms in his hand. The terminal responds ...

>I'm sorry Dave, I really can't allow you to do that ...

The room goes quiet ...

>It's all full of stars ...

[MJB7]

Re: A technician sits back in his chair ...

Malcolm was right! He should have a beer (or three)

[TeeCee]

Informally.

...known formally as the backup payload computer...

That's Mr backup payload computer to you.

[Martin-R]

Re: Informally.

If it's becoming the active machine, doesn't that make it formerly known as the backup payload computer?

[IGotOut]

Hope it's not made by Nortel.

They also would be run for years, even decades without any issues.

But power down more than an hour and you were screwed.

[Omnipresent]

Sad to see End of Life

Truth is they have kept Hubble going longer than intended, It did so well after the lens replacement early on. They are about to send up a "next gen" telescope that will see things we've never seen before. They will do everything they can, and keep it going as long as possible, but I would expect to see more breakdowns until Hubble reaches End of Life.

[Anonymous Coward]

Re: Sad to see End of Life

NASA have always worked to make things work, they are concerned with the functionality of all their devices and code so I expect that this will all go fine. My company hired a former NASA engineer to design a biomedical data collection system back in the early 90's - he did a great design, it worked fine back then and we still have customers using it every day nearly 30 years later - internally it's all TTL and analog chips plugged into sockets.

The problem is that he did such a good job that we did not made a lot of money because our original customers don't needed to upgrade their systems every a few years.

[bigtreeman]

Re: Sad to see End of Life

I've seen gold plated T.I. mil spec memory chips from the 70s in gold plated sockets with the pins corroded out from the inside with just the gold shell of the pins remaining. bimetalic corrosion. Even the very best die.

Proms die, it might be random bit failure, that's why there is ECC memory and controllers, they expect random failure. Random means there could be three bits fail together and the ECC can't correct the fault.

Such is life, so random.

[iron]

Re: Sad to see End of Life

James Webb does not replace Hubble. It can't see the same breadth of wavelengths (UV IIRC) so the plan is for them to complement each other.

[ColonelClaw]

Question

So if the backup computer hasn't been turned on since 2009, and the main computer is borked, then how do they turn on the backup computer and put it in charge? Wouldn't that require... a working computer? Or is there another computer that controls the computers? If so, I hope it works!

[diodesign]

"the main computer is borked"

The main computer is fine, it seems, it's the instrument/payload computer that's halting. So they hope to turn off the payload computer and turn on the backup payload computer.

C.

[Anonymous Coward]

Easy...

Just find a third rate sysadmin that never tries to get a budget for new kit. They're using to firing up kit that hasn't been used for decades.

[jtaylor]

Re: Easy...

I supported a few servers at remote offices. Hardware was replaced when things broke, and my team made sure our stuff didn't break.

One site scheduled a power-off to upgrade its connection to the street. I told them to obtain a replacement DNS server before that date, because the old one wouldn't survive a cold boot.

The remote IT staff had plenty of experience and equipment. After all, they rebooted their Windows servers every month and knew that power-off doesn't magically break hardware. And everything was on a new IP-connected KVM.

Come the day, that 15-year-old SparcStation 5 didn't come back up. It wasn't on their KVM. In full crisis, the local staff called for help. "Is the power light on?" "Yes." "Okay, then the hard drive didn't spin up. Know how to fix stiction with a mallet?" "What's that?" "Never mind. Grab a replacement PC and call me back."

[Henry Wertz 1]

Re: Easy...

Yes, back in end of 1999, I knew someone with some Apollo computers (not the NASA ones.... the Apollo that was bought by HP in the mid-1980s, these ran DomainOS.) The U of Iowa ISCA (Iowa Student Computer Association) ran their BBS off the DomainOS systems.

They actually released a year 2000 update for these, but it involved powering them off and back on. The drives were VERY prone to stiction, they stuck and did not release. RIP several Apollos.

[jtaylor]

Re: Easy...

ISCABBS! Citadel and DaveCode! I knew they had NeXT systems (the deadly sins). I don't remember their Apollos.

Domain/OS was a weird beastie. As you say, the 31-bit OS had its "y2k moment" in 1997. Because file references were based on timestamp, an unpatched Apollo would basically barf its filesystem on the fateful day. We had to patch our DN2500s at Wesleyan.

I met some great people there and even made it to ISCA-nic once. Good times....

[Henry Wertz 1]

At least it's not on the ground

I thought they were (for some reason) having to power up a ground computer for some unusual activity. At least it's not on the ground! I'd honestly be more concerned about one on the ground being lost, maybe missing parts, dust, perhaps rust (some of these systems in the past were not put in careful storage, they were just stashed wherever); whereas the in-orbit one is known to still be there and was stored roughly as well as the in-use one.

Good luck backup computer, NASA, and Hubble!!!

[Anonymous Coward]

Anyone know how to boot Windows 3.1?

Based on that technology, you'll probably see a "help wanted" posting for anyone with knowledge on booting Windows 3.1 or DOS 6.22!!

[T. F. M. Reader]

"back when NASA had the capability to reach it"

Doesn't that summarise an issue a lot more basic than than a borked memory bank or another component?

Sad as I am to read of problems on Hubble (as a former astrophysicist who branched out into other fields I think I've earned a claim to being particularly sensitive here) this loss of an even more basic capability makes me sadder still.