back to article NASA solar mission data recovering after server room flood fiasco

They can put a man on the Moon - but back on Earth, a busted water pipe managed to knock out NASA's solar mission data for months. Data from two NASA solar missions is becoming available again following an outage that began in November 2024. The affected missions are the Solar Dynamics Observatory (SDO), which was launched in …

  1. Andy Non Silver badge
    Coat

    Sounds like they weren't quite expecting

    that type of torrent from a fat pipe.

    1. UnknownUnknown Silver badge

      Re: Sounds like they weren't quite expecting

      Restored just in time for them to be put in the wood chipper by 2 Fat-Heads - one Orange, the other one a bit Musky smelling.

      “Solar - who needs it.”

  2. Dr. Ellen

    Beware Murphy.

    Pipes are everywhere, hidden in walls and ceilings. They lurk, unobserved. And sometimes they strike. I was a curator at the time, in a museum devoted to electricity. One morning I came to work and found the ground-floor exhibit halls several inches deep in water. Museums are often put in historic buildings, and they are at times repaired, refurbished, or extended -- without full attention to the consequences. Some time before the museum moved in, the plumbing had been modernized -- well, some of the plumbing. Which meant that iron pipes were joined with copper pipes invisibly in the ceiling. The joint finally gave away to electrolytic corrosion and let loose the flood.

    The scrambling to save artifacts was intense, even before the pipes had a chance for repair. But we saved the pipe junction and got an excellent artifact demonstrating electrolytic corrosion. Well-documented, too.

    You are never completely safe. Murphy is too ingenious.

    1. Anonymous Coward
      Anonymous Coward

      Re: Beware Murphy.

      Not just water pipes.

      You should also make sure that the insulated air conditioning duct above your network swtich doesn't have an access/clean out hatch D.I.R.E.C.T.L.Y abover your hardware.

      1. Yet Another Anonymous coward Silver badge

        Re: Beware Murphy.

        At least British researchers don't have to worry about such things.

        All our server racks have pre-installed plastic buckets for when the flat roof leaks every time it rains

  3. Paul Herber Silver badge
    Flame

    I do hope NASA survives the current fiasco but I fear it may not.

    1. Chris Miller

      Given NASA is one of SpaceX's biggest clients, I'd be quite surprised if Elon closed it down.

      1. Anonymous Coward
        Anonymous Coward

        Just re-direct the money direct to SpaceX - cut out the middle man.

        1. cray74

          SpaceX is a launch company and rocket builder. It lacks the labs and scientific personnel to carry out NASA's assorted programs, just as NASA lacks the rocket factory and industrial capabilities to replace SpaceX.

  4. Chris Miller

    "If you think business continuity is expensive, try having an outage."

  5. AVR Silver badge

    Months to restore from tape?

    Are they having to source some no longer produced hardware or something?

    1. Gene Cash Silver badge

      Re: Months to restore from tape?

      "Several pieces of electronics are delayed"

      It does look like they went "hey, this drive doesn't actually work and we need a new one" and are waiting on it.

      1. An_Old_Dog Silver badge

        Re: Months to restore from tape?

        If the current tape drive(s) they have are broken and won't read the backup tapes, there's a significant chance that those same drives, used to create the backup tapes, did not write the backup tapes correctly, either.

        1. AVR Silver badge

          Re: Months to restore from tape?

          It's not impossible that the tape drive got more H2O through it that it was designed to handle and needs replacing. Months to replace it seems odd is all.

          1. TRT Silver badge

            Re: Months to restore from tape?

            Yeah... Can get a bit twitchy, especially Sun Workstations.

    2. An_Old_Dog Silver badge
      Joke

      Re: Months to restore from tape?

      PRESS PLAY ON TAPE

      1. Anonymous Coward
        Anonymous Coward

        Re: Months to restore from tape?

        R Tape loading error.

        1. TRT Silver badge
    3. Doctor Syntax Silver badge

      Re: Months to restore from tape?

      It says the process will take several months. I read that as the time it takes to do the actual job, not the lead time. Either its a very large data set or possibly it has to be interleaved with other use of the tape drives such as doing the normal backups. I don't envy them.

    4. collinsl Silver badge

      Re: Months to restore from tape?

      How much data is it and how fast can it be read through? If it's 1000 tapes it may take that long.

  6. An_Old_Dog Silver badge

    Dataset Restoration from Tape Time

    Mega-huge datasets, by their nature, take a long time to restore. Restore a full backup from date A, then restore many incremental tapes to restore the file system to the way it was as of date B. Robot tape changers help, but are not a panacea.

  7. Andrew Scott Bronze badge

    been there. had just occupied the basement in the building. fortunately the server and backups were on the first/ground floor. first flood was caused when the muddy brook which was buried under a parking lot at the time overflowed and put a foot of water into the buildings basement all the way down the block. recovered from that and a week before christmas a water main broke just outside the front stairs putting things under 5 or 6 inches of water. most of the computer drives were above the level of the water so while the computers were dead the drives could be moved to spare machines and brought back up. got the entire office moved to the first floor conference room with networking and printers working as it was the end of the semester and these guys were at the busiest time of the year. both times the flooding happened at night over the weekend.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like