back to article Cosmic rays more likely to glitch out water-cooled computers

When your correspondent is in the pub talking tech, conversation sometimes turns to this 2008 warning from Cisco that some of its line cards can experience "single event upset failures" due to "Thermal neutrons from cosmic radiation of energy less then 15eV." The Register brings that up because researchers from Japanese tech …

  1. Anonymous Coward
    Anonymous Coward

    2008? I'm sure I have read about this in a book dated eatluer than this. The book was about fixing PCs, the solution was to Bury the computer under several metres of concrete

    1. seven of five

      Odd, I have read the same advise about fixing problems with people...

      1. ITMA Silver badge

        And what size overshoes do you take sir?

        1. seven of five

          Special made to fit from Sicily.

    2. Phil O'Sophical Silver badge

      Sun had a problem with stray particles like that back around the year 2000. IIRC the L2 cache on one processor model didn't have ECC, and was prone to occasional bit flips.

      1. John H Woods Silver badge

        ECC RAM ...

        ... should be widespread but unfortunately it looks like it has been used as an excuse for price gouging.

      2. Korev Silver badge

        > Sun had a problem with stray particles like that back around the year 2000. IIRC the L2 cache on one processor model didn't have ECC, and was prone to occasional bit flips.

        So a Sundown then?

    3. Anonymous Coward
      Anonymous Coward

      This has been allowed for in avionics and space electronics for much longer than that.

    4. Bartholomew

      The earliest computers (1613 to ~1954), the ones before the electromechanical devices, were totally immune such miniscule effects.

      But turning organic computers off and on again ("Clear!") or burying them deep under concrete, would definitely have had legal repercussions.

    5. Jellied Eel Silver badge

      2008? I'm sure I have read about this in a book dated eatluer than this.

      In the late '80s, a DEC field engineer showed me a picture of a chip with an impact crater in it. The result of a bunch of DEC boffins trying to figure out what caused this processor to fail, then finally carefully removing the chip from it's packaging and studying it with an electron microscope. Then figuring out the crater was probably the result of a more energetic cosmic ray ploughing into it. For a while, I'd kept a copy of the journal they published their results in, but it helped spawn my fascination with cosmic rays and SEPs. It's also one of those fun things about the dangers of nuclear radiation spread by the Greens. We're constantly exposed to millions of particles of varying types and energys. Mostly, they blat right through us, sometimes we may feel an unexpected pinprick sensation or a flash in our vision, that may or may not be caused by particle impacts. Or we get really unlucky and it knocks off a bit of DNA, our bodies can't fix it and we end up with a cancer.

      But as the component density inside our ICs increases, the odds of a particle twatting something important reduce. Luckily our boffins know about this so have also been developing more resistant packaging.

      1. JohnTill123

        The most energetic cosmic rays have energies that are quite spectacular. In the range of joules: The cosmic ray in the following link would have probably smashed the chip into smithereens.

        1. Jellied Eel Silver badge


          ..equivalent to a 142-gram (5 oz) baseball travelling at about 28 m/s (100 km/h; 63 mph)

          Proton Powah! I remember reading about that when it was published, along with other discussions about where it may have originated and how common these may be given it was observed by chance. Similar things were discussed in the DEC article, ie if the crater was caused by a fault, short or explosion from inside the chip, or external forces. I guess because the packaging was removed, it wasn't possible to demonstrate that by showing a teeny cosmic bullet hole.

      2. TheMaskedMan Silver badge

        "cosmic rays and SEPs."

        Somebody Else's Problems? :)

        1. MrDamage Silver badge

          So that's what I saw out the corner of my eye.

  2. Korev Silver badge

    And that's bad news because, as NTT's summary points out, such neutrons are produced "when high-energy neutrons enter hydrogen-containing materials, such as water, plastics, and electronic substrates, losing their speed."

    Does this include the oils etc. that some people are now cooling servers with?

    1. Flocke Kroes Silver badge

      Watch out for the hydrogen

      Imagine throwing a tennis ball at a bowling ball. The tennis ball is going to keep most of its speed and transfer only small amount to the bowling ball. If you throw a tennis ball at another tennis ball then the high energy tennis ball will transfer about half its energy to the other ball. The same thing happens with high energy neutrons: hitting a heavy atom makes very little difference but hitting hydrogen slows the neutron down. Slow it down enough and it will hang around long enough in you computer to cause problems.

      One third of the atoms in water are hydrogen. For mineral oils the proportion is slightly over one third so they should be worse. Silicone oils are a really bad choice at about 60% hydrogen. There are some fluorocarbons that are used for cooling and contain no hydrogen - just carbon, fluorine and oxygen. The older choices were a global warming problem but that is fixed in the newer ones.

      1. Bartholomew

        Re: Watch out for the hydrogen

        All of the Hydrofluoroethers (non-ozone-depleting chemicals, developed as a replacement for CFCs, HFCs, HCFCs, and PFCs) contain hydrogen but much less than water per unit mass.

        e.g. 3M Novec 7000 Engineered Fluid - 1-methoxyheptafluoropropane - C4H3F7O ; Boiling point: 34°C (96.8°F) ; Freezing point: -122°C (-187.6°F)

        But some of the currently recommended replacements (Fluoroketones) for Fluorinert, used in the early Cray supercomputers, do not contain any Hydrogen.

        e.g. 3M Novec 649 - Perfluoro(2-methyl-3-pentanone) - C4F12O ; Boiling point: 49°C (120.2°F) ; Freezing point: -108°C (-162.4°F)

        1. Bitsminer Silver badge

          Simon says


          ...really quickly five times in a row.

          While in the pub.

          1. Phil O'Sophical Silver badge

            Re: Simon says

            Would you like ice and a slice of lemon with that?

      2. pklausner

        Re: Watch out for the hydrogen

        Doesn't H2O translate to 2/3 hydrogen, 1/3 oxygen? But true, many hydrocarbons have slightly more than 2/3, like the alkanes with CnH(2n+2)

      3. Roj Blake Silver badge

        Re: One third of the atoms in water are hydrogen

        Pretty sure it's two thirds...

  3. Mike 137 Silver badge

    Fascinating and important stuff

    Many thanks for the clear summary of this increasingly important issue -- all the more as the original paper seems to be paywalled. As chips get designed to be faster and less energy consumptive in the interests of compute power and greenness, this will inevitably become a more significant source of soft failures.

    1. Hairy Wolf

      Re: paywalled

      I followed the paper link and found it Open Access. Available as web page and pdf.

      1. Hairy Wolf

        Re: paywalled

        I may have been an SEU on the server.

  4. Doctor Syntax Silver badge

    Add boron to the cooling water.

  5. v13


    "Thermal neutrons from cosmic radiation of energy less then 15eV."

    This is a bit misleading. 15eV can't be the energy of the cosmic radiation as it's very low and wouldn't reach the computer. Nor is there cosmic radiation made of neutrons or else we would all be in trouble. The neutrons are likely produced by gamma rays (photons) that interact with some material.

    That's from the limited nuclear physics I know.

    1. Mike 137 Silver badge

      Re: Hmm

      Whatever the actual values, for soft faults the issue is essentially the ratio of the expected energy for a legitimate state change and the energy of the incoming particle. If they're of the same order, an unintended state change may occur. However as bit state change energy is being designed downwards in the interest of power saving and the bit count is rising dramatically the problem will only get worse as the target becomes both more fragile and larger. I tremble a little to think how these effects might manifest themselves in quantum systems.

      1. Doctor Syntax Silver badge

        Re: Hmm

        "I tremble a little to think how these effects might manifest themselves in quantum systems."

        Wrong and right at the same time.

    2. herman Silver badge

      Re: Hmm

      AFAIK the neutrons come from a gamma ray hitting air, causing a neutron storm.

  6. Crypto Monad Silver badge

    Gate size

    Question: as semiconductors go to increasingly small gate sizes (7nm, 5nm, 3nm...) are they going to become more susceptible to these effects?

    1. Claptrap314 Silver badge

      Re: Gate size

      Yes--and we knew about this in the '90s when I was at AMD.

  7. Conundrum1885

    Gives a whole new meaning to

    Nuking the entire site from orbit.

    Apologies, but this is the best I could come up with.

  8. Claptrap314 Silver badge

    Comic rays? What's next? The Earth's magnetic field? Sunspots?

    1. tonique

      Even worse: The next paper will be published typeset in Cosmic Sans.

      1. jake Silver badge

        "The next paper will be published typeset in Cosmic Sans."

        Are you suggesting they'll be jivin' us with their cosmik debris?

    2. jake Silver badge

      I've used all three of those for answers to questions that would otherwise need to be answered"I don't have time to give you a lesson on computer and networking theory" to "it's a Windows thing" to "I haven't the foggiest idea WHY, exactly, but it's bloody obvious that bit of hardware is dead".

      Deliver the line with authority and appropriate body English ... "It's sunspots ::looks up::". "The Earth's magnetic field shifted overnight" ::waves hand::. "Probably just a stray cosmic ray." ::shrugs::. This isn't strictly honest, but allows you to get on with the job without spending too much time in explanation to someone who probably wouldn't get it even if they took a four year course on the subject.

  9. PaulVD

    Simple solution

    A tinfoil hat will protect your computer from slow neutrons.

  10. david1024

    Significant increase?

    How significant? From one every 10 years to ten?

  11. deive

    I wonder how the oil-based submersion coolants fare?

    1. IvyKing

      If using a hydrocarbon oil, then the effects could be even worse than using water. Hydrogen does most of the work slowing down neutrons and carbon is more effective than oxygen for slowing dow neutrons.

      Mushroom cloud icon for obvious reasons.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like