back to article Lessons to be learned from Google and Oracle's datacenter heatstroke

This year's summer heatwaves aren't just making your average Brit's life a bit miserable, it also caused problems for some cloud providers and server admins trying to keep their gear running. Last month east London datacenters operated by Google and Oracle suffered a breakdown amid the region's strongest heatwave on record. …

  1. hoola Silver badge

    "Finally, Levy emphasizes that steps to mitigate the impacts of these outages need to be taken. Hyperscalers and cloud providers can, for example, migrate workloads to other datacenters or run those workloads across multiple zones or regions to avoid interruptions to their services."

    Anything is possible, it just costs money. So the likely outcome is that the subscription costs will rise.

    Earlier in the article it states that the datacentres are designed for the normal climate they are operating in. If that is changing (and it appears to be increasingly clear that it is) then investment is requires to do all the upgrades, assuming the locations have the physical space and power to do so.

    Just moving workloads to another location is not quite so simple when capacity is already stretched. Equally, the datacentre it is moved to may also be in an unexpectedly hot zone. Unless they are suggesting the workload should be moved out of the region which the creates massive issues on data sovereignty.

    1. Anonymous Coward
      Anonymous Coward

      There may also be legal constraints on off-shoring data.

    2. Yet Another Anonymous coward Silver badge

      So isn't cost also the solution?

      If the data center is overloaded then you charge customers more to be on it or offer discounts to customers to shutdown or move off it.

      The customers know best how vital it is for the to be on a data center in central London during a heatwave

      Isn't maximizing scarce resources supposed to be what the market is good for?

  2. Whitter
    Pint

    Subheading

    Awesome! Well done

    1. BobC

      Re: Subheading

      Yeah, but the heading and subheading should have been swapped.

      Sigh, This no longer is the El Reg of oldendays, that was more aggressively "Biting the Hand that Feeds IT".

  3. Gene Cash Silver badge

    Shite sub-par A/C

    I don't know about the UK, but some of the A/C and heatpump equipment here is absolute tier-1 garbage. Soldering that looks like it could have been done better by a blind 5 year old. Start capacitors that are woefully undersized and go pop on a regular schedule. Fan motors that are similarly undersized and die if asked to run more than 45 minutes. Undersized coils that need a water mist to actually reach their specified capacity.

    But nooooo, the company (or homeowner) has gotta get the absolute cheapest shite, and then doesn't understand why it fails constantly and costs an arm and leg each month to run. And thus the cycle of producing garbage is encouraged.

    1. Brewster's Angle Grinder Silver badge
      Flame

      Re: Shite sub-par A/C

      UK homeowners have none of those problems with A/C. Because none* of us have any A/C.

      * Modulo a statistically insignificant quantity of El Reg readers about to complain.

      1. Auntie Dix Bronze badge
        Flame

        Re: Shite, sub-par AC

        "Shite, sub-par AC" is right! No wonder it fails.

        As for residential AC, sorry, but to most Americans, when it comes to dentistry and "modern cons," the UK and Europe are in The Third World.

        While Google et al. slop on the "green" banners, the truth remains the same, no matter the hype.

        Those cheap-@ss, hippie-dream "green" data houses just got a hot taste of reality.

        1. Pascal Monett Silver badge

          Re: the UK and Europe are in The Third World

          Speak for the UK, but don't speak for Europe.

          I have lived for 15 years in a solid house with air-air converters that could blow hot or cold following my settings. The equipment was Hitachi, and summers were very comfortable, as were winters (with a fireplace supplementing the heat requirements when we wanted a nice blaze).

          As for dentistry, you obviously have no idea of the professionals we have in France (or Germany).

          Europe in the Third World ?

          Not yet - but we may be getting there.

          1. Anonymous Coward
            Anonymous Coward

            Re: the UK and Europe are in The Third World

            I do, my old man had a serious heart condition (he noticed he was getting worn out climbing stairs) when he lived out in France and his doctor told him to drink more red wine.

            He came back to the UK for a second opinion and was whisked off for a triple bypass the same day.

            French Doctors are shit. At the top end though, French surgeons and specialists are very good...but if you can't referred to one because you don't drink enough red wine, then you're fucked.

        2. Anonymous Coward
          Flame

          Re: Shite, sub-par AC

          That's rich, given the USA (whch bans Kinder eggs, because "think of the children!" but won't make guns harder to obtain and retain, despite them causing waaay more child deaths per unit time than Kinder eggs ever did) is so far behind Europe in sensible provision for the well-being of its citizens!

        3. Anonymous Coward
          Anonymous Coward

          Re: Shite, sub-par AC

          Hang on, I'm in the UK, British and I have AC. My dentist is in Seapoint in Cape Town which to be fair isn't very local, but he's a good dentist...British too!

          He's from down the road but his office happens to be on the Southern tip of Africa.

          Since I'm there once a year visiting the wife's family I figured having a private dentist for half the cost on the other side of the world made sense. I can't comment on UK based dentists (who usually aren't British) and the quality of their dentistry because I can't get registered for one (haven't been able to for over a decade) and the private ones take the piss.

          Dentistry in the UK is in desperate need of some serious regulation. £250-£300 to get a wisdom tooth removed is stupid.

  4. Mike 125

    Ok but...

    "a disruption to the datacenter’s supply of electric power or a breakdown in the cooling control system, or a failure to respond to or detect rising temperatures, can set you down the path to an outage."

    etc., etc., etc...

    Is it me, or does basically ALL of this article sound like a long statement of the ultimate in the bleeding obvious?

    What would be more interesting is some explanation for what happens to the heat once extracted, especially in the middle of London. (Texas...? Well, who the hell cares... maybe it keeps the oil warm.)

    As Google reminds us with every search:

    'Carbon neutral since 2007'

    1. Anonymous Coward
      Anonymous Coward

      Re: Ok but...

      Try...

      "A datacenter or cloud outage typically occurs after a long or even short sequence of faults"

      So they're basically saying 'typically $#!+ happens'

  5. Anonymous Coward
    Boffin

    And

    An excellent article but another long term trend is missing since it didn't affect London's Google and Oracle datacenters.

    Just as heatwaves in unexpected areas are increasing throughout the 'temperate zone' so too are unexpected water shortages. And current datacenters need a lot of freshwater for cooling.

    While redundancy, failover, and improved A/C can mitigate this for now, for the future cooling designs need to be rethought.

  6. Anonymous Coward
    Anonymous Coward

    Reminds me of the due diligence on purchasing a ready-built data centre. It was on a flood plain so the building was surrounded by a wall - and large pumps would handle any flooding inside that perimeter. Unfortunately the pumps were wired to the general mains electricity supply - but not the diesel generator backup.

  7. Nifty Silver badge

    And to think that it would take something like a 20x increase in the cost of power to businesses to make data centre designers think about re-using/redistributing, rather than dumping, that excess heat. Couldn't happen in real life of course...

  8. Anonymous Coward
    Anonymous Coward

    Two identical components pushed to the limits

    no not operate twice as reliably.

    But while they can double the number of failures, once it hits 100%, you can't double your failure rate. So there is your silver lining for management when you tell them how much it all cost them.

  9. deadlockvictim Silver badge

    Climate change causes reduction in da Cloud

    Finally something positive about Climate Change: fewer active datacentres.

    Although it'll probably result in Google et al. buying up Baffin island for their datacentres.

    1. Clausewitz4.0
      Devil

      Re: Climate change causes reduction in da Cloud

      "Average annual temperatures at Iqaluit are around −9.5 °C (14.9 °F), compared with around 5 °C (41 °F) in Reykjavík, which is at a similar latitude"

      The temperatures there are quite ok, but I believe there is no reliable power source. So, a nuclear plant would have to be installed as well - high cost.

  10. martin 62

    a look inside data centers....

    if you want a look at how data centers work have a look at this very informative video https://www.youtube.com/watch?v=5eo8nz_niiM&t=481s it goes into details about how cooling/power usage works.

  11. Pietertje

    Disaster waiting to happen

    “The cooling plants are designed for worst-case conditions, and are factory tested as such,” Metcalf said.

    Worst-case conditions? Apparently not that "worst-case".

    "Factory tested"? So they just installed an AC and checked for that "inspected by" label?

    1. Roland6 Silver badge

      Re: Disaster waiting to happen

      However, they were installed on the same roof with the assumption that the ambient air temperature and movement would always permit the conditioning units to work. The high temperatures have proved that to be an invalid assumption...

      Round me with temperatures of circa 41C+ in the shade, on many industrial parks, businesses had removed the side panels and were enhancing the cooling effect by spraying the condensers with water.

      What I found interesting with an ambient room temperature of 38C (outside 41C) was how hot things got in the office. So a client had WiFi AP's crashing because they had case temperatures of circa 43C - obvious really given hot air rises... They also had PoE switches faulting (which also caused APs to fail), I suspect because the temperature was causing the thermal protection to operate. Interestingly, the equipment was only specified for use in 0~40C, but I was recording case temperatures of 42C. My takeaway is that if the ambient room temperature goes above circa 35C, expect equipment to start faulting for no obvious reason. I'm glad I had put temperature probes in all their core data cabinets as they enabled me to quickly link the equipment faults to the temperature.

  12. veti Silver badge

    There is no climate underground

    Bury the data centre deep enough, and your A/C bills will barely change through the year.

    There's a reason wine (and beer) is traditionally kept in cellars.

    1. Tom 38 Silver badge

      Re: There is no climate underground

      Wine generally doesn't produce heat. Try going on the Central Line in London and tell me there is no climate underground.

      1. veti Silver badge

        Re: There is no climate underground

        I didn't say you wouldn't need A/C, I said the need would be much the same all year.

        The Central Line has a *lot* of air coming in from outside. Through the tunnels, through the escalators... in fact I'm pretty sure the stuff is purposely pumped in to keep it fresh. A data centre would be a *lot* more sealed.

  13. Anonymous Coward
    Anonymous Coward

    Water shortages

    It gets even more interesting once you get to the point of water shortages.

    A lot of DCs rely on a local water supply to help with the cooling. If you suddenly get a water usage ban,

    you can go home.

    Similar happened in Texas during the cold snap, because the grid could not cope the local government were forced to shut down

    DCs.

    That smart combined HA/DR architecture will suddenly look very poor in this kind of situation where an entire

    geographical region could go down.

  14. This post has been deleted by its author

  15. An_Old_Dog Silver badge
    Facepalm

    Lessons Learned

    Q. "...what can be learned from these outages to mitigate future disaster" ?

    A. "Do full end-to-end tests o' yer' shit, then fix the problems."

    Q. "But what if it fails? We can't afford that."

    A1. "If you can't afford for it to fail, then you can't afford to be in business."

    A2. "The cost is less when it fails on your schedule, than when it fails on Mr. Murphy's schedule."

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like