Datacenters bleed watts and cash – all because they're afraid to flip a switch

Datacenter power consumption has become a major concern in recent years, as utilities struggle to keep up with growing demand and operators are forced to seek alternative means to keep the lights on. According to Uptime Institute, curbing energy consumption – and by extension lowering operating costs – could be as simple as …

[Anonymous Coward]

Profit.

This will only happen if a data center can make more money from it. If the power and cooling cost is more than offset by being able to charge customers extra for the useless extra performance, they'll happily watch the world melt.

[Paul Hovnanian]

Re: Profit.

True. But for sucking in the training data and building an LLM model, the customer is someone like OpenAI. And it takes a relatively long time compared to executing a customers query.

OpenAI is more likely to understand the resource economics.

[Alan Brown]

Re: Profit.

"This will only happen if a data center can make more money from it."

Not always. I went through our fleet and systematically enabled power savings for the simple reason that we were on the ragged edge of overloading the ACs

Doing so gave us enough headroom to survive until we (finally) managed to get new cooling kit installed

[Persona]

Re: Profit.

They could make money by turning it on. The problem though would be the customer complaining that the server they were paying for was under performing. This could be fixed by giving the customer the option of a full power server or one with power saving turned on. This does however add an level of billing complexity.

[Mythical Ham-Lunch]

I had a job where the rules for preparing any new server were to disable all automatic updates and power-saving features, and install third-party remote access tools.

It's not around any more.

[Mike-H]

In my previous role we would configure OS controlled power saving options, and even VMware DPM to reduce power requirements.

Dell support and VMware support would always suggest setting configuration to maximum performance (ie no power saving) if we raised cases for server crashes (which were rare). Always pushed back on this and fixes were always nothing to do with the power settings. Always fixed by one or more of hardware replacement, firmware/driver update or os updates.

[Mike007]

Sounds like the sort of person who gets a support request about hissing coming from the speakers and their first suggestion is to disable IPv6...

[Anonymous Coward]

You mean IPv6 isn't causing my speakers to become posessed?

[Mishak]

Yep

Drivers aren't always that great at handling changes to the power settings, so not using them often just masks the fact that the driver is badly implemented.

That's a very lazy "fix".

[ecofeco]

Of course max performance is needed

The bloat of modern systems requires everything you can through at it and then some.

[Michael Hoffmann]

Re: Of course max performance is needed

Latter day "Can It Run Crysis": Can it train my "AI" model?

[Nate Amsden]

one situation that may be common

I recall about 10 years ago deploying some bare metal systems for an e-commerce site. Our VMware stuff was all max performance and the bare metal was just the default OS settings (which happened to have some power management stuff enabled by default). Ubuntu 12 I think at the time (though that doesn't matter, situation is the same with Ubuntu 16 and 20, haven't tried 24 yet). Most of the time the servers sat at under 10% cpu usage. It wasn't a huge deal but some folks complained that the performance of those systems was less than the VM-based systems, even though the bare metal systems had 10x the capacity.

Ended up being the power management, given the low utilization the clock speed was just cranked low and stayed low, I assume if CPU usage went way up then clock speed would ramp up as well, but those situations were very rare maybe not even a couple of times a month. (despite that, the bare metal systems were FAR cheaper than the VM based ones due to the software licensing). So I hard set the bios to be max performance, power usage was up a bit but not much and people stopped complaining.

Maybe newer CPUs work much better in that regard, for example maybe they have the ability to have a small subset of their cores to run at max/normal clock speed, and the rest of the cores be lower speed, and ideally the OS is capable of scheduling tasks on those higher clocked cores unless load gets pretty high then distribute the load to other cores. I know Intel has that P core / E core thing, though I don't think that is too widely used in Xeons at this point(if at all?) I haven't tried to check. Of course such co-ordination between the CPU and the software introduces extra complexity and could be more bugs as a result.

[LeeDilkie]

Re: one situation that may be common

The problem is a fundamental one. Measuring performances always a lagging indicator, that is it's a measurement of past load factors. Thus any attempt to tie and automatic increase is based on the past and not the current demand. Hence there's going to be a delay, always, in ramping up performance to match demand. This will, pretty much always, be viewed as an impact to response to inputs

[Mark 124]

Re: one situation that may be common

It's not unsolvable though. In finance, economic numbers or company results are released at known times, so crank up the performance a few minutes before and leave it up until the market settles. Similar for Black Friday in retail or betting on or streaming the Superbowl.

For the average web based system with unpredictable demand but a defined response time target the trailing indicator should be OK. If the target response time is say 750ms then reduce power saving as response time goes above say 500ms. Yes it's an impact to response times but that should be OK if the feedback loop from them to power settings works fast enough.

[EricB123]

Just Ground the Damn Chip Selects

I used to work for a disk drive company, and when my customers started complaining about how much hotter my company's drives were in their systems than the competition, I started looking at the drive's schematics. To my horror, they simply grounded all of the chip select pins. When I asked the engineers why in the world they would do such a thing, they said constantly turning the chips on and off (or sleep perhaps) would result in cracks from thermal cycling. Well, the drives were failing from overheating when installed in actual computers, where not every disk drive gets it's own dedicated fan! Maybe the thermodynamics requirement in EE curricula should be less math based and more common sense based?

I had no idea data centers are doing pretty much the same thing.

[matjaggard]

T series on AWS

Some cloud instances are expected to have worse performance or share more workloads per server. On AWS that's the burstable T series t3 for Intel, t3a for AMD and t4g for ARM. Surely these would be perfect for simple savings on power and cooling for Amazon? Maybe they've done it already.

[Anonymous Coward]

Re: T series on AWS

B series in Azure

[Peter D]

Perhaps...

Labour could consider a levy on data centre power consumption to avoid freezing pensioners to death.

[Mishak]

Better still

Pump all the hot air from their conference* into houses as "free heating".

* I guess you could also use the waste heat from data centres, but that's boring.