Uncle Sam has a datacenter waste problem US government auditors want to save taxpayers' money by bolstering the capability and efficiency of Uncle Sam's far-flung stable of datacenters. Each federal agency's sites have a host of problems, unsurprisingly. The US Government Accountability Office (GAO) kicked off its Datacenter Optimization Initiative (DCOI) in 2016 …
I have no problem believing this is directly associated with the "culture" of US government.
A big part of it is going to be how department budgets "work" within the government. Basically, if you don't use it, you lose it. Any department coming in under budget, very likely will see their annual budget reduced by that amount the next year, because "obviously, you don't need it." This causes all kinds of hijinks around the end of the fiscal year. As a tech in the Marine Corps (yes, hence the screen name), back in the early 90's, I got to see it in stark contrast. We did circuit board repair in our shop. We repaired boards right down to the individual component on the board. As part of that, we had a yearly allotment of powdered gold to use for re-electroplating contacts on the boards. Come the end of the fiscal year, if we hadn't used it, we'd get a reduced allotment the next year. That would really cause issues if we had MORE boards to repair the next year. So, in typical USMC fashion, we were told to improvise, adapt, and overcome. Here's what would happen. EVERYTHING got plated until it was used up. Every circuit board we could get our hands on had their contacts plated. Standard contacts that didn't require gold got plated. Dog tags got plated. Pens got plated. Anything that would hold still long enough to get jammed into the electroplating machine, got plated.
So there's the first questionable practice.
Second comes from the fact that it is a Herculean effort to get rid of ANYTHING in government service. I have a friend that teaches IT courses in the US Job Corps system. It is an absolute nightmare to get rid of excess or outdated equipment. PCs that are 11 to 12 years old sit in stacks gathering dust because they're no longer of any use, but nobody wants to wade through the bureaucratic nightmare to get rid of any of it.
With those two behaviors in mind, is it really any stretch to imagine that government datacenters have racks crammed with equipment that is either surplus to requirement or beyond usefulness? New equipment keeps coming in whether it has a use or not, because otherwise the budget is lost and in the future they MIGHT need additional kit. Nothing gets thrown away so there are probably racks of legacy servers just sucking up power and not actually fulfilling any tasks because nobody wants to jump through the hoops required to retire it from service and dispose of it.
I am honestly surprised it's not actually worse than it is.
And if you do report your datacenter is only running at 49% capacity it would obviously make sense to merge it with another organization that is using 51%. So you will get taken over unless you can get your utilization up above half.
Time for some bitcoin mining or just wrapping all tasks in docker containers till the fans are roaring
You don't want to be running your data centres at 100% capacity. That means when you have busy periods, your computers won't be able to cope.
At the IRS for example, the busy period will be filing deadline day, other departments will have similar busy days.
A big part of it is going to be how department budgets "work" within the government. Basically, if you don't use it, you lose it.
It sounds exactly like over here in the UK. And not just central govt.
When I took over running a local authority education team I found my cupboards full of envelopes.
Because near the end of the financial year there was a rush to make sure that the budget was spent. Use it or lose it.. Which makes no sense unless you think that expenditure somehow follows a 1 year (April to April) cycle. In reality this year our requirements might come to one or two hundred quid less than our allocation, because we bought, say, an expensive new reading test last year so we're set with no big expenditure item for a while. But next year we might need to replace some other test, or an assessment scheme. And of course not only can that surplus not be carried over, and not only would that amount be removed from our next allocation, but our total budget would be seen as fair game- because obviously we're not in need. Whereas,, some department that had over-spent even if it was through poor controls might get an increase if an assistant director decided to like them. There was even a degree of them appearing to think that a team who spent more, even if it was outside their planned budget ( or they'd not planned one), must be deserving more money- because spending was seen as a proxy for doing stuff.
The following financial year was the worst one for this. It was my first full financial year. I set a careful budget to ensure that we used our funds efficiently to get us through the year without wasting anything. Each month was planned so that our expenditure would last the year, we'd have the right resources for the kids and be complete by the end of February. That year they decided to start the claw-back in December because some other teams had over-spent- 10% of my budget was taken- i.e. everything I had left. So we were running out of teaching materials, literally scrounging for pencils etc. Then in April they reduced our budget by 10% - because we'd managed without it.
I tried to buy a gold bar once for this very reason.
Our research grant had £30K in it for a mobile crane to position part of a detector - but it was easier to do it with half-a-dozen slavesgrad-students. We had to spend the budget that year but we didn't have any budget for consumables in the future once the detector was built.
There was a part number for gold wire in the stores catalogue, I wanted to buy a gold bar, about £30K in those days, and then sell it next year.
This was apparently silly/fraud/not-the-way-we-do-things so we bought a fancy oscilloscope for another group and they promised to pay for our stuff, and everyone was happy.
Oh yes, dodgy advance payment schemes. Buy something you don't need and then return it for a credit note
Or help another team buy a big ticket item that needs to be spread over two budgets/years and then in the next financial year they'll buy a load of stuff that you need.
And variations.
5000 data centers?
it was ~2000 data centers 12 years ago
https://www.datacenterknowledge.com/archives/2010/10/13/feds-discover-1000-more-data-centers
"The process defined a data center as any room larger than 500 square feet dedicated to data processing that meets the one of the four tier classifications defined by The Uptime Institute."
500 sq feet seems more like a server room than a data center(data center I'd say should start at 10k sq feet?), but I guess they lump them all together so the non tech people don't get too confused.
This post has been deleted by its author
Another aspect of this is a temptation to gold plate the name of any capability or system. I observed this constantly when I worked for the US Navy.
Example- we needed to perform some modeling and simulation of antenna systems. The appropriate tool was a nice SGI Power Origin cluster. A lovely piece of kit, but nevertheless just two racks of compute capability.
One day I arrived at work to discover a sign on the door calling the system a Supercomputer. The BOFH was now a Supercomputer Center Director. And so on.
I wonder how many of those "data centers" are rooms of excess PCs gathering dust. After all, when it's time for annual reviews would you rather be the Deputy Vice Assistant Obsolete Hardware Wanker, or Datacenter Director?
I should have named myself the High Priest of Computational Electromagnetics. After all, I had the Director of Supercomputing working for me.
Pfft, get with the program!
Get another sign saying 'USN Museum of Antenna Design', apply that to the door of the server graveyard. Then apply for funding from whichever Federal agency doles out money to museums and the arts. In the UK, that would be the DCMS. They fund pretty much anything.
If asked if the museum is open to the public, you could truthfully say 'yes'. Just gloss over the public needing authorisation to be on base, and vetting to TS or SCI to view the exhibits. That would justify the high ticket price, and adequately cover the cost of burn bags needed for application processing. Could probably also justify funding to buy a couple of leopards to guard the basement, because Leopards have antenna and thus are of legitimate interest.
and the Dept of Energy said each facility can use 100 to 200 times as much energy as a commercial building.
So, they are comparing a data centre to an ordinary commercial building such as any office, tyre fitters or machine shop and are surprised that it takes a lot more energy to run a data centre since its rammed full of servers - funny that.
Perhaps if they compared data centres to other data centres then that would be a more meaningful metric, but i assume that their desire was to show lots of zero to justify their analysis and then lead to some other half baked "solution" that saves tons of money, all whilst not understanding the problem.
If I recall correctly, an idle server takes 40% less energy than a server running at flat out, so it doesn't follow either that consolidation results in energy savings.
Turning off redundant workloads or moving old monolithic systems to more modern technologies would however since they could then auto scale depending on demand. That of course will take a lot more money up front to re-engineer things to take advantage of newer technologies before they can think about turning off all the older stuff.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
