How to build a server room: Back to basics Dave Cartwright, an IT operations manager for a telecom company, recently compiled a list of dos and don’t for IT infrastructure buyers for El Reg. The article was well received but… ”the writer obviously moves in better circles than I do”, says Reg reader Hatless Pemberty In response, Hatless - we can call him that, can't we …
Better still use an electronic access control system, either keypad or RFID based, so that not only do you control who has access but you can log who has been in and when.
I'd also say that
"11. A 1 Gbps uplink cannot feed 48 non-blocking 1 Gbps ports."
Is slightly misleading, a 1 Gbps uplink is fine for 48 1 Gbps ports, as long as you don't ever want or expect the total bandwidth in/out of the server room to exceed 1 Gbps.
Providing far higher uplink bandwidth than you're ever going to need at any one time can be very costly.
"12. Metered and managed PDUs will save your bacon one day. Buy them."
Similarly misleading, they're nice to have but if you do your maths correctly, document what's plugged in where, and load your PDUs with a reasonable overhead they're not exactly essential. They're also quite expensive, some good planning and installation management can save you a fortune here.
Both of my points kind of become their own:
17. Don't blow your budget by massively over specifying your build compared to what you actually need.
Try running a five year old UPS at full load and see how long it lasts compared to how long it's supposed to last.
If you have a DR plan that involves waiting for a generator to spin up (or servers to shut down cleanly) it's vital you keep on top of the UPS batteries, unless you're confident that your servers can handle having the mains whipped out from under them mid-operation.
Steven R
Battery life depends primarily on ambient temperature, but also on charging and load profiles. For example, telco quality batteries often have a rated 10 year design life, but that's only at 15-20 C ambient. If you increase that to 35 C, for example, you might only get a year or two design life. It's all on the data sheets for such batteriies and the cooler you can keep them within reason, the longer the life.. Charging regime needs to be right as well and even a few 10's of mV increase in float charge voltage can significantly reduce life. Float charge voltage also needs to be temperature compensated. Finally, every time you load test your ups on these usually cheap underrated gel cell batteries, you typically put 10's of amps load on a battery which may only be rated at 10 ampere/hours capacity. This puts a lot of strain on the battery and reduces it's life significantly, if done on a regular basis...
If you are already running a backup generator system, consider installing a Ride-Through UPS system (example: http://www.hitec-ups.com/p3.php?RubriekID=2798). Especially in countrys with lots of brownouts and short power quality issues this could save you lots of equipment losing their smoke seals.
I've had a demo from that Hitec company on their diesel powered rotary UPS systems and it's impressive stuff. They can switch over power supply from grid to dieselpower fully phase matched, so no brownout or powerspikes. Seems like good stuff for those managing a large site and fortunate enough to have the budget.
With APC the units themselves will bleep annoyingly if you don't replace the batteries when they think they've reached their limit.
Doesn't mean they won't provide power for a while if needed, but they might not provide power for as along as it takes the servers to shut down.
And you may yet have enough runtime but the bleeping will get on your wick, and people will notice the alarm. So you might as well change the batteries. Even if the battery is around £150, it's a pound a week in running costs over those three years.
I can say that I got 7 years out of my UPS that I had sitting at the bottom of my rack cabinet. (No server room, except that it was my second bedroom in my apartment and I only had 4-5 4U boxes in the rack...)
As to cooling. When I was back in High School, the school built a 'server room' (One PDP vaxen) and put a home unit window air conditioning unit to cool the room down. It worked and the drip pan was on the outside of the room. So you can do that today, if you're only talking about a rack and you have ample air flow around the rack.
Yes.
Yes, I am sure.
Yes, I have personal knowledge of what happens when a departmental file server with an ancient UPS which is over double that age is left to fester in service without repair or replacement, and which periodically tests this UPS automatically.
Basically, every time the UPS is tested, the system finds out yet again that it has the electrical storage capacity of a couple of hearing aid batteries (at best), and does the UPS equivalent of "Dave, I'm dying Dave" and promptly turns its self off.
Cue a delayed reaction from the slowly-diminishing flock of users who actually give a shit about the system, and off yours truly trogs to the server room to turn the file server back on again. And, upon my return, a polite whinge to whoever seems to be responsible for the kit this month, requesting a replacement. Which will be ignored, just like the last half dozen repetitions.
I've been through that too. Our UPS batteries were dying. I told my manager. He said we can wait. I said not really, we'll be caught dead, literally. Fast forward a month and the UPS died completely and I told him again. His answer was he didn't want to spend the $350 for a new set of batteries for the UPS.
A week after that the power company did some stuff on the pole. We lost our server as the RAID was taken out. Not one drive mind you, but 3 out of the 5 drives! I was able to recover the server, but the hardware was damaged. The surge/sag destroyed the SCSI controller and did other things which made it unreliable from that point on.
All I said to my boss was I told you so. He blamed me for not fixing the problem and was getting ready to write me up for it! I walked out of our meeting, and I took his emails to me, which I saved like I saved all my emails forever then when I was working, and marched into the General Manager's office. He was on a conference call with the CEO, and I handed him my stack of CYA emails. A few days later my now ex-boss was lead out the door with a folder and a big box of his personal belongings.
If he had authorized the $350 for some batteries it wouldn't have cost him his job, nearly cost me mine, or would it have cost us close to $10,000 to replace the machine.
I worked for a Training & Enterprise Council in the 1990s at a time when there was a policy to merge the Business Links with us. We had to take over the local BL's systems and discovered that their server room had previously been a men's toilet. The urinals had gone (and, thankfully, the smell) but the cubicles were still there! I can't remember whether or not the water pipes were still in dangerous proximity.
It was somewhat less than ideal though it did, at least, have a lock on the door (but only a keypad).
When working for large chemical company on Teesside in the mid-1980s we had a PDP-11/34 (running Mumps possibly appropriately) installed in the site Medical Centre - in the no longer used mortuary. Positioned directly over the blood drain channel.
Just think what the BOFH might make of those opportunities!
From a different era ... when ICI (for that is the company of which I write) built the site after WW2, safety standards were not what they are today and part of the risk acceptance involved in building and operating plant was the fatality rate - I understand that pre-war in other Teesside plants this was measured on a weekly basis. Given that the road network in the area meant that transit times to the town hospital would have been 20-30 minutes, the site had a medical centre capable of handing quite major trauma incidents, and sadly this had to be included.
Having served (prior to recent retirement) as a project / contract risk assessment and quality assurance manager for one of the major IT services companies, it was sometimes salutatory to remind solution development teams that risk sometimes involves more that just financial impact. One of my favorite stories (from a colleague originating from the site discussed earlier) was her discussion with a team solutioning a new IT-based site-wide toxic gas alarm system (I forget for which company). Going through a risk identification brain storming session with the team she asked the question of the team : "what's the most significant risk involved in this project?" Back came the typical responses: late delivery, cost overrun, dissatisfied customer and the usual stuff; until she pointed out that if the system failed to work reliably what about the dead bodies across the site when they were not alerted to an incident?
Obviously there's more to safety critical systems than this, but an area of increasing concern with current developments in the industry.
Hey, cubicles could be handy in a server room. Also fresh water for coffee maker (obviously). Just raise the servers from the floor!
One of my clients are using a (decommissioned) sauna for the few servers they have. The showers in the previous room have been removed, and thankfully the floor drains still work well. (tested during one very rainy weekend)
Figure your power and then double that. Then double it again.
If you have 800 mm deep racks, you can't put modern servers in them.
If you have 1.2 m deep racks, you need 1.2 meters in front of them if you expect to put rack mount servers in them. 600 mm deep racks are only useful for cable termination and sometimes not even that.
You need at least 600 mm on all other sides.
Raised floors are cool but they come in 600 mm x 600 mm (or 2 ft x 2 ft) and you can't cut them and be useful in a small computer room. That means you need at least 5 tiles in a line per rack (600 mm behind, 1200 for the rack, 1200 in the front to load the rack). You need 600 mm on both sides and you better figure on two racks. For 7x5 full tiles is the minimum. Builders cut floor tiles to fit the room, not the other way around so either you have some sort of extra stuff around the edge or your tiles have to go in like a jigsaw puzzle where every piece is rectangular and white.
You need 1 kW of cooling for each kW of servers. A reasonable split system can do 6 kW of cooling but you also need two for redundancy.
Your UPS and cheap generator won't run a 6kW air conditioner but it can run a 2.5kW one.
We have 2 racks that each take about 1 kW and we have two 6 kW A/C and two telco grade 48V systems hooked to 16 truck sized deep cycle batteries. We can run 8 hours (except for that A/C thing) and the solar can extend that for another 4 (except for that A/C thing with a more effecient sun heating load bit)
You want LED lights hooked to the UPS. You want a phone in the room assuming tbe PBX is there too and you might want a way to power if assuming the POE switch isn't in the UPS.
A rack can weight 2 tons. Make sure the floor can cope. Even if it isn't that heavy, it can put massive loads in very small areas on a floor.
Make sure you have the door on the alarm and put in smoke detector in the room.
Did I mention that you need to double your planned power? And then double it again?
"You want a phone in the room assuming tbe PBX is there too and you might want a way to power if assuming the POE switch isn't in the UPS."
Speaking from experience, a phone in the server room may not be useful with all the background noise created by the cooling fans.
Those fans aren't a problem, just shut things down. We had (past tense... gone soon after) VP who insisted she needed a key to the server room. She went in to take a private call on the cell phone (only room she was close to) and decided the fans made too much noise. Not as much noise as the screams from the employees, followed by management up and down the food chain, though.
She didn't hit the power switch... just started unplugging all the power cords.... switches, servers, RAID, PBX, Aspect phone system... took me and the manufacturer's techs almost a week to get everything working properly again. This was 16 years ago... upper management sticks by the mandatory "only credentialed IT allowed in server rooms" which was implemented two weeks after the event.
With our SME installation - one rack moved from a previous site - the CAT5 networking was terminated in panels installed in the back of the rack.
As the rack had been positioned but the back doors not fitted, the cable guys fed the 30 or so cables through the back frame rather than down through the hole in the top of the rack.
So now the back doors can't be shut.
Now I though I knew that they knew what they were doing but I was busy with other stuff to do with moving site. So c'est la vie.
Subsequent cable installers, CCTV engineer etc have since followed the path of least resistance. There's a multi-coloured elephant trunk sized bundle of cables now.
For those who want a secure SOHO in a single family home...
You want to have re-enforced walls. So if you're building a new house, you can pour the walls as part of the plan, or buy prebuilt walls that contain metal flakes. (Just make sure your foundation can carry the weight. (assuming that you have a poured foundation.) )
Next put on a safe door. (essentially a metal door that looks like a regular room door. )
Now you have your room, add your A/C and power. (running both through thick pipes)
Add a Faraday Cage around the servers. And then the fire suppression unit (optional) that you will want after to ignite the det cord you put on top of the drives in your server. Please note that this last option is only for those who are truly paranoid.
Now if you skip the fire suppression unit, this room also makes a great safe room too.
Sorry but you can just follow the cable afterwards if it is a PatchSee network cable. You put the handy tool thing over the cable and vola the other end is illuminated making identification easy peasy. They also come in lengths from 60cm to 3.1m in 30cm increments all with the length printed on the plug. So no excess cable, and no guessing on the length when reusing the cable.
Yes, indeed, check and double check and thrice check your UPS equipment state daily and consider the location at all times.
Also consider the consequences of NOT checking the above.
See here > http://www.bbc.co.uk/news/uk-england-nottinghamshire-11959403
This was caused by a UPS.
I used to work there, therefore, I know. I can't obviously put more details there for confidentiality reasons but I'm sure people do know about the "Freedom Of Information" request system if they REALLY wanted to get the documents from the investigation.......
I worked for a rather large UK ISP, whom wanted to move some of their hosted equipment into a small server room, in their office building. Needless to say the 4 racks + those already there + office equipment would have taken 105A. The boss didn't understand that the CEGB main fuse on the building was only rated at 60A. Guess why I stopped working there?
You could if you're a huge company, simply move all the local servers to a datacenter that does indeed adhere to all these rules and a whole lot more. Then when all the users complain that the link to their folders (redirected, but most of the users haven't a clue what that means and everything is datacenter hosted for that matter) is slow, you simply tell them "LAN latency, it's a fact of life, live with it".
Add environmental monitoring to the list. Temperature, humidity and leak detection are a must IMHO.
It all goes a bit smelly when the CRAC stops working, either because of a fault or because of a power outage and the CRAC doesn't start up again after power is restored.
Add leak rope on top of racks if you've not been able to divert overhead wet services and/or wet contact sensor under the floor tiles if you've got a void under your expensive equipment and priceless data.
Some metered PDUs (dare I say it, APC) have built in environmental monitoring if you're very tight on space or want an "all-in-one" solution.
You are trolling or miss the point in this discussion or I did not pick up on intended sarcasm. Other then that you are also wrong because of inductional loads give rise to "blind power' or cos phi related currents. Excuse me if/when I am using wrong technical terms/words since I am not a native english speaking person.
After Googling translate I might have had to use the phrase "electrically reactive power".
Just to be clear, you get an extra load in current besides the power the equipment will draw just for quenching it's power thirst.
It's called power factor here in the uk and iirc, the US. Bad power factor means that the current is not in phase with the voltage in terms of the load, which can appear as though the systems are drawing more amps than the watt hour meters would suggest. It means that heavier cables must be installed to carry that wattless current. Older IT equipment was very bad in that respect and often presented a bad harmonic load to the supply as well. However, modern servers nearly all have power factor correction within the power supplies and it's not anything like the problem it once was...
If you need more power from the cleaner's hoover socket you simply plug in extra '6-way mains lead' things which supply more wobbly amps. And for super power security, there are 'mains adapter blocks' which indicate their functionality by glowing and giving off acrid exhaust fumes as they generate more current ... a 6 way with 6 three way blocks - that's 18 times the power!
We all have to start somewhere.
I've also seen floors collapse because "the guys who did our office floor, you know, the one with the carpeted tiles, did such a good job they might as well do the computer room".
I guess the point I was trying to make is that building a proper Data Centre is easy beacuse they costs a chunk of money and often are designed from scratch so even the worst PHB can see the need to have proper project management, buy proper equipment and hire proper professionals. And so, all the points from the original arcticle (all very true) are less likely to be missed (YMMV)
The real problem is at the bottom end because the cost of the "extra stuff" can be disproportionate compared with the cost of the IT you want to buy (plus it is never budgeted for). Every university has some researcher that bought a couple of blade chassis only to discover they cannot plug the bloody things anywhere.
In these cases, hosting and/or collocation can be a better solution but these are the same people who cannot bear the thought of keeping their precious servers off-site, don't have the staff to manage things remotely and cannot work out that depreciation is a real expense. And so, the poor bastard with the root password has to make lemonade.
It shouldn't be like that but that's life.
P.S. My typing skills suck.
I can testify with regard to floor loading problems. Even if it has an access floor already installed. Actually, especially if there is an access floor -- they come in different strength levels and add an appreciable amount of weight to a structure.
Decades ago, I was helping some friends of mine (different company) and our IBM field engineers install a new IBM 3033 mainframe. As we were pushing it into the computer room, we noticed that we seemed to be going "down hill" a bit. Caution ruled and we pushed it out on their loading dock while we investigated.
We opened up the suspended ceiling in the employee's cafeteria below and found chunks of concrete which had spalled from the deck. Imminent structural failure.
IBM had to take the machine back until they built a new data center (about two years later!).
Oooops. I wonder how much an IBM 3033 depreciates if it drops a floor? :-)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
