Tempting, but dangerous
In 1992, I was migrating a server room to a different floor in the same building. The company was expanding, so they'd rented the full 10th floor, and were abandoning the half of the 3rd floor they had occupied for about six years.
The new server room was built, the power, A/C, HVAC, and halon systems were all approved, tested, and good to go, so now it was just a matter of moving 150+ machines. They were mostly Sun IPCs and IPX machines, but there were a few HP machines, and even an SGI box or two. All the networking and power connections were set up, so all that should be required for each machine was doing a proper shutdown, unplug all the wiring, throw it in the elevator, go up 7 floors to the new server room, find the properly labelled bench, plug everything in, turn it on, and everything would be hunky dory.
It was booked for a long weekend, all users were informed, so all user workstations were shut down, there were no open files or pending requests on the servers, etc. Everyone left on Friday night, and Tuesday morning, they'd go to their new office, where everything would be the same.
We had a fireman's line with one group disconnecting machines and loading them on a cart, a second team moving the cart to the new floor and unloading the machines, and the third team putting the machines in their new location and setting them up. In theory, it should only take a few hours to move all machines.
We all know the difference between theory and practice.
As it turned out, nearly 40% of the machines failed to turn on. We tried bringing them back down to the old floor, but no luck. So, we powered down some of the machines, and tried to power them back up again on the same floor, and the 40% number held. Almost half the machines wouldn't power back up after a shutdown.
Fortunately, all of the machines that were DOA were Suns, so we only had to deal with one vendor. Even more fortunate, it was a big customer with lots of money, so it had enough clout to get a Sun tech person on site within an hour (3am on a Saturday morning).
The culprits were the power supplies. Having never been power cycled for half a decade, many had faults occur that were never detected. Well, they would be detected on the next power cycle, but because the admins were competing to see whose subnet of machines could have the highest total uptime, most of which were over 2,000 days at this point, they went to extraordinary lengths to never reboot. Patches were hotswapped in, and other voodoo was performed to keep the uptime going.
Fortunately, it wasn't the hard disk, and the machines were off the rack configuration, so they could pop the disks out of a unit with a bad power supply into a new machine and get it up and running quickly. All that we had to do was get about 80 new Sun workstations at 4am, have them delivered, swap out hard drives, and that was it. Easy peasy.
The Sun rep saw a multi-million dollar account being lost on his watch, so he moved heaven and earth, and actually got us the machine in under four hours. I think he aged a decade in those four hours. We were amazed, and speculation was that he had pictures of Scott McNealy with a sheep, or something. But he managed to pull it off.
Some security configurations that used MAC addresses had to be modified, but other than that, it was surprisingly uneventful.
Our "should only take a few hours" migration that was supposed to be complete by Saturday noon, leaving Sunday and Monday for testing ended up taking until 9pm Sunday. We got everything working, but we were glad that we're migrated the small server room of 150 machines, not the major data center with over 2,000 machines.
Before doing that undertaking, a new rule was instituted that all servers would, on rotation, be cold booted every 60 days, specifically to avoid a repeat. And that data center with 2,000+ machines? It's admins didn't have uptime competitions, so failure rates were under 40%, but they were still around 5%.
Multi-year uptimes are great, but make sure you've got redundancy, and factor in the possibility that you may have to take everything off line at the same time at some point.
Biting the hand that feeds IT
