Junior techie had leverage, but didn’t appreciate the gravity of the situation Welcome once again to On Call, The Register’s weekly column that tries to balance your diet of industry news with your peers’ experiences of the messes they confront at the coalface of IT. This week, meet a reader we’ll call “Robert” who told of his time working for a local government organization in the UK. Robert gave one …
What if I took a swim in a typical spent nuclear fuel pool? Would I need to dive to actually experience a fatal amount of radiation? How long could I stay safely at the surface? - Courtesy of the 'What If?' section of XKCD
Err no.
The relationship between VA and W is very simple and depends only on the power factor. A resistive load draws the same in VA as it does in W, and the power factor is 1. Historically, computer equipment tended to have a lousy power factor, and a lousy crest factor, and well - just a generally very poor current waveform that caused all sort of issues (especially burning out neutral conductors in large buildings.
A capacitive load, or an inductive load (such as directly connected AC motors), have a power factor less than one - and then, because the V and A are not perfectly in phase, the W figure is lower than the VA. In an extreme case (pure capacitive or inductive load) the power factor can fall to zero and you can draw lots of VA without drawing any power - the term for this is VAR (Volt Amp Reactive). Power companies don't like VARs - it loads up their network without you drawing billable W - so they meter large consumers on the VAR consumption as well as the W consumption.
But back to the case in point. Because historically computer equipment had poor power factors, UPSs were generally built to handle a higher VA rating than their W rating - and still do even though modern computer equipment (to comply with legislation) generally has a fairly good power factor and crest factor due to active front end correction circuits in the power supplies. But UPSs still tend to have a lower W than VA rating - typically around 10% to 20% - and this is something you need to watch out for when speccing up a system.
I agree completely. This is the fault of management. An experienced person should have been present to ensure that the junior understood the requirements and pitfalls of the task.
It would seem logical to install a heavy equipment as low as possible - I know I certainly wouldn't have thought of getting it to the top of a rack, even when I was only 20. But the human mind is a curious thing, and who can say what criteria the junior had in mind for the task ? Maybe he was thinking that it would be more practical to plug something into the top rack, instead of crouching down to the bottom rack. It's an argument that is not without merit, but you have to discount the weight. Experience tells you to never discount the weight.
Not to mention that a 65kg 6kVA UPS must have a certain cost associated to it. An experienced engineer should definitely have been overseeing the operation.
"where to dispose of the used electrons?"
You collect them in the bit bucket and dump them back in at the highest point for re-use.
Hint: Use a funnel. You don't want to miss and spill them all over, they are a bitch to clean up. NEVER OVERFILL! Don't forget to replace the cap when you are done.
And there you were, thinking bit buckets were only good for collecting chad.
It would seem logical to install a heavy equipment as low as possible
It would, but when the then-new City hospital was being built in Belfast many moons ago the architect put the X-ray unit on the top floor of the tower, seemingly unaware of the weight of lead screening. That was a wee bit harder to fix.
I was going to say the same. In general, big shiny machines go in basements, not just because they are heavy, but because that's where the vibrations typically aren't. The same goes for temperature fluctuations. Anything doing imaging at high resolution, like an X-ray, or MRI, or CAT scanner doesn't like vibrations (although they are typically vibration-damped, especially the ones with big spinny bits in them like CAT scanners). The same goes for the machines found in university basements, like NMR machines, thin-film diamond vapour deposition thingummies and suchlike. Put them where the temperature is (more) stable, services are available, vibrations are minimal, and so on. Just not below the level of any sewerage services...
There's a building on the south coast of England, that is a training facility and has a round swimming pool inside for said training. First time I went there I stepped out the lift on the 10th floor and thought it was a silly thing to put a swimming pool on the 10th floor. I looked over the edge into the water and quickly realised it was on the ground floor.
https://en.wikipedia.org/wiki/Submarine_escape_training_facility isn't round, but is above ground, possibly for similar reasons.
Actually reading the parent post properly the pool is round, so maybe it is the same one.
I can think of reasons why you might want the escape chamber at the bottom to be above ground, but I would be speculating.
It's perhaps a bit too easy to flog Robert, too hard, for his part in this. Smart people frequently have a hard time imagining how non-thinking less-intelligent people can be ("Ain't i' fookin' obvious t'at ya don' put t'e heavy t'ing in t'e top 'o t'e rack 'cause t'at makes it EXTRA-tippy?!!").
That said, Robert's job was supervisory, and he should at least have evaluated the minion's previous experience and typical thought processes (or lack thereof) before turning him loose. Perhaps Robert did previously evaluate the minion, with RESULTS==OK, but the minion had two brain-farts that fateful day.
From TFA: "The minion had decided to pull the UPS out on rails at the top of the cabinet before getting the server crane into position, resulting in a huge imbalance," Robert revealed.
"Pull(ing) the UPS out on the rails" implies he was un-installing it from the first, incorrect, place where he had previously installed it.
Yeah - badly
I'm not saying He was installing further UPSs in unsuitable spots. We know he was uninstalling it , but after being told WHY , how could he let that happen ? he didn't even have the 'crane' present .
Did he think "Hmm, I've been told installing it high makes it unstable , I'll just yank the rails out to exponentially magnify that effect while I wander off to get the trolley to uninstall" ?
They may teach Physics but in most schools you'd be hard pressed to find a physics teacher with more than an A level in any science, STEM teachers starting pay isn't enough to compensate for kneecapping any future non teaching physics based career if you're a recent graduate.
Could be worse - in the US these days, what is important is how you feel about your answer. If 2+2=794, it's the correct answer as long as the answer makes you feel good about yourself. I am dreading when these guys start building highway bridges and decide that a 1 inch diameter plexiglas rod is strong enough to support a 10 lane highway because it just feels like it should work.
No it fucking isn't. A single "Combined Science" GCSE if the kids select it, means that they get a single grade for their three science exams.They do less total science in single than double or triple science- which give two or three GCSEs respectively, so is a good option for those that aren't very motivated to it. But they still do exams for each section. Physics, Chemistry, Biology.. And the level for kids doing triple let alone choosing three single sciences is extremely high.They're mostly the ones who will want to do it at A level and university.
No, if the school offers it, and the children select it.
My youngest got A's in everything including "Combined Science", but only because she was not in the first 30 drawn from the lottery who were the ones allowed to study triple science.
I saw the head of science at a recent STEM event and she asked why youngest had moved to a different school for 6th Form when she had done so well in the GCSEs at their school. I just said "triple science lottery". The head of science looked like a beetroot and shot out of the room!
Their latest OFSTED has moved them down to "Needing Improvement", quel suprise!
Hmm. I've never come across that with science before. Rationing by grades, possibly. Discouraging kids from doing triple if they aren't planning to go to sciences in 6th form, certainly- my youngest (top grades in GCSEs and A*A*A in the subjects she did do at A level ( the one she only got an A in turned into a First class degree last year)) was advised that there was no value to doing triple. Better to get two As than risk overstretching for 3 Bs- or whatever numbers they give them in the latest iteration.
But it is rationing. A shortage of suitably qualified science teachers and possibly lab space, too, I assume.
Back in the 80's my school was vehemently against triple science.
I ended up shunted into German Language instead of Biology, and was only able to pick it up at Higher level after A's in Physics and Chemistry. Even then I had one class where I was left sitting on my own with the textbooks 3x a week
"You don't need to be *taught* levers and moments, it's part of the physical functioning of the human body! It's like expecting people to be have to be taught how to foooking BREATHE!"
While a nice catchy soundbite, people don't learn about levers and moments from their own body on a rational and logical basis. It's almost exclusively trial and error. Tell an average person that the process of walking is falling forward so they're about to fall over and then putting a foot out to stop the fall, and repeating that action to create the motion of "walking" and they'll look at you as if you are stupid. Add in comments about tipping points, levers and moments and they'll be utterly convinced you are stupid.
Actually, stop and think for a moment.
It does not matter where in the rack you put the "'kin 'eavy" lump. Whether it's at the top, at the bottom, in the middle - when you roll it out the moment tipping the rack is the same (weight * distance of CoG beyond the front pivot point, measured between verticals). By the time the rack has tilted enough that the front edge of 'kin 'eavy lump touched the floor, you'll have a lot more moment pulling the rack over.
There's a reason some racks come with a big plate you can bolt to it at the front (to extend the fulcrum point forwards, as well as making sure the rack can't tilt at all before it's engaged with the floor).
Where it does make a big difference is when considering dynamics. In ${DayJob} our equipment has to operate in an environment subject to shocks, quite substantial shocks, and still function. When the rack/cabinet is bottom mounted, the effects of having the deck move rapidly under the rack do get substantially "more interesting" the higher up the heavy lumps are.
But I find it a bit hard to fathom why a weight of "only" 65kg was able to topple multiple racks. He might have rolled it out a bit sharpish and added some kinetic energy to the mix when it hit the stops, but even then it seems a bit questionable. I think the racks would have had to be "mostly empty" (i.e. not full of lots of heavy servers), and possibly have been on castors or feet set back from the front edge of the rack, and thus moving that fulcrum point rearwards.
With a previous work hat on, with just 3 racks, and only about 1/2 full, I reckon there would have been more than enough weight within the footprint to have survived even rolling out a 65kg device (some servers can weigh that) "with gusto".
It would seem logical to install a heavy equipment as low as possible
Indeed, but the minion was "told to put the UPS in the lowest free space in rack number one of seven". They may have taken "lowest" as meaning lowest number with slot numbering going top-to-bottom rather than lowest meaning nearest to the ground.
I know from experience of dealing with people who are 'on the spectrum' that they can tend to interpret instructions in a way which is technically correct but not what others would intuit.
No it isn't. Many people assume the autistic spectrum is from high-functioning to low-functioning, but that would be a continuum —shades of grey.
there are many areas in which people can diverge from neurotypical: think of each one as a colour of the rainbow, and of each person with autism as having their own colour palette. THAT'S the spectrum.
"I know from experience of dealing with people who are 'on the spectrum' that they can tend to interpret instructions in a way which is technically correct but not what others would intuit."
Also known from experience, if instructions can be interpreted in multiple ways, then when things go badly, the person doing the job will be blamed, not the imprecise instructions. Look at you, Project Manglers!!
Long ago,in 1975, I was on a Civil Service job that was being rescued by a gang of freelancers and others of us hired from small software houses. This was after the original implementation team, newly trained and inexperienced Civil Servants filling the roles of both analysts and programmers, both teams with zero previous experience, had predictably, screwed up. It was quite a complex job, running on five ICL 1904s distributed round the UK, all running George 3 and programmed in COBOL. They interchanged data by writing it to tape and sending it between data centers in vans.
Not long after I'd joined the project, another new programming hire rocked up. It turned out that he could not understand that a divide by zero error would simply be trapped by the program, which would report the error and stop. Nobody could convince him that divide by zero in one of the running programs wouldn't immediately bring the whole machine to a screaming halt and probably damage the CPU as well.
Needless to say, he didn't last long on the project. We never discovered who hired him either.
"It would seem logical to install a heavy equipment as low as possible - I know I certainly wouldn't have thought of getting it to the top of a rack, even when I was only 20. But the human mind is a curious thing, and who can say what criteria the junior had in mind for the task ?"
I could see a situation where an inexperienced junior whose job it is to "service the racks" might see it as logical to put the stuff least likely to need changes in the most inaccessible places, leaving the more easily accessible spots for kit that might need pulling out or havering cables swapped around more frequently.
Ah yes absolutely true, but you forgot "working for a local government organization in the UK." The usual modus operandi here would be to quickly look around for someone to pin the blame on, and ideally the most junior person they can find.
On the plus side, it was a good test of their disaster recovery procedures! I wonder if the UPS survived to be used again?
This post has been deleted by its author
"working for a local government organization in the UK." "a good test of their disaster recovery procedures!"
Nope, sorry, lost me there. Local government? DR?
AC as I am currently "working for a local government organization in the UK" and have spoken up several times about the DR procedures that are so out of date that they are useless.
(And before anyone shouts, yes, I am actually trying to do something about it. But as I am "working for a local government organization in the UK", you will understand that Sisyphus had it easy.)
I am currently "working for a local government organization in the UK" and have spoken up several times about the DR procedures that are so out of date that they are useless.
To borrow a phrase from someone else (I did mention I would in an appropriate case), those DR procedures are state of the ark.
you will understand that Sisyphus had it easy.
And there you win the award for understatement of at the very least the decade.
I used to work for a very large, but unknown, IT firm who did lots of this sort of stuff on behalf of well-known firms. They had this lovely weasel way of making it your fault by saying things like "now, this part is marked as heavy, do you want someone to assist?" so that the decision on how many people were needed was pushed onto the hapless field guy, with the unspoken implication being that you didn't need help. Asking for a second pair of hands would usually result in a more senior manager ringing up to double check, rinse and repeat until you hit the top area manager who would usually say "yeah ,we haven't got anyone free, can you do it today yourself?"
> ...with the unspoken implication being that you didn't need help...
"If it's marked as 'heavy' then 2 people (or lifting equipment) are mandatory, as is a handling plan. I don't want to be prosecuted for breaching HSE regulations and I'm sure you don't either, so please draw up the plan"
Had anyone ever bothered to show Robert how to supervise? Dispatching people on missions is not a natural thing.1 Delegation and [shssh] leadership are skills which come mainly from experience.2 It's not particularly difficult when you know how but people are different and situations change. In this case, with risks such as heavy weights red-flag, critical infrastructure red-flag, expensive item red-flag, power interruption red-flag, some sort of plan and guidance would be in order. (And some sort of debrief after a success, or a woah! conference if there's something not going according to plan.)
1. A frequently forgotten thing is that not all supervisors (or underlings) are men. Men 'go on quests' while women dig-in and make the most of what they've got. Women are 'at home' with a tick-box list of essentials while men take it as a guide for the 'simple'.3
2. The sort of experience that comes a few moments after you needed it.
3. Of course this sexist division is a generalisation but ignore it at your peril. By the way, if you want an example of a girl doing heroing the man's way (ie. Monomyth) look at Dorothy in the Wizard of Oz. Have you ever wondered why Dad's Army remains popular? Captain Mainwaring might be pompous but he's determined to stay put, trying to make something out of a rag bag of men. That's doing heroing 'the woman's way'. The 'man's way' is the successful Hollywood formula.
... are one of the most-valuable tools in my box, when they are for procedures. For personal use, I'll make a list if it's for tools, supplies, or clothes, and am willing to hand-wave it when I'm at the stores, depending upon what I find.
I do not hand-wave technical procedures. Emacs' add-in, now a built-in, org-mode is super-handy for creating technical procedures with tick-boxes. The resultant file can be "ticked-off" electronically in Emacs, or printed on paper and ticked-off by pencil or pen.
*Especially* where processes are sequentially dependent and/or there are time lags between steps, so you absolutely must have some form of documentation to tell you where you've got up to so you can do the next step.
* Create account, wait for it to propagate
* When propagated, do X, .... etc.
* When X completes, do Y, etc...
At the very least I think Robert didn't do his own risk assessment for his supervisory role correctly. A job like this (especially after the underling already demonstrated a lack of understanding of the physics involved) should have involved some closer monitoring. Can be very hands off, from a distance, but closer monitoring nonetheless.
That is possibly the dumbest sequence of events I've read about in "On call" or "Who me?"
I gathered from the headline what the issue would be, little could I imagine the scale of the fuckup and , dumbassery that caused it.
It beggars belief .
The kid is told exactly why you dont put weight high up and to move it down , and then proceeds to let that happen.
...and to cap it all brings several racks over like those "Forklift in a Warehouse" fail vids you see online.
The fact that it was a heavy item of kit should have been enough to require two people to install, or remove it. Even with a hoist no one should be expected to be able to move that amount around safely on their own. I'd have to look but 18kg was the limit I think when I was racking and mving stuff in datacentres. Also we were not allowed to take all the HDD's out to reduce the mass either not after the Great Hard Drive Shower fiasco where we managed to spill a stack over over a hundred onto a concrete floor.
Robert I think is lucky that he also didn't get a interview without tea and biscuits, as there seems to be a distinct lack of the supervisory element here...
Yeah, but that only removes about half the weight- you still have a great fricken big transformer in the guts of the machine, and that weighs a fair amount. And then, it's off-center, so having a second (or third) warm body to do the lift and shift is easier on everyone's bodies.
The largest UPS I've ever dealt with was a pair of 5kVA Smart-UPS units, and even with two others with me, I used my handy-dandy lift table to rack the bloody things, because we wanted them 5 units up from the floor in case the place flooded. AND we pulled the battery modules out first.
(There's a damned good reason why I have that lift table- You can only rack so many disk shelves by yourself before your body says "screw this, no" and something important in your body goes pop.)
:: wanders off humming We All Lift Together from the game Warframe ::
" so having a second (or third) warm body to do the lift and shift is easier on everyone's bodies."
My experience with racking jobs is that if you need a second body to lift something into position in a rack then you just quadrupled the risk and a 3rd is just plain reckless from the outset
There's a reason lifting equipment exists. It's not that expensive in the overall scheme of things and is a LOT cheaper than injury compensation claims
Not a "minion" but someone who was very junior and lacked experience and possibly skills.
This is not a "He should have known better" issue ( though a bit of common sense sounds like it's missing), but a failure to manage the training and development of a junior staff member.
Had said "minion" installed a bloody heavy device before? - Apparently not.
Did his boss make sure that the "minion" knew what he was about? -clearly not.
Were there risks attached to moving and lifting something that heavy? Bloody obviously there would be.
Can the "minion" be blamed. No! It's not his job to know what he doesn't know. It's his boss's job to know what the "minion" needs to know and bloody well make sure that he did.
"I've bagged groceries a few times!??"
If you look at peoples shopping trolleys going around the supermarket, or the customers bagging up their goods at the tills in front of you, you may begin to think a little differently about how much people know about weight distribution and stacking heavy items on top of lighter, more fragile items!
Rewind to your first experience and you might not know that it would make the rack so top-heavy and that the rack couldn't handle it. Sure, having a heavy thing at the top is a problem, but not if the thing in the bottom is also heavy. Those big servers are down there. I've never lifted one of them, but I did see someone go get another person when they were going to move one, so they must be heavy as well. This rack is built to hold lots of really heavy things, and it's not even resting those heavy things on the floor or on each other, meaning it was clearly built to take a lot of weight. They built it that high, so presumably they have built it to take weight at the top and it was installed to deal with that, for example being well anchored in the floor or wall, the way you would anchor a shelf if you wanted to put something heavy on it.
We know that those things are not true, but if you've just started on plugging in servers from working with desktops as a student, would you know all these things? I didn't know the typical weight of rack servers until I lifted my first one; there had been no reason for me to look it up.
"Rewind to your first experience...."
Well, for any normal human that will have been around the age of nine months when you extended an arm and positional feedback told your brain that gravity was a thing. If you didn't manage to get past that stage, you have a great deal more wrong with your life. YOU. DO. NOT. NEED. TO. BE. TAUGHT. HOW. TO. BREATHE.
If you read past the first sentence, it would become clear that I was referring to experience with server hardware and racking to know things like the weight of the equipment and the general stability of racks. I have a feeling you did read past that sentence. Thus, your reply fails to make any meaningful point.
Throughout childhood, you also learn that it is not as simple as "heavy thing in high place, always bad". For example, a heavy item placed on a countertop which is elevated usually doesn't cause the wall to topple over because the counter is attached to the wall and floor. A heavy item placed on a sturdy and stable table is also usually fine because the table helps to spread the weight. That's even true if the heavy item is not centered on the table. Of course, there are numerous examples where it would cause a problem. Life would show you that there are cases where that works and ones where it doesn't, but unless you saw server racks in your childhood, it wouldn't tell you which one they would be.
A person who assumed, incorrectly as it happens, that server racks, being designed to hold really heavy things, would be designed more like a counter than a cheap bookshelf in the weight management department is really not that outlandish. If they went the other way, I could easily imagine someone not wanting to install a server in the high part of a rack because that's kind of heavy too, so maybe we should reserve the top slots for if we ever get some lightweight fan array or something. I am asking you to consider what things would be like if you were missing some important information that you have gained through experience. There was a time in your life when you didn't have that experience. We don't know whether this particular person was a new starter who had never worked with a rack before or just clueless, and there's a chance that the correct answer is both. There is also a chance that the answer is only one, and it's less stupid than you make it sound.
Absolutely. If there are racks designed to hold heavy stuff then it's OK to put heavy stuff in them. If this included high racks the natural, reasonable assumption is they they are fixed to enable storing heavy stuff. At which point all the comments implying that said minion is an idiot are just plainly wrong and unfair.
It is not reasonable to expect anyone to suspect that the big high racks full of expensive kit aren't securely bolted to a solid wall
And this is true.
I do vaguely recall reading some time ago, that in theory at least it is 'best' to have the UPS's at the top of the rack because of the heat they generate* can then rise up and away rather than heating up the equipment above them.In practice, though this is not recommended because of obvious problem, unless you know that the rack is really, really securely fixed to the floor and/or has brackets securing it to a convenient wall so it really can't tip.
Off course in the vast majority of cases, the rack is simply sitting on the floor on it's dinky little castors** and is about a secure as a jelly nailed to the ceiling.
However, I think we can assume that 'the minion' wan't aware of this. But what is inexcusable is that having done it and the 'so-called' supervisor had seen this and simply told them to reposition it and left them to get on with it, without any, well supervision. You don't need to be a genius to realise that this person doesn't understand the potential consequences of what he has done, although they did manage to get away with it, and hence what is quite likely to happen when they move it!
Surely as a 'supervisor' your job is to, well, supervise and make sure that the people working under you are doing their jobs safely*** and properly.
I am genuinely interested, in hearing from our American colleagues here, if such a thing were to happen to a US company or local government organisation, identical situation and systems were down for a couple of weeks, resulting in losses of the order of many, many millions of dollars, who would be in the firing line?
* Not 100% sure this is a real thing, how much heat do they give off compared to a server? Maybe was true in the dim and distant past.
** And I am always a little wary and prepared to jump back whenever I have to pull a 2U server forward and out on its rails, even mounted in the bottom half of a 42U rack. I look at it and think, 'this doesn't look too stable to me, I think I'll pop the cover, add the extra RAM etc. and push it back in ASAP.
*** And what would the consequences of this happening if the 'minion' was standing under the UPS when it fell? Health and Safety, not to mention AmbulanceChasingLawyers4U would be all over this like flies around a dog turd.
I think that, no matter where they were, the blame could go anywhere the internal politics wanted it to go. It wouldn't be hard for someone to decide to blame either or both of those people. A charitable manager could probably have insulated both from too much blame and reduced it from firing to an unpleasant meeting. It comes down to the question of how high on the organizational chart did it go, how angry was that highest person, and did they want to do something to either of the people involved.
It's rather hard to bolt a row of racks to a wall when you need access front and rear
Bolting them to the floor might seem the obvious move, but most server rooms use tiles on a raised subfloor, so it's not as secure a method as you may think
Some racks (eg: Prizm) have an option of pull-out stabilising arms in the plinth to mitigate the risks when sliding heavy kit out: https://www.dcdi.co.uk/server-cabinet-plinth-with-stabilizing-arm
«He knows it was a mistake to let the minion do the job unattended.»
Whether it was a mistake depends entirely on how much he disliked the minion. For minions high on the spectrum of personal dislike, giving them an improbably difficult task ("job empowerment!") and watch them fail from afar (pure, evil joy aka schadenfreude) is the way to go.
And before you jump to conclusions or worse, action, this post may contain irony.
As someone who's been on the receieving end of missing training/supervision and then have gotten a bollocking for not following a procedure "that was obvious" and "I should have just known",
I have a lot of sympathy for juniors making a mess when seniors should have been checking their work.
But when you've been told to redo your work because it's unstable, you do not then proceed to make it more unstable, and then leave it to itself.
The minion should get the bollocking for putting it in a silly place in the first place. His manager should get the bollocking for not closely supervising the corrective action from a minion who has already demonstrated that they don't know what they are doing in regard to physics and heavy things. At the very least, there's a shared responsibility there.
I've deployed some brand new server racks in my day. Every one of them came with an anti-tip mechanism. Either the ability to bolt them directly to the floor, or a big lip to bolt on the front of the rack. I totally admit to not installing the anti-tip mechanisms and just remembering to be cautious about where I put the weight.
It sounds to me like the senior admins failed to build out the server room properly in the first place.
The level of fundamental stupidity of an adult that cannot figure out not to put heavy things up high indicates an eternal intellectual challenge to work around. Keep it away from me and anything that could be made hazardous to others. This one sounds like the interviewer missed the bits of crayons in the applicant's teeth.
not everything goes on rails in a rack.
Why was a 65KG UPS sold with a rail kit?
i've installed a number of cisco 6506's mid rack back in the day solo.
helps when there is something to rest it on like a blanking plate or other switches. it would never occur to me to put it at the top of the rack as it is so heavy & obvious heavy things go at the bottom.
yes i did ask questions before installing mid rack.
at 65kg's even with a hoist its a 2 person job.
i guess if it had rails and you where told it was a solo job & there was other stuff in the rack like fibre and copper in the middle or bottom & there wasn't enough room at the bottom & the jerk asking you to do it thought you where not capable then sticking it at the top would be the obvious choice.
The next obvious choice is to go back and say where in the rack do you want this.
Having seen it at the top of the rack, why did the experienced people help in removing it if the reason to relocate it was that the rack would be top heavy.
The story teller should have lost their job.
The kid who did the installation said (paraphrasing) "Well, we don't want the equipment to get hot, the UPS can generate lots of heat, and heat rises, so ... "
Got to admit, he actually thought about it. And it's almost impeccable logic ... Though maybe not in Earthquake Country.
We did not tip the rack over while fixing the issue. Forklifts are your friend.
Back in the late 90s where they were using HP K class PA-RISC servers. Those were fairly hefty, a quick look at specs on the web shows them at 60kg and something like a third of a rack in height. They were generally installed two per rack, with UPS on the bottom and JBODs below each unit. So the "top" unit was generally lifted in with two people as you had to raise it to shoulder or so. And yes we were doing that - the datacenter at this place wasn't "union" like some I've encountered, so in the name of getting stuff done faster than the facilities people would if you wait on them, we'd take it upon ourselves to tear open the pallets when new servers and (fully assembled) racks were delivered and put things together.
One of the other half dozen consultants on the project was rather large - 6' 8" and had to be at least 300 lbs. He had a gut on him but clearly had once been in shape or stayed in shape despite the excess weight he was carrying around the middle. He had already placed the UPS and bottom server when I came in so I was going to help him lift the top server into position, but he just lifted it off the ground and slotted it into place like it weighed almost nothing. I'd helped others lift a few of those into place when the previous shipment that had come a few weeks earlier, so I knew how heavy it felt. If pressed I think I could do what he did - but not without a bit of struggle and not without risk of immediate termination if it turned out I was wrong! He made it look so easy, like he was lifting a pizzabox server into place. Didn't even bother to bend his knees, he lifted with his back but it was so light for him I don't think it would have occurred to him do otherwise even if he had a bad back.
Scott, wherever you are, have yourself a beer!
During a job interview in 1996, i asked the candidate to write a function in C that would take an array (int *p) and length (int len) and move all the zeroes to the end of the array, so nonzero values came first. When they got done i asked them about their fierce search for the first zero ...
Me: What if the array was entirely nonzero, would you like to make a change to your code (note: their code would loop out beyond the end of the array)
Candidate: No need!
Me: Why?
Candidate: it will eventually find a zero ... Somewhere in memory!
Needless to say they did not get the job!
We are not going to fix blame from a one-sided tale filtered through a cutely clever editor.
But we can brain-storm.
In serious work, Utility Power Systems, if there is more than one person on the job it starts with a Tailgate Meeting. The boss calls the others to his truck tailgate, outlines the job, asks for comments objections or better ideas, then says how we will do it, assigns specific tasks. 11,000 Volts, half-ton, 40 feet in the air or down in a manhole, out of sight of the other team members..... you gotta all know what you are doing! (Even on one-man jobs, that man may pick his path and then call his supervisor for heads-up and what-if and FYI.) As some here said: multiple fingerprints on the mess-up. Blame shared is sometimes blame spared.
Tick-Box Lists can be done on scrap 2x4 wood.... how carpenters do it. Yes, there are many high-tech ways but whatever. WRITE DOWN the rack-slot so both boss and grunt are on the same page and the same screws.
> I've bagged groceries a few times!??
That's become a lost skill. The local takeout/away consistently puts the hard heavy items on top of the soft mushy item. (They also turn the frozen dairy treat upside down, but that is a bit of snark at McD's which can't even get that right.)
If this was junior’s first time doing anything like this, their mentor’s the one who should’ve been suspended (or had a UPS hooked up to their ‘nads). The fact that junior tried to install the UPS at the top of a rack suggests they weren’t adequately instructed.
"let the junior tech do something alone that we knew might be dangerous without watching over him, even though we knew he'd never done it before. Then when it all went wrong we gave him a bollocking (because we knew it was our fault for not supervising). And to add to that, we made him do a walk of shame as he'd been suspended for making a mistake.
We left out the part where the junior tech then said. "Fuck this job, if you're going to suspend me over a mistake, a mistake you probably should of been helping me with then I don't want to work for you as you all sound like entitled cunts."
Thats what I'm getting from this story. First one where I'd blame the senior techs.
to my other comment. Upon starting in local gov, I hadn't been there long and was sent on a job I said I had no experience with but I'll take a look. Went into town to look at a display LED board that wasn't working. Connected up to an office on the sea front. This this was really old, something from the 80s I guess and we were now in 2010.
I had no fucking clue, senior techs weren't helping so I said "Fuck it. No one monitors what we do. All they know is I'm in town trying to fix the sign.". So I just spent two hours wondering round the shops in the end.
Several years later, after I'd finally joined the senior techs and knew them well I look back. I moaned at one for giving me no help but understand, to a point, he was already demoralised from working there. None of them were getting help from management for anything. No training programs were offered, they'd get blamed for shit that wasn't there fault. And the final part he said "None of us really knew how that sign worked".
Over the years I've passed that sign every lunch and remember it was my first ticket there. In all those years it was never fixed. Its finally bee ripped out.
Many years ago now, we moved offices. The least breakable stuff had been entrusted to a removal firm, but we had to keep a small cabinet for last, as the machines in it needed to face both ways.
I was carrying a 1500VA UPS (just a little 2U one) under each arm. One of the lads (who had sometimes given me a lift home on his way to the gym .....) offered to help me. I put one down and passed the other to him, and he sagged visibly under the weight of it. "What have you got in that thing, Jules?", he groaned, "Lead weights?"
I just smiled and said, "Close enough". I wasn't about to dock him any points for missing the big chunk of copper and steel at the opposite corner from the lead.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
