back to article BOFH: Something's consuming 40% of UPS capacity – and it's coming from the beancounters' office

BOFH logo telephone with devil's horns It's completely dark in the server room and the building is dead silent. Out the windows of Mission Control I can see, off in the distance, the warm smoky glow of a transformer fire. The building's generator is stopped and the redundant UPS units are in OVERLOAD BYPASS. The only …

  1. Jess--

    Aaahhh the world of IT, where turning it off and on again really does "fix" everything.

    1. JimboSmith Silver badge

      We discovered a fan heater in the server room that someone had obviously plugged in because they were cold. We never discovered the culprit (had a few ideas though) but sent round an email to the entire company stressing that people should not plug anything especially heaters into "IT power sockets".

      A few days later I picked up a call from a very timid sounding lady I'll call "Jane" who worked at a branch office. She'd recently reorganised her office to get her out of the stream of cold air coming in from the leaky window. Apparently Jane had moved her desk from one side of her room to the other. She wanted to know how we found out that she'd plugged her heater into the socket the computer was previously in. Jane also wanted to know if she could actually use her heater again as the radiator in her room still didn't work and it was very cold.

      After determining that this wasn't a wind up I asked what had happened. Jane told me that she'd finished moving her desk and then plugging her computer equipment back in. Having checked that everything worked she'd plugged in the fan heater and then gone to get a coffee.

      When she got back there was an email sent to everyone not to plug heaters into IT sockets.

      Genuinely worried that we had some super spy capability/hidden cameras for detecting the power use…….she'd unplugged her heater immediately.

      Jane had then spent the next few days in a very cold office with her coat on because she was scared of a telling off by IT.

      I explained that she'd been a victim of an unfortunate coincidence and so long as the heater wasn't:

      a) Pointing at the computer

      and

      b) Plugged into an extension strip

      Then we didn't care what she did to heat her room in the offices.

  2. Giles C Silver badge

    Similar problem

    Had a power failure in a comms room, this was about 15 years ago the new distribution board lost power, and the ups apparently failed to start.

    The answer was found poking around in the wiring cabinet behind the server room - for some reason the ups power went out of the room through a breaker and back in - no I don’t know why.

    The problem was the dis board had a nice fat 100A breaker which was on, but someone fed the board from a 40A breaker. The power load got to probably about 42A and tripped the breaker, stopping all power to the dis board (and cutting off the ups).

    That weekend the site had a power down whilst the breakers were upgraded along with the cabling….

    The ups well it was (I think) a 20kva unit so it could have happily support 90A of load….

    1. Alan Brown Silver badge

      Re: Similar problem

      "...Someone fed the board from a 40A breaker. The power load got to probably about 42A and tripped the breaker..."

      <Non-bofh mode>

      Breakers (and fuses) trip more or less instantly at TWICE their rated current and hold forever at their rated current

      Between N rating and N*2 they take X time to trip which is related to the square of the power draw (ie: at 125% it's usually about 18-24 hours, 150% is ~8 hours and 175% is ~1-2 hours)

      </>

      It's worth knowing this stuff (even roughly) for when someone pulls overload stupidity games.

      A marginally overloaded breaker(*) won't trip instantly, but it will probably go eventually. They're primarily there to prevent the wiring catching fire, not to protect anything else. Surge tripping (magnetic tripping) is a bonus feature

      (*) Anything less than twice is "marginal"

      As for the overloaded building UPS: I recommend a chainsaw - applied gently to the user's ribcage so as not to make the brake kickback

      1. Martin an gof Silver badge

        Re: Similar problem

        figures

        I don't think that's quite right. It's a while since I did "the regs" and I don't have my copy of the tables to hand, but I did find this resource which gives the trip curves about half way down in a section headed 1. Type B Curve, and are more like the sort of thing I remember.

        From that graph, a type B MCB (almost all domestic and most office breakers are type B) will trip "instantaneously" (that is, magnetically) at some point between 300% and 500% overload. For a 200% load it should trip between 10 and 40 seconds (thermally) and for 150% between 40 and 400 seconds (just shy of 7 minutes), not "about 8 hours". 150% is around about the minimum overload where it's always guaranteed to trip, eventually. Down to about 125% it might or it might not, while below that it almost certainly won't.

        M.

        1. KBeee Silver badge
          Meh

          Semi Joke

          Fuses... MCB's... WIMPS! Solid connections only!

          "What doesn't kill you makes you stronger"

          (remembers when LV interconnected was all solid links - if you hit it you died, but nobody went off supply) fault blew clear. Bloody Health and Safety!!!!

          In rememberance of my mates who died

      2. Mark 85 Silver badge

        Re: Similar problem

        As for the overloaded building UPS: I recommend a chainsaw - applied gently to the user's ribcage so as not to make the brake kickback

        Just make sure it's a petrol powered one and not an electric chainsaw.

    2. Blackjack Silver badge

      Re: Similar problem

      Turn off the power

      Unplug EVERYTHING.

      Then turn on the powe back on and try plugging things one at a time to see if one of the things is what is causing the problem. Also don't keep things plugged in during this "testing". Eventually you wl find what is causing the problem and or you end damaging everything.

      Also do not this if the problems happened due to rain.

      1. Martin an gof Silver badge

        Re: Similar problem

        Unplug EVERYTHING

        In the process of which you are very likely to discover the culprit - "why is there a sandwich toaster and a coffee machine plugged into the protected circuit?" - so no further testing required. Other, probably than a bit of Portable Appliance Testing which will either kill the kit through "accidental" misapplication of the insulation resistance check or condemn the thing outright, resulting in chopped-off mains cables, so problem solved.

        Unless it's a really nice coffee machine, in which case it'll find itself relocated to a specific staff kitchen, nowhere near the protected circuits...

        There is a potential problem with the unplug-everything route though - some places are so "organically" installed that it may not be obvious where something gets its power or what appliance (or appliances) connect to a specific plug. Hard-wired appliances are one thing - they should be on isolation switches anyway - but sometimes a cable disappears behind a desk into a rat's nest of other cables with the actual socket (or appliance) three extension leads and half a room away.

        M.

        1. wjake
          Holmes

          Re: Similar problem

          Half a room away?

          Try another whole floor away!

          Renovating 3rd floor. Equipment stripped, room stripped, shutting off power...

          Computers go off on 2nd floor!

          Hilarity ensues.

          Took them days to trace it back upstairs!

          1. PC Paul

            Re: Similar problem

            We have several buildings on a small business park. The local power company substation had to be worked on but we got plenty of notice for a change so arranged a full outage of the one affected building, with enough time for an orderly shutdown. It all went really smoothly and as Friday evening drew to a close the whole building was off and the main breakers isolated to avoid the self-same "turn it off and on again and off again and on and off and on and off again and...".

            Then early on the Saturday morning they threw the power. Guess which building feed wasn't labelled correctly at the substation...

  3. Hazmoid

    Beancounter central would not be happy on Monday

    I think the phone system would be on bypass as well :)

    1. Korev Silver badge
      Flame

      Re: Beancounter central would not be happy on Monday

      Or connected to the UPS for when some "Bright spark" picks up the electrified phone on Monday...

  4. Hot Diggity

    The security system

    If they kill the security system, then the load everywhere will drop to zero.

    The new zero cost computing paradigm. No power = no cost.

    1. Mr Sceptical
      Flame

      Re: The security system

      As an installer, most systems will fail open by design - hence why we state they should not be the sole lock on perimeter doors.

      Only specific locks fail secure and they normally have a manual release on the secure side so you can get out when power is out.

      Presumably the beancounters office both fails secure and EMPs their phones to prevent them calling for help before starting a small fire...

      1. Anonymous Coward
        Anonymous Coward

        Re: The security system

        Ah yes the Door Access system argument where it was classified as a security system, when it got dumped on us (Because random people got given access to the IT Suite and we then failed a security audit) we already knew the mag locks would let go during a power outage. We opened the can of worms throwing it at the department who dumped management of the system on us (Revenge :muhahaha:), they found the buildings insurance did not clasify them as locks. This meant the building still had to be locked even though the manager who put it in said it didn't. Said manager was ok though, changed jobs just before it went in (As they do).

        1. Dr Dan Holdsworth
          WTF?

          Re: The security system

          Yes, the senior staff seem to have a thing whereby they get to the senior level then have a series of 3-year contracts until they retire. Basically this means that the top management layer ends up being mostly parasitic since they have never been in post long enough to know what they are talking about.

      2. Marcelo Rodrigues
        Devil

        Re: The security system

        "Only specific locks fail secure and they normally have a manual release on the secure side so you can get out when power is out."

        Surely, in the beancounter's case, the safe side would be the OUTSIDE of the room?

      3. Anonymous Coward
        Anonymous Coward

        Re: The security system

        In the building then owned by a cable service provider north of Winchester, the IT wing mag locked doors would all lock firmly during fire alarms. This was raised with the manglement numerous times, but never fixed in the time I worked there.

        1. Will Godfrey Silver badge
          Facepalm

          Re: The security system

          There's no way I'd even enter an area like that (assuming of course I was able to discover that was the case)

          1. Anonymous Coward
            Anonymous Coward

            Re: The security system

            Oh, as long as it has very expensive smashable windows and there are enough things inside to smasjh with I'd be OK. Surely it's not my fault that I panic quickly when I can't get out?

            :)

        2. Doctor Syntax Silver badge

          Re: The security system

          A tip to the local fire service that an inspection might be a good idea should have fixed that.

        3. Anonymous Coward
          Anonymous Coward

          Re: The security system

          Had the opposite in a location (floor in multi-tenant building) that was meant to comply with, ahem, certain security requirements. Every Saturday morning at around 2am the building fire alarm circuit into the access control system activated (going O/C) releasing all - ALL - the doors for a minute or so. Including the small-scale DC.

          In my time there, the building management never tracked down why that was happening.

      4. H in The Hague Silver badge
        Pint

        Re: The security system

        "Only specific locks fail secure ..."

        That reminds me of an anecdote a long-retired friend told me, probably happened in the 1960s:

        "When I started with the company the oil industry wasn't doing very well so they built 'austerity refineries' on the cheap. One day there was a fire at my refinery. After a short time the site power went down, which caused the main gates to close - just as the local fire brigade were approaching."

        Almost that time of day -->

        Have a good weekend.

        1. Anonymous Coward
          Anonymous Coward

          Re: The security system

          "main gates to close - just as the local fire brigade were approaching."

          That's when the chief says something over the radio like "truck 41, I need a crew at the gate. Bring tools" and a fire crew gets to do some impromptu gate modifications. Probably made their week.

        2. Alligator

          Re: The security system

          It may be that the refinery (or chemicals plant) did this by design. BP Baglan Bay and Llandarcy certainly had their own fire brigades on site and routinely locked the gate in case of a fire to stop the idiots from the fire brigade doing something stupid again. This was certainly still the case in Baglan in the early '90s. The fire brigade had got into the plant during an incident in the late '70s (IIRC) which the on site brigade were carefully dealing with, gone hung-ho having not understood why the fire was being dealt with carefully and slowly, and caused an avoidable full shut down of the plant which cost over £1m per hour and lasted a couple of days.

      5. Glen F

        Re: The security system

        I can recall working at a data and call center for a major ISP and long distance telco. No UPS except maybe on the servers, but maglocks throughout the building and exterior doors. If there was a camera system, it wasn't on UPS either. Summers meant frequent thunderstorms and transformer outages, amplified by drivers with water soluble intelligence that would often collide with utility poles. We came to work one morning to find the server room stripped of much of its gear. Some enterprising folks took advantage of the power outage the night before to load up a truck while all the doors were unlocked.

        1. WolfFan Silver badge

          Re: The security system

          Ah. Almost certainly an inside job. That’s the way that half a million dollars worth of assorted equipment, including a company car allocated to replace the MD’s car, vanished from Stores one night at a place where I once worked.

  5. Jabba

    Designed to fail

    An UPS is designed to work perfectly until you actually need it...

    1. Captain Scarlet Silver badge
      Coat

      Re: Designed to fail

      hmm I've only had them fail during self tests (Including one which tripped and then melted its power input cable when I reset the trip), I must be lucky.

    2. Peter2 Silver badge

      Re: Designed to fail

      Excluding the time that somebody spilled water over one at a remote site and phoned in asking if a UPS should create an artificial lightening bolt (and is it likely to be damaged as a result) the only times I have had a problem with UPS's is when people have attached too much to them.

      Assuming that you only keep them for 5 years a time and then replace, as opposed to "run until dead".

      1. Doctor Syntax Silver badge

        Re: Designed to fail

        We had the building UPS taken out by a lightning bolt. The rest of the building was OK & it took ages to get the UPS fixed during which time we ranthe servers on the uninterrupted interruptible power supply.

      2. Anonymous Coward
        Anonymous Coward

        Re: Designed to fail

        I wish... our 800kW unit is pushing 18 years old whilst the 1.2MW one is only slightly younger

        The problems with both of them are "overload" closely followed by "Ancient SCADA interfaces that not even the makers understand anymore"

        1. Peter2 Silver badge

          Re: Designed to fail

          Those would have a reasonable excuse for not having a more frequent replacement cycle. If I had commercial units built in then i'd happily maintain those rather than replacing them every 5 years.

          With the lower end UPS's most people have though given that the cost of the batteries is 90% of buying an entire new unit you might as well just toss and replace the whole lot.

          1. MachDiamond Silver badge

            Re: Designed to fail

            "With the lower end UPS's most people have though given that the cost of the batteries is 90% of buying an entire new unit you might as well just toss and replace the whole lot."

            The other issue is at the point where the batteries have that dry rattle when shaken, something else in the UPS that doesn't like not having functioning battery installed lets out its Magic Smoke. For Trade Secret reasons, all of the markings on the semiconductors has been buffed off and the manufacturer doesn't supply schematics and troubleshooting docs. Never buy a UPS where you can't get a full set of service information (pretty much rules out APC).

      3. Antron Argaiv Silver badge
        Thumb Up

        Re: Designed to fail

        It's usually only the batteries that die after 5 years.

        I have three UPSs at home, all discards from our IT department after upgrades, and they all have been through several sets of batteries, meaning they are from 12 - 15 years old. Still working fine as far as I can tell. The newer models have poor reviews, so I'll keep replacing batteries in these until the electronics go bad...

      4. trindflo Bronze badge
        Devil

        Strange smell

        Had someone comment that my office smelled strange, and initially just thought it was just that my semi-annual shirt change was overdue. In a couple of weeks I commented that I thought I was smelling sulfur. After listening to the predictable jokes "well of course your room", I went on a hunt and found one of the batteries had begun to leak in the UPS. So I can site at least one instance where a UPS went out without being murdered by the beancounter's office. It was a clever piece of APC gear that would regularly run tests to make sure all was well, but apparently no corrosive liquids in search of victims detector.

      5. PC Paul

        Re: Designed to fail

        There's another option?

    3. Adrian 4 Silver badge

      Re: Designed to fail

      Fail-safe systems fail by failing to fail safe.

  6. Terje

    I was almost certain that the extra load would have been the the mining farm.

    1. Mr Sceptical
      Thumb Up

      Simon's no dummy - the farm is being fed from the two adjacent buildings' mains supply and has the bosses new electric car hooked up as the UPS ;-)

  7. Michael H.F. Wilkinson
    Coffee/keyboard

    I love the smell of burning beancounter kit in the morning

    Absolutely brilliant episode!!

    Batenburg LeydenJar conjecture

    ***DUMMY MODE ON***

    Hadn't seen that mode engaged in a while. Love it

    1. TRT Silver badge

      Re: I love the smell of burning beancounter kit in the morning

      The Batenburg LeydenJar conjecture... genius. Now if we just reverse the polarity of the neutron flow and get it working again, we can have a nice cup of tea and a slice of cake until some ham-fisted bun vendor breaks the spatio-temporal hyperlink again.

    2. Richard IV

      Re: I love the smell of burning beancounter kit in the morning

      The Batenburg LeydenJar conjecture is a keeper...

      Presumably it's an effect where 2 UPS's of inverted polarity are connected in parallel and partially fail leaking a pale emulsified dielectric material.

      Not to be confused with the Viennetta LeydenJar conjecture with many parallel UPS's of matching polarities and the leak is a pale liquid dielectric due to coolant failure.

      1. The Oncoming Scorn Silver badge
        Joke

        Re: I love the smell of burning beancounter kit in the morning

        I'm Sorry I Haven't a Clue as I'm hungover & have apparently have stumbled into a game of Mornington Crescent!

        In the hope that that is indeed what I find, I call Cockfosters!

        1. Korev Silver badge
          Stop

          Re: I love the smell of burning beancounter kit in the morning

          Sorry, we're currently playing Queen Victoria rules, so only a quick half-shuffle to Baker Street is acceptable

          1. The Oncoming Scorn Silver badge
            Go

            Re: I love the smell of burning beancounter kit in the morning

            My my what a school boy error to make, in my defence as stated I'm still hung over.

          2. Jesthar

            Re: I love the smell of burning beancounter kit in the morning

            In that case, I call crabwise to the Strand...

        2. Martin-73 Silver badge
          Pint

          Re: I love the smell of burning beancounter kit in the morning

          Careful you don't yell Cockfosters within the first 2 mins of a YouTube video, all hail the algorithm!

          Icon: Weekend, cheers!

    3. Just An Engineer

      Re: I love the smell of burning beancounter kit in the morning

      I love the smell of burning beancounter in the morning

      TFIFY

    4. steelpillow Silver badge

      Re: I love the smell of burning beancounter kit in the morning

      What about the smell of burning beancounter in the morning? Maybe next episode, when the culprit is identified and their kit hooked direct to the replacement transformer...

      1. herman Silver badge
        Devil

        Re: I love the smell of burning beancounter kit in the morning

        Their kit hooked up… - ouch! You are very cruel minded indeed.

  8. Mr Sceptical
    Facepalm

    You'd have thought that pros would do it better...

    Was involved in a migration of on prem Exchange servers to a C&W datacentre in the mid-2000s.

    The blurb on the DC was it's got two mains supplies from separate cites, UPS up the wazoo, state of the art control room, etc.

    Cue a power cut the weekend after the bulk of the migration - there was an actual grid outage and both mains feeds died. The dual UPS architecture immediately failed due to a cascading overload. Unfortunately, due to the design of the control room - all their kit was on the same UPSs so they were dark too.

    Come Monday morning, lots of screaming to the C&W team as to why the hell they didn't bother alerting anyone that all our kit was down and why several thousand staff now had no email...

    1. Andy A Bronze badge
      FAIL

      Re: You'd have thought that pros would do it better...

      We had a site with two sets of pylons, running from opposite directions, and a "ring main" feeding the other site substations.

      Once a year, during holiday fortnight, the sparks moved the Big Breakers over to use the other pylons. Check the other substations were still live, then after a couple of hours move the breakers back to the original (cheaper) supply.

      We took the opportunity to test the server room UPS. Servers all powered down, and just core switches left running. I estimated 3 hours runtime according to the UPS specs.

      It lasted 15 minutes. Not even enough time to warm up the second set of pylons.

      1. Potemkine! Silver badge

        Re: You'd have thought that pros would do it better...

        Testing UPS duration for real is always enlightening. Because in theory, practice works like theory. In practice, it doesn't.

    2. Admiral Grace Hopper

      Re: You'd have thought that pros would do it better...

      I read that as "the Country & Western team", in which case they may have been too busy having a hoe down to get round to alerting anyone.

      1. Dave Pickles

        Re: You'd have thought that pros would do it better...

        Clueless and Witless.

        1. WolfFan Silver badge

          Re: You'd have thought that pros would do it better...

          Completely Worthless.

      2. WonkoTheSane
        Trollface

        Re: You'd have thought that pros would do it better...

        Bunch of cowboys!

    3. Doctor Syntax Silver badge

      Re: You'd have thought that pros would do it better...

      So the UPS really was up the wazoo.

  9. Anonymous Coward
    Anonymous Coward

    Doh!! moment

    Big data center, two massive UPS units each capable of running the entire center by themselves.

    During a failover test, the utility company was disconnected, and whilst testing each UPS individually, the active one was switched off before the inactive one was enabled.

    Everything lost power, whole kit and caboodle. A/C silent, lights out, several hundred switches and servers all off. Big multinational company too. Millions of dollars in revenue lost.

    1. TRT Silver badge

      Re: Doh!! moment

      Ouch! Of course, the testing regime should cover all possible scenarios.

      1. A Nother Handle

        Re: Doh!! moment

        Yes but... a scenario you expect to fail (remove ALL power supplies) perhaps should not be tested in production.

        1. TRT Silver badge

          Re: Doh!! moment

          Test to recovered state is perfectly valid. It's obviously a scenario that should have covered by, say, failover or distributed load with a second location DC.

        2. Pascal Monett Silver badge

          That is a problem with the testing procedures, not with the equipment.

          The one who wrote the procedures should be chastised. The one who approved the procedures should be fired. The manager who signed the procedures without looking them over and checking should be fired as well.

  10. Spanners Silver badge
    Pirate

    A possible solution

    Reroute all those "vital" beancounter" device supplies to a special mains supply. Just don't tell them that it is no longer a sine wave, a steady 50Hz or even between 220-240v anymore.

    1. Is It Me

      Re: A possible solution

      No need to do that, powercycling 200 times has already done the job.

    2. Mike 137 Silver badge

      Re: A possible solution

      Another possible solution. MK used to supply (maybe still do) 13A wall sockets and plugs with a round earth pin and you could get straps that clamped the plugs in place.. If the UPS circuits are terminated with these sockets in the offices, only kit with compatible plugs can be connected. And there's another benefit. I once worked in a small IT co. where the main '24/7' servers were plugged into a 13A wall strip. Every so often a cleaner would unplug the server in order to plug in their vacuum cleaner after hours. Such a round earth pin socket would have eliminated this problem.

      1. Dazed and Confused

        Re: A possible solution

        I occasionally used to visit a site in an old fashioned manufacturing company. They had a policy of using very obviously non-standard plugs for any important kits to stop cleaners and maintenance staff unplugging kit. If it don't look like a 13amp plug they ignore it.

        1. The Oncoming Scorn Silver badge
          Headmaster

          Re: A possible solution

          Went to site that had lost networking in one external school building "Down all day, kids can't work style of screaming" in the call notes.

          Get to site, sure enough there's a three pin plug looking sad & lonely on the ground, gazing up at it's rightful home (Much like Utred does at Bebbanburg) in this outlying building wondering when someone will make the connection.

          Left site with the advisory to the rather embarrassed teaching staff\school contact:

          "You might want to put a sign on that plug so that the cleaners don't unplug it again!".

          1. The Oncoming Scorn Silver badge
            Childcatcher

            Re: A possible solution

            Actually my powers of recall missed the edit window, the plug wasn't on the floor, the cleaners had thoughtfully & neatly left it draped over the top of the wall mounted power socket when they were done.

          2. Andy A Bronze badge
            Unhappy

            Re: A possible solution

            A place I worked in the early 80s had the entire comms rack running from a single 13A socket - the building had a huge alternator and generator waiting to kick in should the mains fail.

            I was squatting by the comms rack looking for a problem (oh all right - it was a hot day and I wanted to be in the cool) when the whole rack went dark.

            Brown trousers time - there are going to be a hundred angry calls from users.

            I look round to see that the cleaner has unplugged it all from the mains to plug in the vacuum.

            Next to the socket they had purloined, actually in the same faceplate, was another, fully functional, and EMPTY socket.

            We moved the disk racking two inches to the left so as to ward off a repeat.

      2. Korev Silver badge
        Terminator

        Re: A possible solution

        Every so often a cleaner would unplug the server in order to plug in their vacuum cleaner after hours. Such a round earth pin socket would have eliminated this problem.

        Although, the cleaner probably would only see the special socket once they'd unplugged the server...

        1. MachDiamond Silver badge

          Re: A possible solution

          "Every so often a cleaner would unplug the server in order to plug in their vacuum cleaner after hours. Such a round earth pin socket would have eliminated this problem."

          The cause of the issue is hooking up an important bit of kit with a standard 3 pin plug. It needs to be hardwired. I can't think of any situations in the past where unplugging something that important was ever needed.

      3. Doctor Syntax Silver badge

        Re: A possible solution

        "Such a round earth pin socket would have eliminated this problem."

        Not really. It isn't apparent that the vacuum cleaner won't fit until the server has been unplugged.

        1. Terry 6 Silver badge

          Re: A possible solution

          I did think this, but it's not something I'd come across myself. However, "act first think later" is something I've come across - multiple times.

      4. Martin-73 Silver badge

        Re: A possible solution

        These days it's a T shaped earth pin, we have installed some for cleaners' supplies in blocks of flats, to stop tenants stealing power.... doesn't work because they have those special MK shutters that open with just the L&N pins, and some of the tenants got wise to it.

      5. JimC

        Re: Vacuum Cleaners

        There are two things to do.

        The first is to label all the sockets you wish used for vacuum cleaners with a distinctive* label saying vacuum cleaners only.

        The second is a regular patrol to remove every device plugged into a vacuum cleaner socket.

        *Needs to be a distinctive label since in these times not every cleaner will be able to read english**.

        ** I worked in the same suburban office for 25 years. A sociological study could have been made of the successive ethnic groups being exploited as cheap labour for office cleaning.

        1. Pascal Monett Silver badge

          There is a third thing to do : make sure that the cleaning personnel understands the Golden Rule : to not unplug anything in order to plug in the vacuum cleaner.

          The first time the cleaner makes a mistake, report him to his superiors and forbid him access to company grounds. The second time the cleaner makes the mistake, sue the cleaning company and tear up all contracts with them.

          1. JimC

            > tear up all contracts

            I'm not sure that sort of aggressive combative approach actually works in the real world. We are talking about minimum wage workers with no training and probably minimal command of english. If you get one sacked the next one is just the same, and as for tearing up contracts, well the amount of trouble that causes is out of all proportion, even if you won't be overruled by the bean counters. And in any case the next cleaning company will be employing much the same sort of people, because that's all you are going to get for the money you are paying for the contract.

            No, the right approach in this sort of circumstance is to make it as difficult as possible to get things wrong. It starts with building design too. I particularly admired the office layout where the clean power sockets for the office equipment were at floor level and feeding power strips on desk islands, and the vacuum cleaner sockets were on ceiling pillars at waist height and in open air. The person responsible for that office layout knew what they were doing. I just had to patrol every few weeks to make sure no-one was using the vacuum cleaner sockets as part of a general safety inspection.

            In my opinion we don't think nearly enough about in the IT industry about making it easier for people to get things right instead of wrong. And its just as valid for intelligent people who see the IT as an obstruction to doing their job as it is for the zero skilled end of the industry. Instead there's a tendency to revert to a script driven process where every trace of humanity is removed from the poor peons at the sharp end.

            Oops, sorry, rant over...

    3. iron Silver badge

      Re: A possible solution

      The chance of their mains supply being 50 Hz and 220 - 240V in the first place is surprisingly slim.

  11. Version 1.0 Silver badge

    Working from Home

    ... two small fridges, the entire kitchenette – including a big fridge, microwave, dishwasher, and an under-sink hot and chilled water unit... so the beancounters are working from home in the office. I'm seeing this occasionally these days, so many working areas now have a toaster, a microwave, and empty beer cans in the trash buckets - it's the working from home environment.

  12. Anonymous Coward
  13. Johan-Kristian Wold 1

    About 25 years ago, I was involved with changing over to new facilities for a small company.

    Some custom systems were involved, with a bunch of different servers, so there was a number of UPS systems in the server room. Everything fed from a distribution breaker panel inside the server room.

    A power failure outside normal business hours showed that a critical point was overlooked by the leccys.

    The combined start current when the mains returned, and all the UPS systems starting to recharge the batteries was enough to trigger the quick-acting fuse (in the main breaker panel) feeding the distribution panel in the server room. So the UPSes ran out of battery and shut everything down. A change to a slo-blo fuse fixed the problem.

    1. knottedhandkerchief

      I remember discussing with a fellow student, in deliberate earshot of an over-amiable young female student, the pros and cons of slow-blows vs quick-blows.

      That passed for student humour in the 70s, maybe not so PC these days.

      1. Martin-73 Silver badge

        If the female student is of the morally upstanding variety, the correct term is 'time delay' :)

        1. M.V. Lipvig Silver badge
          Joke

          It's much more fun to say fast/slow blow, wait for them to become morally outraged, then say"We're talking about electrical fuses! Only one of us here thinks it has to do with carnal activities and that's you, you pervert!"

      2. Anonymous Coward
        Anonymous Coward

        I remember my Dad having a similar conversation with business associates in a bar, where the subject was appropriate sizes and shapes of breasts.

        It's very important for portion control in chicken sandwiches.

    2. Antron Argaiv Silver badge
      Facepalm

      The time when...

      ...our (married) young and very attractive female intern confided in me that she could never remember which was the male and which was the female connector.

      Cue a few seconds of careful thinking, then a very carefully phrased reply on the order of "the male is the pin and the female is the socket", while hoping upon hope that this would prove both a sufficient answer, and insufficient to attract the notice of HR.

      She thanked me and continued working, to my infinite relief.

      ...but I did get a story to tell here...

  14. ecarlseen

    I have literally seen some of this happen.

    Floor heaters plugged into the red "backup power" plugs? Been there, done that.

    1. bombastic bob Silver badge
      Unhappy

      Re: I have literally seen some of this happen.

      at one company I was at, each desk got its own small UPS which is not a bad idea... until the cleaning service plugged vacuum cleaners directly into them.

      (NOT a "power strip" - and if it were, and it had a line filter, the results could be JUST as BAD)

      1. The Oncoming Scorn Silver badge
        Coat

        Re: I have literally seen some of this happen.

        Had that at one place & the complaints of the UPS's beeping when yet another of these aged units batteries finally died under a desk.

        Slaughterhouse floors, even in the offices are never entirely clean (They would benefit from a damn good scrub), as they are covered in a slightly sticky film of what I shall politely describe as "grease" as its "walked" from the main passageways.

      2. Adrian 4 Silver badge

        Re: I have literally seen some of this happen.

        Don't you have those incredibly annoying desk-bound socket strips where each outlet is independently fused with a 3A fuse ? All of which, obviously, have blown ?

        1. Andy A Bronze badge

          Re: I have literally seen some of this happen.

          Once had to install a whole batch of those for "refurbished area".

          I explained that should a fuse blow because someone thought a heater was a good idea, they would have to negotiate for themselves with the site sparks for a replacement.

          The behaviour common at the place they moved from ceased once they knew I kept my word (and had no spares of my own).

  15. norsni

    Remembering ALL the vital things to go on ups.

    I got a nice big ups at home, external batteries for about 5 hrs runtime at full load, powering my proliant server & network kit. Our utility announced a maintenance outage. No problem, been there, done that, i power down the unneccessary stuff before heading to work that day, and monitor the situation remotely. About an hour into the outage the proliant goes down. ILO tells me its having a heatstroke. Whoopsie, i forgot the cooling air fan; the server is a bit noisy, so after complaints from the missus it sits in a silencing cabinet, therefore requiring external forced air to keep it running. And i forgot to put the fan on the ups circuit. Thank you ILO, after the power came back i could power it on remotely. And rewired the fan after i got home. Lesson learnt.

    1. MachDiamond Silver badge

      Re: Remembering ALL the vital things to go on ups.

      "And rewired the fan after i got home. Lesson learnt."

      This is where a simple hand drawn circuit diagram can be of great use rather than winging it. I had a great mentor years ago that taught me the value of writing things down and drawing little pictures. Having to do cost breakdowns and BOM's has helped too. I have a ton of projects on spreadsheets that I can "save as" and repurpose to not have to create a new template each time. I also have files on projects that were done by hand on <gasp> paper.

  16. Eric Kimminau TREG

    During a recent DR test

    It was discovered that Facilities routed power to all 4 datacenter HVAC units through the UPS on a secondary power connection to each unit. They didnt tell us this until we had a power maintenance planned and shortly after we switched over to UPS, we saw 4 very large spikes in power draw as they manually switched over each HVAC unit. The 3rd unit switch occurred just as the generator kicked on, which was not rated to handle all 4 HVAC units.

    Key blown breaker.

    1. Dwarf Silver badge

      Re: During a recent DR test

      @Eric

      I've seen an alternative approach, where the CRAC's are just fed off of grid power / generator power. The logic being that the thermal mass of the room is sufficient to cover the point where the mains fails and all hell breaks loose and the generators kick in.

      The approach works fine, until the generators don't kick in and its nice and toasty and hard discs start dropping like flies.

      1. Roj Blake

        Re: During a recent DR test

        If the generators don't kick in you're stuffed anyway, as you most likely have no more than 30 mins before the batteries on the UPS fail.

  17. You aint sin me, roit
    Pirate

    BOFH with Hans Gruber aspirations?

    We all know why he wants power to the building cut...

    1. Martin-73 Silver badge

      Re: BOFH with Hans Gruber aspirations?

      You asked for miracles? I bring you the B. O. F. H!

  18. Anonymous South African Coward Silver badge

    Hah.

    Had installations wire up a hefty YPS to a 10A breaker.

    All was ok until I decided to test the UPS. Flipped the CB to off, and let the UPS run for a while.

    All good. So I flupped the CB back on, which lasted 1 minute before tripping.

    Turns out the 10A CB was borderline, and when the extra load for charging the UPS batteries was added, it could not handle the load.

    Site Tech managed to move the UPS feed to a more chunkier CB, and all was well.

    1. Adrian 4 Silver badge

      YPS might have been a typo, but I read it as YTS.

      1. Anonymous Coward
        Anonymous Coward

        The average YTS was too thick to draw anything like 10A, even with a direct connection.

      2. SusiW

        Ahh. The old Youf Training Scheme.

        I remember our department having a seemingly non-ending stream of PFYs foisted upon us by gubmint stooges.

        I also remember them leaving almost as quickly when they discovered they actually had to "do stuff."

        Good times.

  19. xyz123

    Once had a company with a gigantic UPS. CEO's office was wired direct to UPS power because he was running a from-the-office website "of an adult nature" that he hosted himself.

    1. Phil O'Sophical Silver badge

      Uninterruptible Porn Supply?

    2. really confused

      Tractors ?

  20. Richard Jones 1

    Ah The UPS Overload

    Many years ago, the main frame and a few terminals were on our no break UPS. I needed more data collection terminals for in-process data collection and other statical functions. Each time I added more collection points, (PCs) I told the power engineer and never had a response. Finally, I saw him on the stairs and questioned if all was well with his UPS and 'my loads'. A light bulb went off in his face, my activities were close to drawing 50% of the company's power draw. His comment, I wondered where all our power was going, bear in mind I was sharing the power with the mainframe running the company and several other 'useful' functions.

  21. MachDiamond Silver badge

    Desk heater = death

    The average person equates power needs based on the size of a device. While they might have lived long enough to realize that a kettle draws a lot of power, they don't think a dinky little under desk heater is that big of a deal. When I had a manufacturing company employees were told that bringing a heater for their work area was not allowed. I'd either bump the HVAC up a little or advise them to dress warmer during winter months. I didn't have a fancy dress code so layered creativity wasn't a problem. I put on my thermees when I knew I'd be out in the test lab in winter time. There was no way to efficiently heat that space.

    Another issues is people with business degrees sometimes have a hard time with math. Yes, plugging the humidor into the UPS might preserve a couple of grand of imported cigars during an extended outage (domestic cigars are never good enough), but tripping the breaker on the UPS and shutting down the company could be tens of thousands an hour in profits going up in smoke. I had a girlfriend many years ago that worked at an auto manufacturing plant. At the time, she told me that a production line going down was $125,000/hour in lost profits. The discussion was why it was useful to hire a Lear jet to bring in parts if they were running out. The express cargo shipment might be $20,000, but that was less expensive than the $125K/hr and still being required to pay the staff that was sent home.

    Unless it's a hospital or some other critical enterprise that must stay lit in a power outage, the point of a UPS is to allow a graceful shutdown of computers and the ability to have people leave a building with some lights on. If the outage is expected to be brief, it allows the business to continue until the mains come back on. The purpose of the backup needs to be well defined and if the needs change, the UPS has to change as well. If the higher-ups want the UPS to run the whole building without interruption or just hook in non-essential crap to a temporary system, they need to see what the cost of that will be. Maybe the beancounters office refrigerator can do without back up if they are shown that adding it means a couple hundred thousand in new gear and an expansion of the electrical room.

    1. Martin-73 Silver badge

      Re: Desk heater = death

      Yes, as a sparky, customers are very much like this... One guy the other day, melted a socket feeding his outside lighting. He had replaced one of the light fittings (decorative dangly things with warm white 9w LEDs) with a similar sized, 3kw heat and light unit....

  22. Blackjack Silver badge

    All this to hide that th hidden gamer rig that was taking like 56% and to justify getting the money to upgrade it since is not windows 11 compatible.

  23. Ordinary Donkey

    Wasn't part of this a classic BOFH plot?

    I'm sure back in the days when the series was passed around on public ftp services one of the stories had someone given a sequence of directions that included turning it off and back on at the case exactly twenty times, so there was nothing for him to support any more.

    Obviously technology has marched on and twenty times is nowhere near enough any more.

  24. Anonymous Coward
    Anonymous Coward

    F'UPS

    One of our team is a former UPS engineer, he installed two of the 4 systems in our building, he has a range of stories from stupid managers to corporate madness from his past work, but the worst UPS stories we know are from much closer to home. From illicit desk-top units being hidden under piles of paper ideal for starting a fire when the batteries finally fail from years of charging, to a 300KVA 3 phase unit delivered from Germany having been flown first from Japan to the USA then on to Germany before being trucked to the UK wheeled into position, plugged in and simply switched on by the users, without removing covers and more importantly having a trained installer check none of the battery bolts had come loose in transit, that one caused a fire estimated to have cost close to £200,000,000:00... Anon for obvious reasons.

  25. Unicornpiss
    Alert

    Similar but not entirely different..

    At a facility about 800 miles away, that we support remotely, the big UPS died. A manager with access decided it would be best to shut down the facility before a major storm and when the UPS was powered up again... well, it didn't. So the quick-thinking office manager gathered every power strip she could find and plugged all equipment in their small server room into 4 power strips, going to all 4 wall outlets in the room. And it held until I made a trip down there a month later.

    One thing she didn't know is that 3 of the 4 outlets in the room were daisy chained on once circuit. When I found the breaker, it was noticeably warm, but holding. She had somehow hit the magic combo by sheer luck--if she had put all the switchgear on the same circuit as the servers, the POE providing current to all the phones and cameras would have surely popped that breaker. As it was, the power strip was also comfortably warm and its internal breaker was probably barely hanging on by its electronic fingernails.

    As it was going to take several months at best for the PO to be cut, the purchase made, the new UPS sourced, and then installed by a local electrician, I purchased 2 fairly beefy consumer-grade off the shelf units, suitable for a well-endowed home office, and balanced the load as best I could between them. The transformer box for the old dead UPS was still good so I was able to connect these to it and not use the wall outlets. Some months later they didn't have the new UPS yet and were experiencing some power problems again.

    Diagnosing the problem over the phone with the office manager and the photos she sent, I finally discovered that some exec visiting from corporate had complained about how messy the server room was so after hours some of the staff decided to neaten up all the wiring. they had taken my interim solution and had one UPS plugged into the other now, with most of the load on it, and some back plugged into a wall outlet again "because it was more tidy."

  26. Danny 5
    Happy

    Dummy mode on

    My eyes lit up when I read that..... Been a while since it was invoked!

  27. Curtis

    Have I been doing it wrong?

    Meanwhile, gnerds like me are asking IT which are the UPS sockets and which aren't for plugging in our "extra" gear. Like my personal laptop, docking station, test gear, soldering station etc. (try explaining to manglerment types why "phone tech support" has a soldering/reflow station at his cubical...)

  28. Richard Pennington 1
    Flame

    Years ago ...

    ... perhaps some time round about the mid-80s ... I read a report about a fire at Thomas Cook in Peterborough. After the local fire service had serviced the fire, the damage report noted that they had got off remarkably lightly.

    The UPS was not so Uninterruptible any more. It was the only casualty.

  29. Anonymous Coward
    Anonymous Coward

    butter fingers

    The most fun with power outages I can recall was when a contractor working on our power systems dropped his spanner/wrench/antenna across two phases of a three phase power supply. I was mighty impressed with our UPS during the bang and light flickering that ensued. Didn't lose any network or hosts!

  30. Glenn Amspaugh
    FAIL

    Our college server room (1990s, a few racks of Sun Sparc 5s and 10s, Novell file server) had bright orange plugs for UPS, beige plugs for mains.

    New guy decided to rearrange server room for some unknown reason and plugged the chain printer (pale green and white fanfold printer) into the UPS.

    Save us from the eager clueless.

  31. Celeste Reinard

    It's 01:46 ... I love your nightmares, Simon...

    ... I do...

    ...

    Let’s elope...

  32. choleric
    Happy

    Can't believe I missed this episode when it was published.

    This PFY is a keeper, and therefore one to watch carefully!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022