Company that made power systems for servers didn’t know why its own machines ran out of juice Each new Monday ushers in a week during which you might shine or flatline. The Register celebrates the times you end up doing the latter with a new instalment of Who, Me? It's the column in which you admit to making mistakes and execute cunning escapes. This week, meet a reader we'll Regomize as "Cole" who told us he once …
It's wild to me that the likely cause of outages was identified correctly, but no one bothered to find out when and how they occured!
Even if it was an issue with the grid, i'd personally have a chat with them about our power dropping for extended periods every weekend.
Must be nice to work at a company where issues are solved by throwing money at it.
Wouldn't you plug something in on a non UPS protected outlet that's logging to a file every minute, so you could figure out when it goes down? Interface with the UPS to determine when it sees its external power going offline?
Had they done even the barest minimum of diagnostics, they would have determined when the power is cut off and it would have quickly become obvious that it is happening when the "last person to leave" has left for the night.
They spent who knows how much on two unnecessary UPS upgrades because no one had the brains or curiosity to investigate anything. Why can't I hammer that nail in? Guess I'll use a bigger hammer, rather than noticing I'm trying to hammer it into stainless steel plate!
You thought Americans would be less likely than yourself to understand a quotation from a book written by an American?
On the evidence here and other places, most Americans don't understand the quotation.
I note that in the original, the reference is to Topsie's ignorance of God and religion. The conventional understanding, and original quotation of the phrase, was to indicate Topsie's lack of parents: she assumed she "just grow'd" by spontaneous generation.
The way it is now most often used by Americans is to indicate rapid or explosive growth of a company or social enterprise.
Absolutely!
As we expanded we took over the adjacent buildings on either side. Not wanting to incur three utility bills, also needing a substantial three-phase power supply, we organised a new single supply and fed all three units from the same source. The interconnections were such that breakers were in one unit controlling the supply in another. The 'labelling' was poor and I spent considerable time sketching the scheme as it was, to ease diagnosis by those that needed to know.
Of course, the electricians who came in to add a minor circuit ignored all this and because they couldn't get into one cabinet, which was locked, padlocked and had a sign stating 'Isolate in Unit 2 before opening this door', I found them prising the door off its hinges in order to gain access to what they assumed was an 'unused junction box'. They had even taken the handle off the local door-mounted isolator as it was preventing the door from release. Instead of looking at the 'scheme' they had 'followed the cables' and made a mistake where the ducts entered the building.
They thought a pair of four-core, 120 mm2, three-phase, armoured cables would be the very thing to power an outside light.
Currently in 2 adjacent units which we mainly use for warehousing. I'm not sure what the previous occupants did, but when we had the electrical circuit inspection done when we moved in, we discovered the building has 2 x 100A 415V 3-phase supplies which then split out to 9 different circuit breaker boards around the building and we had a total of 108 circuits to be inspected. I suspect the largest part of our electric usage is for lighting in the warehouse and offices, even with LED lights installed everythwere
Unfortunately the 3 things we have which needed 3 phase power couldn't be sited near where there was existing power so we had to have more 3 phase distribution cabling installed for our electric forklift charger, car lift and EV chargers. When the car lift and forklift charger were installed (2 separate installations) both times the electricians managed to take out all of the power to the offices on the opposite side of the building because a couple of the distibution panels are daisy chained.
Fuses? Luxury!
Joking aside, I have seen some, err, 'entertaining' ways to hook up power when I was in rural Thailand, especially where they have welding shops. They also use US style plugs and cables for 240V circuits. Add to that the heat and humidity which makes RCDs unreliable but which deteriorates insulation at quite a clip and yes, hearing that quite a few people got electrocuted there didn't come as a surprise :(.
This has nothing to do with nightmare electrical systems.
They had the system logs, thanks to various values of UPS. As such, they knew exactly when the power went down. Extrapolating the why from the when has been child's play in every RealLife scenario I've ever been a part of.
It's not clear whether they did have logs. As I read the article, it's not clear if :
• The servers shut down (as in, triggered by the UPS, did a clean shutdown) in which case that would mean the UPS properly configured and they probably have logs
• The servers just powered off without a shutdown, meaning the UPS not set up well and probably no logs
My own favourite was a client at a previous employer who had their server room in the corner of an otherwise almost unused floor of their new building. One day we got a call from their manager, screaming about the rubbish UPS we'd sold them, how it was a pile of s**t, and really laying into the helldesk guy unlucky enough to take the call. So we logged into the server that had the UPS software on it, to find it had gone onto bypass due to overload. "Had anyone plugged anything in in the server room ?" they were asked ...
After a short pause long enough for someone to run upstairs and check, the UPS returned to normal, and the manager had to admit that someone had been working upstairs (in the unused floor space) and was a bit cold. So they plugged a fan heater into the nearest socket they could find - in the server room, and VERY clearly labelled (I did it) "UPS Maintained - IT Equipment Only". No apology for their rant though ...
UPS would die on Monday morning then each weekend slowly regress towards Saturday night or Sunday morning. Doesn't look like any monitoring was in place, though it does look like backups were in place.
Did the UPS service team inform the IT manager?
Unlikely?
In military housing, there was a regular white light switch in my basement. I was never able to figure out what it was for, until one night I turned off all the other lights and flipped the switch. The street light on the opposite side of the street went out. I only saw it as it as it was directly in my eye line out the basement window.
Interface with the UPS to determine when it sees its external power going offline?
Most UPS manufacturers will charge you extra for the network card that allows you to use SNMP (or whatever) to monitor the UPS. Although, it's not exactly expensive compared to the cost of the UPS, but given the level of joined-up thinking on display in this story, I wouldn't be surprised at someone OKing spending ££££ on a new UPS, but balking at spending the extra hundred quid to monitor it properly...
But thanks for reminding me I need to chase up with Eaton why one of our UPS's just dropped offline at 57% battery during a recent power failure, it's been a couple of weeks since they last replied, I think they're stumped.
> Most UPS manufacturers will charge you extra for the network card
Or the trivial piece of cable that will connect the USB data lines from the deliberately non-standard socket on the back of the cheaper units (hint: after you've found online the connections needed, when you spend the couple of minutes to make your own frankencable - it really is just a cable - connect ground and data but *don't* connect the +5V line on the USB, it probably isn't!).
I wasn't around during that time, but i do remember reading plenty of horror stories surrounding those APC serial cables!
I believe something catastrophic could happen if you used a normal serial cable to link up UPS units?
Update: I read into it and apparently they still use these cables for backward compatibility?!?!?
Also, it seems like plugging in a regular serial cable will simply shut down/reset the UPS, which may or may not be catastrophic!
Yes, it was a sort of Frankenconnection. It's quite a few years since I last had to deal with them, but the serial lines were just that - RS232 serial and used if you used the "smart" cable and software. The other lines were "sort of" like the RS232 signalling lines, but worked differently - such that if you plugged in a standard serial cable, the UPS would just power off as the default state of one line was the "power off NOW" signal to the UPS*. A proper connection for the basic signalling lines involved a couple of transistors and resistors IIRC.
* I reckon there are 2 types of IT people - those that have found this out the hard way, and those that have never worked with UPS hardware.
" I wouldn't be surprised at someone OKing spending ££££ on a new UPS, but balking at spending the extra hundred quid to monitor it properly..."
Employer decided to save £100 on a £1.2 million flywheel UPS/generator setup which resulted in the generator not starting because the starter motor battery was dead
Eh, you don't even need the SNMP.
You have event recording on the UPS. It will tell you that the rectifier lost power. It will also tell you when the power comes back on. This is basically true for ANY large UPS(>60kVA).
So, the troubleshooting for this problem is pretty simple.
1. Did we lose input power?
2. How long did we lose input power?
3. Did the facility lose power? Don't know? Put a logging voltage meter as far upstream as possible. You don't need a dranetz. You could just as easily do this with a good fluke
4. If facility and UPS lost power, contact utility. If only UPS lost power, figure out why. It almost certainly means that a breaker was opened/closed.
Many years ago I worked somewhere that had a program doing exactly that kind of logging. It sat on two machines, one on the UPS protected power and one no. Both produced identical versions of the report except that one would stop when the power to the rest of the building went out. No one reslly read the reports they were each stored in an indiviual text file, so there were 1,440 files in every 24hr period.
One day however someone did have a look because they’d just taken over as department head,technology. They wanted to see what this small form factor box they’d found labelled reports actually did. For the most part it was standard stuff inside the report about power being nominal, network connection okay, system uptime etc. There was also a curious line about number of administrators on system.
He queried it with a senior IT staff member who didn’t have a clue, nor did anyone else. The program that produced the reports was written in house so he looked up who had done so. Dismayed however to discover that the author had retired about a year before. Someone still had his number so the pensioned techie was duly called and he explained that he’d added that as a little security twist.
If the number of accounts with admin privileges increased without anyone else being officially added then they could investigate further. What nobody knew and he explained, was that in the program there was an option, (not enabled) that allowed notification of this by email. Also there was an export to Excel function so you could select a period of reports and export the data. All very clever and sadly not really being used.
It was two boxes that had the reports prog on them this was so that if the main power went out, the non UPS protected box would stop reporting. The one day I was there and the mains went off for 20mins overnight, We had a record of everything.
Im really bothered by this entire post.
This company made power equipment. That means they almost certainly had things like PQMs or at least Fluke multimeters in the workshop. Are they legitimately arguing that NO ONE looked at the log on the UPS nor did even a tiny amount of troubleshooting? My Fluke 289 has a recording function!!!
But even without that, nearly every UPS including tiny desktop units, makes an alarm when power goes out. Did this idiot just turn off the power and ignore that that UPS went into alarm and was making noises and then walk out of the shop?
"Must be nice to work at a company where issues are solved by throwing money at it."
Throwing money at the wrong thing(s), then saving money by erecting signs instead of getting an electrician to put in a dedicated feed for the servers.
I've done that at several customer premises - nothing as drastic as server power, just power to sensor lights in entry corridors to save people breaking their necks if they needed to come in after dark. Massive "DO NOT SWITCH OFF!" signs in bold red lettering. You think people noticed?
She has a story to tell that nobody else does.
Many moons ago I took my daughter to SLAC on take your kid to work day. At the ripe old age of 9, she had been there many times before and knew the ropes, but I figured she deserved a day out of school.
She told me as we were walking in that it'd cost me ten bucks for her to not push any buttons. I gave her the money.
On the way back out, I told her that it'd cost her ten bucks for me not to tell her mother she was running a protection racket. She made a face and paid up ... and promptly told her mother as soon as we got home. They both still laugh about it :-)
I've expereinced staff (usually security guards) ripping off the tape to turn things off.
We even had one switch off the AC in a small server room with predictable results 6 hours later - apart from the question about why he was in there, it was impressive because the switch was 8 feet off the ground.
I generally move the output live wire (L1) to the second connector on the input live side (L) of a switch if I want it to stay on no matter what (usually because the light fixture is "smart" or motion activated). I label it accordingly, of course, but it's safe against people who don't read.
Clean, easy to revert, and relatively foolproof (until the universe invents a better fool, so 5 ms maybe?)
This is why IT in the company I worked for in the 1990s said (figuratively), "Fuck it. If you want a networked printer, the department requesting it will have to come up with the money to buy an HP JetDirect card, or an external JetDirect box* if their printer won't accept a JetDirect card. Too many 'printer not printing' trouble tickets had a resolution of, 'turned Novell NPRINTER host PC on'."
I had one 'printer not printing' ticket where the problem was the department had physically disconnected the NPRINTER host PC and moved it somewhere else, all without consulting/informing IT. I arrived at the physical location of the printer to find just the printer, the PC end of its parallel interface cable lying forlornly on the table, and an empty network jack.
*For serial interface devices, we used Castelle LANpress boxes.
I've worked at companies like that! Often it's been when something could be quickly bodged in-house to work round the immediate problem.
"I need a shelf for..."
"No problem, we'll make you one!"
This ignored the fact that plastic bending and welding the shelf cost 3 or 4 times as much in materials plus the labour than it would to have just bought something from the DIY shop over the road.
We moved into a house that had some interesting shelf brackets. They were solid aluminium. The top and wall-facing surfaces were smooth, the opposite faces were saw-cut. Somebody had cut the end off a billet of aluminium and cut about 95% of the slice to waste. The previous owner was an aircraft mechanic at the local RAF airfield.
I have a grill in my garden, made by the apprentice workshop of a chemical plant. This thing will survive not only myself, but generations to come - massively built out of pharmaceutics-grade stainless steel pipes and 4mm thick sheets, lovingly welded. It's over 30 years old and has never been protected from the elements in any way, nor needed such protection.
"Ah having stuff made for you by various engineering departments - we called those jobs "foreigners" and they took more care with them than the normal jobs !"
Dad told me a story of someone he worked with who had come from Rolls Royce. His Morris minor had slowly had all of the engine components replaced over time with stainless steel, copper piping and brass fittings and apparently was a beauty to behold when the bonnet was open and ran like a Rolls too :-)
When I was an apprentice I had to do the usual 'experience all the different departments' thing and in the Industrial Engineering department they had a frightening example of this that everyone had to learn by heart.
One of the mobile weather radar systems had inspection lamps with 30 foot cables that clipped to the battery for use as general lighting. These are the things that were common back in the 70's/80's in garages and were a bulb in a cage attached to a clip.
Because these were for a military product they had to meet certain specifications which had all been worked out and approved and resulted in lamps costing just over £300 each. Then the bulbs they were using went out of production so they raised a 'minor change request' (MCR) to use a different bulb. Sadly the new bulb needed a new socket so another MCR was raised. The new socket didn't fit in the existing cage, another MCR. The cable needed a larger gland for weatherproofing the new socket, another MCR.....
In total there were at least 8 MCRs raised at a cost of over £1500 per MCR, only 15 lamps were sold at a new cost of over £400 each. Civilian versions in the garage across the road were selling at £5 each
> "It's wild to me that"
Side topic but I was listening to some presenter on the radio literally just this morning and wondering when and why people made the sudden collective decision to start using "wild" everywhere in this manner/context?
It's just wild that it's taken off like this! (Sorry)
"I do not think it means what you think it means."
I don't know, it seems quite succinctly defined.
> The path to the similar meanings is different between sketchy and dodgy, but the end destination is the same, near enough.
A sketchy curry is merely something like a curry; an outline of a curry if you will.
A dodgy curry is a genuine curry that might give me the shits.
Sounds like aerospace, defence, mining or oil and gas related - someone was asking on Reddit what people's "authorised maximum spend" was in their industry, everything from £50 to "what do you mean maximum spend?" Where various people then commented that person had to work in one of those sectors, which was confirmed and then then others chipped in with their accounts of working in those sectors and being told if they REALLY needed a max spend to call it X million but don't stick to that if it's an emergency and don't bother the boss below that amount or wrecking X hundred thousand worth of materials and the boss just shrugging their shoulders as if it was just a few bucks worth. Different world apparently.....
You cannot have a too big UPS but selling manglement on that might be tough. :)
So if Cole inadvertently and unknowingly provided a pretext for a UPS capacity upgrade (or two) discretion might have been the better part...
In most contexts obtaining an allocation of resources to solve an existing problem is always several orders of magnitude easier than getting even an in principle allocation to prevent a critical problem arising in the first place.
We had a major site with a large MCC running HV equipment. Because of the site's importance, and a very vocal customer, there was a lot of monitoring equipment which was supported by a dedicated UPS. The specification detailed that the UPS was to power the monitoring gear for at least 2 days in the event of a power failure for statutory reasons (to avoid major fines). Our proposed contractor to supply the MCC put in a series of weasel words to avoid supplying the batteries but eventually agreed to comply with the specification. Only they skimped on the batteries, providing only enough power for a controlled shut-down, hoping no-one would notice. When the testing came to be done, the system worked as designed, except that we left the site over the weekend in this shut-down state....
Monday morning revealed the system inadequacy and the contractor was summoned to explain..... and in particular, explain how they were going to rectify the omission/shortfall in battery capacity. Eventually, the correct battery pack was supplied. It was enormous, requiring its own additional enclosure and a dedicated monitoring package...
It's a few years ago, now. I don't think the battery pack has ever been required, even for a few minutes apart from testing. It will cost a lot of money to replenish and I think the end-user will choke on the cost.
I'll have to assume "Cole" will be reading this and moderate my thoughts
YOU COULDNT FIGURE THAT OUT ?? JEEZ !
We arnt actually told Cole's role , just
Cole eventually figured out the problem: The switch that controlled power to the company's servers was the same switch that powered his workshop.
So perhaps he was a mechanic for the factory floor , and went into I.T. later , possibly due to this incredible diagnosis he pulled out of the bag , without admitting he was the cause of it : )
We use them for everything, including electrical dodads! We howled when we found out one of our cabinets was on the same group as offices. I was lucky to work on a well balanced team, outages like this one would have been on one of the radars. Especially the hardware guys, right I was in that group!
If the servers do a clean shutdown over the weekend they (should) log why, and it would have been found out.
But if such a log entry is missing the servers were not shut down clean, which meant they were not configured for checking and logging the UPS state and shut down before the power is too low. And that is probably the reason for "our IT guy just shrugged". Whether it is his fault for not implementing, or <blink>the managements fault for buying UPS-es which cannot be monitored</blink>... I suspect the latter, "shrugged" equals to "I gave up telling them what they should do".
...I think many underestimate factory IT.
Usually it's one or two people that have to do EVERYTHING. Going by the post, it sounds closer to the one IT guy rather than two.
Servers, Networking, Desktops, Printers, Factory Machine interfaces, the Website, phone systems, probably even the TV in reception doing PowerPoint slides as welcome info for visitors. You don't get chance or opportunity to learn like SNMP trap logs (or may not have the ability to do so)
A lot of the time it is genuinely trial and error
I had a customer that suffered from power outages crashing their system. After a couple of rebuilds of their DB, I convinced them that they had to buy a UPS (this is in the 80s). So they told me that they bought a UPS... until the next power outage when they called me again to repair the system. They were adamant that they bought a UPS and it must be something else.I got to their site to discover that they indeed purchased a UPS - which was plugged into itself and the server plugged into the wall socket. NEVER underestimate the problem between keyboard and chair.
We had a similar problem with the phone systems at an ironsands mining site. It turned out that the radio-linked remotes were wired to the non-essential power on the mining barge
It only took 20 years of fault calls and me showing up on a Friday evening (my second ever visit) to figure out the issue.
It was a similar story with the batteries. Lead-acid cells don't like deep discharge and it had gotten to the point that they only provided 5 minutes backup vs the entire weekend when first installed. Fixing was a matter of moving the PSUs to an adjacent outlet and adding a label. The company investigated, confirmed the designated outlet (dedicated circuit, vs the cleaners' sockets) was wired to the wrong supply and decided not to bother dealing with it as the mine was due to be shut down in less than 2 years
20 years later in a different career, I discovered that a network switch was plugged into an outlet in a workshop that the staff diligently turned off as they left at 4:30pm. Maintenance staff in other parts of the building worked until 5pm and "noticed". Labels to the rescue again
One of my old colleagues used to tell a story of when he was a young, newbie support person. We used to have a PBX and the extensions were patched in using the structured cabling used for the computer network with each patch room having a dedicated voice panel. Each socket on the voice panel was assigned a phone number.
One weekened, my colleague was in tidying the connections on the patch panel. He'd assumed that patching the phones was the same as patching the network. IE as long as you plugged each patch into the right switch, it didn't matter which port on that switch you used. He thought it was the same for the voice panel.
It wasn't. As he discovered when, having removed all the cables from the voice panel, he plugged one back in and tried to ring the extension he thought he'd plugged in.
He spent the whole weekend plugging cables back in, then rining extensions and seeing where he could hear the phone rigning. Of course, had he done this during the week, he could have contacted the switchboard, who did have plans of the all the patch cabinets showing what was patched to what.
We currently have a VOIP switchboard, with extension numbers assigned to IP addresses, so it wouldn't matter which port on which switch you plug the cable into, as long as it's assigned to the right vlan.
A bit late to comment, but since the subject of PBX has been raised:
In the days of the late unlamented Assad Senior, the Syrian Military (and Secret Police) imported our PABXs .
One site had the usual dual 3 phase mains feed to the telephone exchange power.
In those days everything telephone ran from a battery floated at about 54V DC.
Being a military site, the spec would be for a decent standby battery.
Of course, both mains cables ran in the same trench, which was inevitably breached by the first cut on site.
Also inevitably, all was overspecced to the point where the job lot ran for the whole weekend (Thur & Fri),
not just the 8 hours in which the problem should have been discovered (and fixed) but naturally was not.
As an aside: to show how reliable the mains supply was, the socket outlets were 4 pin jobs,
with earth/neutral in the middle and 4 phase pins in a triangle. If your power went off, just unplug, rotate 120 and try again! Simples.
