I don't know but it's been said, Amphenol plugs are made with lead Tear your gaze away from the retreating rear of the weekend and instead look forward to another week filled with opportunity, adventure and, of course, The Register's Who, Me? Column. Today's cock-up comes courtesy of reader "Terry", who told us of his time working in a data centre that served a number of high-profile …
Why ? There is nothing that tells me what happened, except that the cable apparently went "between the guard rails". And ? Did it hit the circuit board ? Did it somehow fall in the power supply, making a short-circuit ?
Could someone please explain why a cable falling on a router makes it "go dark" ? I don't have a clue why this happened.
Further, the supplies are recessed slightly so objects dropped from above cannot hit the power switches
Maybe they should have gone for a toggle switch with a flip-up cover which means it can't be activated if anything does hit it, and it it has to be flipped-up to be activated?
You know; the sort of thing the military use all the time to stop switches and 'bombs away' buttons being activated accidentally.
Halfords, Amazon and eBay stock them. Any decent elctronic component supplier does and they cost all of a couple of quid.
It seems my Raspberry Pi reset switch is better protected than that router.
An RJ21 connector swung down and hit the power switch, probably on a Cisco 7000 of some description (the prior AGS and AGS+ couldn't switch at the rate mentioned). Some folks had (optional) protective bars fitted, in the hopes of avoiding this kind of accident.
Murphy was an optimist. (If you're British, substitute Sod or Finagle for Murphy.)
I stand corrected. I knew John Campbell used it, but I didn't realize he invented it. I always thought Finagle was one of yours because I've never heard it "in the wild" over here (outside of Astounding, of course), but I have heard it over there.
Likewise, when I used Murphy in Yorkshire way back when I got odd looks, but I soon found that Sod was the goto in such situations. Strange how epigrams vary so much from region to region. Noted, and ta.
Beer. Seems like a good way to start the week :-)
"Likewise, when I used Murphy in Yorkshire way back when I got odd looks"
Maybe they were just giving you odd looks on principle.
Over here we also use Murphy to trap unwary seekers after umbrage. A comment such as "Murphy have dug up the road again" is apt to flush out those who take umbrage on behalf of Irish labourers. Then we can point out that the Murphy Group are well known public infrastructure contractors.
Oh I had one of those. Back in the late 80s - was demonstrating the new Telegram Retransmission Switch to BTI to get their sign-off. Was based on a Sequoia fault-tolerant unix box. We had the test traffic running as we were casually pulling out circuit boards and disks - all that happened was that it slowed down. We were just getting to the end of the demo when Paddy and his JCB took out power to the whole building - and the demo system wasn't on the UPS.
Underwear changes all round. Fortunately the BTI guys hung around for long enough for the power to be restored. At which point the whole thing just carried on where it left off. The force of exhaled breath probably started a major meteorological perturbation..
We call them diggers. Or after the UK market leader JCB.
They did the first dedicated tractor style one or as well call them JCB.
From Wiki
In the same year, JCB launched the first European hydraulic loader, followed by a backhoe with a 180° slew, fitted to a tractor, in 1953. In 1957, CASE developed the first integrated tractor backhoe loader at the Case New Holland plant in Burlington, Vermont, helping contractors complete their work more efficiently.[8] JCB introduced the first dedicated backhoe loader, incorporating the excavator and the major loader in a single, all-purpose tool. [9]
Because of the long-time predominance of the JCB marque in the United Kingdom and Ireland, it has become a genericized trademark there, and backhoe-equipped diggers are commonly called JCBs, while the term "backhoe" as an excavator component is almost unknown to the general public in this context. The founder of the JCB company, Joseph Cyril Bamford, is the only Briton to be honored in the Association of Equipment Manufacturers Hall of Fame.[11]
There is a book about them.
https://en.wikipedia.org/wiki/The_Nome_Trilogy#Diggers_(1990)
And I used it to make sure that the totals for number of clients recorded as seen=total for all staff members records= total male+female=total for each location=total of all ethnic categories. Because it never did otherwise. Not once ever.
The trick was to find two that did match and adjust the others proportionately.
' to make the result of a lab experiment match the required result.'
<sigh> I remember rubbing out the actual values I had measured and fudging them in my pencil-on-graph paper curve to make the experimental results fit the chemistry book 'S' curve. As had legions of children before and since no doubt.
The experiment was dropping zinc granules into HCl and graphing the release of hydrogen - starting slowly, rapidly reaching a steady high rate then dropping off to zero as the Zinc was exhausted.
The slow-down I had measured and eliminated midway or higher on the steepest part of the graph was no more.
Many years later I heard of a truly scientific chemistry teacher observing this (probably having ignored it before) and carefully re-doing the experiment. The rate *did* slow then speed up.
It turns out that the hydrogen bubbles forming on the surface of the zinc and clinging by surface tension before floating off prevent the maximal access of the HCl to the surface as the reaction progresses. (My best understanding - if anyone can improve or correct this please do.)
Hat's off to that anonymous teacher.
My recollection of the Cisco 7000/7500/7600 is that they either have hardwired DC PSU's with no power switch or a "clunky" turning knob type power switch to power them on.
Having said that, Cisco produced so many different PSU options for these beasts that there is likely something exactly as OP described.
We do? First time I've ever heard of it, as far as I'm aware its Sods law or Murphy's law. Sod's law being you'll get the worst option, whilst Murphy's law is what can go wrong will go wrong at the worst possible time, and even if it can't go wrong it will anyway..
Ah those infamous Cisco 4500-E super sensitive power switches. They could flip by simply breathing at them. Cisco 6500-E PEM power switches were a way more secure and better designed.
https://www.cisco.com/c/dam/en/us/td/i/000001-100000/75001-80000/79001-80000/79138.ps/_jcr_content/renditions/79138.jpg
>Based on descriptions, it was likely a Cisco 6513. I suspect they may be referring to the handles on the power supplies as guard rails.
Nope Cisco 6500-E had very difficult to turn off accidentally power switches, 1950's-style. This surely was a Cisco 4500-E switch which power supplies had a very sensitive power button only protected by a thin barrier on each side which protected nothing - without any sort of a plastic cap as on some other models. It could flip by literally knocking on the PSU.
Did the DoD not think this critical router was worth having a spare power supply on site so they were waiting for Cisco to show up with a replacement? Was it not properly monitored so he only found out it had a dead power supply after? Or did he know had a dead one, but was not authorized to touch it, and whoever who was authorized was not available on site yet?
it doesn't do much good to have redundant hot swap power supplies if you knowingly let a device operate with a dead one. Maybe they need three power supplies to handle such a scenario!
I thought this was going to be similar to some of my war stories. For redundant systems, the biggest cause of dual failure is an engineer arriving with a replacement unit and then trying to swap out the good one instead of the failure.
But in one machine room I was responsible for, we had the usual three-phase power supply and the critical kit had dual power supplies. At one point we lost a phase and no-one noticed, except for a group of servers (not my responsibility, thankfully) whose dual power supplies were both powered from the same phase.
Well at least that protects against failure of the power supply. More than a few data centers in the US don't have redundant circuits to the individual cabinets. Still should have two separate cords though, so accidentally unplugging the wrong cord from the PDU doesn't kill both PSUs.
Very true. In my specific case, our datacenter had intermittent issues with the plugs slowly creeping apart, causing power failure warnings and more rarely total dropout of the server if this wasn't caught. After requesting some photos for an unrelated issue, we noted the cables to be hanging under their own weight; guess they hadn't discovered tie straps. They did eventually outfit the racks with additional PDUs, but I don't think they ever pursued redundant circuits.
BTDT, albeit with a SAN switch - one PSU failed and took the fuse with it, so the other PSU ended up without power as well, taking out half of the SAN. That happened on a Friday evening with me on call, but no call went out.
Monday morning the fault was discovered, the only effect being that a multiple-hour long billing run during the weekend had taken a few minutes longer than usual.
The SAN had been planned and built properly, with redundancy in mind, so the second half still worked flawlessly and the PSU could be replaced and rewired to another fuse/phase without undue haste.
I can't tell you how many times I've run into places where the PTB would happily spring for the redundant power supplies to power the kit, but would refuse to pay for actual on-site spares. Yes, including in .gov and .mil installations.
I didn't say it made any sense, I'm just reporting what I've seen.
Sometimes monitoring and/or alerts are missed or mis-configured too. I got sent out to a data centre to swap an HDD for, erm, a well known World-wide Payments system. The HDD was obviously part of a RAID. It had two hot spares, one of which had already failed over correctly to replace the faulty drive. All good so far. But when I got there, as instructed, called the admin who flashed the LED on the faulty drive as per procedure to make sure I got the correct drive (obvious anyway as it was the only drive with a red fault LED showing rather than green ready/busy), I mentioned that there was another red flashing LED, this on one of the two redundant PSUs. Cue a short silence followed by the tappity-tappity of a keyboard and a "Are you sure? There's no logs or alerts showing". They sent me back the next day to swap out the PSU and confirmed the logging had reported the swap out correctly.
"Maybe they need three power supplies"
That won't always help. We use several HP Blade enclosures at work. Now, these enclosures can take up to six hot-swap power supplies, and depending on how many blades you're running (etc.) it'll run on only two.
I was busy cabling in a new UPS, and also using the opportunity to tidy some cabling. To do this, I was carefully checking that various machines had redundant power, and then moving a single power cable at a time. This was going fine until I reached the blade enclosure which was running on four PSUs, however, as soon as I removed one, the whole thing went dark, taking half a dozen important customer servers with it.
It eventually turned out that one of the power supplies was faulty, and when the blade enclosure tried to draw more than a few hundred Watts, it failed, putting too much load on the remaining two supplies which also tripped.
These days that enclosure has five redundant PSUs, and they're all the high wattage ones, so it should be able to cope with two (fingers crossed).
These days that enclosure has five redundant PSUs, and they're all the high wattage ones, so it should be able to cope with two (fingers crossed).
"Should be" is a pipe dream... a faint hope. If it's not been tested, assume the whole thing will fail. Finding this out the hard way is a bitch. Don't ask how I know.
I always tell clients that they need to schedule regular tests of any redundancy (both physical and logical) they depend on, because the tests done once when things are installed right before going into production aren't sufficient. ESPECIALLY where software is concerned i.e. clusters, but hardware tests of redundancy where feasible is a good practice if you are already taking down the hardware for regular patches, configuration changes, etc.
Some are smart enough to schedule the time, others aren't. You can only tell them, you can't make them perform the tests regularly.
When I've worked with HP blades before they seemed very good about reporting module failure, so I'm surprised that failed power supply wasn't alerted previously. Or perhaps it was and it was sitting in someone's queue ("no need to rush on this because it still has power")
Having the ability to have a lot of power supplies, of different types, to meet a wide range of needs is nice but the monitoring needs to be able to tell you how much redundancy is left so replacement can be properly prioritized.
The faulty PSU reported as good in every possible way, even after it had failed. I think it was capable of producing a small amount of power (1-200W), but as soon as it had any big load it just died, in such a way as to kill the whole enclosure.
I tried it in another enclosure afterwards and it still showed as good, and working. IIRC we did get a free replacement from HP though.
+1, I call this the Magneto Principle. Light aircraft used to have (maybe still do, I don't know [1]) the very simplest engine equipment, because: fewer things to go wrong. For ignition there would be two redundant magnetos to generate the HT spark. Part of pre-flight would be to run up the engine and switch off each magneto in turn, to ensure that they were both working well enough to support ignition independently.
[1] It seems unlikely, though. The first pageful of search results for 'magneto' all refer to an X-Men comic character of whom I've never heard. Mumble grumble.
My mower (mid-90s?) uses a magneto for the spark. (It also has a separate alternator for charging the battery.) The only part of that circuit that is ever likely to fail would be the spark plug itself. Simplicity is a wonderful thing.
When I first got it, used, I spent some time trying to figure out why it wouldn't turn off when the key was removed. Apparently turning the key to "off" connected a lower-voltage wire from the magneto to ground, preventing spark - and that wire was disconnected. Pulling the spark plug wire with the engine running was definitely no fun.
In my experience, when it comes to lawn mower ignition, a couple of things are more likely to quit before a quality spark plug. The first is generic corrosion of the wiring caused by dis-similar metals exposed to moisture. The second is work-hardening of the copper wiring in the coil(s), leading to breakage. The third is timing issues caused by hitting a large object with the blade, bringing everything but the flywheel to a sudden halt. Momentum keeps the flywheel moving, which bends or breaks the key that keeps the crank & flywheel positioned properly with respect to each other. The forth is the magnet(s) in the flywheel becoming dislodged by the same sudden stoppage (rare, becoming more common, see below). Lastly, the magnets can actually lose their magnetism over time (I saw it only once before the rise in Chinese motors, now I see it a couple times per year).
No. The (D)ARPAnet was just a research network designed to research networking. Redundant paths were included for one simple reason: The hardware of the day was flaky.
.mil already had SLFCS and then MEECN for the kind of thing you are suggesting.
There were two separate routes from east to west in the original ARPANET. A north and a south one. A single back hoe took them both out as the phone company routed both east/west links through a single cable. That was when they discovered that it is important to specify different physical routes.
News to me ... I remember three links as early as '73 ... Rand/USC to the Pentagon, Stanford to Harvard, and Berkeley to MIT ... Each link made several stops along the way, and there were several mid-continent cross-connects. All three of the west coast schools were connected directly.
That said, in late 1977 I managed to take down all the PDP10 kit at Stanford and Berkeley with a software upgrade. Effectively split the West coast ARPANet in half for a couple hours. Not fun having bigwigs from Moffett and NASA Ames screaming because they couldn't talk to JPL and Lockheed without going through MIT ...
Been down this road... and back up it a few times. No spare for the gear, it went down, howls all around. When we replaced it, bought the spare PS, sat on the shelf, 6 years later, replaced that item, power supply still sits on the shelf. These are the costs that drive my manager batty... That said, we've not has a PS die in many a year. Luck?
The spares do not work even though the part number in the database says it should. Then you look down and notice the part numbers are different. I'm looking at you IBM and your so-called equivalent FRU parts. Or worse all your spares are faulty and the last spare kills the entire unit. Now not only do you need a new PSU but new board :).
Never ever say we have not had a failure in x time and do not expect one to ever happen.
"When we replaced it, bought the spare PS, sat on the shelf, 6 years later, replaced that item, power supply still sits on the shelf"
It's called insurance. Even if you never use the spare and end up junking it when the kit that it's supposed to cover is retired, it's still worth it. Ask the beancounters if they would ever skimp on, for example, building insurance. The case to be made is simple - what is the cost* of this item not working for a few minutes / days / hours / ever again. Then based on those numbers you devise an appropriate backup / redundancy strategy.
* Although not all cost is strictly monetary, for beancounters everything can be reduced to (currency of choice) amount. Missed sales, man-hours needed to troubleshoot, emergence equipment sourcing / shipping cost... even intangibles such as customer confidence are in a company's balance sheet as 'Goodwill'. Sometimes you just need to know how to talk to beancounters in their own language
The problem is that the decision NOT to buy spares is always made when the kit is first installed, spares are plentiful and the number of processes running on it are not too bloated, so cost of downtime isn't too great. So a beancounter looks at the numbers and veto's having spares because "the cost outweighs the benefit".
Give it a few years and all those processes have grown in importance and a few got added. Meanwhile the manufacturer has come up with new shiny, shiny kit and the spares for your stuff now have to come from the "long term support center" at triple the cost and four times the delivery time. Downtime is now absolutely killing and guaranteed to be 4 times as long, but since the beancounters originally calculated you don't need spares, you can't have spares.
"Downtime is now absolutely killing and guaranteed to be 4 times as long, but since the beancounters originally calculated you don't need spares, you can't have spares."
You then point out the revised cost/risk situation and that the only way to mitigate this is to replace the equipment now, before it fails and this time why not buy the spares.
If it's IT kit and from a reputable manufacture, you head on over to the local tat bizzare that is eBay and buy the part as a used item from an IT recycler. The older method is to ring up a seller of refurbished parts and buy it from them direct though this tends to be more expensive in my experience.
<quote>Sometimes you just need to know how to talk to beancounters in their own language</quote>
I have always found that walking into a beancounter's office carrying a clue-by-four slung over my shoulder, gets their immediate attention.
Well, it is a gamble. Manglement needs to be reminded of 1) the stakes and 2) the probabilities, and 3) possible repercussions regarding bottom line / payouts / penalty payments - and preferably you need something in writing that states "yeah... nah, we won't buy replacement to set aside just in case"
This post has been deleted by its author
"bought the spare PS, sat on the shelf, 6 years later, replaced that item, power supply still sits on the shelf."
PSUs are cheap. Callouts are not.
Anything containing a moving part should be considered a high risk potential point of failure
I lost track of the number of times as a field tech I set out on on a job with "You only need XYZ" and found that I needed ABC JKH and NOP too. The days when I packed the wagon with EVERYTHING imaginable were the days when I found that someone had wired the power to the wrong circuit (non-essential power, etc) on the customer premises and my responsibility consisted of running a temporary extension lead so our batteries didn't go flat and telling them it needed to be fixed within 5 days, when I'd be back to checkup & sign off the fault or have them charged at full callout rates.
Big Cisco box, recessed dual power supplies? Capable of shifting 720Gbps around? Sounds like a Cat6500 with a SUP720 controller to me.
If they'd gone with the SUP720-3C then they could have set up a pair of 6500s as a VSS pair to avoid this kind of thing - the old adage of "why have one when you can have two for twice the price".
As a slight aside - does anyone know of a good way to remove redundant part numbers for old Cisco gear from using up space in my brain? I've tried beer but it hasn't worked so far...
Ever thought that's EXACTLY WHY people enjoy beer? It's as close to water without actually BEING water, AND it still provides a nice buzz (some people don't really want to get drunk--those that do--usually out of desperation or deep depression--resort to cheap liquors and bum wines). Always consider the environment: a lot of Americans are experiencing 40-degree weather right now and could really stand a cold one, not JUST for the buzz.
If they'd gone with the SUP720-3C then they could have set up a pair of 6500s as a VSS pair to avoid this kind of thing - the old adage of "why have one when you can have two for twice the price".
Maybe the Pork Barrel wasn't quite so deep that year ?
It's gets better when the equipment is over spec'd enough that it lasts for years... And years and the company that made said kit gets bought out. And changes its name.... Twice (as does the company that uses the kit).
Before finally a battery dies in the system and no one notices. Or has a spare. Because the equipment is 15 years old (and 10+ years since it was last DR tested).
Yeah. Been there before (battery was a raid array battery and because it wasn't my own kit I couldn't juryrig some batteries to get it working with a soldering iron and persistence). Spare part was located in a different city and shipped over same day and fitted within 30 mins of going to fail over systems.
Also happened to be part of a billing system for a very large teleco. Fun times.
Anon because I'm still working for them and I'm pushing for replacement of the kit should it die again (once bitten, twice shy)
The trick is to explain what you want done in small words, in writing (so they can't argue about what was said), have a copy of the regs to hand where they can see you have it and most importantly, make sure they understand that if they don't do it to a working, certifiable and satisfactory standard, they don't get paid.
This last bit is crucial to getting their complete undivided attention AND have them ask questions if they think the task is odd, instead of just bodging it.
For added attention you can add that if someone else has to fix it, you'll bill them for the rework.
Reminds me of a situation I once encountered with a piece of network kit that had two (redundant) PSUs. Each PSU had a small handle at each end to allow it to be smoothly pulled out of the chassis.
The problem was that whoever had installed them had used the handles on PSU B as cable management for the power feed to PSU A. There was insufficient slack in the cables so, if PSU B failed, it was impossible to replace without first disconnecting PSU A.
A certain corporate entity delivers 4 prepack racks of proprietary exalent (no, not a spelling error) servers. 2 10Ge fibre cables, 2 Infiniband cables, 2 10Ge ethernet cables, 1 100Mb console port cable, 2 power cables. Fitted to 1U servers, 36 to a rack.
Issue: replace one of the 1U infiniband switches on the *bottom* of the rack, WITHOUT taking out the entire rack of servers. Go for it. anyone? Bueler?
Representatives of the military "showed up the next day and ripped out their circuit.
"They weren't smiling."
I dunno what it is about modern militaries; all seem fairly humourless. Prolly more efficient than in the past, but as far as I can see --- unless they were a naturally surly bastard of an NCO, most such emissaries in the British Armed Forces back in 1925, 1945, or 1965 would have been fairly sanguine and said things like, "Well, there you go..." and "Worse things happen at sea...". Grimness was left to the tedious Ivans...
Separate power sources protect against a different set of failures from separate power supplies. In this case, the E450 was protected against single PSU failure. If you also wanted to protect against the failure of a single phase, transformer, UPS, etc. then you would also need to use 3 separate power cords and 3 independent sources. The fact that most of the time we want both does not mean there's no value in the one.
Many hardware designers operate under the naive belief that moving parts are the most common source of failure, so they provide redundancy in anything that has a fan while usually ignoring everything else. In reality the most common sources of failure are operator error and firmware/software defect, in that order, so most of the time that's not enough.
"In reality the most common sources of failure are operator error and firmware/software defect, in that order, so most of the time that's not enough."
Fans come in third though, and most of them aren't field-replaceable items (Says me, who's had to resort to changing bearing races in PSU fans for one model of disk array that Xyratex stopped making 7 years ago, thanks to budgetary constraints making it impossible to replace the arrays - yet)
No network with that kind of throughput isn't without multiple paths to get layer 3 from one place to another. If the phone call wasn't to ask why the standby router/path had taken over, I'd be asking why such a high throughput system wasn't asking me that question.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
