Techie banned from client site for outage he didn’t cause

Welcome to Monday morning and another instalment of “Who, Me?” - the weekly reader-contributed column in which we share your stories of what not to do at work, and how to get away with it. This week, meet a reader we’ll Regomize as “Patrick” who told us he once installed an extra shelf of storage for a NAS at a local council …

[Pascal Monett]

“They still banned me from site”

Well at least, that way, when they finally found the problem (or didn't until it happened again), they couldn't blame him any more.

[MiguelC]

Re: “They still banned me from site”

Correction: whenever they found the problem, and whatever it was, they still blamed it on the external guy that wasn't there anymore to defend himself

[Michael H.F. Wilkinson]

One of Murphy's Laws at play

"No good deed ever goes unpunished"

[HXO]

Has a client or colleague blamed you for a problem you didn’t cause?

Oh yes. There are ignorant people in all industries. And that is one of the big reasons to stay with a manager (or group), that is intelligent and has technical understanding. In BigCorp even official root cause analysis was finding the first and easiest 'victim' for demerit and firing. Some frustrating fights over the obvious, and *never* a 'sorry', when the real cause was eventually accepted.

[Anonymous Coward]

Sounds like gold bars

I once got banned from a site because their insistence on buying clone toner cartridges ruined a bunch of fusers and I refused to clean out the printers for free and replace the fusers under warranty (it was absolutely obvious what caused the problems)

Tightwads (with a large presence in the IT contracting market)

[ITS Retired]

Re: Sounds like gold bars

I have not so fond memories of clone toner cartridges that upchucked all over the inside of the printer. Reams of paper in the drawers damaged from being buried in toner.

[meataxe]

Re: Sounds like gold bars

My good friend has been known as Sooty for almost 30 years now, due to some overly excited toner that he was involved in as a newbie tech

[IGotOut]

Re: Sounds like gold bars

Pah toner.

I'll take you that and raise you a Techtronix Phaser. Unplugged it, picked the bastard heavy thing up to move it to another desk. Molten wax fucking everywhere. Took me hours to clean up the printer, desk, floor, me.....

[Scene it all]

Re: Sounds like gold bars

My worst experience was a very old Epson printer that had an ink hose come loose inside the case. What a mess. I had to buy a new one. (The new Epsons are much better.)

The riskiest think I ever did was replace the ink ribbon on an IBM 1403 printer. They put those plastic gloves in the box for a reason...

[I could be a dog really]

Re: Sounds like gold bars

I see your IBM, and raise you ... a Risograph ink duplicator (forget what they were called).

For those who've never seen them, they look like a photocopier. But when you hit the copy button, it wraps a sheet of some paper-plastic sandwich around a drum, having removed the plastic where the ink is required to make a "master". Inside the drum is a cartridge of ink - with the consistency of "gooey toffee" and very very black. It then spits out one copy.

If that's OK, then you dial in how many copies you want, and it runs them off - very fast and very cheap. Where I worked, it was used for the price lists, thousands at a time, and the model we had could do 130copies/minute ... which was ... interesting if those sheets of A4 coming out the end at a little over 2/second didn't drop neatly into the collecting tray and sprayed themselves around the room.

OK, so that's the basics. Are you thinking "what happens to the masters when you do a new one ?" No ? Well, the machine is *supposed* to unwrap this piece of paper liberally coated with black gooey toffee ink off the drum and put it into a waste bin. Ours was faulty, and this bit didn't work.

We had a sign on the machine, clearly instructing to manually remove the old master before making a new one. Done right, it was clean as there was a strip across one end free of ink, so you could grab that, unwrap it off the drum, and keeping it well clear of your clothes, drop it into the bin. Did users do that ? Did they f*** ! So the machine would shred the old master, so removal now meant getting multiple strips of black gooey toffee coated master off the drum. Needless to say, the person responsible would never do this as it was "too dirty" for them to go near - I really should have dug my heals in and told them "you foooked it, you fix it - then perhaps you'll try following instructions next time".

And we didn't have a supply of plastic gloves.

[Anonymous Coward]

Re: Sounds like gold bars

I have a co-worker now known as "Smoky", due to finding the nearest outlet to plug a 120V printer into... which turned out to be a 240V outlet. (Now properly labeled.)

[Korev]

I save an engineer in the same situation once. He was upgrading some instrument controller Macs to the wonders of OSX. One of the computer's hard discs decided that was the time to die (maybe the amount of IO didn't help). I managed to dig a compatible disc out of the "I'm sure this'll be useful one day" cupboard and he installed the OS afresh.

I then had to convince our scientists that it was just unfortunate timing on his part and not anything he did wrong.

[Anonymous Coward]

I've had servers die because of unfortunate disk failures (second disk failed shortly after the first and the client hadn't noticed or bothered to report the first), it was always fun explaining they needed to restore from backup then finding their ultra speedy backup had only catalogued a tape and their last, valid, backup was from a month ago

[FrankAlphaXII]

As I always say, unless you validate it, it should be assumed there is no backup.

[HXO]

Things I have learned to do:

SW/OS change on server or labPC: Shutdown. Remove power. Leave to cool for as long as possible, at least a tea break. Power on. Check that HW and old SW/OS is still fine. Then do your changes and finish with an OS restart.

Programming: Before making even a trivial change, see that the application builds as is, so you can tell if someone elses change broke the build.

[that one in the corner]

> Programming: Before making even a trivial change, see that the application builds as is

So much this*

> so you can tell if someone elses change broke the build

Even - or especially - if that person would have yourself! These days, I'm only coding for my own needs, but before leaping in to bash away at the recently spotted bug in my notes-taking program, I made sure that the working copy was updated from VCS (and had no local mods) then fired off a build from the top of that copy. 'Cos who knows what state I'd left it in when it was last fiddled with, months ago: I know I'd added a neat feature, the commit message reminds me of this, but was I distracted and left something part done in that working copy? Only one way to be safe...

* and you can have the all-important tea break whilst waiting for the build to complete

[BartyFartsLast]

Programming.

Especially if there's a chance the toochain has been updated

[Will Godfrey]

Don't remind me!

We've had to scramble to get our project back in operation as three libraries we depend on made changes that were incompatible with their previous versions around the same time - one of which still isn't properly sorted out, so our only recourse has been to rewrite our code to use an alternative route.

[An_Old_Dog]

Bob's Laptop

see that the application builds as is

"Don't worry about those error messages. The product only builds properly on Bob's laptop."

"Where is Bob's laptop?"

"With Bob, at a convention in Geneva. He's got his laptop with him so he can do emergency support."

[AtomicWombat]

Re: Bob's Laptop

Particularly when the only reason that the build works at all on Bob's laptop is that it has a hardcoded dependency on a custom nightly build of the entire FreeBSD userland (on a Linux box!), which Bob has not bothered to write down or tell anyone about.

Not that I've ever had that happen. Nope.

[Cheshire Cat]

When applying a new system configuration change, make sure the system unit tests all pass *before* you make the change, and if they don't, dont change anything. Else you'll be spending ages wondering what you broke when the tests are failing *after* you make the change.

[Anonymous Coward]

Counterexample:

We had a Macbook that got a major OS update. Suddenly all hard drive access was at 1% of previous speed - but with no errors, just painfully slow. After a WEEK of backing it up, I took it to the "Genius" bar, who promptly declared the drive had failed. I pointed out that (1) this is not how a failed drive looks, and (2) it started on the FIRST BOOT after a major update. Nope, computer automated tester says no, so it's failed. Could not convince them otherwise.

[Korev]

(Pat)rick

It sounds like they should have PAT tested the power supplies better

[Anonymous Coward]

Re: (Pat)rick

The needn't have got in a paddy about it.

[Korev]

Re: (Pat)rick

I'll give you a pat on the head for that

[Korev]

Speaking of Patricks, is this the same Patrick as last week or has the Regomiser got stuck?

[Inventor of the Marmite Laser]

So it's all YOUR FAULT is it. You're banned from El Reg

[Korev]

Superb, have a breakfast pint

[Inventor of the Marmite Laser]

Cheers.

[Giles C]

Sop?

You went in there last week therefore it is your fault.

Standard line from a lot of people.

However if you think it was you own up people will treat you better we all make mistakes.

[TheOtherPhil]

Re: Sop?

Had a few of those customers...

"Customer's sites keep falling over the day after your team installed the new frontend system, so it must be your fault." (Yeah, and two days after the backend software house installed the new database and set a reboot for Saturday evening in the process.)

"Customer says their <insert-random-machine-here> doesn't work; you were the last engineer on site so it must be your fault..."

I've even had "Customer's network is down. You were in reception at the time, so it must be your fault..."

[Anonymous Coward]

Re: Sop?

Oh that's as old as the hills, my career started with a part time job repairing radios and television sets, the number of returns we got (usually months after a repair) with "it's never been right since you touched it" or "it's gone again" for some totally unrelated problem (needed new set of legs on the cabinet, set is now on fire)

[Flightmode]

Re: Sop?

Ah, the classic "According to the firewall log, the last config change was made by you three weeks ago..."

[Contrex]

Re: Sop?

In the 1980s I had downstairs neighbours who were convinced that my Sinclair QL was 'making their TV go funny'. I said 'Is it doing it now?' and was told 'Yes'. I said 'Funny that, I sold it two weeks ago'.

[BartyFartsLast]

Re: Sop?

Back in the days of analogue telly it used to be the same with CB, Ham radio etc.

Didn't matter if you transmitted or not, it was your fault their telly picture was screwed up

[Anonymous Coward]

Re: Sop?

I recently passed my entry level radio amateur’s certificate. Our instructor was adamant to tell us that any interference on neighbours’ TV, stereo or other equipment needs to be solved on two fronts; the electromagnetic and the psychological.

73 de SH4ROA (QRT 4 now)

[BartyFartsLast]

Re: Sop?

Always, the most important part is how you deal with the complaint, nowadays though it's more likely their kit interfering with yours, the number of QRM generating devices in the average home, light bulbs, phone chargers, pretty much anything with a power supply or processor will screw you, EV chargers and even the EV itself can be absolutely awful.

One of the nice reliefs of the past few years has been the move to FTTH broadband services which has removed a lot of ADSL noise

[I could be a dog really]

Re: Sop?

Going back a while, a good friend was a radio amateur.

He told me how a neighbour would come knocking on the door complaining about interference on his TV. So my friend would recommend going down to the Post Office and filling in a form and [forgets name of agency now] would come and investigate - I did say it was a while ago now.

Multiple times the neighbour would come round, each time more worked up and threatening violence. Eventually my friend went down to the Post Office and filled a form in on his behalf. Violence did very nearly occur after that - can you guess why he'd not reported the problem ? Yup, no TV licence ! By chance, my friend was back in the Post Office for something a week or two later - and the lady behind the counter commented that she'd sold quite a lot of TV licence in the same street during the last week.

My friend was also a computer enthusiast - that's how we go to know each other. Remember when it was common to use an RF modulator and a TV as a monitor ? And how these tended to be quite leaky ? As an aside, I recall finding I got a better picture with no cable at all - using a cable meant too much RF and I guess it swamped the tuner front end. Neighbour at the time was called gerry, and was one of those who'd occasionally going twiddling to see if he could get any other channels - where we were, if the atmospherics were right, we could just about get S4C from a North Wales transmitter. One day, through the wall, my friend heard Gerry exclaiming "What the **** is going on" - when he picked up my friends computer output, where he'd left a message "Hi Jerry, you old coot"

[Anonymous Coward]

Re: Sop?

Interfering electromagnetically with nearby hams is also not recommended. Many decades ago I built my own heterodyne (NOT superhet) SW receiver out of gash TV valves and hand-wound coils, and my local RT ham neighbour was not at all pleased and said so in no uncertain terms on air, but he couldn't find me, fortunately. I soon realised I needed to do a bit more work and build a superhet instead. Main excuse: I was only about fifteen.

[BartyFartsLast]

Re: Sop?

Do you mean regenerative?

[FrankAlphaXII]

Re: Sop?

Yet as soon as the Hurricane happens they're the first at your door begging you to yammer at FEMA or the State to do something to get the power back on.

I do miss my radio, but I don't miss that shite at all. Very few Hurricanes in Central NM at least.

[jdiebdhidbsusbvwbsidnsoskebid]

Re: Sop?

Years ago, we were doing some oceanography work involving high power low frequency sonar. There was a lot of interest in our work from environmental groups because of the the concern about impact on whales and dolphins. (We were bothered too, hence we had operating rules for when marine mammals were around, but the science was so uncertain in those days we were never really sure how effective those rules were).

On one expedition (several weeks at sea), there was a mass stranding of whales on a nearby shore. It never quite made the public news but we got the blame in certain open reporting. We knew it was nothing to do with us because at the time of the stranding, we hadn't yet got any of our kit in the water. We never felt the need to challenge the blame claims, they probably wouldn't have believed us anyway. We just shrugged our shoulders and moved on.

[I am David Jones]

Re: Sop?

With hindsight, maybe matchsticks were not the best choice of legs….

[ColinPa]

Don't blame me - they virtually banned me

One of the service help desk came round to my desk, and said an important customers was blaming me on a major problem they had.

They had told their management, "we did xyz as expert abc said, and it caused this problem".

I looked into it.

The advice I had given was

If blah blah blah blah then do xyz. They ignored the if statement and did it anyway.

I had words with the salesperson who worked the account, who did some digging.

I offered a good will(free) call with the customer to discuss the problem and solutions.

The technical team had screwed up, and wanted to blame someone else, so they were definitely not keen on a call.

[Doctor Syntax]

Re: Don't blame me - they virtually banned me

Short attention span. By the time you get to the end of what you're saying the first part has already left the building.

[NXM]

Re: Don't blame me - they virtually banned me

Eh? What? Missed the first bit.

[I am David Jones]

Re: Don't blame me - they virtually banned me

Next time try rephrasing: “Don’t do xyz UNLESS blah blah blah”

[technos]

I wasn't even there.

Decades ago I worked for an ISP, and one of our customers reported their connection was occasionally dropping overnight and screwing up their off-site backup. Everything on our side looked fine, outside of there being times of 90% utilization, but they were squeaky enough the boss scheduled me to go over and have a peek at their hardware and look at the backup software.

But the morning of the appointment I get a call from the guy I'm supposed to meet, saying he will be too busy to walk me around. Cool, whatever, next week? Oh, and maybe we can save some time if you email me the logs before then?

The next day get called up on the carpet by my bosses' boss, who asks me to tell him 'my side' of what happened at my site visit. "Uh.. There wasn't one. They called to reschedule."

To say he looked relieved would have been an understatement. See, he'd just gotten off the phone with some executive at the squeaky customer, who claimed I was the cause of an outage that was costing them a shed-load of money.

But since I hadn't been there he pushed back hard, and in the end it shook out that the accusation was a CYA attempt. Their file server, which ran the backup software, had a pair of disks fail. And since he thought I had been there to touch the machine, he thought he could blame it on something I did, and not the fact he'd told his guys that they didn't need to keep RAID spares sitting around because "What are the chances we have two drives fail? Like zero.".

(Oh, and the backup failures? They had a very short timeout and a small number of retries set, figuring there'd be nothing else using the office network at 2am. They were wrong, they had at least two numb-nuts downloading movies and fifty machines grabbing email over POP3 every five minutes.)

[Anonymous Coward]

Re: I wasn't even there.

> they didn't need to keep RAID spares sitting around because "What are the chances we have two drives fail? Like zero.".

A couple of years ago, our network performance monitoring system died. We could see that the ports connecting the servers were up, but the application GUI wasn't responding, so we sent down a PFY into the data center to check it and power cycle the box. Turns out there were THREE red lights on the RAID controller, so it was well and truly gone (as was all historical data and customizations we'd done over several years - and those weren't free, by the way). At which point some smart person said "oh yeah, that one has had red lights on it for weeks. Maybe not three but at least one or two!".

For some reason, the issues weren't picked up in any logs or fault monitoring systems. But you'd think that if you saw a RAID controller with red lights, you'd probably at least ask a colleague "hey, is this anything?"...

[phuzz]

Re: I wasn't even there.

Unless it was a gen 9 HP Proliant server, in which case the red lights might have been the 'do not remove' lights. When there's a disk failure, that light goes out, and different, orange light comes on. (IIRC, this was a few years ago now)

I found this out after swapping out the disk with the red light on it for a failed mirror array. It was even more failed when I pulled the good drive :(

[PB90210]

Re: I wasn't even there.

What do you do when the manual says the status light should be either off or green but currently is a salmon pink colour?

[Red Or Zed]

Re: I wasn't even there.

Consult a decorator and repaint everything else to match.

[imanidiot]

Re: I wasn't even there.

Powerwalk in the other direction and look to power on the SEP field.

[StargateSg7]

Re: I wasn't even there.

I fully UNDERSTAND and utterly SYMPATHIZE with the old adage 2 is 1 and one is ZERO! Always keep TWO EXTRA hard drives fail as they will do so at the most INOPPORTUNE TIME (like TODAY in January 2026!) -- Good thing I have extras lying about! 20 TB drives ain't all that cheap but a failed one, two or even three is even costlier!

V

[NetMage]

Re: I wasn't even there.

Once had a server lose a hard drive from a RAID setup on a Saturday. A second one failed before anyone could respond. Drives manufactured and purchased at the same time.

[David Hicklin]

Re: I wasn't even there.

That's one drawback with RAID5 , your are good if one drive fails but screwed if a second one goes before you can either replace or *rebuild* the failed one.

I used to have RAID5 on my 4 bay Synology NAS but later switched to 2 disk mirrored as I did not need all that space and yes if 2 disks on the same mirror go then oh well time, but at least the other mirrored pair will still be happy and half my data good. All 4 disks would have to fail to destroy everything.

[I could be a dog really]

Re: I wasn't even there.

Drives manufactured and purchased at the same time

Which is a common problem with arrays - if there is a lifetime related issue, or a problem with the manufacturing batch, then it affects all the drives around the same time. For my own use, I like to buy drives separately - partly this is cost, build up the array as I can afford the drives !

[Doctor Syntax]

It's always the DNS...

... until it's the UPS.

[theDeathOfRats]

Re: It's always the DNS...

And sometimes (as mentioned in the interlude in the story) it's the OOOPS!

[Anonymous Coward]

Typical council

I used to work in a local council. I asked why they don't use more open source - e.g. not spending money on a full Photoshop licence just so people could resize images. Their answer was "but if it all goes wrong, who do we sue"?

[ben_s]

Re: Typical council

I don't know about getting sued, and using Photoshop to pivot resize photos does seem like overkill, but the disadvantage of using open source software is that it usually doesn't come with support, so if something does go wrong there's nobody to raise a support ticket with.

That's fine if you're happy to take the risk, but I think decision makers should be aware that there is a risk.

[Doctor Syntax]

Re: Typical council

1. There are companies who specialise in supporting FOSS.

2. Do you get to raise a ticket with a huge S/W vendor unless you have a support contract with them?

[ascii bandit]

Re: Typical council

I made an error report on mercury transmit not working in premiere pro in 2016. As far as i know, the ticket is still open. So having a paid adobe account definetly doesn't help with anything.

... and the woirk around: scaling on all connected monitors must be set to 100% - then everything works, in case there is someone out there still toiling with this issue

[NetMage]

Re: Typical council

Can you find a company that supports GIMP?

[CountCadaver]

Can I find a company that supports GIMP? - well....Yes actually

https://www.openlogic.com/supported-technology?st-title=gimp&field_integration_weight_value=All

[Doctor Syntax]

Re: Typical council

The contra to that line is "What would the costs be if you lose when you sue Adobe?" Although they'd probably ask "Who's Adobe? We use Photoshop."

[spireite]

Re: Typical council

In my experience, there are mostly GIMPs employed in Council offices.

[Anonymous Coward]

Re: Typical council

We had a VP who said at a meeting that outsourcing IT work was good because it gave him someone to sue.

The question I should have asked: suing might get your money back, it doesn’t get you a working system

[H in The Hague]

Re: Typical council

"We had a VP who said at a meeting that outsourcing IT work was good because it gave him someone to sue."

Getting a judgment against an underperforming outsourcer is one thing. Collecting the money from them is an entirely other thing.

[Anonymous Coward]

I was once working slightly late and my colleagues had already gone to the pub. I walked past the server room (glass-fronted, it was from the dot com boom), and could hear the wail of all the UPSs, so I went to investigate. It turned out that an awful lot of the equipment in the server room was running off a (now melted) 13A mains socket - how it had coped until then I don't know. Obviously I didn't get any thanks when I eventually arrived at the pub having hastily rewired the servers to spread the load.

[Anonymous Coward]

IF a 13amp socket had melted, there was clearly a problem with the fuse in the plug!

[excession]

Nah, with the right overcurrent you can melt the back of the plug, especially if it has one of those plastic fuse holders, way before the fuse actually pops.

[that one in the corner]

Look back through the work logs:

September 28: mains plug fuse blew, replaced

October 03: main plug fuse blew again; testing found that machines had been running at a load of 279 and that drew too much power

October 04: CTO issued Change Order: load balancer set to a maximum of 200

October 05: CFO issued Change Order: load balancer set to a maximum of 278 "So we get the most out of all the money we spent on this"; can we set a minimun of 278 as well? Why not?

November 11: New backend processes come on line; waiting job queue fluctuates but is now never empty. CFO happily reports to Board "no money is being wasted on idle CPU"

[Giles C]

He should have taken the Tom Lamb approach

https://youtu.be/9jkcdXwGbkM?si=Rkp5G-oEIoL81GBF&t=600

The 500A slow blow fuse,,,,,

[Anonymous Coward]

Dangerous muppet

Good way to risk a fatal electric shock, melt the extension lead to its reel and lastly start an electrical fire....

That youtuber is an idiot who should be kept WELL away from tools and as far as possible from structural work....

This is why health and safety rules in the UK USED to be so intensely strict and fiercely enforced....the "bonfire of red tape" and growing attendant unchecked idiocy as shown in this video will likely lead to SE Asia style building and bridge collapses....

[Francis Boyle]

Compete amateur. Everyone know that you're supposed to replace blown fuses with live rounds. The way he's going the only person he'll kill is himself and what's a Darwin Award worth these days?

[StargateSg7]

So MANY server rooms, or even local council or office workspaces where a backroom closet is used as the so-called "Server Room", no-one has done ANY power supply and voltage/amperage calculations because it seems they no longer teach Electronics in upper forms (or High School where I live!) and that means people have NO IDEA you just cannot willy-nilly plug too many items into a household or workplace electrical socket system.

In Europe, with the 230 volts at 13 amps used for most home and and workplace plugs, your maximum continuous Mains wattage is only 2,990 watts per circuit and in 2026 where we now have personal computers or servers that have 1200 watt power supplies along with big inkjets and laser printers, plus WiFi and LAN routers and switches and any local lights or other systems plugged in, each circuit will be QUITE OVERLOADED by the time all is said and done!

No one has thought of spreading out the load to OTHER independent circuits as MANY PEOPLE simply plug everything into a single big powerbar and EXPECT everything to work just fine and dandy!

The EASIEST way to figure out power draw is to LOOK at your breaker panel and see which ones are labelled as truly independent from each other and use that calculation of Watts = Volts and Amps and ADD UP the power usage of all your computers, LAN routers and switched, peripherals, displays, lights, etc which is labelled on a sticker located at the back or underneath your devices and systems, and then see if one single group of them EXCEED the rated value of that circuit! For safety reasons, you SHOULD ONLY USE 80% of the actual available wattage in each circuit, so for MOST European sockets, 2,392 Watts is the MAXIMUM SAFE AMOUNT of electrical load you should be putting on each electrical circuit.

For North America, our much-lower wattage plugs running at 120 volts at 15 amps sockets gives us 1800 watts which REQUIRES that we only use 80% of that available wattage which lets us SAFELY draw 1440 watts for a single circuit!

That is just ONE single computer with a printer and a wifi-router attached to it! Anything else added REQUIRES that we plug it into a DIFFERENT fully-independent circuit that is NOT simply strung off another plug! It MUST GO DIRECTLY to the mains electrical box and cannot be not connected or bridged to any other circuit!

V

[CountCadaver]

wrong, wrong wrong and....wrong

Various Areas you are wrong frankly (coming from someone who held BS7671 qualifications and was conversant with CENELEC regulations)

Apart from the UK most of europe's 3 pin plugs are rated at 16Amps

I've seen equipment racks that are hardwired or are supplied via 32/63/125 (or higher) blue industrial plugs or red 3 phase (400/415v in similar current ratings)

PCs generally have an initial current spike on startup which drops down rapidly, circuit design should include the principle of diversity (electrical diversity before some right winger foams at the mouth) where the cable size for the circuit is calculated based on a realistic sustained loading

UK socket circuits are wired as rings not radials, with 2 x 2.5mm2 cables forming a ring and where the circuit is protected by a 32Amp type B breaker

Wrong on the 80% - that is a bizarre Americanism, likely due to your archaic wiring standards and equipment (seriously instead of shrouding in non conductive material and recessing screwheads to prevent accidental contact, the American approach is just to wrap the whole thing in insulating tape) (Also example unlike most of the world you don't have different "trip curve" breakers to handle items with varying inrush current - domestic is generally on Type B (3-5X rated current to instantly trip out), Commercial and IT with large inrush current demands on startup is type C (5-10x rated current to instantly trip - mitre saws for example can and do trip 16A type B breakers but generally are fine on 16A type C) and heavy industrial gear is run on type D (10-20x rated current for instant trip).

To reiterate: The 80% rule does NOT apply in the EU (or in fact most of the world)

[Anonymous John]

13 amps is {approximately} the maximum that won't cause the fuse to overheat and damage the plug. It takes about 50 amps to blow the fuse instantly. 21 amps for 30 minutes.

[mtp]

The unpredictability of fuses

Watch this and you will never think about fuses the same way.

https://www.youtube.com/watch?v=WG11rVcMOnY

[Jimjam3]

Re: The unpredictability of fuses

Video not available anymore (UK)

[Glenturret Single Malt]

Re: The unpredictability of fuses

You have to sit through the ads before the actual video starts (UK).

[Roland6]

Re: The unpredictability of fuses

Video is available again.

[Will Godfrey]

What fuse? There this thing that keeps failing so we wrapped it in silver foil, and it never fails now.

[Anonymous Coward]

My parents moved into a new house many years ago. I did a routine check of the fuseboard, and found that indeed the correct rating of fuse wire had been used in each holder.

Unfortunately they clearly wanted to make a good connection, so they had wound several turns (of the right gauge wire) in parallel between the screw connections for each fuse...

[Electronics'R'Us]

Unusual fuses

When I was fixing two way radios (and other stuff) in Florida, we also had the contract with most of the local police for their blue lights and sirens as well as the radios.

One day, a deputy comes in and tells us the blue lights on his cruiser need fixing as they were blowing fuses and he put a temporary measure in place in the fuse holder which was the clamp type.

The temporary measure was a bullet.

[The Oncoming Scorn]

Re: Unusual fuses

Thats a typical Merican approach to problem solving.

[Kimo]

Re: Unusual fuses

The Audible Warning fuse.

[CountCadaver]

Re: Unusual fuses

Why did that seem totally unsurprising to happen in Florida???

[NXM]

13A plugs

Not always true. Kettles and heaters quite often have overheated plugs because said plug can't always handle 13A for extended periods of time. Sometimes it's the fuse holder, whose clips don't hold the fuse properly and warm up. Or as Anonymous John points out, the fuse itself warms up because it has to have a resistance, otherwise it wouldn't ever blow.

I dislike those bits of card you get on new plugs telling how to wire it (from the days when you didn't get a plug on new kit and had to put one on yourself, which many people were incapable of doing properly). If the plug overheats, the card catches fire.

[Number6]

Re: 13A plugs

One of the common causes of overheating is dirty contacts. That bumps up the contact resistance, which generates a bit of heat and a bit more oxidation, which bumps up the contact resistance, which...

I suspect part of the problem there is that most of us plug the kettle into the wall socket and never unplug it, so the wiping action of inserting the plug never has a chance to clean off the crud.

[jdiebdhidbsusbvwbsidnsoskebid]

Re: 13A plugs

"One of the common causes of overheating is dirty contacts. "

Had that before, with a bit of kit at home that runs via a 15A fuse. One day the fuse holder overheated and blew. The fuse was fine but it was the contacts that overheated and melted. The replacement (identical design) fuse holder is fine, so I put it down to dirt in the fuse holder. It's now protected a bit better from dirt and dust.

[CountCadaver]

Re: 13A plugs

you are meant to remove the cardboard and chuck it out....

[Jon 37]

In an act of massive stupidity, a dual 13A socket only has to be designed for a TOTAL load of 13A, and can have a 32A fuse at the consumer unit.

Good brands will exceed the legal minimum and design for 26A total, which is 2x13A.

(Disclaimer: I found this out a decade ago. Regs may have been fixed in the meantime).

[CountCadaver]

actually for a double socket its 20A total (14A on one socket and 6 on the other) I'm aware MK rate theirs to 26A though

[Zakspade]

Tried to remote onto a server. Failed. Turned out a laptop user in the office unplugged it to plug in their charger... (it was a small office server).

They powered it back on for me, but the dirty shutdown (and the on/off process having been going on for weeks as the user unplugged the server every time they were in the office) had the server come up once more, but unable to read the disks. The RAID controller had failed (I seem to recall).

Cue booking engineer to attend to fix AND perform the restore once fixed.

Turned out the backup device never came back on once the server was initially turned of by the laptop user in the first place - so the server had never been backed up since the day they first charged their laptop from that outlet.

The office (and company) made a formal complaint against me for wrecking their system.

Oh how we all laughed - but it still required me to undergo the formal process - despite everyone taking part - HR included - knowing it was a crock...

[Bebu sa Ware]

Machine Room Video

A good reason for installing video camera(s) in your machine room and keeping several days' recording.

Surprising how many people have access that ought not — bloody cleaners with skeleton keys for one.

Keeps people honest and points the finger of suspicion more equitably.

A simple setup on an isolated network with a cheap Chinese cam (ethernet only) connected to a [*ix] box running VLC is cheap enough.

[PRR]

Re: Machine Room Video

> camera(s) in your machine room

Or a "game camera". We strap them to a tree to log deer in the woods. Or kids vandalizing signs. Really fancy ones (color, cellphone access) have come down below $70. They run on batteries for months (5 months on 8 AAs once when I was monitoring vandals). A memory card holds thousands of still sightings or many video clips. They all log time/date.

[GlenP]

Surprising how many people have access that ought not

When I started at the current employer I discovered that the key to the supposedly secure machine cupboard was in an unlocked keybox in one of the offices that anyone could access. I couldn't even secure the cabinet as access was so tight the sides of it had bene removed.

I solved this later when an office move gave me a former strongroom, complete with reinforced walls and ceiling and a Chubb safe door! There were only two keys, I carried one and the other was kept in a lockbox in my office with only a couple of trusted senior people knowing the code.

[Unoriginal Handle]

I worked for a ( back then ) large American manufacturer of networking kit, but on the network management side of the business, supporting said NMS. We had a lab with all sorts of kit, and a key on a small fob which we used to control access - otherwise kit would get robbed for customers despite use needing it.

The problem of the key going missing was resolved once the 6"x8" lab sign fell off the door, and was split pinned to the key. Never failed to make its way back to us reasonably quickly after that.

[Philip Storry]

This brings back bad memories

I don't recall ever being banned from a site, but I do remember a colleague of mine getting very close.

I was working as the head of the support & implementation team at a small company that did software development and consultancy. We had a client we'd recently migrated to a new mail system. They then reported a problem.

I was working on another project so I sent my colleague John* to fix it. (This was in the early 2000s before remote support was commonplace.) After some basic checks John was very confused by what he saw, called the office to confirm that the system had the wrong settings, and then fixed it.

A day later, the exact same issue. I considered sending someone else, but I figured I'd send John again as he now had some familiarity with the site. When he arrived they were unhappy, but he cracked on and found... the exact same issue. We spoke briefly on the phone, and confirmed through date/time stamps on the configuration that this had been changed, and that only certain admin accounts had access to change it. John fixed the issue, reported all of this to the client, then left.

The next day they had the exact same issue. The client was livid. They were also demanding we send someone "more competent" than John. I happened to have finished my other work so I went onsite to give a show of "seniority", in the hopes it would calm them.

Same issue - the configuration had been changed manually. Before leaving the site I asked if they had a CD burner. They did, so I created copies of the configuration and burnt them to two CDs - one for us to keep, one for the client. This was so that we could prove a change was definitely happening.

Whilst I'd been on site poor John had been back at the office in a rather awkward meeting with one of our Directors about his performance. This was John's first job, and he'd not long been with the company, so this was not a fun experience for him. He was very worried due to all of this, fearing he'd have a career outlived by most mayflies. I reassured him and made sure to speak to the director the next day, making it clear I trusted John completely and didn't trust the client.

Two days later, the client has yet another failure. I went out to the client, as they were still unhappy with John. It was the exact same configuration change. But this time I invited their IT manager to compare the backup that they had with the current configuration, and showed the date/time stamps on the current configuration. I made it clear that someone with admin access to their environment was making this change and sabotaging their system. With no other explanation available, he accepted that this. I fixed the problem, took another backup for evidence in the future, and went home.

A few days later our director spoke to me about this. After some internal investigation the client found that one of their technicians didn't like the new system, and had been sabotaging it in the hopes that they'd roll back to the old one.

I made sure that John got an apology from the director, but was less than happy that they'd not contacted myself or John to apologise.

When the same client had some deployment work a few weeks later, I allocated the visit to John. He didn't want to go, but I told him that I expected them to apologise to him in person, and to be more professional with him from now on. If that didn't happen, I'd never send him there again and would tell our management that this client wasn't worth the hassle and we should not renew their contract when it expired. To the client's credit, they apologised to him and treated him well.

Some clients simply aren't worth the hassle that they bring you. Being banned or unwelcome at a site is one of the (many) danger signs.

Looking back on it, I'm mostly happy with how it went. I should have asked John to take a backup after his second visit, but didn't know that they had a CD burner at that point. We did change our standard operating procedures after that, making sure everyone took spare CD-Rs onsite and offered to create a backup of configs before leaving.

--------

* Name changed, of course.

[Kevin Johnston]

Naming co-incidence

I got the blame all the way up to an HR admonishment because I was working on something which had the same name as a process in the automated account creation which suddenly started failing.

I tried pointing out that what I was doing had no link and relied on people logging into a specific system before it even took any notice of them (so well after account creation had happened) but I was not flavour of the month/year with either my manager or his and they needed a scapegoat urgently.

I took it on myself to checked the process which was failing where the cause was easily identifiable and presented all this to everyone involved. Two weeks later they finally agreed that I was correct and my work had been totally innocent but strangely I got no apology from either manager and my HR black mark remained...

[Anonymous Coward]

Re: Naming co-incidence

and my HR black mark remained

This ! HR types can be quite to throw you under the policy bus when it suits them, but strangely reticent to fix things when they are at fault. Me, bitter ? What makes you think that ?

[ComicalEngineer]

I got into the habit of recording whatever I did on my phone after a colleague got wrongly accused of breaking a rack server unit. Turned out that the rack bolts hadn't been inserted properly and the weight of the server plus assorted gubbins caused a physical crash when a bolt came out of the rack during some routine cabling changes.

I went into one place and found that their UPS was well undersized for the duty and would have lasted about 10 minutes instead of the required and specified hour and that in the event of a power loss this would probably cause a major data loss issue. As the bearer of bad news I was told to leave the site - until their senior engineer pointed out that I was doing a job on site that was mission critical and involved a complex iterative calculation which took between 40 minutes and 2 hours to run. The following day a new UPS was delivered, nonetheless a certain manager was unhappy with me especially with the cost of having to install a new UPS suitable for the job.

[Electronics'R'Us]

Been there

Some time ago (decades now) I was writing test / diagnostics software for smart payphones (basically a microcontroller with supporting bits that acted like a payphone).

In this case, I had written the code running on the test machine and the target hardware.

One day, the director of operations came over to my desk screaming that my tests had all gone wrong and we had 100% failures at test at the build house (who had one of the testing systems).

I went to said build house (not a short journey) and looked at the failures and they were all failing one specific test: Escrow relay. For those who may not know, this is the beast that will either take or refund your money. In this specific design, it was powered by a static inverter that generated 120VDC and dumped the energy into the relay when required.

I took a few of these failed units back to the office and the first thing the director of engineering said was 'test it on <other engineer>'s test setup'. Those in hardware will know this is hardly a scientific test, but the relay did operate (although to me it seemed sluggish). <other engineer> declared the unit was fine.

I took said board to a test bench and hooked up a scope to the drive circuit and fired the circuit into a known load. Rather than seeing the expected 120V peak and discharge curve [1], the signal went up to around 80V and abruptly dropped back to 40V from where the normal discharge curve was observed.

A close inspection of a couple of capacitors (which were carefully desoldered) showed they were delaminated and were shorting out at the above 80V. That meant this was a build issue. While I was doing those tests, the director of engineering, director of operations and <other engineer> were watching and once they saw what I had found they were rather red faced and left without another word.

I talked to the build house and they found the build process was heating up that area of the board to a temperature well above what it should get to, so the problem was solved.

In this case, my tests were working just fine.

Note 1. I was sampling this signal multiple times in this test to capture the energy under the curve.

[abalamahalamatandra]

Post Tenebras Lux

Post hoc ergo propter hoc!

[Anonymous Coward]

Re: Post Tenebras Lux

Romanes eunt domus

[X5-332960073452]

Re: Post Tenebras Lux

Romani ite domum

[jake]

Re: Post Tenebras Lux

Ramones via ad ruinam

[Rtbcomp]

I worked on a mainframe site where the noisy printers were in a room of their own rather than the computer room. At random times the machine crashed, this went on for a couple of weeks and the customer was blaming every one of us for not fixing the fault. We eventually noticed that after each crash a particular operator left the printer room.

It turned out he was deliberately tripping the circuit breaker on a printer during a print run to cause a crash. Nobody apologised to us and the operator was asked to resign.

[phuzz]

I have, more than once, gone to move a cable, and inadvertently brushed a C13 power cable which had been just making contact, and disconnected it. I have a feeling that years of vibration from fans might cause them to slowly back out. Often, you don't know anything has happened until someone comes running it to the server room wondering what you've just done.

I try and insist on the C13/C14 cables with latches on now. (eg). Equally, RJ45 cables that have lost their locking tab get thrown in the bin.

[Anonymous Coward]

Old man dodged a nice one.

My own father, being a Civil Engineer, dodged an Iowa-Class 16 inch bullet.

He was asked to do a look over on a Dam that was being traded, its purpose being power generation, river flow control, and maybe uprate the power generation by the buyers.

He looked it over, and testified on his paper "this and this needs to happen, or the whole thing collapses and floods a town below it on the next hard rain, potentially killing 30 thousand people in the flood."

The guys selling tried to "rea$on" with him to change the report.

He refused and stuck with his guns, so to speak. He digitally signed the PDF and delivered the written warning.

3 months later, he got the news that it had happened, the dam ruptured and flooded the town, from a reliable colleague that didn't agree with any trans$action either, and they TRIED to blame him anyway, but he had not just digitally signed the paper, he also notarized it, er, verfied signature by a notary public, if google translation is working properly.

Not IT related, but an attempt to shift blame was made.

[azander]

Workig for a Dialup ISP in the 90's...

I was tech support for a Dialup ISP. Often times I would be dispatched to a home or business to find out why people were getting poor connections.

I was careful to use a modem that I knew worked well on a good line. Every single time I would get a 53K connection. Since I was dialing in to the same modem bank myself;l that everyone else did I usually knew when we had a problem long before anyone else noticed. I could tell what kind of bad line someone had by the speed in which that modem would connect. Then when I go to their location I could see using that device if it was the phone lines (call teleco) or their modem.

I went to a local Bowling alley and Motel combo. I walk in and the place is dead. Nobody around. No customers and no staff. I wait at the counter, in full view of a camera (later learned it was a fake). I unpacked my laptop, and was unpacking all the cables. I had not attempted to get behind the counter or anything like that and as far as I knew at the time was in full view of their camera. A guy comes out from the back and asks who I am. I explain I am from the ISP they used and was here to look at their connection issue they reported as chronic. He proceeded to tell me to get out that they won't have someone telling them it is the hardware when they know it is the internet provider. I smile, mentally shrugging, and start packing up my laptop and cables. Not even a minute later he is asking me what I am still doing there and I replay, nicely, that I am packing up my equipment and leaving like he ordered. He tells me to get out now or he is calling the police. I smile again and tell him that he can go ahead and call the police, I'll wait outside on my car for them. He clearly is nuts but I am trying to be polite. The police show up and tell me I am to leave, that I am trespassed from the premise and not to return. I thank the officer and file a counter claim of not being able to pack up my equipment safely before being thrown out (laptop value was over $3000 at the time). I my be trespassed but the business owner is fined $1000 (obscure law in the city at the time) and now has a public business violation on the 'public' books.

I inform my boss at the ISP the next morning. He tells me not to worry about it. I later find out from one of that business' employees later that my boss called them, terminated the contract with prejudice (aka no refund) and added them to the black list and added the actual owner''s name to the list as well. Preventing his home from getting the local access. We were, at the time, the only local ISP. The next ISP was long distance, into another state at the premium Long Distance fees of the time.

I really wish my boss, the owner, hadn't decided to sell out in 2000. He was wonderful to work for, and he took care of his employees.

Az

[Taliesinawen]

Re: Workig for a Dialup ISP in the 90's...

Is this an A.I generated hallucination :o

[theDeathOfRats]

Re: Workig for a Dialup ISP in the 90's...

A couple of grammatical and ortographic errors. Soo... Maybe not?

And, then again, maybe yes if the poster is a chatbot prompt warrior.

The story itself is not too nuts. If you leave some USofA specific details apart... It might be one of me own tales :p

[theDeathOfRats]

Re: Workig for a Dialup ISP in the 90's...

Too late to edit, so...

Going by his previous posts, I'd go with NOT A CHATBOT.

[Doctor Syntax]

Re: Workig for a Dialup ISP in the 90's...

He may not have many posts but his handle goes back way beyond the start of all this nonsense.

[KarMann]

Re: Workig for a Dialup ISP in the 90's...

ExecPC, by any chance? Sounds about right, especially with the selling to Voyager.net in 2000.

[Rob F]

Guilty by proximity

I once was doing some work in a co-lo datacentre where I was putting a NetApp VMware Lenovo system into a couple of racks and just as I was leaving I got a "What did you do?" phone call from my MSP manager because a client's systems had crashed hard whilst I was in the same room. I had to prove that a) that client was in a rack that was in a completely different row and b) I didn't even have the keys to those racks on me. Time has robbed me of the specifics, but I wasn't sure at the time if I had actually caused the issue. Never got an apology for the accusation stressing me out big time.

[Rob F]

A famous story from a previous company I worked for

One of the Systems Engineers was a bit of a liability at an MSP I worked for. He was asked to do some work at a customer site and for reasons never clarified he DCPROMO'd the only working Active Directory Domain controller and then, unsurprisingly, everything stopped working.

We thankfully had well implemented backups of the system state and AD and were able to recover from last nights backup.

The customer asked that the System Engineer in question never worked for them again.

[Doctor Syntax]

My experience was not being allowed to leave. I'd installed some reporting stuff I'd written on site in Italy and the system ?started crashing with bits of memory content being left in a lost&found directory. The software worked fine on my client's system back at base and I'd no information as to whether there were crashes before. They didn't want me to leave until it was working and (a) I was booked to leave that evening (b) I was running out of Lira and (c) I had a meeting lined up with a potential new client in England on the following Monday. OTOH it looked like a H/W problem. Fortunately they had a good run & I escaped. I later heard it was indeed a faulty memory module. Quite possibly the extra job just pushed the memory usage up to reach the area with the fault in it which hadn't been reached previously.

[Anonymous Coward]

Some years ago I visited a customer at a power station that we sold some video kit to. It was a video capture card that plugged into their PC and they were complaining that it was slow.

Rebooted the PC and it came up with an error message something like "SMART has detected an error with the hard disk drive, backup before you lose data"

Their response was "Oh, it's always does that!"

Spoiler alert - tried a different PC - worked fine.

Couldn't help but wonder if they did the same with the generators...

[bigtreeman]

Council fall guy

A not so small town council mid-north-coast-NSW. We had the contract to supply an upgrade of workstations pre-Y2K ( a major contract in our town).

Our workstations started failing, I was called in and found their "techs" randomly stripping machines, trying to find the problem.

I found out of spec pci video card edge connectors which caused random bad connections and failures, especially on machines roughly handled.

My fix was replace all video cards. The council canned our contract and gave it to a competing supplier.

Small town, the story came out later that the council head tech was mates of the other supplier and pushed for the change of contract.

[Anonymous Coward]

I run my own business and am the only support provider in my state for a particular s/w package.

I got a job for a new install & training. All went well until, during the training, I turned the client's screen so they could see better. Apparently, this pulled on the video cable and the screen blanked.

I had a look and the PC (on the floor at the side of the desk) had been pushed back against the wall too many times, resulting in the connector on the cable being bent/broken. I didn't have a spare on me, but they said they'd get one from their regular IT support company. So, job all done, I left.

A couple of weeks later they sent me a copy of an invoice from their IT supplier for the cable and a new video card, demanding I pay it! When queried, they said I caused the damage when I "pulled" the cable. It's obviously not for much, but out of principle I refused to pay.

I called their IT Guy and found out the new video card was actually for a different PC!!

So, I called the client and told them I would no longer provide support. In future, they would have to pay to fly in a support person in from interstate.

They tried to "back-peddle", but I don't want that sort of client, so I held my ground.

I'm guessing it ended up costing them a lot more in the long run ;)