BOFH: Nothing says 'business continuity' like a dry wooden broom

Official BOFH logo (red rotary telephone with devil's tal and devil's horns) BOFH logo EPISODE 9 It's 3:30 am and I'm at work, having been woken up by numerous outage notifications. The Boss – as useful as Jason Statham's method acting coach – is also on site, presumably to offer moral support. The Building Manager – who's so …

[Hot Diggity]

Outages

I was patching a server in a restricted room that contained connections to servers on the DMZ late one Friday night. I rebooted the server and it didn't come back.

Suddenly the whole floor was alive with other staff turning up which seemed odd for just one server in the DMZ being offline.

Then I found that the cable supplying electricity to the main data centre had been hit by lightning and the DC was now completely offline. The staff were turning up to move everything to the DR site.

It had been a nasty 20 minutes though thinking that it was something I had done

[Neil Barnes]

Re: Outages

What? You can't call yourself an engineer if you can't summon the lightning!

[Caver_Dave]

Re: Outages

I've been hit directly once and indirectly once - I call myself an Engineer

[The Oncoming Scorn]

Re: Outages

I call you a lightning rod*

*If you meant you've been stood on mountaintops during thunderstorms in wet copper armour shouting "All the Gods are bastards'".

[Phil O'Sophical]

Re: Outages

Kill the wabbitt....

[Tim99]

Re: Outages

"Well, what did you expect ... A happy ending?

[Inventor of the Marmite Laser]

Re: Outages

Uovited for the Sir Pterry reference.

[Potty Professor]

Re: Outages

Christmas Eve 1976 we had an enormous thunderstorm. We had only recently moved into our 1840's cottage and were settling down to watch some variety show on the (B&W) TV when there was an enormous flash and a deafening bang, and the house was plunged into darkness. I looked out of the window and saw that the whole village was similarly dark, so we went to bed. Next morning (Christmas Day), we awoke to find still no power. I checked our main fuse, still OK, no power on the incomer. Went outside to see the pole transformer hanging drunkenly from the top of its pole with half its guts dangling outside the can, it had presumably been struck by lightning. We decided that it would probably take a considerable time to get fixed, so we decamped to the In-Laws' back in town for the rest of the festivities. The replacement took four days to arrive, during which the village was dark.

[Caver_Dave]

Re: Outages

Caves give off a column of negative ions and walking out of one the lightening hit the rolled up metal ladder hanging from my belt, went down the wet seam of my wetsuit leg and into the ground. The seam vaporized and left a relative small burn on the skin, and I threw away the ladder just in case. The only lasting issue was about 6 months of deafness in the ear on that side.

My parents live at the top of a large plateau style hill (used as a WW II bomber airfield). We heard thunder and I was crouched over unplugging the TV aerial when the lightening struck (about 3" out at the time). My legs spasmed and I hit the corner of the ceiling and wall 14 feet horizontally away and dropped down onto the sofa next to my mother. I think that she was more shocked than I was.

[CrazyOldCatMan]

Re: Outages

Many orks ago I worked for a borged company that was migrating their (fairly small) computer room into a larger purpose-built one, along with all the servers from the borgee company.

I'd done the thermal and power load calculations and mentioned that the 3 aircon units [1] would be running full-time just to keep up. And the UPS was no-where near the capacity required (could do either companies stuff, just not both).

I was ignored. Oh well, I'd got my CYA material so just got on with it.

We moved our stuff in, life was good. Borgee-company started moving their stuff in. About 1/3 of the way through, the UPS tripped due to too much load. So I was told to run it on bypass for the moment (to get the migration done as we were giving up the lease on our old building) and they'd look at upgrading it afterwards.

Ho hum. Again, stored carefully in my CYA material store. We finished migrating and had a few days of uptime.

Then one of the PFYs noticed that one of the aircon units was dribbling water onto one of the racks. We couldn't shut it down (thermal capacity didn't allow that) so did a Heath Robinson duct with plastic sheeting and duct tape to carry the water away from the rack - forgetting that duct tape isn't particularly waterproof and, if soaked, ceases to be sticky.. And, after a week or so, promptly came unstuck.

So we went round, shut down every server that wasn't business-critical and got the aircon serviced at eye-wateringly 'emergency callout' rates (because, yes, they hadn't bothered to get a maintenance contract in place..)

Then, one day, the building had a complete power outage. And, the UPS, being in bypass mode, did nothing other than beep. And the room got very, very quiet (other than the UPS beeping..)

Needless to say, every server was set to 'resume previous power state' so, when the power came back on, every server tried to come back up at the same time.

UPS gives up in disgust as the oversurge protection kicks in.

Eventually we get everything back (manually power off each rack suppy board, reboot UPS, turn each rack back on one by one) and the IT Director starts looking at upgrading the UPS. Which can't be upgraded because they decided to buy a smaller unit and exapnded it to the max. So the whole UPS has to be replaced.

I left there a couple of months later.

[1] Original idea was that we could run on two units with the third on standby to be used when one of the others needed maintenance. Not only were all the coolant pipes running directly along the top of the racks rather than across them but the units themselves were not powerful enough.

[CountCadaver]

Re: Outages

Buy better quality duct/gaffa tape and it's then both waterproof and stays stuck

[Tim99]

Re: Outages

They broke a basic law of the technical universe*. Never skimp on duct/gaffer tape - It's like second-hand-discount-store fire extinguisher, If you need it, it has got to do the job, otherwise you are better off without one. You will waste time trying to get it to work, instead of just evacuating the building.

*If it's not meant to move, and it does, apply sufficient duct tape. If it's meant to move, and doesn't, apply WD40.

[phuzz]

Re: Outages

Zip-ties have their place too, and they're water proof.

(To be clear, their place is NOT attaching network cables to a rack, and definitely not if you're going to cut the tail off at an angle, leaving me a sharp edge to cut my hands on)

[Democrab]

Re: Outages

The trick is to not just cut the tail off at one angle, but two so that you create a V-shaped spike and ensure any techs working on the rack know that A) you've been there and B) you have a firm and unshakable hatred of humankind.

[swm]

Re: Outages

When I was working for Xerox in research a major transformer that powered the building blew up. The transformer was in a room opposite to the guarded entry. When asked, the guard said, "oh yes, there was a hell of a bang last night."

"Did you report it?" "No, I thought it was someone else's problem."

When they finally installed a new transformer, they did it live. Two men stood with crossed 2x4's in front of the transformer and braced themselves. A third man reached over the 2x4's to connect the transformer. The job of the two men with 2x4's was to pull the third man off of the transformer/wires if anything went wrong. Everyone was very serious. Fortunately, all went well.

[DS999]

Early in my career I used to not understand this sort of thing

But once you're in the situation of making a budget and/or contributing to the making of a budget, it all makes sense. The managers above have your budget on theirs but higher level managers have bonuses based on PLANNED budgets (i.e. minimum growth over previous budget, bigger bonus if it can be cut, etc.) so preventative stuff gets cut. They know it will cause problems later, but they'll have made their bonuses.

Now sure there will also be bonuses based on actual spending, but often the idea of "who could have expected this would happen, we had no choice but to spend all this money to keep things running" allows those items to be EXCLUDED so they may still make their bonus based on actual spending despite going over their original budget. Spending in an emergency "doesn't count", so which encourages emergencies and is why you'll see unrelated spending happen if they can get away with it during said emergency.

This is what happens when managers get evaluated based on numbers that are disconnected from the actual reality of keeping things running, especially when those "numbers" are budgets that spend or don't spend the money required to keep things running.

[azander]

Re: Outages

About a decade ago I was working for a small ISP. We had rack space in a big data center outside of Detroit, and I was about 2.5 to 3 hours away.

The power poles just outside the datacenter were struck, several times in fairly rapid succession, with lightning. The system did what it was supposed to do and fail over to the generator. That went mostly well. According to the after action report one server in the entire building failed. The failover triggered an alarm that meant I had to make my way to the datacenter, just in case.

When I started out, things were running, but anything noncritical was in the process of self-shutting down while on battery and/or generator power. About 2 hours later (I pushed the limits), at 2:45am, I get there to a completely dark data center. Generator not running, Fire department standign there looking nervous with special attachments connected to their hoses all watching the obviously smoldering generator. Seems that during my drive, the generator was struck several times by lightning as well. The after action report showed that the lightning rods on the generator melted during the strikes and that a fuel line leaked. It was 4 hours waiting for the all clear to go in and see what the damage was.

We lost 2 out of our 10 servers. Backups were "live tested" during the next few days restoreing systems. The plan was to be there in case of problems, and the plan worked as intended. The backups were good. But it was a long 4 days getting things restored for everyone.

The datacenter electrical system? Total meltdown all the way to the big secondary breakers at the top of each of the racks. That took most of the 4 days to replace. They were lucky, they had just purchased and received new ones for an expansion into the building next door.

[Xalran]

Re: Outages

That reminds me of a story I heard (I know it happened and was not embelished, as I spent some time on that site later and saw the telltales) in my previous $ORK.

One of the main public exchange building in a well known wine area got stuck by lightning....

Public exchange ( especially the ones that were PSTN... but since mobile was in it's infancy they were the most ubquitous ones. ) at that time were beating when it came to floor space occupancy.

So basically one exchange that served 60K+ landlines got a on target hit by lighting... Obviously it stopped delivering calls... especially since the massive amount of (acid/lead) batteries in the basement just boiled (which should have been good for a few hours of uptime), the backup generator fried before even starting to show a sign of life... It took almost a day for another generator to be hooked on a brand new power line (because that fried too) so that the process of assessing the damage (curiously only a few non critical board had fried) and starting the hour long process to bring back the exchange to life.

(at that time such an endeavour was betweek 6 and 10 hours from the time power was restored to the time a call could be handled... not counting the time to replace all the fried boards in the cabinets.)

[Benegesserict Cumbersomberbatch]

I nominate "betweek" as word of the month.

[Ian Johnston]

Re: Outages

Before I got FTTP at home, I kept a supply of configured routers because about twice a year the local telephone exchange ("local" as in "in the village two miles away") got hit by lightning and fried my router. Luckily Netgear DG834Gs were about a fiver a pop n eBay.

[phuzz]

Re: Outages

I got through three noname PCI modem cards, all fried by lightning, but they all did a great job of taking the hit and leaving the PC just fine.

[Xalran]

Re: Outages

I (ab)used a pair of DG834 and DG834g (the difference is easy : no wifi when there's no g) for as long as I had an ADSL connection.

And every time I had to call the *customer support drones* the first question they were asking was :

what lights are on on your box ?

My answer was :

I don't have a triple play box... I have a router.

This inevitably led to a minute of blank while the *customer support drone* performed a hard reset, shrugged off and asked me the next question in his list.

It was always fun, since everytime I was telling them exactly what was not working in the network (in technical terms that sadly the *customer support drone* hasn't been trained to parse as : this require a person more competent then me )...

After some time the *customer support drone* finally blanked out completely and just handed me off to a techincal guy that took a few seconds to concur with my analysis of the issue and told me they were going to work on it.

[Elongated Muskrat]

Re: Outages

I can recall a similar call some years back to $telco to request some replacement hardware after a nearby lightning strike (the induced ground current in the coax line had neatly blown up just the TV box, and not the router, both attached to the same cable via a splitter, I'm guessing that was the low-impedance route).

"No, there aren't any lights on on it, it has been struck by lightning. There was a flash and a loud bang outside, and it stopped working, now it rattles if I pick it up and shake it."

[TSM]

Re: Outages

I had a difficult time trying to get my ISP to accept that I needed a replacement router when the original one died. Support were adamant that I had to try every possible troubleshooting alternative to eliminate any chance that the problem was with the line or the computers. I never did get an explanation of how, for example, the line being down would result in the power LED not coming on when the router was switched on (for some odd and apparently inexplicable to them reason, none of the other LEDs came on either). We had to get a telco person to come out and physically verify that yes, the line was live and reachable from the port we were using before the ISP would acknowledge that the problem possibly could be with the router. And even then I had to argue with them to get a new one sent out free of charge - although they were happily offering one to any new customers. I was quite happy to commit to the 2 year contract, having been with them for 10.5 years at this point, and until then not having had any real complaints.

Sadly during that period the quality of their tech support had fallen off a cliff after being acquired, but in general they've been pretty good and I haven't needed to call technical support very often.

That was my worst episode with them until I decided to try their 5G service, which they quite aggressively pushed when it (supposedly - I can't say for sure either way) became available at my address. They advertised a no-cost return to (my then) ADSL connection within 14 days if I wasn't happy with the 5G service so I figured I might as well give it a go. I filled in the appropriate forms and in due course the box arrived, so I tried setting it up. Followed the instructions carefully, my devices could see the router but were not getting any internet. Called support and did troubleshooting. Turned out the box was seeing absolutely no 5G signal anywhere in my house. I don't know whether this was because we in fact didn't have any 5G service, or the box was broken. (Support of course did not entertain the second possibility.) OK, this is not going to work, no problem, switch me back to the ADSL plan please. Oh, support can't do that, I have to call customer service in business hours. Of course this was on a Friday night, so wait until after the weekend. (I should note that our ADSL connection was still working - we weren't deprived of internet access except when we had it turned off to try to configure and/or troubleshoot the 5G rig.)

Customer support were not keen on the idea (I don't remember all the details) but eventually were convinced that this step was necessary. Then they informed me that I would be charged for both the 5G service and the ADSL service for the month. I was not happy about this and asked about the 14 day thing. Oh, that's expired, they told me. How could it be expired when I'd only received the kit a few days earlier? Well, the timer started running when I filled out the form on the website. So almost all of the 14 day trial period had expired by the time the kit actually arrived at my house. After some further expressions of, shall we say, rather pointed (but still politely expressed) unhappiness, they agreed that I would not be charged for the 5G service and all was well. Or nearly all, because even after repeated prompting they never did manage to send me return instructions for the 5G box, so I still have it sitting on the top of my desk. I should probably chuck it out one of these days, but I think my subconscious is convinced that the instant I do, they'll suddenly notice and charge me for it.

[Anonymous Coward]

Re: Outages

In the very early days of such endeavours, a watery company in the east of the country had a small desktop computer (ISC3651) doing a spot of data logging. Logging was to a pair of 8 inch floppy drives in a separate enclosure (I did say it was early days).

The site had a nearby lightning strike, which upset the logger,,which promptly stopped logging.

My chum was duly despatched with a bootful of spares. He arrived in somewhat inclement weather and diagnosed the issue as being with the dual floppy drive, so he fitted his spare.

The weather was getting very inclement again, with more rumbles of thunder. And behold, a nearby strike took out the replacement disk drive.

Muggins here was duly despatched with a replacement² unit, while my chum made a few careful alterations to site cabling.

I now live not far from that part of the world but I'm retired so I don't give a rats arse.

[Anonymous Coward]

Re: Outages

I've had similar happen, but was remote when it did. I just knew that I was going to be fired the next morning, until I got to the data center and the room was dark and quiet.

[Bebu sa Ware]

Some BoFH Classics in the making here

"We're not talking 'three-hour Richard Stallman monologue' horrible" Gruesome. Cruel and unusual.

"Only if you want to save your loved ones the cremation fees." This isn't just a joke… it happens.

"…pretend we've never been here..." Discretion often the better part of valour even if a killer rabbit isn't involved.

[Democrab]

Re: Some BoFH Classics in the making here

Unfortunately, because the BOFHs solution included turning off the PFYs Bitcoin mining rigs and passing the problem onto him (then likely telling him that it was the Boss' idea) there absolutely is a killer rabbit involved here.

[Anonymous Coward]

Sadly very close to a real power issue

Many years ago back in my early career I was working at the R&D labs for a major telco we had something similar.

One afternoon we suddenly lost lots of systems including the phones, upon heading to our labs down in the basement there were two key observations, they were all dark and boy did they stink of fried electronics.

Working with our newly outsourced facilities team they worked out there had been an issue with one of the main power boards in the basement plant room, this being a room accessed much as the article describes! This place had air handlers that you could do indoor skydiving in, ducts you could drive a van through and masses of electrical control gear from the substation transformers outside.

The team showed us the issue a 2" piece of Neutral bus bar on a 1000A+ board missing, now vaporised and coating the insides of the board. Turns out no-one had checked the balance on the three phase that board was supplying, it was massively imbalanced towards one phase so we'd been putting high currents through the neutral that it wasn't designed for over a number of years and it had been running hot for years before finally giving way. It was known that control board was running hot but it had critical equipment on like Cell Site controllers and a cable TV head end, so they'd been told not to touch.

We should have shut the doors, turned our kit off and walked away, but sadly we had to spend several days working out all of the PSUs we needed replacements for and all of the other kit like CRT monitors we needed to replace.

[David Hicklin]

Re: Sadly very close to a real power issue

unbalanced three phase loading ... UPS systems don't like it either

[The Oncoming Scorn]

Re: Sadly very close to a real power issue

"They don't like it UPS them"

Corporal Jones - I'll get me coat.

[Michael H.F. Wilkinson]

Is it just me?

Bit of a cliffhanger here. I was expecting a "KZZEEERRRT" moment involving the boss at any time.

Maybe next episode?

[UCAP]

Re: Is it just me?

This is probably just a lead-in to a fairly spectacular, but mostly load and bright, termination of the Bosses employment contract.

Probably also the PFI's if his Bitcoin farm is the root cause of the problem.

[Caver_Dave]

Re: Is it just me?

With this being electrical - 'load' and 'loud' could be equally appropriate in that sentence.

[M.V. Lipvig]

Re: Is it just me?

New here, are ya? If the PFY's bitcoin mine is the cause, the company will remain dark until The Boss signs off on a full power upgrade, which will include a new set of top of the line miners. The only one on his way to the jobs markets is The Boss when the new power system costs come in way over budget.

[Throatwarbler Mangrove]

Re: Is it just me?

The PFY seeks vengeance and is persuaded to go after the Boss instead of Simon ...

[Anonymous Anti-ANC South African Coward]

Re: Is it just me?

What if the PFY was first at Mission Central, and duly noted the existence of the old switching gear, then put his phone on silent and buggered off, leaving the problem for the next unlucky bastard?

[GlenP]

Not major but...

Many years ago the factory where I worked lost power on one phase, unfortunately the one with most of the offices on. It wasn't a major issue, just the main fuse blown but getting it replaced took several hours:

Engineer attends, can't get the dead fuse out, calls it in and leaves.

The heavy mob are sent from another job and remove the fuse in about 30 seconds, but aren't authorised to work on live equipment to put the new fuse in so they call it in and leave.

A senior engineer arrives to check our load balancing across the phases as they suspect that may be the cause (it wasn't, the issue was a slightly loose fuse overheating) which he can't do without the fuse in but fortunately he was authorised for live work and happened to have his kit with him so he fitted the new fuse then did his balancing check.

He reported there was an imbalance that we should investigate but it wasn't major - that was simply that as soon as he'd inserted the new fuse all the hand driers had started up!

[UCAP]

Re: Not major but...

Not quite at te same level, but the first couple of years after getting married my wife & I lived at her house out at the Essex end of London. Very nice semi-detached, probably 1920's build with super-high ceilings.

One day, we lost all power to the house. A fast check by yours truly showed that all of the fuses in the fusebox were OK, but nothing was getting so much as a watt of power. Nothing to do but call out an electrician (fortunately covered by house insurance) - we where a priority visit since we had a baby (#1 sprog) in the house. The electrician duly arrives and after about 45 minutes of head scratching finds two things: (1) the fuse on the input power feed into the house had blown for no readily apparent reason; and (2) nothing in the house was earthed.

The sparky called the electricity board to get them to sort the fuse out while he busied himself getting everything earthed as per regulations. When the engineer from the electricity board arrived, he inspected the blown fuse (remember on the power line coming into the house) and stated that he could not repair it since they had stopped making that type 50 years before - he had only ever seen them in a museum. So, instead of an easy repair he had to spend a couple of hours stripping out the old fuse and fitting a modern one to the power line.

[Paul Crawford]

Re: Not major but...

Thankfully rare but not that rare - some of those cut-outs had neutral fuses as well as line - a definite no-no now, and often the building earth was done via the cold water pipe, in the days when they were always metal (usually lead) and soldered joints so usually good conductivity, thus it was allowed.

Fast forward a good few decades and most of that has been replaced/upgraded but you will occasionally find a house that has not had major work done since the 40s (and that was often the fitting of 13A sockets for the new safer arrangement) and little since. But some houses will now be on plastic water pipes and thus no earth, etc.

[swm]

Re: Not major but...

Where I worked I saw several 3 phase fuse boxes with a copper bar in the neutral line.

[The Oncoming Scorn]

Re: Not major but...

Had something similar to that old house 1890's at least, burning insulation smell, power company had to fit a new fuse at 11.00 pm on Sunday night.

[Anonymous Coward]

Re: Not major but...

We (me an' the missus) moved into our first "proper" house, a '30s built semi, in about 1975/6. The mortgage offer had no conditions, only a recommendation - just that, a recommendation - that the electrical system should be checked.

Well, I checked it and it was a) absolutely minimal and b) bloody frightening.

The lounge and dining room had been knocked through and had just one mains outlet between them.

The wiring was old black rubber insulated cable, which was beginning to crumble away inside ceiling roses.

When I lifted floorboards to investigate further, I got a waft of warm rubber - cables were lain in the same notches as the central heating pipes.

We'd been in there a fortnight and I'd rewired the whole house: new consumer unit, the lot. AND I was complimented by the man from the board who came to check the work and reseal the board fuse.

Bloody hard work it was. I decided to do a proper job of channelling and mounting sockets a foot or so up the wall - lots of sockets. And all the wall, - external and internal partition, were brick. Hard brick.

I learned a few new words doing that too.

[Caver_Dave]

Re: Not major but...

!930's house, but owned by a bodger before us.

On the first day of ownership an Electrician friend came in to check the whole system for us.

"Well, the fuse board should have MCBs, but I'll check out the easily accessible wiring first."

Que a lot of banging coming from the attic.

"I'm condemning it all. Some of the upstairs light are running on bell wire and half of it is wired up backwards. I've had two electric shocks, just moving around up there!"

We knew we had to replace windows, doors, boiler, etc., but the number of unknown jobs that had to be sorted either before or after the trades was much larger than anticipated. I took advantage of the floors up for plumbing and electrical, to flood fill the house with CAT5 and sockets.

We found out that the previous owner had got various tradesmen in, told them their prices were too high and tried to do the work himself. All the local tradesmen seemed to have the house on a blacklist, until I could persuade them otherwise.

[An_Old_Dog]

Sometimes, the Dry Broom Handle will Not Save You

"Arc flash hazard."

At one chip research/manufacturing complex, they have a room with a mega-huge circuit breaker. Humans are not allowed in the room.

They have a specially-built robot-like device in there whose sole purpose is to operate that breaker.

[Anonymous Coward]

Re: Sometimes, the Dry Broom Handle will Not Save You

Reminds me of the time I visited a sub-station just outside one of the UK's major cities. There were notices within the main area saying to maintain a 2m air gap at all times. I saw a maintenance crew working with a cherry picker just below some live buses - I tactfully pointed out that any fault in the hydraulics (such as a speck of crud finding its way to a valve) and we might see some spectacular arcing displays. The cherry picker was promptly removed. (I was there conducting an H&S audit for the main network distribution company at the time - I believe the mention of the incident in my report led to some serious discussion with sub-contract management).

[The Oncoming Scorn]

Re: Sometimes, the Dry Broom Handle will Not Save You

Freely admitted as a repost....hence old git icon.

Big power supply with lots of Caps to store a charge for a echo sounding boom being towed by a trawler up & down Gare Loch performing a survey for Faslane who wanted to put degaussing coils into the Loch for subs to travel through.

Survey over one of the team goes to hit the main shutoff\isolator for the power supply with a wooden boat hook, the boats owner commented.

"Och aye, Wassa matter wi' him, is the fella scared to turn it off by himself!"

"Damn right he is, there's enough stored charge in that thing to kill him, if it arcs over!"

"Ohhhh!"

Icon - Details may be a bit vague as it was 37 years ago, I was watching the boat go up n down the Loch from a hotel bar at Gareloch head most of that week.

[Paul Crawford]

Re: Sometimes, the Dry Broom Handle will Not Save You

"Arc flash hazard."

For my sins, which are many and varied, I was once testing the supply to our site off a busbar chamber in an old sawmill that had its own 0.5MW substation, one of the guys in there was saying "Be careful with that, it is 415V!"

I pointed out it was not the 415V that frightened me but the prospective 10,000A fault current...

[Andy A]

Re: Sometimes, the Dry Broom Handle will Not Save You

One of the requirements when I was supporting London Underground was to undergo Track Training. An important part of this was how to get the current switched off in an emergency. Tube trains use DC -210V on the centre rail and +420V on the outer rail.

Favourite method was to phone the signallers who could do the task.

Second option was to get hold of station staff who would phone the signallers.

Third option was the use of a shorting bar - a plastic coated aluminium rod with angles at each end. These are stored in many places. One in each driving cab, others under marked seats in carriages.

We practised the use on a section of dummy track. Place the shoe against the nearest live rail and lower the bar towards the other one. LOOK AWAY. Arc welding uses a couple of hundred amps and causes eye damage. A tube train draws 500kW and you need about double that to trip the breakers.

It is someone else's job to chip away the remains of the rod.

[frankvw]

The PFY's absense

"Had any of the alerts been from his rack of Bitcoin mining machines, however, he'd have been in the office in a flash."

I'd expected the BOFH to cause said rack to generate such an alert in this case...

[Mr Humbug]

Re: The PFY's absense

I assumed that was the noisy equipment mentioned at the end

[Anonymous Coward]

Re: The PFY's absense

But how can it be the PFY's equipment if the boss's signature is clearly on a purchase order which went around normal planning and acquisition channels?

[You aint sin me, roit]

Re: The PFY's absense

And the equipment isn't for bitcoin mining, it's for training the new management AI system...

[Anonymous Coward]

That ominous silence

In a former role, we had a "server room" in the building (which was the usual slightly-large cupboard with AC and enough space to cram in a cabinet). I was doing some work that involved re-routing power cables for the servers (all of which were dual-PSU), and all went fine until I reached the last one. Unplugging the secondary PSU cable went fine, but when I plugged it in again, I was suddenly aware of the complete absence of any kind of noise from the 40U of kit. And also it was pitch dark.

Luckily I did not get eaten by a Grue, but for a while I wondered if that might have been preferable!

It turned out that several years before I started, there had been a plan to put the server room on its own supply, but that work had never progressed beyond someone updating some documents to say that it had been done. As a result the entire ground floor of the building was running off the same circuit, which must have been right on the edge of tripping. The act of re-connecting the PSU was the last straw.

Funnily enough, within a week, the server room was on its own circuit.

Oh, and that's also how we discovered that the super-awesome-bargain UPS the boss had bought was not, in fact, such a bargain after all.

[Korev]

Re: That ominous silence

I worked somewhere that had a data centre in a building built in the 70s, there was an outage as the cable burnt out. Turns out that the company putting the cable in had put in a lower capacity cable and billed them for the full capacity cable...

[Screepy]

Re: That ominous silence

Upvote for Grue reference.

I was eaten a lot by Grues in my youth.

[Bill Gray]

Re: That ominous silence

Some days, you get the grue; some days, the grue gets you...

[Anonymous Coward]

I was once visiting one of the distribution transformer huts on a north Wales sea front (a periodic visit to review ongoing work practices). As we entered, I was told not to touch any of the bare metalwork on the wall beside me - it was live at 440V. Needless to say (evidenced by the fact I'm typing this) I didn't. But the local contractor did - albeit standing on a rubber mat and wearing hefty rubber gauntlets.

[Giles C]

Dodgy batteries

First of two tails here this afternoon.

Early 2000s I get called out to the server room has gone down.

So I trudge over to the site (it was a weekend) and walk into the room to be greeted with silence, hmm the UPS is offline, no problem I will just reset the machine and everything will be back online.

Big mistake, it turns out it had gone off due to a battery failure and when I turned it back on the failed battery started venting sulphuric acid gas into the room.

Not something to be repeated.

[Giles C]

Breaker problems

2nd story….. we had just had the server room refitted and the electricians had installed a no CD e new distribution board with a 100 A breaker as the main release. We had been waiting for a while for this to be done and another person was installing new kit now we had the power available or so he thought.

He plugged it in and turned the power on only to plunge the comms room into darkness.

So we went in with torches and looked at the new board everything was good all the breakers were in the on position.

It took some head scratching it we found the problem round the back of the comms room was the power feed for it. This went through a 40A breaker so the extra load was enough to trip the breaker outside but not the panel inside…

[original_rwg]

Back in my yooof, I worked in a shop that sold wines and spirits. Our store was upstairs and there was a conveyor on the stair with a control box at the bottom,(up, down, stop) and two control boxes at the top (up,stop - down, stop but the stops were interchangeable). One day, a fault developed in one of the top control boxes in that when the stop was pressed, it blew one of the 100A fuses (the shop had three-phase). The remedy was to call out an electrician or it may well have been the Electricity Board to replace the fuse. For reference, a 100A fuse is about the thickness of a 6 inch nail and makes quite a loud bang when it goes. This failure took out half the lights in the shop and also meant we had to wind the handle on the cash register repeatedly.

We got around the problem of the faulty stop button by sticky-taping a piece of paper over it so we knew to use the other stop button. However mistakes happen and after the fuse had been popped a couple of more times, head office took action.

They inserted a few of RSJ's and knocked a bloody great hole in the floor to fit a new conveyor going directly from the store to the back shop. During the fitting of that, one of the fitters was happily drilling into the floor next to the wall to create a fixing point for the one side of the new conveyor when to his complete shock, he drilled through a pyro cable that had been covered in the screed and floor tiles when the place was built. Cue another loud bang and the poor bloke requiring a change of underwear!

The poor fellow feared he would get his books but he did not. The contractors however, had to replace several metres of cable at their own expense.

[Blofeld's Cat]

Now Ygor, while the storm is at its height ...

Years ago when I was an apprentice, we did a lot of work in old factories and mills in North West England. Nearly every switch room you went in had something to avoid:

"Don't get too close to that busbar chamber with the missing lid. The bars used to be insulated with asbestos, but they removed it as it was dangerous."

"We think that the ebonite board with the brass knife switches is left over from the mill's generator set."

"Most of the stuff on the ceiling hasn't been properly supported since the air raid in 1941."

"I'm pretty certain turning this valve off isolates the halon system. Just don't hit any of the sprinklers."

[Boris the Cockroach]

Back in the days

of yore when I was a mere PFY (yeah it was a loong time ago) our customer had a bit of a problem of exploding electrical panels when certain equipment was started, given there was a live body in front doing the switching did'nt help either.

So it was decided to replicate the problem at a safe place, however , even the PFY objected to standing in front of a live panel under test.

So we rigged a remote switch to said panel to a test booth 100 yrds away.... however it was the PFY's job to run said cable through the underfloor cable duct( its amazing how clean and tidy duct work is when shown in holly wood movies.... its not like that in real life).

But I did get the reward of pushing the button....... and destroying a perfectly good electrical box in a shower of sparks and bangs.

My BoFH at the time liked people who shared his love of dangling 40 feet in the air while trying to feed a 200A armored 3 phase cable under, then through an experimental platform......"look on the bright side" he said "you'll only fall to your death once"

[The Oncoming Scorn]

Re: Back in the days

"look on the bright side" he said "you'll only fall to your death once"

Bastard!

[Anonymous Coward]

Re: Back in the days

And that's why a DNR order on-file is a must.

[Alan J. Wylie]

Dry broom

A long time ago I worked for an engineering company that had an ancient poster on the wall: "What to do in case of electrocution". If your colleague has been electrocuted, take a dry broom, wrap your flat cap around it and pull him away from the supply.

[Excused Boots]

Re: Dry broom

Well it’s fairly good advice, it just needs to be updated,

If your colleague has been electrocuted, firstly go out and purchase a flat cap, make sure you get a receipt otherwise you will not be reimbursed......

[stiine]

give that man a raise!

Thanks Simon. I haven't laughed this hard in years! In fact, I'm still laughing.

[CA_Diver]

Dry Broom? Try a Wet Mop

Decades ago, I responded to a loss of air conditioning alarm in the company's server room. One of the technicians had the false floor open and was mopping up a/c condensate spill, standing a puddle of water. Said puddle containing 440 power cables, among the communications lines. I mentioned that it would be a good idea to get out water just in case one the cables wasn't properly insulated, and my CPR card was expired.

[Tim99]

Re: Jason Statham's method acting coach

I find his films quite comforting, I know pretty much what's going to happen. Not comforting enough to endure the inconvenience and expense of going to the cinema, but on a wet evening in front of the TV with a suitable self-consumed libation, pretty good.

I'll also whinge about the new site - my muscle memory can't cope, and some of the links from "Comments" back to the article don't work for me. A shouty retrograde step.

[The Organ Grinder's Monkey]

Re: Jason Statham's method acting coach

"Self-consumed libation," is there any other kind? Even the obscenely wealthy don't yet pay someone to do that for them (though I don't move in those circles, & nothing they do would really surprise me.)

[Tim99]

Re: Jason Statham's method acting coach

"A libation is a ritual pouring of a liquid as an offering to a deity, or in memory of the dead." Self consumption is probably better than pouring it onto the multimedia system? I believe that this could have been modified such that the deity’s earthly representative consumed the good stuff on their behalf?

[Meph]

Average airspeed of an unladen sysad

Back in my formative years, working as a service desk tech at an educational facility. Our local power distribution site (after being starved of replacement parts and spares budget by successive state governments) decided to detonate in a spectacular fashion.

All the power went out in our building, but being a tropical town, this wasn't super unusual. What was unusual was the bang of the front door, and the sound of one of our system administrators breaking the sound barrier on his way towards the campus generators.

It turns out that someone had forgotten to refill the diesel tanks, and the generators failed to start due to being bone dry.

IT geeks aren't normally a physical bunch, so seeing one moving at that speed was both surreal and hilarious at the same time.

[Anonymous Coward]

Re: Average airspeed of an unladen sysad

If that ever happens, something has gone very, very, wrong.

[chivo243]

We were never here!

I was so tight with my ex colleagues, that we didn't need to say it, one look between us, and We were never here!

[Cliffwilliams44]

Wow, how clueless can you be!

Interesting story written by someone who has no clue how electricity actually works!

The only reason a main breaker would be tripped, and all the downs stream breakers are not, is the fault occurred behind the downstream breakers!

And a broom handle! Pfft! Electricity deserves your respect, not your fear! Most often it's the fear of electricity that gets people killed!

[Strangelove]

Re: Wow, how clueless can you be! err no !!

breakers are not fuses and they don't discriminate like that. If the fault is a dead short or at least well above the de-latch current of the highest breaker in a chain, they will all start to trip. and more than one may well have passed the point of no return before the fastest in the chain actually opens, and that may well not be the lowest rated one.

yes, to begin with they get faster as the fault current increases, but only up to about the point the contacts are moving at the speed of sound, much above that and the time to break stops getting faster and remains almost constant.

Fuses however depend on energy dissipated to heat the metal of the fuselink, and fuses of comparable thermal design, but ratings in a ratio of 3 or more can usually be relied upon to trip smallest first regardless of the actual fault current. but only usually. This only works reliably if both are slow blow, or both fast blow or whatever to the same standard. If not then you need the makers time current curves..

regards Mike.

[Judge Mental]

The golden rule of BC.

Not if, but when.

[thosrtanner]

On a slightly different topic, what happened to the feed for this page? Ever since the major site update it's disappeared.

[Luiz Abdala]

You can experience a breaker tripping due to a surge overload safely in Satisfactory!

Great game that one.

You can purposely build a power system and overload it happily, and watch the breaker trip over and over as all machines try to run at once.

The coat icon matches the aesthetic of the game.