back to article Exsparko-destructus! What happens when wand waving meets extremely poor wiring

Welcome to another edition of On Call in which a contractor's shonky job and a guard's Jedi-like abilities result in an impromptu pager party. Our story, from a reader Regomised as "Andrew", takes us back to his time working for a certain telecommunications company, "still well known in the ISP/Telco hardware world," he said …

  1. Steve Kerr

    Hands up

    My first job in IT as a 19 year old learning the ropes.

    Helped the IT manager with some cabling twinax cabling work for AS/400 that ran down 4 floors in a conduit shaft, problem was, we didn't have access to 2 of the floors as they were owned by other companies, somewhere they were cable tied very firmly so couldn't move them,.

    had the bright idea of cutting the cables at both ends to which we had no access so they could be moved.

    Then, by stripping back the plastic, twisted the cables, soldered them together and covered them in insulating tape and left it at that.

    Wasn't any complaints on the Monday morning from the AS/400 people so job well done and dodgy cable soldering hidden.

    I was the 19 year bright spark who said to cut the cables and resolder, just fortunate there was enough slack at either end to do it.

    1. Anonymous Coward Silver badge

      Re: Hands up

      I remember the cabling risers I dealt with from that era. 19 year old me would've shimmied down the riser with some cutters; they were certainly big enough. They were a little intimidating as they ran the full height of the building and I was based on the 8th floor.

      1. Roger Greenwood

        Re: Hands up

        You are John McClane and I claim my £5

        1. ChrisC Silver badge

          Re: Hands up

          Yippie ki-yay, cable cutters!

      2. Mast1

        Re: Hands up

        Wow, they were big cutters, 8 floors long : how do you get them to open in a riser ?

        1. Flightmode

          Re: Hands up

          Ah, Fridays.

      3. Anonymous Coward
        Anonymous Coward

        Re: Hands up

        Did that* as a university student, and slipped (was on the 8th floor of ~11 at the time).

        Hanging by one hand grabbing a wire mesh wasn't the most pleasant of sensations... But that particular bit of wire mesh did have one very small hole surrounded by a set of somewhat larger ones for at least the rest of that year.

        * Technically it was a staircase and it was just easier to scramble up the middle than keep passing cables round the edges. I wasn't aware until afterwards that the blockage between the 5th and 6th floor was basically a single sheet of plasterboard or similar.

    2. TeeCee Gold badge

      Re: Hands up

      I'm not sure whether to be horrified that you spliced Twinax or astonished that you got away with it.

      It carries a balanced signal and any imperfections in the cable almost invariably result in the attached terminals dropping out at random intervals.

      1. Pascal Monett Silver badge

        Cue next week's On Call about a company whose terminals suddenly start randomly disconnecting . . .

  2. Neil Barnes Silver badge

    Less poor wiring than poor building

    Working on a studio many years ago in the UN building in New York... while climbing through the ceiling voids looking for cable runs I discovered that the 'secure' wall between the studio complex and the presumably highly secure bank branch who had the next suite, er, stopped at the ceiling tiles.

    They were not amused when I told them...

    1. Mishak Silver badge

      Re: Less poor wiring than poor building

      Hmm. You would really have though that was something the bank's security team should have checked well before moving in!

      1. Anonymous Coward
        Anonymous Coward

        Re: Less poor wiring than poor building

        Not an issue in modern housing due to the way it's thrown up, but in older terraces It's not unknown to have dividing walls between premises not go into the roof space (and a fire risk as the dividing wall is the main firebreak). Offices are a different matter as they're rarely occupied in the way originally planned.

        1. MrReynolds2U

          Re: Less poor wiring than poor building

          Yeah, I once had a terraced council house with shared attics. There was talk of people climbing into neighbouring houses and helping themselves to TVs and the like. I augmented my access panel with large lockable bolts just in case.

          1. Anonymous Coward
            Anonymous Coward

            Re: Less poor wiring than poor building


            That was very well known after a particular terrace housing one of the local 'Dodgy families' had a few items 'disappear' without any evidence of break-in. (From TV's to money etc)

            This happened until someone heard scratching in the loft and caught someone stepping over the joining wall working their way across the terrace.

            Bricks had been removed between the houses so you could 'visit' any house via the loft with no trace.

            High level 'negotiations' took place in the loft space and the next day most of the lost items where returned. :)

            The family moved out quite quickly !!!

        2. Doctor Syntax Silver badge

          Re: Less poor wiring than poor building

          For a few years we had a house like that. The insurance or building society insisted in the installation of a firebreak, some sort of mineralised panels which didn't look very substantial.

          At about the same time I wasn't very pleased to discover that at work someone from facilities had visited the machine room to do work on the riser and had drilled a few holes in similar panels leaving a good deal of mineral dust about the place, this being back in the days of tape backups.

          1. adam 40 Silver badge

            That would be white asbestos

            and the dust would be lethal.

        3. A.P. Veening Silver badge

          Re: Less poor wiring than poor building

          That was also the main reason 11 prisoners lost their lifes in a fire at the Schiphol prison complex on Oktober 2005 (Dutch only unfortunately).

          1. ICL1900-G3

            Re: Less poor wiring than poor building

            Poor sods. Tragic article.

      2. Anonymous Coward
        Anonymous Coward

        Re: Less poor wiring than poor building

        I worked with a company whose deals were in the multi million pound order. They had 2 meeting rooms which could not be used for negotiations for the same reason, not only did the partition finish at the false ceiling because all the other walls did go all the way up the reproduction in the other room was perfect. They discovered this after some negotiations went the wring way.

    2. Mast1

      Re: Less poor wiring than poor building

      Cheaper version of this. My office backs onto a lecture classroom. The wall is plasterboard & studs, but with offset studs for better sound isolation (decouples the walls). I complain that I can hear the bass of every lecture (despite loudspeaker not being mounted on that wall) , so during re-vamp of the classroom they put in a third, offset stud wall.

      I can still hear the lecture, so do a bit of prodding myself.

      Yup: stud walls go up to beyond the (early 2000) ceiling tiles. But only as far as the 1950s ceiling studwork for attaching an earlier set of now-removed acoustic tiles. Sealing that gap sorted the problem. (too thin for students to crawl through to escape the lecture)

      1. Dabooka

        Re: Less poor wiring than poor building

        Similar here.

        My old office suite (I was in one of three) was created by dividing a meeting room which is used for, er, meetings along with student seminars etc.

        The sound echoes through the void above the stud partition. Warned them at the time but what would I know. I does work both ways though as they discovered when I was on the phone to a car insurance company that had incurred my displeasure.

        The centre manager and I also tried to get some services (network and power) installed at the same time but were overruled by the Facilities Dir. Now I've moved office and they want to rearrange and put two extra desks along that wall but guess what's missing?

        1. Mike 137 Silver badge

          Re: Piss poor planning

          I once worked in a company handling personal data that had a very large open plan office with partitioned off meeting rooms as one end. They had an area of the open plan office by the meeting rooms dedicated to a "secure" service and secure entrance contols to the entire floor.

          One day they had the bright idea of leasing part of the open plan office to another company, so they put up another partition with a locked door. Not between themselves and the other company, but between the majority of the space shared by both and the "secure" service area. Unfortunately that put the meeting rooms (shared by both companies) on the "secure" service side of the partition.

          So both companies worked shoulder to shoulder on the main floor and both had access to the "secure" area. I made myself very unpopular by pointing out the (rather obvious) problem.

        2. Rob Daglish

          Re: Less poor wiring than poor building

          Carlisle Hospital, a PFI-flagship, was built with missing/dodgy firebreaks, and after a relatively small fire that should have been easily contained, it transpired that the fire alarm didn't work properly either. The solution? Well, for quite a while, until the contractors could be brought to heel to fix the issues, they had people in purple hi-vis walking around the hospital as fire-wardens to check all was well.

          Mind you, the warning signs should have been there - the fact that they have a hospital where the general public goes into a large central atrium, and to get patients to/from wards/treatment areas, they need to transit through this large central atrium, the corridors off the atrium aren't wide enough to take two beds side by side, and the walls tend to fall apart if a bed rolls into them.

    3. Peter Gathercole Silver badge

      Re: Less poor wiring than poor building

      Saw that in a segregated shared server room. The wire security cages neither went above the ceiling tiles nor below the suspended floor.

      1. The Oncoming Scorn Silver badge

        Re: Less poor wiring than poor building

        At another branch of the same three letter pharma company, this time in Stevenage....

        I was happily doing my rounds of dropping off equipment & heading back to my car, when the head of security came in the opposite direction & tackled me on the frequent subject of laptops disappearing & how secure were Kensington locks, as over the prior weekend someone had got into a locked office area, via the roof void from the neighbouring unlocked office & taken said laptop. The evidence for this was a pair of dirty boot prints on the desk where the miscreant had dropped himself onto.

  3. GlenP Silver badge

    Not quite the same but in a previous role we had a Notes server supplied and managed by a sister company's IT people who were very precious about it.

    Following an office move, when I was just seeing out my redundancy notice and handing over, the server power failed, cue annoyed calls from the Dutch, what's wrong, why didn't the UPS kick in, etc. It turned out that when things had been recabled in the new office (not by me) they'd plugged both PSUs on the server into a single C14 socket strip connected to the UPS, it was the socket strip wiring that had failed. Why it was cabled that way I don't know, given that I'd deliberately installed two such strips to allow some degree of redundancy as budgets didn't run to a second UPS*.

    *Ideally we'd have had one PSU to the UPS and one direct to the mains but that wasn't considered acceptable

    1. Dave K

      I submitted a "Who Me?" to El Reg a couple of years back where I experienced a similar issue. It's always great when someone installs a server and doesn't realise that the redundancy offered by two PSUs is limited somewhat when you plug both of them into the same power distribution unit.

      1. Anonymous Coward
        Anonymous Coward

        I recall having to explain to a senior manager why we needed a second UPS when he'd signed for a replacement server with two PSUs (for a small office, ~20 years ago). I had to point out that one reason we needed a new server was because the other one (basically a Compaq tower PC with a bigger HDD than normal) had lost power when the UPS failed!

        (I hasten to add I had only taken on the IT role part-time and temporarily when the IT manager, who had previously worked for a multinational, and probably had had limited hands-on with actual kit, decided to retire.)

        1. the hatter

          Really depends on your circumstance, but how often does wall power fail ? If you don't have UPS redundancy, then one PSU plugged into the wall and the other into the UPS gives you redundancy for PSU failure, for mains failure, and for UPS failure, Dual feeds cross-feeding dual-input UPSs, feeding in twin-psu boxes is nice and all, but a small office with one server sound like 9 times out of ten (if not 99//100) would be better off with mains+UPS, and if anything, putting extra budget into a better or bigger UPS rather than a matching pair.

          1. phuzz Silver badge

            The only wrinkle with that set up is that it's very difficult to get an accurate UPS run-time in this way. The UPS knows how much power it's supplying, but depending on how the PSUs in the servers are set up, that might only be half the total draw when the external power goes off.

            Mind you, unless you have two equally sized UPSs, and manage to perfectly halve the power draw between them, you're probably going to run into the same issue. At work we have one large UPS, and three smaller ones and devices have just been plugged into whatever was closest, so our actual runtime is pretty hypothetical. That said, the longest the power has actually been out for, in the last ten years, is just thirty seconds, and I'm pretty sure our UPSs can last at least ten minutes. Probably.

            You're right though. Best practice means nothing without the budget to match.

            1. Dave314159ggggdffsdds Silver badge

              "Best practice means nothing without the budget to match."

              Yes. If you're worrying about UPS run time, you need bigger batteries.

            2. Andy A

              One place I worked at had two strings of pylons running towards the ends of the site from different suppliers. Once a year, during the official shutdown, they tested the switch-over.

              That meant that the stuff in the server room had to be shut down just in case.

              One year I decided to check the UPS runtime. From its control panel, the runtime with the servers should have been about 2 hours, but I limited the load to just the switches.

              The lights went out on time. The UPS gave up after only 20 minutes.

              Still, I got to look, properly supervised, inside one of the big substations and see what happened there as power was restored.

          2. kmceject

            In NYC we split to wall power for some low risk items but we often had brownouts (during high load days they might drop as low as 95v on a 120v circuit!) that could trip a power supply. I think it was HP servers where this could ruin your day as you had constant alerts (the room could have twenty screaming alarms going) and they were a pain to reset. Beyond that some of them required a power cycle to clear the alarm lights on the panels, preventing you from knowing to check for other hardware faults...

            1. Robert Carnegie Silver badge

              Does a UPS, or two, save you from brownout? I suppose so... at first. I thought another possible issue would be, wall power isn't conditioned, UPS output is conditioned, but a set up that notionally connects wall power on one PSU with UPS conditioned power on the other PSU makes trouble. And if the power actually is off, then it could come back on with wall power at opposite phase to the UPS's phase...?

      2. Stoneshop

        It's always great when someone installs a server and doesn't realise that the redundancy offered by two PSUs is limited somewhat when you plug both of them into the same power distribution unit.

        Then yours wasn't an Alphaserver DS20 or DS25.

        Three PSUs, can run on one but needs two for startup (and the third one for that n+1 redundancy during startup), but they MUST be connected to the same phase else they go frrzttt. Now of course you can run the power cords to different PDUs as long as you keep track of which phase you're using, but it's still an invitation to things going frrzttt when another sparkie is doing some impromptu replugging. With all three power cords going into the same PDU you can more easily ziptie them in place with a warning label attached (which will of course be ignored, but at least you've then done your part).

      3. NoneSuch Silver badge

        Or powerbar with a twitchy surge protection. *coff*

      4. John Brown (no body) Silver badge

        I remember when someone first showed me redundant PSUs in a server chassis. Look, he says, they're hot-pluggable, you just press the lever and yank it out.

        Alarms ring out. Oops.

        Yeah, it had hot-pluggable redundant PSU bays. Three of them. It needed two installed to keep running. It only had two. They'd not paid for the full complement of three. And yes, they were both plugged into the same power strip.

        Since then, I've also seen many places with dual or more redundant PSUs in server chassis where the only redundancy they can protect against is a PSU failure. If there's a UPS present, it will be powering both inlets.

    2. Dave314159ggggdffsdds Silver badge

      Power strips are usually a sign of bad planning. I particularly hate seeing them installed with more sockets than needed. Someone will eventually plug a high power something into the empty socket and let the magic smoke out.

      Single socket extension leads exist, as do longer power cables. A no-power-strips rule would have highlighted the issue with plugging both power leads into one UPS, if it only had one output.

      As far as redundancy goes, people sometimes overdo it. Servers for external services obviously need to stay up, but there's no point doing more than gracefully shutting down servers that run offices in which no-one can work during a power outage.

      1. G.Y.

        In NYC, the law of the Medes&Persians restricts the number of sockets, seemingly on the assumption that each socket powers a 'fridge+-.

        When you have a few laptops, iPhones, ..., the only way out is lots of power strips

        The apartment building's super does this in spades

      2. Anonymous Coward Silver badge

        That's exactly why, in right-pondian land, we have fuses in our extension leads (and everything else)

        If you overload it, you'll blow a fuse and everything on that strip loses power.

        There's still a risk that a poor joint somewhere will gradually overheat, but that risk would be the same for a single high power load vs a loaded extension lead.

    3. Stuart Castle Silver badge

      Years back, when I were a strip of a lad, our “server room”

      was a long bench on my office wall. Taking advantage of the fact my boss was away, I bought in my pocket cd Walkman and power adapter, with the idea being to have music all day.

      The problem was that the circuit that side of the room was overloaded, and me plugging my CD player in was the straw that broke the camels back.

      The circuit tripped, everything that wasn’t ups protected (which was most of the equipment, bar a couple of servers) switched off and the ups went mad.

      Thankfully, I got everything working, but could not reset the alarm on the ups . So, instead of pleasing music, I got a tortuous alarm sound ..

    4. Blank Reg

      Years ago when I worked for a large wireless telco we bought an expensive fault tolerant Tandem. Sure enough it was wired to a single circuit

  4. tip pc Silver badge

    Bodge it and quickly

    Facilities had a moan once because the crew I’d organised to run Ethernet through a campus building had pinched / snagged a power cable in a metal surface trunking. I asked why there was metal surface trunking in the first place. Weeks of blaming with them insisting they wanted to run the Ethernet with me countering, it was agreed the electric wiring wasn’t safe and the metal trunking would be replaced with new electrical wiring and Ethernet re termination.

    Everyone was just lucky the metal trunking was earthed.

    On inspection there had been evidence of arching in lots of places and after plastic was installed the stories came out how that building was plagued by electric circuit trips for years previously.

  5. Anonymous Coward
    Anonymous Coward

    Not so much poor wiring, just a sparky drilling a hole through a wall into the comms room and managing to get the drill bit smack in between 2 cables, stripping the insulation and shorting the wires as he went. One loud bang later (not the Paris kind) and we had a room full of dead comms kit...

    1. Jimmy2Cows Silver badge

      And a charred, smoking sparky? Or was he somehow spared voluntary electrocution?

      1. phuzz Silver badge

        Unless it was a really crappy drill (which is still possible), if it wasn't just arcing between the two cables they drilled, it would have down the drill's power cord rather into the sparky.

        Humans don't conduct electricity very well, stray electricity will earth via metal or wires before it tries your arm.

        1. H in The Hague

          "... it would have down the drill's power cord rather into the sparky."

          As far as I'm aware most electric drills (at least in Europe) have long been double-insulated, so no earth/ground wire in the cord. Furthermore I think they usually have an insulating component in the motor shaft, so that even if you hit a live wire only the chuck (which you normally don't touch in use :) will be live, and not any other part of the drill.

          Almost that time again -->

          Here's one for the weekend.

          (Is this the right place to whinge about the lack of a wine icon?)

          1. Stoneshop


            The more serious electric drills (and the not-serious ones before cost became a major factor) have metal gearboxes that tend not to have an electrically isolating covering. You're supposed to hold them by the front grip, not the gearbox, but occasionally one needs to take off the grip when it gets in the way.

            The better electric drills have an indication showing you're closing in on live wiring.

            1. Gene Cash Silver badge

              Re: Gearbox

              > The better electric drills have an indication showing you're closing in on live wiring.

              Seriously? I've never seen or even heard of that. Cite? I'd love to see one.

          2. Dave314159ggggdffsdds Silver badge

            You're spot on about double insulation. I've seen someone do as you describe, but get the drill stuck, so went to tighten the chuck. Fortunately the fuse had blown on the circuit they shorted, otherwise they'd have got a proper shock - they had wrapped their hand round the chuck to grip it :/

            These days wires inside hollow walls aren't supposed to be fixed in place, so drilling will push them aside. If they are fixed such that a drill could penetrate, they must be protected by trunking. If you manage to drill into a wire, you can at least blame the dangerous state of the building's electrical installation.

      2. A.P. Veening Silver badge

        As the drillbit shorted the two cables, I wouldn't expect the sparky to experience anything more than a nasty surprise due to unexpected fireworks within the wall.

    2. Anonymous Coward
      Anonymous Coward

      My late father managed to perform the domestic version of that.

      Spend a few weeks redoing the bathroom, and all that was left were a few hooks for towels and so on. So, he drills the first pilot hole with a 4mm drill and an impressively loud bang sounds as the lights go out.

      The bit of the drill that was originally in the wall? Gone..

      I grabbed a chisel and took a closer look while we kept power off and surprise: it turned out he had found a main supply line and drilled into the exact centre of it, shorting the steel mantle (earthed, which is why he didn't feel a thing) with ALL three live wires in there.

      This was a house with an old style electric cooker, which meant the house master fuse was a 3 phase set of 40A lumps, and this distribution cable came straight off that for a fuseboard situated elsewhere in the house. This also explained the missing bit of drill - that had vaporised and small metal pearls were still stuck to the inside of the covering tiles.

      Anyway, long story short , I dug this thing out, isolated it so we could reset power and then got materials to repair it. Needless to say, we then got a metal detector :)

      1. NXM Silver badge

        Reading that, I was so relieved that the short wasn't the cause of his lateness!

  6. Anonymous Custard Silver badge


    Shouldn't the tagline be "Hello. My name isn't Andrew. You killed my data centre. Prepare to die" ?

    And was the guard left-handed?

    1. Precordial thump Silver badge

      Re: Tagline


      1. DailyLlama

        Re: Tagline

        You use that word a lot. I do not think it means what you think it means...

  7. Blofeld's Cat

    Er ...

    Not quite the same, but I got called out to a site where their in-house "electrical genius" had installed a number of "Emergency Stop" buttons demanded by Health & Safety.

    Shunning the usual methods, he had wired cheap push button switches across the mains at strategic points. Pressing the switch shorted the supply, tripped the circuit breaker and cut off the power.

    Eventually one test welded shut the contacts of the 32V rated switch involved, making it impossible to reset the breaker.

    1. Denarius Silver badge

      Re: Er ...

      sounds like a datacenter in center of Oz crapital with some, er, response sensitive departmental servers in it and an ISP ITIRC. Plumber tightening a cooling pipe dropped a spanner which with the unerring accuracy such situations demand, across the terminals of 415 V going into the center. Cue loud bang, some of steel wrench becoming gaseous and rapid silence on servers as UPS due for maintenance failed with enthusiasm, or so the story goes. Not there, heard it second hand

      1. Anonymous Coward
        Anonymous Coward

        Re: Er ...

        I know of someone who did that in a telephone exchange, but it was across the 48v DC busbars.

        Spanner vanished noisily in a cloud of steel vapour, two large bites were taken out of the busbars. Telephone exchange continued working happily as if nothing had happened. Sparky had to change his underwear, though.

        1. Stoneshop

          Re: Er ...

          A painter, and his tin of paint.

          The result was not unlike Mr.Bean's way of painting his living room, save for the wrapping items beforehand.

      2. Emir Al Weeq

        Re: Er ...

        "Failed with enthusiasm"

        I'm going to keep that one for future use.


    2. Denarius Silver badge

      Re: Er ...

      also reminds one of the simplest project a merkin electronic magazine published in 1970s, but to a higher standard. Metal box with one single pole switch, rated 110 V 10 Amp. Plug it in flip switch, blow fuses. Why ? GOK

      1. Robert Sneddon

        Little Jiffy Fuse Blower

        A box sitting on the table with a sign reading "DO NOT PUSH THIS BUTTON." Then the lights go out.

  8. This post has been deleted by its author

  9. DS999 Silver badge

    RFID wand

    They should have known giving a terminally bored security guard something that could be used as a sword or light saber would eventually bite them in the ass somehow!

    1. MJI Silver badge

      Re: RFID wand

      Anything really of that type.

      They play cowboys with barcode guns.

      I once went through a log of failed scans and created custom messages after I IDed the barcode.

      With 2 way display guns....

      1. Anonymous Coward
        Anonymous Coward

        Re: RFID wand

        Yes. It was a barcode or magnetic strip tag. Magnetic ones were in fashion in the nineties. Sometimes hard to scan, those buggers.

  10. js.lanshark

    If it works, it isn't stupid!

    Contractor was engaged to install a bit of networking kit (Centillion 100 for those old enough). When it came time to go Gig-E, it all had to go. We found a C100 cargo strapped underneath an AC duct in the ceiling of a room. It was the only place where all of the Ethernet cables would meet.

  11. Will Godfrey Silver badge

    Nothing New

    If I had a pound for every bodged bit of wiring done by 'professionals' that I've had to put right over the years I'd be a very rich man!

    {we won't talk about the bits I did myself in the early years}

    1. Sam not the Viking Silver badge

      Re: Nothing New

      One of my very first commissioning jobs concerned a large panel of MCCs (for motor starting) plus some smaller panels for auxiliaries. The main panels were built by a shipyard electrician. The auxiliary panels were a bit ad hoc....

      The main panel interiors were exemplary; everything was clearly placed, routed, clipped and marked. It worked.

      The auxiliary panels would not have been out of place in a bowl of spaghetti. There was no consistency in wiring, colours, wire-sizing. Over-long wires were just bundled up. No wires had terminals. A few had numbers. We sent it back for 'tidying up' and on its return their 'engineer' came to demonstrate how it worked. It didn't, and when we showed him an example of the expected standard, his face just dropped: a flatlander seeing another dimension.

      We asked the shipyard to re-do the panel; they built a new one. It worked!

      1. Boris the Cockroach Silver badge

        Re: Nothing New

        Yupp... been there.. seen that... had to fix it anyway because the boat was leaving in 4 hrs regardless....

    2. Imhotep

      Re: Nothing New

      Up vote! This could describe me.

      Also, moved into a ten year old house here and had to redo so many circuits. I think the standard of construction 'electricians' must be pretty low in this area.

      1. Stoneshop

        Re: Nothing New

        Also, moved into a ten year old house here and had to redo so many circuits.

        Mine was a 90 year old house at the time; it had been fitted with that electrickery stuff in the 1930s (found some gas piping for lights in the sitting room and the back room), then redone to less ancient standards in the early 1970s, and creatively modified in the 1980s including some rather unwise structural remodeling.

        There was a hidden switch, or rather a dodgy joint, halfway up the stairs controlling the light on the landing; knocking on a plasterboard panel in the right spot had a fair chance of changing the light status. There were two junction boxes where switched live and neutral came in via different conduits (they were arranged in some kind of grid), even where their upstream boxes had the same switched live and neutral available. Oh, and using ground (yellow/green) wire for neutral (blue) was deemed no problem.

        It was all taken care the next weekend.

        And that collection of bodges was eclipsed when two years back I had to deal with a workshop that had been in use by a small-time weed grower. Junction boxes that couldn't be closed due to the number of joins in them. Junction boxes and the wiring inside charred but somehow still functional and not shorting. Yellow/green used for *live*. One fuse panel was to be taken out anyway so I didn't bother to trace each circuit to their fuse but simply shorted it (two circuits out of a dozen had to stay up to provide lighting while the replacement was installed).

  12. Anonymous Coward
    Anonymous Coward

    Cage nuts and cable monkeys...

    We've outsourced our WAN, and this works... mostly...

    (after we got rid of the Frame Delay setup that was delivered at first)

    And it's the company we outsourced it to that sends out techies to replace routers and stuff when there's an upgrade.

    One of these 'techies' placed the cage nuts on the front of the rails, mounted the router onto them, and placed the modem and PSU on top of the router.

    1. Charlie van Becelaere

      Re: Cage nuts and cable monkeys...

      (after we got rid of the Frame Delay setup that was delivered at first)

      Perhaps I've simply missed it, but I love this term and will use it henceforth.

      Have one on me!

  13. chivo243 Silver badge

    My guess

    Either a Monday morning job, too bleary eyed to know or care, or Friday afternoon at Pub 0'clock!

    1. A.P. Veening Silver badge

      Re: My guess

      Probably a combination of either the first two or the latter two.

  14. Giles C Silver badge

    Dodgy circuit breakers

    A few years (and jobs ago) we had just had some new distribution boards in the computer room.

    Some new kit had been installed and powered up but it was only idling.

    A while later it powered up fully and at that time the whole room went down. Nobody was in it at the time fortunately.

    When we went down to investigate the first thing to check was the new dis board - everything was switched on and none of the breakers had tripped.

    So out of the room and off to find the switchgear running the room - which had tripped.

    We had a 40A circuit breaker feeding the new board which was protected with a 100A breaker.

    So the load tripped the breaker outside the room and killed the power. The electricians were in that weekend swapping the supply cables out….

  15. RockBurner

    "bitten on the behind by the leavings of a shonkmeister"

    My own shonky code (over 10 years of PHP, there's plenty of shonk to go around) aside, the worst I'd ever seen was (ironically) right at the beginning of my PHP 'career'.

    Joined a small marketing company as a Front-End web dev (with not a lot of 'server-side' knowledge), with the proviso that I'd 'learn up' the server-side as well. After about 6 months I'm working on the front end of a website that is completely AJAX driven (client's demanded it - don't ask), and the content management system is being built by the self-proclaimed PHP Guru in the office who's been there a year or two. He'd been working on this project for 6 months apparently, trying to get the (home-brewed) CMS working correctly (IIRC - it was again AJAX driven with PHP backend over a MySQL db).

    For some reason known only to himself and Beelzebub, he'd decided that instead of saving html markup into the db cell for each page of the site (or some other normalised system), the text was saved as "text-string|paragraph-type|text-string|paragraph-type|etc|etc|etc" eg: "Page Title|h1|Page intro copy|block-para|Paragraph 2 copy|para|" etc..... Lists were particularly fun to decipher.

    I had to parse these strings to make them display on the site - he had to ensure that each individual string, and it's paragraph type was editable (and without fucking up special characters) - which is what he'd been struggling with for 6 months.

    Eventually they had 'discussions' with him, he left, and I got lumbered with it.

    Finished the CMS off 2 weeks later (albeit still on the same horrendous database structure), including fixing the encoding issues (still not sure why it was hard just to check that everything was configured to use the same character encoding configuration!) and learnt a hell of a lot..... only for the client to go under a week later!

  16. Hero Protagonist

    Tip of the hat

    for working in references to three different movie franchises in the headline, subhead, and first paragraph – tip back one of these as reward!

  17. Anonymous Coward
    Anonymous Coward

    My name is Inigo Montoya

    I killed your server...

    1. Anonymous Custard Silver badge

      Re: My name is Inigo Montoya

      Prepare to die...

  18. Mike 137 Silver badge

    Far from unique

    At one site I worked on, there was an intermittently occupied environment that became hazardous once or twice a day. it was equipped with a hazard warning siren that went off each time, a few minutes before the event. But the safety system was "home built".

    A colleague had installed a critical relay (the one that supplied power to the warning signal) on a panel by gluing its top to the panel instead of wiring in a socket and plugging it in. Then he soldered wires to the pins of the relay. Consequently it would be very hard to replace. He wired the relay contacts so they directly fed a massive capacitor that was used to carry on powering the warning for some time after the mains were switched off for safety reasons. So every time the warning signal was triggered a humungous current rushed into the capacitor through the relay contacts.

    Quite soon the relay (and thus the warning signal) ceased to work. When I investigated I found the contact arms (originally springy) had softened with the repetitive heating until they could be bent like tin foil. Analysis of the power source suggested a current in excess of 50A for up to some 100ms through a pair of 2.5A rated contacts each time the warning was triggered. The silly thing was that the hazard warning siren only took a few milliamps, so the addition of a single resistor to limit the current into the capacitor would have prevented the problem entirely.

    This, as in the example reported here, is a combination of Dunning Kruger and lack of interest in the outcome.

    1. Anonymous Coward
      Anonymous Coward

      Re: Far from unique

      an intermittently occupied environment that became hazardous once or twice a day.

      Yeah, we had a gents toilet like that as well.

      1. Mike 137 Silver badge

        Re: Far from unique

        "Yeah, we had a gents toilet like that as well."

        This one was more interesting and less scatological. It was walk in a climatic simulator where it rained typically twice a day, but it contained 240V AC power sockets so researchers could plug their equipment in. The purpose of the siren was to warn anyone inside at the time of "rain". I'm amazed it ever passed safety inspection (supposing it actually got inspected).

  19. -tim

    There can't be anything wrong if it isn't even hooked up

    I worked for a place with a Sun E10K and it of course came with redundant power supplies. The problem is no one ever plugged in one side. The one that was hooked up had a nice short curved lead near the corner of the server that went to a rather large plug. Someone managed to get their foot in that loop while walking too close to the very expensive computer and the power went out.

    1. Anonymous Coward
      Anonymous Coward

      Re: There can't be anything wrong if it isn't even hooked up

      I still remember the monstrous Sun Enterprise 10000 machine that sat in the centre of a large, caged off area at Telecity in London. It was next to our far less impressive racks of Proloiant servers, and I always wondered who it belonged to as it must have been costing them a fortune to host there.

    2. Anonymous Coward
      Anonymous Coward

      Re: There can't be anything wrong if it isn't even hooked up

      I remember the E10K well - we had 3 Telco customers worldwide using them.

      E10K machines provided "domains", a form of virtualisation/isolation, where each domain was its own SunOS instance. After one of our customers had a couple of production downs due to the whole E10K chassis going down we narrowed it down to being triggered by "lsof" which we supplied as part of our team's Operations toolpack ("lsof" was not part of SunOS).

      Eventually with the help of the lsof author it was determined that it was lsof *reading* a certain SunOS kernel structure inside a E10K "domain" that caused the whole chassis to die. The author raised a ticket with Sun and all Sun would say was "you should not be reading that kernel structure" and indicated they did not consider the chassis dying to be a bug).

      So we had to stop using lsof on our E10K customers machines (I don't believe back then there was any adequate alternative to lsof that we could use) which made debugging of operational issues with our software a lot harder.

      The Sun Netra ft 1800 was a lovely machine. Worked with one of our customers on the ever first deployment of it in Europe (Sun sent a senior Sun trainer come over from USA to give his first ft1800 course). It ran a modified SunOS and our customer was not happy a couple of years ago when Sun told them the only upgrade path was to move to non-Fault Tolerant Sun machines as they'd canned that product line (after never porting Solaris to it).

  20. Anonymous Coward
    Anonymous Coward

    Employee micromanagement via technology contributes to sudden server emergency

    Good on the guard, I say. Making the usual overbearing employee monitoring into something somewhat personally enjoyable. The likeable personality of the guard's bosses may well have contributed to the sharpness of the raps, as well.

  21. Dave314159ggggdffsdds Silver badge

    What was the guard's name?

    A 'he'? Shame, I was hoping she was a Molly guard.

  22. Bruce Ordway

    Reversed polarity

    After powering up a test system I heard an awful humming, shut everything down before any damage.

    Turns out that all extension cord reels in the shop had been installed with reversed polarity.

    I stuck with wall outlets from that point without any problems.

    I also notified the building manager but I'm not sure they were rewired (or if he understood what the problem was).

    1. Is It Me

      Re: Reversed polarity

      I have seen more wall sockets wired up the wrong way then extension leads

  23. Anonymous Coward
    Anonymous Coward


    We moved into a new office and I was asked to put in structured cabling and floor ports. Pulling up the floor showed that it was made of a polystyrene sandwich between two layers of flooring. Whoever had laid the power cables had obviously never heard about the adverse reaction between polystyrene and PVC. They had just cut channels in it and put the wiring in without using conduit. The PVC had gone hard and brittle. Disturbed one cable and a chunk of the outer sheath fell off exposing the copper.

  24. Nifty

    I worked for a company that made process control systems. It was in the days of film cameras with a big electronic flash on top. The installation technician would start up the system for the first time with the processor cabinet doors open. Check that all processor lights were blinking rhythmically as expected. Check the SCADA graphics (and have a good listen to the factory noises) to see that all valves were in the correct position and motors off so that the system was 'at rest'. Now was the time, before locking up the cabinet, to take the photo that went into the project album. These were used by the salesmen or for reference before service techs went to a site.

    Flash, then bang and wallop. All processor lights now frozen and valves and motors could be heard activating at random across the factory. Turns out the EMP from the cameras flash was enough to upset the processors so they crashed. One processor crashing was enough to corrupt the shared RAM so the others would crash. In the process, I/O sent signals that caused plant activations*. Normally the risk of this happening even in a lightning storm existed but was low, because the steel cabinet doors are kept locked.

    Previous photography had been done the same way without crashes but there must have been a probability that every Nth photography flash would trigger a crash. The procedure was modified after that to ensure that the rack was powered down in the presence of a flash gun.

    *For those worried about the implication that factory fires/explosions can occur if automation goes doolally, there were additional direct safety interlocks that override to (theoretically) prevent overfilling/over pressure, overheating, etc. But things can still require a lot of cleaning up and checking after an incident.

  25. Potemkine! Silver badge

    Taling about contractors and server room

    We had to install a humidifier in a server room which was too dry, there was a risk of electrostatic discharge.

    This humidifier required to install a condensate drain. We asked a supplier to install it. Some months later, we saw traces of humidity on the wall through which the drain was going. After a detailed inspection, we discovered the drain was cut just after reaching the wall, and the water was dropping freely into the wall. The supplier just "forgot" to connect the drain to an evacuation.

    1. Nifty

      Re: Taling about contractors and server room

      I was testing a shower to see if water pressure was good while viewing a show home once, and discovered the builders had done the same thing.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like