back to article Server broke because it was invisibly designed to break

The week, and indeed the year, may be ebbing away to their respective conclusions, but The Register continues to toil away at On Call, our weekly reader-contributed tale of techies triumphing under trying circumstances. This week, meet a chap we'll Regomize as "Kris" who arrived at his desk one Monday morning to find the phone …

  1. KittenHuffer Silver badge
    Childcatcher

    Shoulda got a ....

    Magic switch then!

    ------->abra.cud@bra

    1. Will Godfrey Silver badge
      Thumb Up

      Re: Shoulda got a ....

      That one is an absolute classic

    2. trindflo Bronze badge
      Devil

      Re: Shoulda got a ....

      Funny story!

      I note that nobody thought to put an ohmmeter onto the switch. I knew of an instructor of a hardware class who took apart a relay with a clear plastic shell and painted clear nailpolish on the contacts. A student could visibly see the relay switch, and If you didn't verify your assumptions, you would never work out the problem.

  2. DS999 Silver badge

    A service provider that doesn't bill because their attempted fixes failed?

    Surely this is a service provider which no longer exists, because the ones that last are the ones that nickel and dime you for everything. Even when they are wasting your time with failed fixes, they are billing you $300/hr for that service if it isn't covered under the normal contract/warranty.

    At best if a replacement part doesn't fix the issue they won't charge you for the part, though I've seen them insist on putting the original part back to avoid such a charge!

    I recall at one consulting gig the company I was working for had a few obsolete Clariion arrays that were under T&M support. I'm not sure of the details, but apparently IT typically did internal billing at a set price per gigabyte to the various departments, but these arrays were legacy arrays that had come when they acquired another company and the manager of that unit didn't have to pay the internal billing so long as they kept using the arrays they "owned".

    So something went wrong and the IT support people called EMC, who sent out someone who said he hadn't been trained on hardware that old so he said they'd send someone else. Then he came back the next day because they didn't have anyone in the local field office trained on such outdated hardware! So he muddled along, running up huge charges, and finally and belated fixed the issue. IT of course made the "owner" of that array pay that hefty bill, and it wasn't long before those legacy arrays were gone from the floor! I imagine one big bill can wipe out months or even years of "savings" on that per gigabyte charge.

    If EMC didn't charge for all the failed attempts at fixing it, I'm sure that bill would have been a lot smaller. But T&M is T&M, and EMC is legendary for making you pay every penny of that charge when you are not on a support contract!

    1. Anonymous Coward
      Anonymous Coward

      Re: A service provider that doesn't bill because their attempted fixes failed?

      Not hardware but...

      When I took over my current role one of the things I inherited was a non-working, but critical, VAT report which was preventing sign-off of a major project.

      The consultant asked if they could bring a trainee with them, which was fine by us, we're always happy to support youngsters and their development. After a day on site the report still didn't work correctly and was still preventing sign-off, yet they billed us for 2 people's time for a full day! I passed it back to the finance people who'd arranged all of this (before I started) stating, in polite language, where I felt the provider could stick their invoice but I suspect it was just paid.

      In the mean time I'd fixed the report anyway - seeing *0.15 in a VAT report was a bit of a giveaway, especially as the rate had changed recently. Why the **** of a consultant had hard-coded the calculation when the database had a TAX field with the correct value is a mystery. It was never going to work anyway even if the rate had been correct as some items, e.g. fuels, carried a different rate.

      1. Elongated Muskrat Silver badge

        Re: A service provider that doesn't bill because their attempted fixes failed?

        Fuck me, Anyone who has never heard of VAT codes needs to be forbidden from touching any software that is remotely financial.

        $Deity only knows that there are some horrible kludges in the dim and distant past in our software, but I can confidently say that hard-coding VAT as 15%, on everything, VATable or not is not one of them.

        1. C R Mudgeon Bronze badge

          Re: A service provider that doesn't bill because their attempted fixes failed?

          Once upon a time, Ontario had both Canada's federal GST (a value-added tax) and a provincial sales tax (not value-added). [1] My then employer got in a new sales order system that, among numerous other problems, had ongoing trouble with doing tax calculations correctly.

          I never worked much with that system (dodged a bullet for sure), but my colleague who had primary responsibility for it remarked on the folly of buying a system from a company based in a jurisdiction with no sales tax at all (New Hampshire) for use in a jurisdiction with two of them.

          [1] They've since been "harmonized", i.e. rolled into a single larger tax, HST, which the two governments divvy up between them.

          1. The Oncoming Scorn Silver badge
            Pint

            Re: A service provider that doesn't bill because their attempted fixes failed?

            Smug mode - GST (5%) only in Alberta.

            1. Montreal Sean

              Re: A service provider that doesn't bill because their attempted fixes failed?

              Lucky Albertans!

              Over here in Quebec we've got GST (5%) and then QST (9.975%) added on top of that. :(

              1. Yes Me Silver badge
                Headmaster

                Re: A service provider that doesn't bill because their attempted fixes failed?

                Surely it's illegal to mention that in English. En francais s'il vous plait.

              2. Mostly Irrelevant

                Re: A service provider that doesn't bill because their attempted fixes failed?

                Quebec does a lot of things to sabotage themselves. The only people it hurts is people and businesses in Quebec. Stuff like the contest rules that mean all national Canadian contests are "excluding Quebec" and you can only use computer systems that support the assinine "tax on tax" system. All it does it hurt Quebecois.

              3. Scott 26
                Alert

                Re: A service provider that doesn't bill because their attempted fixes failed?

                > Over here in Quebec

                Good fishing in Key-beck.

                1. jake Silver badge

                  Re: A service provider that doesn't bill because their attempted fixes failed?

                  Good fishing? Nowt but tiddlers in Ka-bek ... that's all you can pull through an 8 inch hole.

                  Although you can occasionally pull a hoquet player out from under thin ice.

          2. jake Silver badge

            Re: A service provider that doesn't bill because their attempted fixes failed?

            "rolled into a single larger tax, HST"

            Nicknamed "Hard Shit, Taxpayer"? Curious ...

        2. Anonymous Coward
          Anonymous Coward

          Re: A service provider that doesn't bill because their attempted fixes failed?

          You would be surprised or maybe not.

          I've had it a couple of times in companies where they built their own accounts packages and hard code the VAT. Why? I think it's because they are in a specific industry like mobiles telecoms and don't realise it's not a good idea. In fact it's a very bad idea as it causes you problems. However the very worst award goes to the programmer who not only hard coded the VAT but they didn't keep it in just one place. If they are reading this and it's entirely possible I hope you have stepped on the same number of upturned UK plugs in the night as you hard coded each instances of that VAT rate. If that has happened I doubt you can still walk to be fair. Still to this day I don't understand the thinking or logic behind it.

      2. C R Mudgeon Bronze badge

        Re: A service provider that doesn't bill because their attempted fixes failed?

        ISTM that even if the consultant had fixed the problem, the trainee shouldn't have been billable.

        That's rather like me asking if I can store my car in your driveway for a couple of weeks, giving you permission to use it occasionally -- and then bil[lk]ing you full commercial car-rental rates for the entire duration.

        Of course, if the trainee had spotted the problem (entirely possible -- fresh eyes and all), pay the invoice without a quibble.

        1. Anonymous Coward
          Anonymous Coward

          Re: A service provider that doesn't bill because their attempted fixes failed?

          Absolutely they shouldn't have been billable. Frankly, as the provider had already charged a significant sum for this useless report, we should have charged them for my time to actually fix it. Ultimately I suspect it was paid as the difference on a six-figure project wasn't all that significant.

          1. Anonymous Coward
            Anonymous Coward

            Re: A service provider that doesn't bill because their attempted fixes failed?

            Oh, I don't know, I've always found that the finance people are more than happy to oblige when I tell them "don't pay this invoice" or "only pay for these line items on this invoice". I always get "no problem, will do"...

        2. Tim99 Silver badge

          Re: A service provider that doesn't bill because their attempted fixes failed?

          We live in a retirement village An elderly neighbour has a "care option" plan with a local supplier. It can be used for a range of things like, taking her shopping, cleaning the house; or in this case, providing equipment to help. She had broken her hip and was immobile, so the nice care options lady suggested an iPad to keep in touch with her family, many of whom lived interstate. The nice lady ordered a WiFi only iPad. When it was delivered they realized that she would need a WiFi base unit to connect to her existing modem router (used with her existing smart TV). The nice lady then arranged for a WiFi access point and an installer. I received a phone call saying that the neighbour and the nice lady couldn’t get the iPad to work. The access point had not been set up correctly, and was acting as a bridge, but without WiFi. A quick prod with a paperclip and 3 minutes later everything was working.

          A few weeks later the neighbour got a bill from the installer for $490 for the installation - The access point was not included as it was supplied by the care provider. I asked the neighbour why the bill was so high, she said that the installer was there all afternoon. "What was he doing?" I asked. "He spent most of the time sitting on the floor looking at the paperwork, and on the phone trying to set it up. He seemed overwhelmed, so I gave him coffee and biscuits". It looked like he was learning how to do a simple install, and was charging her for it! All that was required was to unbox the item, check which port on the patch panel went to the wall socket, plug it in and turn it on; then connect the iPad to WiFi and type in the password. I wondered if the installer had spent time teaching her how to use the iPad - No, he said that was the care option providers responsibility. I phoned the installer’s company and the care options lady - After an exchange of views with the installation company, the neighbour didn’t have to pay…

      3. Anonymous Coward
        Anonymous Coward

        Re: A service provider that doesn't bill because their attempted fixes failed?

        That's reminded me of the time when, many years ago, the software develpment company I worked for dabbled in training people in C. Working as a consultant to a large bank who had contracted in a team of developers from a large and well known company, it was somewhat alarming to find that half their team had been on our beginners C programming course the week previously...

        1. yetanotheraoc Silver badge

          Re: A service provider that doesn't bill because their attempted fixes failed?

          "somewhat alarming to find that half their team had been on our beginners C programming course the week previously"

          Perhaps more alarming that the other half of their team *hadn't* been on the beginners course.

        2. YetAnotherLocksmith Silver badge

          Re: A service provider that doesn't bill because their attempted fixes failed?

          They had to earn that money back ASAP, right!?

        3. RockBurner

          Re: A service provider that doesn't bill because their attempted fixes failed?

          "half their team had been on our beginners C programming course the week previously..."

          And what was the "training" like??

          When we moved into our new house earlier this year, we ordered up a new oven, including delivery and fitting....

          Pair of likely lads turn up (very pleasant guys actually), heave the old one out, heave the new one in, and then spend an hour fiddling with the electrics.....

          Suddenly there's a BANG and the entire house goes very quiet, and dark.

          Now....

          I'm NOT an electrician.. but I DO know that the live cable goes to the live connection - often marked with an 'L' , neutral to 'N', and earth to 'E', or ground symbol.... There's only 3 wires, all uniquely coloured.... how hard can it be?

          Apparently the training that these lads , well, one of them, had had, didn't really make that distinction. Allegedly it was a 2 hour powerpoint about not lifitng things with your back, but using your legs, and not much else....

          The live was (iirc) wired in nicely to the ground connection on the oven, the neutral to live, and the earth cable to neutral.

          To add to that, they'd stripped back something like 3 feet of the outer sheathing and about 6 inches on each cable so there was bare metal everywhere.

          We told them, politely, to go home, and succeeded in getting our delivery and installation fee completely re-imbursed (that was a fun phone call I have to say), and got a proper electrician in, mainly to make sure the 'delivery boys' hadn't made any other messes.....

      4. Alistair
        Windows

        Re: A service provider that doesn't bill because their attempted fixes failed?

        IBM s/36, MAPICs II. It was the second install of MAPICs in Canada. IBM had bought the underlying core code AFAIR, from someone in Europe. It did not deal well at all with the Import tax/Federal manufacturing tax/fuels tax issues, took about 6 weeks to get that sorted, and I can't recall how many attempts at a patch. It introduced me to RPGII, and some weird compiler issues on that machine.

        It also predated GST by a few years.

      5. J.G.Harston Silver badge

        Re: A service provider that doesn't bill because their attempted fixes failed?

        Well, when I first started coding in the early 1980s, I thought "VAT" *meant* "15%" in the same way "fortnight" *means" two weeks. After all, absolutely every instance of VAT is a identical match to "15%" innit. If it was supposed to be variable, it should be known to be variable, like, I dunno, different values of VAT existing. It was regularly on the news that "this year's income tax rates..." or "this week's interest rates" so clearly they were variables, but clearly "VAT" is some strange form of English that means fiVe And Ten. You'll be wanting me to declare var pi next.

        It was only in 1992ish when VAT went up to 17.5% that I discovered it wasn't a constant.

        1. Anonymous Coward
          Anonymous Coward

          Re: A service provider that doesn't bill because their attempted fixes failed?

          McVities had a running battle with the tax man over Jaffa Cakes and VAT.

          If they were biscuits, as the taxman claimed, then they were liable for VAT as the chocolate made them a 'luxury' ('plain' biscuits are exempt)

          If they were cakes, as McVities and the name suggested, then they were tax-exempt.

          It ended up with McVities baking a giant Jaffa Cake and allowing it to go stale. Like a cake, it went hard, rather than going soft like a biscuit

          1. prandeamus

            Re: A service provider that doesn't bill because their attempted fixes failed?

            Read "The Fourth Bear" by Jasper Fforde because <$SPOILER$>

    2. EVP

      Re: A service provider that doesn't bill because their attempted fixes failed?

      ”Surely this is a service provider which no longer exists”

      My thought were that this story is improbable, if a service provide doesn’t even try to bill. Maybe there were some pre-emptive yelling by the company’s directors?

      The article says: ”at least it didn't cost Kris's employer a cent!” but the server run a piece of important software. The incident probably cost the company quite a lot of money.

      1. FatGerman

        Re: A service provider that doesn't bill because their attempted fixes failed?

        Depends on the size of the client. Some service providers won't bill if they know they've effed up and they know the client knows that. They'd rather take that small hit than risk losing someone who will definitely pay them more in the future,.

      2. TeeCee Gold badge
        Headmaster

        Re: A service provider that doesn't bill because their attempted fixes failed?

        Depends whether that's "important software" as in the company cannot function without it, or "important software" as in somebody important can't populate his spreadsheets and produce his quarterly presentation without it.

        1. EVP

          Re: A service provider that doesn't bill because their attempted fixes failed?

          Indeed, a very good point you have. It was probably the former one, though. There was no mention of bosses going ballistic over the incident and having everybody to drop whatever they were doing to fix it, but just mere "three days into an incident that deprived the company of an important application Kris was under more than a little pressure."

          Logic clearly dictates that PHBs' spreadsheets were not affected >:)

        2. jake Silver badge

          Re: A service provider that doesn't bill because their attempted fixes failed?

          Or "important software" because it was one of the C*'s favorite pR0n viewers.

          Yes, I've seen it. The viewer, I mean. Although ultimately I had to see the pR0n, too. Fired him for it.

      3. Anonymous Coward
        Anonymous Coward

        Re: A service provider that doesn't bill because their attempted fixes failed?

        Yes, this was quite an incident never to forget. I'm "Kris" in this incident and it did happen, however with a bit more complexity which was distilled down for the article. I was hired in 1998 as a contractor and became a full-time employee when this incident occurred. When I came in, I inherited this old Proliant 6000 and an even older server running Novell Netware.

        The company I work for was being spun off from Polaroid. Yes, you read this right. You know, that big company that made cameras and stuff. The fledgling company wasn't big by any means. I think we had about 250 users at the peak of their operation and only got smaller as time went on.

        As Polaroid was imploding, they cast off their truly awesome tech support group and brought in this outside service provider whose name completely escapes me now, that was made up of total idiots. Totally escapes me since I've been retired about a decade and that incident occurred in 1999! I hated dealing with the group because they were so slow in responding to incidents, critical or not and in the end, I usually fixed the problem anyway even though I wasn't supposed to.

        When it came to any charges for services, we were no longer under Polaroid's accounting, although we still used their infrastructure such as their network and other tech including the T1 down to some remote locations where products were shipped from. Service calls, and other things not related to the infrastructure were always invoiced separately and since I was responsible for verifying and approving invoices, this one never appeared on my desk.

        The more critical stuff such as accounting information was still being done on the big SAP server and was in the process of being transferred to Scala 5 and the service department and R&D kept their small databases on the server that crashed along with other user-data. This data was transferred to brand new hardware once the sale was finalized and the poor old Proliant 6000 was retired. The new servers were still in operation a decade later when the old spin-off finally closed thanks to the Great Recession.

        While this incident was occurring, I truly could absolutely not believe the unprofessional nature of this tech. He came in completely disheveled as if he had been out the town the night before and smelled like he slept in his ashtray. When he showed up days later, his appearance was the same, so this wasn't a one-off thing. My biggest complaint back to the company when asked for feedback was that "mission critical" hardware wasn't treated like "mission critical" hardware. This was a server, regardless of whatever content was on it, and not some user's PC or Mac that had failed.

        The old 6000 was heavy. I think it weighed more than the APC UPSs we had and required two people to slide and support it from the racks. And yes, much to my shock and horror as well, the fixes we thought would fix the problem didn't do it.

        The classic thing was his line when he powered it up. "Yup, it's broken", and no I didn't enjoy this all that much. It was quite stressful. I was under constant stress until the old beast was finally retired because I was afraid something else might die.

        1. MachDiamond Silver badge

          Re: A service provider that doesn't bill because their attempted fixes failed?

          "As Polaroid was imploding, they cast off their truly awesome tech support group and brought in this outside service provider whose name completely escapes me now, that was made up of total idiots."

          Why do companies think that outsourcing something has any chance of saving money? If there is enough to employ one full time employee, it's worth having in-house. If it was a whole group, assuming they were all needed, the only way for the outsourced company to make any money is by paying their employees far less. That often means a bunch of turnover, which leads to nobody having much experience with each customer's set up.

          When I had a manufacturing company I did outsource metal machining. I really wanted to have it internal, but the capital cost was prohibitive and I didn't have enough work for a couple of full time machinists. By having a whole machine shop on call with loads of machines and lots of well trained staff, the material I ordered would be made into sale-able parts much faster and often times before the invoice was due.

          1. MJB7

            Re: A service provider that doesn't bill because their attempted fixes failed?

            I can see why IT support might seem like a sensible thing to outsource. Instead of having one person who needs to be able to handle DB admin, network configuration, hardware wrangling, etc (and who can't go on holiday), you can have a share of a full-time DBA, and a networking guru, and a hardware experts - and with cover so you don't have to worry about holidays or sickness.

            In practise of course, it never seems to work. I remember when I foolishly deleted a file and asked IT (a three man in-house team) to restore a copy from backup if possible (if not, I could regenerate from scratch). Three days later I got an apology for having taken so long - they had been struggling with sickness in the team. AT THE SAME TIME, my customer had corruption in their SourceSafe database. The only fix was a restore from backup. Until this was sorted, a ten person team were effectively unable to work. It took their (out-sourced) IT over a week to restore it.

            1. DS999 Silver badge

              Re: A service provider that doesn't bill because their attempted fixes failed?

              I agree with that outsourcing logic for smaller companies where you need one or two DBAs. You have to cross train people to do each other's jobs poorly in case one leaves, or screw them out of vacation "sorry you have to reschedule your vacation since the other DBA just gave his two weeks".

              For a big company that has large teams where losing one or two doesn't have much impact, it is really hard to see the savings other than outsourcers paying a lot less by having a bunch of cheap overseas labor.

          2. A Non e-mouse Silver badge
            Mushroom

            Re: A service provider that doesn't bill because their attempted fixes failed?

            If there is enough to employ one full time employee, it's worth having in-house

            And that is why you fail.

            What if that one employee goes off ill, or has a holiday? Or what if, $Deity forbid, they quit and all their knowledge disappears with them?

            To consider running something in house verses outsourcing, I reckon you need a bare minimum of two staff but preferably three. And don't forget their overheads when working out if insourcing is cheaper: Pension, cost of desk space, equipment, training, etc.

            I'm not being paid by an outsourcing company, but people who always assume that "Outsourcing is bad" are just deluded. Like anything in life, there is rarely a standard good answer. Outsourcing has its place, just as insourcing has its place.

            1. J.G.Harston Silver badge

              Re: A service provider that doesn't bill because their attempted fixes failed?

              "Or what if, $Deity forbid, they quit and all their knowledge disappears with them?"

              I had that. I did a "cover" job at a company between the previous IT Bod leaving (well, escorted off the premises might be closer to the truth) and until the replacement was available to take over. I spent my three weeks doing a "deep dig" constructing process documentation that did not exist anywhere outside the previous chap's head. I feel that I write this stuff clear enough for a random stranger to pick up and run with, but I dispair at how crappy most organisation's process documentation is, if it exists at all. My principle is: can somebody take over within ten minutes of me falling under a bus?

            2. MachDiamond Silver badge

              Re: A service provider that doesn't bill because their attempted fixes failed?

              "What if that one employee goes off ill, or has a holiday?"

              What if the company you chose goes out of business, burns down or has a data breach? I'd never have only one person that could do the job, but it might be just one that does do the job on a regular basis. I gave an example of where I was outsourcing and why. I do not forget the total cost of having an employee and am not naive enough to think that my only cost for them is their wages. What YOU might be missing is that there are substantial risks when you outsource that you can't control. That other company may only have one person assigned to do the task for your company that can go off sick, on holiday or get hit by a bus. Did they cross-train anybody to step in? Did that employee move to another company where you know neither who the employee was or the company they went to along with your proprietary information? As I had a manufacturing company for 17 years, losing an employee also meant losing a big chunk of training and being able to get feedback on our processes. An outside company could change how something is done to save time without realizing that the change makes a big hit on quality. I had one soldered part that required a very specific high-temp solder. If that got swapped out for a more common type, I'd have had thousands of products coming back under warranty. The in-house staff didn't have access to any other type of solder on that line and I could monitor that. Who knows if that outsourcing company is doing the same thing.

    3. Bitsminer Silver badge

      Re: A service provider that doesn't bill because their attempted fixes failed?

      $300/hr?

      How old are you?

      1. Anonymous Coward
        Anonymous Coward

        Re: A service provider that doesn't bill because their attempted fixes failed?

        How about you?

    4. C R Mudgeon Bronze badge

      Re: A service provider that doesn't bill because their attempted fixes failed?

      "Surely this is a service provider which no longer exists"

      That depends. If you only rarely screw up, you can afford to make it good. If you screw up all the time, well, you *should* make it good anyway, but affordability becomes a problem...

    5. scott2718282828

      Re: A service provider that doesn't bill because their attempted fixes failed?

      When I was in college 50 years ago I had the pleasure of spending a year working in a lab with an ancient DEC PDP-8. Apparently the total contract with the university was lucrative enough that it covered maintenance even for relics like this that were kept busy by researchers whose budget did not cover getting a replacement.

      There are no chips in a PDP-8. All computer logic is built from individual components, including the core memory. If you've never seen this, each individual bit of memory is a separate tiny magnetic doughnut. This machine had the expansion memory, a glorious additional 4K words of memory that required an additional rack. A lot of effort goes into getting a coherent signal to go down a memory bus through two cabinets and suppressing noise and echoes.

      There were components used to tune the impedance of each line in the memory bus in several places to enable stable operation. There was still someone at DEC who knew how to do this. He was a vice president by then, but occasionally to honor the contract he still had to come out and get his hands dirty. Fortunately this was all pre-paid maintenance without the extra hourly charges.

      1. jake Silver badge

        Re: A service provider that doesn't bill because their attempted fixes failed?

        "When I was in college 50 years ago I had the pleasure of spending a year working in a lab with an ancient DEC PDP-8. "

        In 1972 (50 years ago) the PDP8 was hardly an "ancient relic". They had first been released a mere 7 years earlier, and in fact were still in production.

        "There was still someone at DEC who knew how to do this."

        There were many people at DEC who knew how to do this. The PDP-8 was still rolling off the production line.

      2. MachDiamond Silver badge

        Re: A service provider that doesn't bill because their attempted fixes failed?

        "All computer logic is built from individual components, including the core memory."

        They had the guts from a US space shuttle?

  3. KittenHuffer Silver badge
    Holmes

    It then occured to me ....

    .... that this was an open and shut case!

    -------> Never had an open and shut case!

    1. Greybearded old scrote Silver badge

      Re: It then occured to me ....

      You don't usually get quantum superposition on an object this large. I'm sure there should be a Nobel in that.

  4. Disgusted Of Tunbridge Wells Silver badge

    > Which is a fine idea because nobody should be electrocuted while working on a server.

    'Elf and safety gone mad

    1. EVP
      Devil

      “nobody should be electrocuted while working on a server”

      Where has all the fun in the world gone?

      1. b0llchit Silver badge
        Coat

        Where has all the fun in the world gone?

        It got electrocuted?

      2. KittenHuffer Silver badge
        Devil

        It has been sucked out of the world by a bunch of vampires that were rejected by HMRC for being 'too nasty and petty to be tax employees'!

        -------> The AVERAGE H&S employee!

        1. heyrick Silver badge

          Yesterday I had to spread salt around where people walk so that nobody slips on the ice, as happened a few days ago when the freezing fog descended.

          It's -4 at night, but it's bone dry and there's no ice at all. No snow or rain is forecast, until Sunday afternoon when it'll rain and be about 8C.

          Oh, and nobody is working there until Monday morning.

          But, alas, the Health and Safety person wanted it done so it had to be done by someone. They will "review" whether or not it needs to be done again on Monday morning (forecast: raining and about 9C).

          Typical H&S, nothing happened when it was needed, and after the fact it's followed by pointless gestures "just in case".

          1. YetAnotherLocksmith Silver badge

            Just in case you or the other 10 up voters see this: black ice causes more issues than a little snow. Because it is invisible. That rock salt stops it happening.

            1. DropBear

              I have that reminder permanently engraved on an old set of tailpipes (and my leather jacket). I'm still amazed that there appeared to be precisely zero nanoseconds between riding normally and sliding on the asphalt behind my bike throwing off sparks like crazy. The human brain is a funny thing, and apparently impact sensitive, even in a full-face helmet.

              1. jake Silver badge

                Zero nanoseconds?

                In my case, almost an eternity ...

                I high-sided a bike at Thunderhill once. At about 140 MPH ... a friend & I were practicing drafting, and swapping the lead back and forth ... He cut in a trifle early and my front tire hit his rear as we were accelerating out of a sweeper. Not good with hot sticky race rubber. I remember thinking "Well THAT was a daft thing to do! This is going to hurt. Pull in your arms & legs & get ready to roll. Shit, the wedding is in a week, SWMBO is going to be PISSED! I wonder if I'll be able to get a beer in the ER? Hopefully Doug will get the bike back to the house for me." and then I hit the deck and was rolling through gravel. I wasn't in the air for more than a tenth of a second or so.

                No hospital, just a badly bruised knee. I drove myself+bike home. Made the wedding.

                SWMBO still laughs at my limp in the videos.

                That reminds me, I must call and ask if one can get a beer in the ER ... and if not, fix it.

        2. pirxhh

          Tax men

          Years ago, I ate at the tax office's canteen about once a year. It was incredibly bad - a basement room, greasy unappetizing food staff calling Methuselah young man... the works.

          It was so bad it had to be intentional, lest the taxmen be pleasantly sated and forgiving after lunch. Can't have that - lads need to keep their edge by nurturing low-grade ulcers, apparently.

      3. mantavani

        It died in a freak HVAC accident.

      4. ttlanhil

        Surely the ones working on the server should be left alone, and the electricity supplied to all the shoulder-surfers and manglement interrupting to ask why it's not fixed yet?

        1. Strahd Ivarius Silver badge
          Devil

          Just hand them a pair of wires.

          When they get fried, say with a sad nod, "not yet fixed"...

    2. TeeCee Gold badge

      Also counterproductive.

      Anybody who opens one up to work on it without turning the power off first needs to be electrocuted before they do some real damage. At worst they probably won't do it again and at best they won't be causing any more problems.

      1. jake Silver badge

        "Anybody who opens one up to work on it without turning the power off first"

        Shirley you mean "without unplugging it first"?

      2. martinusher Silver badge

        >Anybody who opens one up to work on it without turning the power off first needs to be electrocuted

        Anything high voltage is now required to be shrouded which is why you find power supplies in a perforated metal box and mains wiring to switches and the like covered. So there's no reason to cut the power when you lift the lid; it may not be a good idea to work on a plugged in chassis but it definitely won't harm you. (Where this is voltage expect to find warning stickers.)

        You need this overkill because most engineers and techs don't work with voltages higher than about 30 or so. They're likely to hurt themselves on vintage equipment with its open terminals, fragile valves and the occasional 'live' chassis (not to mention stored/restored charge in smoothing capacitors!).

        1. MachDiamond Silver badge

          "So there's no reason to cut the power when you lift the lid;"

          There can be. If lifting the lid kills the forced air cooling path, you won't want the computer doing heavy work or it might lead to lots of expensive magic smoke.

        2. mbrinkman

          You are correct in that high voltage (over 48v) is covered, but that is not the only hazard. Systems are required to have what is called 240va protection, which says any power rail that can deliver over 240 volt amps (similar to watts) must either be also covered or have a lock-out to cut power. This is because if one of these power rails gets shorted it can heat up significantly and burn. Better servers get by this by having multiple power rails that each can only go up to 240va, but this adds cost.

    3. phuzz Silver badge
      Unhappy

      Most of the really dangerous voltages are sealed inside the PSU(s), so usually the biggest danger of working inside a running server is getting mauled by the fans.

      Server fans are no joke, I had one finger chopped right down to the bone by one. Then of course I had to take half the server apart to replace the now damaged fan, and to clean up the blood splatter.

      Never got electrocuted though.

      1. Disgusted Of Tunbridge Wells Silver badge
        Mushroom

        I would imagine the case open switches are to prevent you from breaking the thing if you fat-finger the electronics while they're powered. Eg: shorting a +12v with a +5v.

        1. Anonymous Coward Silver badge
          Boffin

          It's more to do with the controlled airflow, or lack of it when the case is open. Power off so the thing can't overheat.

      2. breakfast Silver badge
        Coat

        Server fans

        Honestly servers are something of a niche topic so you'd be wise to be careful around people dedicated enough to label themselves as server fans.

      3. jake Silver badge

        "Server fans are no joke"

        There's a reason that ties were fair game for anyone with a pair of scissors at most early Silly Con Valley companies ... hand-built one-off prototypes often had voracious cooling fans. The theory was that if we starved 'em of ties they'd be too weak to do much other damage. Not even IBM Field Circus folks were safe from the shears ... HP, somewhat wisely, decided ties were pretty useless fairly early on, as did DEC's Palo Alto contingent. Most of the other big names followed. Some of the Military Brass working out of Ford Aerospace, Varian & etc. had special dispensation to do without neck-ware "so they'd fit in with the locals" ... We had high hopes that it'd become a world-wide movement and we'd be done with the useless things for good.

        1. Potty Professor
          Headmaster

          Ties are dangerous

          At school in the sixties, doing "A" Level Metalwork, I was using a pair of hand shears to cut thin sheet steel whilst working at a bench. I heard a strangled cry from behind me, and turned around to see one of my classmates desperately trying to prevent his face being dragged into the chuck of the lathe. He had been threadcutting, and the end of his tie had become wrapped around the workpiece and was being wound in. He had his left hand on the headstock casting and his right on the tailstock, but the tension on the tie was still pulling him in.

          I took a couple of steps and sliced through his tie with the shears, and he shot backwards across the aisle and ended up flat on his back on my workbench. I then had to use the shears again to cut the tie around his neck, the tension had tightened the knot so that it couldn't be undone.

          The Metalwork master (Trevor Jones) decreed that no ties were to be worn in the workshop ever again, which prompted fierce opposition from the Deputy Head, but Trev got his way, and the Woodwork master followed suit in a show of support.

          1. YetAnotherLocksmith Silver badge

            Re: Ties are dangerous

            Damn, that was lucky he was tapping, otherwise he'd be dead.

            Rule 0: don't be on fire.

            That covers things like this and loose clothing, hair, etc when using machine tools!

        2. Hazmoid

          ties and their uselessness

          Having worked everywhere from a Stock broker to a small 2 man outfit, I can safely say that ties in Australia are only on salesmen, politicians and lawyers. Most of the workplaces in Perth ( where summer temperatures regularly exceed 40 degrees celsius (100 F for the seppos)) have loosened their workplace dress standards to polo shirts, or open collar business shirts, simply because ties just end up a mess or dragging in something.

        3. Robert Carnegie Silver badge

          Ah neck ties... I thought you meant cable ties.

      4. Anonymous Coward
        Anonymous Coward

        Should have just replaced the fan and left the blood as a scrafice to the server gods!

        1. phuzz Silver badge

          The server was destined to go to a customer, and they tend to be unhappy when their new server arrives with blood splatter inside.

          1. jake Silver badge

            Doesn't matter.

            99.999% of your customers will never see the inside.

    4. Death Boffin
      Angel

      I did see someone get branded once when they got their bracelet across a large 5V power supply.

      Remember folks, rings and watches off.

      Icon better take that shiny thing off before working around electricity!

      1. Old Used Programmer

        As an EECS student at UC Berkeley, I oncce got a tour of Lawrence Berkeley Lab. When we got to the 188" cyclotron, they shut the beam down, slid a shielding block aside, and we were invited into the actual cyclotron "room". We were also sternly told to leave things like watches on the tray outside, even if said watches were supposedly anti-magnetic. Inside the room, there was a constant magnetic field of 37.5 Kgauss.

        Our guide took one of the old style AAA pen light flashlights out of his pocket and stood it on end on his hand. He then pushed at it with the other hand. It didn't fall over.

        1. anothercynic Silver badge

          Magnetic fields are a fun thing to demonstrate, and they can be a bitch too... especially if you have... ahem... body modifications, and if you ever experienced some shrapnel.

          This is why when you are given an MRI, you are asked if you have ever been shot, been in an accident, have had rods, pins or screws inserted, and if you have taken out any rings, studs, bars, etc before they let you into the room.

          6 Tesla is not something you want to experience pulling at metal in your body...

          1. MarthaFarqhar

            Some old tattoo inks result in a burning sensation as they used metals in the old days for pigmentation.

            A 7T research scanner wasn't the place to be to find that out.

      2. jake Silver badge

        A friend of mine reached behind a large bank of relay racks and managed to get his Rolex watchband across the 48V supply ... The resulting loud "CRACK!" and fans spinning down, coupled with the smell of roasting/burning pork, were rather disturbing. To say nothing of the screaming. I managed to calm him down & get him to the ER ... Xrays showed little balls of gold melted into his wrist behind the 3rd degree charring. The surgeons later told him he was lucky to still have full use of his hand. Today, 25 years later, the scarring is still impressive, despite skin grafts. He got a new band for the watch, and now wears it on his other wrist. It still works.

        And people wonder why I always take off my wedding ring when working on electrical stuff. Yes, that includes cars, trucks, boats etc.

        1. Potty Professor
          Windows

          Watch Band

          I was on holiday with my parents, touring northern Italy. One of the hydraulic cam followers in the six pot Chevy's engine failed and was making an annoying clattering sound. The rockers on the Blue Flame engine still had adjusting screws, so we removed the rocker cover and screwed the adjuster down so that it was acting like a solid lifter, with the correct clearance.

          Unfortunately, the battery live post is only about an inch from the hold down clamp, and Dad managed to short them out with his expanding metal watch strap. He let out a howl of pain and used his right hand to pull the strap away from his left wrist, burning two fingertips and his thumb in the process. He let go of the strap, which immediately snapped back onto a different part of his wrist, and this was repeated a third time before the metal had cooled enough not to cause another skin burn.

          He was in such pain, and was taking so many Tramadol, that he couldn't drive, so I had to do the driving for the rest of the holiday, and he had to have a lot of treatment on his wrist when we returned to England.

      3. MachDiamond Silver badge

        "I did see someone get branded once when they got their bracelet across a large 5V power supply."'

        I recall a story of somebody badly injuring a finger due to RF radiation burns from a wedding band. They seemed to have reached up into a magnetically charged device and the ring became a highly conductive shorted single turn coil. Got really hot, really fast.

    5. Tim99 Silver badge

      You’re not a BOFH then?

  5. Admiral Grace Hopper
    Alert

    It is a fine idea

    Having walked in on someone doing the 50 Hz boogie, fortunately with no long term impact on the 220 V AC disco dancer, it's not a fun experience for anyone involved.

    He (and I) were fortunate that there was a wooden-handled broom near to hand to allow me to nudge him free.

    1. KittenHuffer Silver badge

      Re: It is a fine idea

      I still have the scar on my thumb that resulted from an inadvertent connection to the national grid. I'm just glad my Mum hadn't spotted me on the way into the house and demanded that I remove my wellies! If I hadn't been so well insulated then my feet (along with the rest of me) would probably be six feet under at the moment!

      1. Anonymous Coward
        Anonymous Coward

        Re: It is a fine idea

        At the age of 9 I was flung across a room when my Dad asked me "just hold these multimeter probes in this socket while I figure out which breaker I need to switch off".

        Some years later, while changing a light fitting with the power still on, I connected live to earth via my thumb. No shocks, but the smell of cooking meat. I had a small area of my thumb that was completely dead for several years.

        I learned. Eventually....

        1. usbac Silver badge

          Re: It is a fine idea

          When I was about 13, I learned why they make special high voltage test leads when I tried to measure the cathode voltage on my vacuum tube o-scope with normal test leads!

          1. KittenHuffer Silver badge

            Re: It is a fine idea

            ZAP?!?

            1. usbac Silver badge

              Re: It is a fine idea

              Yeah, I woke up on the other side of the room a minute or two later!

          2. MachDiamond Silver badge

            Re: It is a fine idea

            "When I was about 13, I learned why they make special high voltage test leads when I tried to measure the cathode voltage on my vacuum tube o-scope with normal test leads!"

            I waited a few years longer before doing something rather similar.

          3. heyrick Silver badge

            Re: It is a fine idea

            When I was about that age, one weekend at boarding school, a bunch of us decided to have a go at fixing the huge broken CRT TV in the dining room. It would have been 1987 or 1988. One of those horrible things in wooden frames with a cardboard back, inches of dust, and radiating heat from all over. It's a miracle it never self combusted.

            I was the one who foolishly decided to move that thick black cable out of the way so we could better see how the rest was wired up.

            I woke up in sanatorium aching like hell, apparently having backflipped over an entire table (don't remember that part), with matron trying not to have a heart attack over it.

            Took a long time getting over the pain.

            A little later the physics teacher pointed out that although it was rough on me, it was perhaps the best outcome considering that we really had no idea how to fix it and six boys crowding around a live chassis to poke and prod it while it was switched on (to see if it made anything better) was a massive deadly accident waiting to happen.

            Why didn't we know about the chassis being live? Inches of dust over the pathetically tiny warning label and zero previous experience with televisions.

            There's one thing I completely refuse to do in life, and that's go anywhere near the back of a CRT.

            1. jake Silver badge

              Re: It is a fine idea

              "backflipped over an entire table"

              And now you know why they call it a flyback circuit ... Seriously, kiddies, don't dick around with old CRTs without getting expert instruction. There be dragons there. This is true even if the box has been powered off for a long, long time (years).

              1. DropBear
                Facepalm

                Re: It is a fine idea

                At some point, my old CRT TV suddenly just stopped turning on - investigation revealed that a large transistor in the PSU simply crater-exploded for some reason. I replaced it (and any suspicious passives around it), but I was the whole while scared shitless of working around an (even unplugged) CRT, for fear of touching something still charged. TV works just fine to this day, btw.

        2. Gene Cash Silver badge

          Re: It is a fine idea

          I remember replacing a light switch in the wall as a kid. I clicked off the breaker for that room beforehand.

          I did it carefully with insulated pliers as I was just paranoid, put it back in the wall, flipped it, and the light came on!

          That's when I learned nothing in the breaker panel was truthfully labeled.

          In my "new" house, I flipped all the breakers off, then on one-by-one to map everything. Imagine my surprise when the back bedroom lights came on... along with a cupboard in the front kitchen. Or when the garage lights came on... along with the second-bathroom lights.

          1. jake Silver badge

            Re: It is a fine idea

            That's actually common here in the US. A logically wired house has three walls and the ceiling wired to one breaker (which also feeds the fourth wall (and maybe the floor) in another room)), and the fourth wall (and possibly the floor) wired to the three walls and ceiling in another room. The theory is that in the event of a tripped breaker, every room will still have power/light. Exceptions in this house include hallways/stairwells, bathrooms, garage, basement, laundry rooms, mudroom, etc. The machineroom/museum/mausoleum/morgue has it's own solution, soon to be up and running. Hopefully.

            Took a week to plan, a week to rip out the old wiring, a week to pull new wire, and a week to button it back up. Not a job to be taken lightly.

            1. MachDiamond Silver badge

              Re: It is a fine idea

              Putting lighting on one breaker isn't a big deal for built in fixtures. It's even better now with LED lighting as it takes a whole lot of installed LED lighting to trip a breaker. I like to have the lights stay on when I pop a circuit breaker. My house has a couple of wiring WTFs that I need to sort out.

          2. An_Old_Dog Silver badge

            Truth in Breaker Labelling

            When we moved house, I was 9, and my dad asked me to help him figure out which breaker controlled the garage overhead light. I did a visual scan, saw that a plug-in below the light was fed by the same wires supplying the light, made a "dead-short" plug -- 18 gauge lamp wire shorting the two power prongs -- walked up to the outlet, lined everything up, put my left hand over my eyes and jammed in the plug. The relevant breaker tripped, I said, "It's that one," and pulled the plug out of the socket. There was a chunk missing from one of the prongs due to the brief electrical arc. It's the one-and-only time I used that technique.

          3. J.G.Harston Silver badge

            Re: It is a fine idea

            Yes, I've just surveyed my Mum's house. One breaker controls the living room lights and the cellar head, but not the cellar itself. Another breaker controls EVERYTHING ELSE. One breaker controls the sockets in the kitchen and the cellar, another everything else except two sockets in the back room. On breaker didn't seem to do anything, so I had to follow the cable, and found it eventually ended at a junction box with nothing else connected. So that I didn;t have to document it, I just removed the whole circuit.

            When I moved into my house 30 years ago, the first thing I did was replace the consumer unit and re-arrange all the circuits logically. Up Lights, Down Lights, Up Sockets, Down Sockets, Heating Control, Outhouse.

        3. An_Old_Dog Silver badge

          Re: It is a fine idea

          He didn't explain to you the importance of holding the probes by the insulated part?!

    2. wi94e&*L2Xm?

      Re: It is a fine idea

      There is no exposed high voltage in an open server case. That’s not to say there are no hazards, but high voltage isn’t one of them.

      1. Anonymous Coward
        Anonymous Coward

        Is this a 220/50 thing or did they stop putting these things on server chassis at some point?

        While we commonly use 220v UPS units over here, most of our servers are specced out for 120v PSUs. I have seen plenty of CID(intrusion detection) switches, but never one that cuts the PSU. That's across a few decades of server chassis from multiple vendors. While that totally seems like something I would have found on an old COMPAQ back in the day(along with saving the BIOS data to the HDD, and putting torx screws in everything), I never did see something like this.

        Considering the number of hot swappable components, I'd hardly look kindly on the box dumping on me if i cracked the case to swap one. It seems like the sort of thing that would cause me to slap a bright yellow warning sticker the size of a pie plate on top of the cover for the next guy.

        That said, yes, I also shut the damn things down before I crack the case open whenever I can, take ESD precautions, all that jazz, but having to down the server to swap out something plugged into an internal USB port? No thanks.

        1. mbrinkman

          Re: Is this a 220/50 thing or did they stop putting these things on server chassis at some point?

          Power supplies are now auto sensing. They can switch between 240 and 120 on their own.

          1. Anonymous Coward
            Anonymous Coward

            Re: Is this a 220/50 thing or did they stop putting these things on server chassis at some point?

            Not so much sensing 120/240, more that they will accept anything from 100-240v (I've even seen down to 90v and up to 250v)

            1. Anonymous Coward
              Anonymous Coward

              Re: Is this a 220/50 thing or did they stop putting these things on server chassis at some point?

              Just come across a couple of Cisco PSUs...

              85-264 VAC, 47-63 Hz

              and a HV AC/DC PSU: 85-305 Vac/47-63 Hz or 192-400 Vdc

    3. John Sager

      Re: It is a fine idea

      I've had several belts over the years, the first at age 2 on my Gran's bed unscrewing the ovoid switch hanging from a flex that such houses still had then. My other Gran then promptly knitted a cover for the one in her bedroom.

      1. C R Mudgeon Bronze badge

        Re: It is a fine idea

        "... the first at age 2..."

        Uh huh.

        I can remember my first zap too -- and the grey Westclox alarm clock whose power plug gave it to me when my very young self touched the prongs while pulling it out of the wall. I haven't seen that clock in something like half a century, but when I searched for them just now, I *instantly* recognized the model in question among the results. Some lessons don't fade...

        You know the famous Santayana quote: "Those who cannot remember the past are condemned to repeat it"? I looked it up in context, and I believe this is what he was talking about. He meant "remember" literally, not metaphorically. The line isn't about learning from history, or anything quite so highfalutin, but about the more mundane idea that all the parental warnings in the world about plugs and sockets don't carry nearly as much weight as that one first electrical shock, personally experienced.

        It seems to me that this is why we seem stuck in the same loops: timesharing, PC, diskless workstation, more powerful PC, thin client, still more powerful PC, cloud, rinse, repeat.

        Or outsource, bring it back in house, offshore just-in-time, get bit by supply-chain issues and pass a CHIPS act, rinse, repeat.

        The new people who have come up into positions of authority haven't *been* there the last time around, so whatever theoretical understanding they might have of the risks, the knowledge isn't visceral.

        Same with all the "Who, me?"ish "rm /*", and didn't-make-a-backup, and hit-the-off-switch stories, and what-could-go-wrong'ness in general: the single best way to learn not to do that is to have already done it, possibly several times, and paid the price.

        1. YetAnotherLocksmith Silver badge

          Re: It is a fine idea

          Though if you pay the ultimate price, you're not useful for recalling anything.

          So stickers and interlocks do have a place.

          1. C R Mudgeon Bronze badge

            Re: It is a fine idea

            "So stickers and interlocks do have a place."

            Agreed. I wasn't attempting to claim otherwise.

  6. Wanting more

    had a printer with the same fault

    I had a old dot matrix printer that wouldn't print. The interlock switch (just a leaf spring pair of contacts) thought the lid was always open. Fix was to bend the contacts a little.

    1. Anonymous Coward
      Anonymous Coward

      Re: Fix was to bend the contacts a little.

      I've got a dehumidifier whose "full bucket" sensor is like that. Every now and again I have to bend the contact strip out a bit, and the fault is resolved. Eventually it'll just go "snap", and that'll be it, I suppose, but it's done quite well...

      1. aidanstevens

        Re: Fix was to bend the contacts a little.

        Then just bypass the switch if it breaks?

        1. Rob Daglish

          Re: Fix was to bend the contacts a little.

          If it's like our dehumidifier it will fill a tank with water, and you want it to stop when the tank is full or you'll have a dehumidifier sitting in the middle of a puddle, so maybe not bypassing the switch would be a better idea?

    2. C R Mudgeon Bronze badge

      Re: had a printer with the same fault

      There are occasions -- rare, but they happen -- when I want my washing machine to operate with the lid open. I'm planning to add a switch to bypass the interlock.

      But it will be a momentary-contact switch, so that I can't accidentally leave it in the wrong position.

      1. YetAnotherLocksmith Silver badge

        Re: had a printer with the same fault

        That's wisdom, talking.

        1. C R Mudgeon Bronze badge

          Re: had a printer with the same fault

          "wisdom"

          For which, read: "a lesson learned the hard way" :-/

    3. MadDrFrank

      Re: had a printer with the same fault

      Similar experience .. DEC dot-matrix printer not moving paper, under contract so I was not supposed to mess with it, but I sneakily put an ohm-meter across the (readly visible) terminals of the paper-sense microswitch and established it was permanently OC.

      Put in a service call. "Engineer" duly turned up, changed circuit boards, changed PSU, changed paper drive motor, said he would return to base to get a new wiring harness "because there was nothing else left to change". I suggested ohm-meter check on the paper-sense microswitch, he was scornful -- "it clicks so it must be working".

      I offered to fetch a meter, and he admitted he did not understand what I was talking about. Eventually I persuaded him to humour me, took seconds to demonstrate that switch remained OC whether clicked or not.

      I then demonstrated with a piece of wire that the paper moved when terminals were bridged. He said this was not in his training course. He had minimal understanding of anything electrical or electronic, he had been trained as a board-changer.

      He returned next day with a microswitch and watched me remove the old one and solder the new one in -- he had never seen such a process.

      1. Rob Daglish

        Re: had a printer with the same fault

        As a "temporary" job, I spent about four years working for a company that handled a fair bit of warranty work for some very large Server and Desktop manufacturers, along with at least three printer manufacturers. Their field engineers were generally paid around £19k, and staff turnover was very high, and as a result most of them, at least as far as printers were concerned, were purely component swappers - all diagnostics were undertaken under direction from technical "specialists", ie. the call centre staff in Romania reading the service manual for whatever equipment you happened to be working on at the time.

        1. DropBear
          Devil

          Re: had a printer with the same fault

          Now, I would be inclined to take this as a personal affront if only were I not aware that (completely useless) checklist-reading tech support was equally bad regardless of physical location.

  7. Tony W

    Safety interlocks and fault protection

    I had many call-outs to an FM transmitter that was extremely reliable apart from safety interlocks and the airflow detector for the cooling fan, one of which had a tendency to switch the whole thing off at a very unsociable hour. So now I tend to think of interlocks first in any totally-off situation.

    1. MachDiamond Silver badge

      Re: Safety interlocks and fault protection

      "I had many call-outs to an FM transmitter that was extremely reliable apart from safety interlocks"

      KOTR in Cambria, CA?

      Man that was years ago but I got all of midnight to 6am on Thursday mornings to myself and a friend of mine. Might have had all of 2.3 listeners. At least we know we had one person since somebody would call when the transmitter went off.

  8. Sam not the Viking Silver badge

    The Interlock and Health and Safety

    We supplied some equipment to a well-known process contractor in the food industry and their procedure was that all MCC's had to be commissioned by their own staff 'for safety reasons'. No problem, I had a trainee with me and it would be good to see someone else doing the job "with the highest regard for safety of personnel and equipment". We had to undergo a two-hour H&S 'induction' to cover the general, local and specific hazards, which included not carrying two cups of coffee and never carry a tray on stairs etc. as well as the usual hazards of electricity and the like. We were examined on the knowledge so transferred.

    Eventually, suited, booted, gloved and glasses, we meet the 'Appointed Person' who would be doing the hands-on pre-checks etc. before starting the plant and after further words about safety he proceeds to open all the panel doors, then switch on the isolating circuit breakers using a spanner....... My trainee and I, stood well back...... Of course, nothing worked and after a frustrating half-hour or so where the expert would take no advice, I tentatively suggested that the door-switches might be causing, correctly, an interrupt in the circuit? He proceeded to by-pass the switch whereupon the breakers slammed in and the interior became live. It was mind-bogglingly dangerous. We couldn't work or even observe under these conditions so we went back to the project manager who sighed and said they had 'a bit of a problem' with this guy.

    We returned the next week to repeat the exercise, albeit with a reduced H&S Induction, this time successfully, with a different 'expert'. I still have the first guy's name in case our paths ever cross again.

    1. I ain't Spartacus Gold badge
      Megaphone

      Re: The Interlock and Health and Safety

      If I can't carry a tray on stairs, then how am I to get my cup of tea from the kettle in the office kichen (ground floor) to my desk on the second?

      I warn you, there are serious health and safety implications here! If you stop me from getting my tea, you are likely to require immediate medical attention.

      1. yetanotheraoc Silver badge

        Re: The Interlock and Health and Safety

        Are you allowed to carry the kettle?

      2. jake Silver badge

        Re: The Interlock and Health and Safety

        I always had a coffee pot and tea kettle in my office. Mini-fridge, too.

        I suspect I only got away with it because Security saw me coming in first thing in the morning, sans coffee, once too often ... Early morning emergency calls are fun for nobody.

      3. Roland6 Silver badge

        Re: The Interlock and Health and Safety

        H&S isn’t stopping you getting your tea, the rule just means you can’t take your singular cup of tea back to your desk on a tray.

        So from the information provided you can either carry your (singular) cup or just take your tea break away from your desk…

        1. heyrick Silver badge
          Happy

          Re: The Interlock and Health and Safety

          Singular cup?

          Amateurs.

          1. Roland6 Silver badge
            Happy

            Re: The Interlock and Health and Safety

            >Amateurs.

            Agree... looks like the rules don't apply to teapots etc.

            This (plus tea cosy) also has the benefit that you can have several really good cups of hot tea at your desk and not have to make a second trip for the biscuits...

        2. Chris 239

          Re: The Interlock and Health and Safety

          Carrying the cup of tea in your hand! That's a burn hazard!

      4. C R Mudgeon Bronze badge
        Happy

        Re: The Interlock and Health and Safety

        I just an hour ago read this line in Roger Zelazny's "My Name Is Legion":

        "The following morning, during the ambulatory, coffee-tropism phase of preconsciousness..."

      5. Robert Carnegie Silver badge

        Re: The Interlock and Health and Safety

        Use the lift?

        I don't remember rules here but I think that a tray to carry with one hand is all right for stairs, you can still use the hand rail to support yourself. If you drop the tea tray anyway... that's messy but mendable.

        I have a tray of small size; a design also exists where a platform is hung below a carrying handle.

  9. Prst. V.Jeltz Silver badge
    Flame

    guesswork

    the server remained broken and that the motherboard must be the real problem.

    So just take another guess then? and wait three days for parts to find out? (leaving customer without their "critical application" )

    This is the difference between "Part Fitter" and "Engineer" , whatever type of hardware is involved.

    Car dealerships often do this and charge you truly eyewatering prices for each wrong guess - despite being in a far better position than anyone else to do the job properly due to having all the knowledge and proprietary diagnostic equipment on hand.

    1. wub

      Re: guesswork

      "Car dealerships often do this and charge you truly eyewatering prices for each wrong guess - despite being in a far better position than anyone else to do the job properly due to having all the knowledge and proprietary diagnostic equipment on hand."

      And access to information about "silent recalls".

      My girlfriend owned an Audi Fox that was about 2 years old at the time (late '70s). One day, we drove into town, shopped, and then tried to go home - car simply would not start.

      Had it towed to the nearest Audi dealer (in Chicago area), they scratched their heads, finally replaced the fuel pump and billed us for that plus 3 hours of diagnostic effort. I looked at that, called bullshit, and asked to keep the old pump, which I boxed up and shipped back to Audi national headquarters to be checked, along with a copy of the repair invoice.

      In the meantime, we started a 3 week road trip to the Rocky Mountains. Yeah, why would I be typing this if the car didn't fail exactly the same way at a campground 125 miles out of Jackson Wyoming? Had the car towed to the only foreign car dealer in town (Subaru, btw). When we rolled into the parking lot, a guy outside looked up, saw us, whistled at another guy, gave him a couple of minutes of instructions. By the time we were doing the sign-in in the office, they already had the car off the truck and into a bay.

      Fifteen minutes later, the car was fixed! No parts, and they only billed the 15 min labor!

      The head mechanic kept up on trends in car problems, and knew about the faulty connector design Audi had committed on the power circuit feeding the fuel pump, so he told the guy to simply pull the wires out of the connector and solder them together. Never had another problem (with ignition) on that car.

      Oh, and when we got back from the trip, Audi had sent us a letter stating that the fuel pump was just fine, along with a check reimbursing us for the full, original, dealership "repair". About six months later, there was a formal recall of that connector.

      And my then girlfriend had had the foresight to pay an extra $2.00 on her car insurance to cover emergency towing situations, so the repair in Jackson, Wy was our only expense - insurance covered the 125 mile towing charge. Clever girl!

      She is still my wife - I'm glad she thought I was good enough to keep around. We drive a Subaru now.

      1. Anonymous Coward
        Anonymous Coward

        Audi electrics, oof

        Literally every person I knew that drove an Audi had buyers remorse because of electrical gremlins. I can't imagine buying an EV from them.

        Subaru's electrics are pretty good, but they use a ton of soy based plastics that brittle up after a few years, so be careful under the hood when slipping of those ten year old connectors or you may end up splicing the wiring harness :)

        Other than that just remember to change the oil, swap the axle boots before they crack, and do your head gaskets and timing chain/belt on schedule and it will literally run till the body falls off the power train. I just wish they made a decent PHEV. Hoping to see one from then in the next couple years. Glad you had a good experience with the dealer, but too many of them are pirates these days, and of course get used to hearing "that parts not in stock but they have it in Washington, three days."

        1. C R Mudgeon Bronze badge

          Re: Audi electrics, oof

          Interesting. I had a series of Volkswagens back in the 80s, and at least one of them was prone to electrical problems -- the late 70s Rabbit [1] I think. I hadn't realized that was a thing though.

          Fortunately there was a mechanic in town who specialized in VWs. [2] One of that car's electrical problems turned out to be a burned-out trace on the main fuse/relay panel's PCB. It was a common problem I believe. He gave me the option of buying an $x00 replacement part, or he could just solder on a jumper. I went for the jumper; it worked fine.

          That guy had an amazing ability to imitate car sounds. I'd describe a (non-electrical) problem. He'd say, "Does it sound like this?" and make a noise. "Yes." "I know what the problem is."

          [1] aka Golf. They couldn't call it that in Leftpondia for a few years, though, due to a trademark collision.

          [2] The shop still exists, and still bears his name, but I have no idea whether he still works there.

          1. Rob Daglish

            Re: Audi electrics, oof

            Audi electrics are still crap.

            I've got a 2 month old Q7 that fails to even try and start the engine a couple of times a week, the first time being after we'd had it three days. Dealer can't find anything wrong, but has so far suggested that:

            1) We aren't pushing the start button hard enough

            2) We aren't pushing the start button long enough

            3) We aren't pushing the brake pedal hard enough

            4) It might be because the car is suffering in the recent cold weather

            Unfortunately, every time it goes in for a week or so, we need to use two cars to go anywhere as we can't get 3 kids seats and two adults in the back of the courtesy cars (An A6 and an A3 so far)

            And to top it off, they've managed to put two six inch long scratches down the side of the car when they last had it.

            I don't think I'll be buying another Audi anytime soon...

          2. usbac Silver badge

            Re: Audi electrics, oof

            I had a '77 VW Scirocco with the same great fuse box design. The one where the PCB traces would burn out before the fuses. It required replacing most of the wiring harness under the dash to change it. Some genius decided that instead of having a few connectors on the back of the box, the wiring would all be soldered to the back side of the PCB.

            They were horrible to replace. As a teenager with no money, I had to do it myself. I sold the car shortly after that. Unfortunately, my "upgrade" was was a '78 Triumph Spitfire!!

          3. DropBear
            FAIL

            Re: Audi electrics, oof

            One evening I parked my VW B4 Passat just fine, only to have it crank but completely refuse to start the next morning. Turns out it's an openly known secret that the relay supplying the ECU (109 I think...?) has a bulk fabrication defect that WILL fail on you at some point, no exceptions. Replaced the relay, car started right up. Opened the old relay - there's this massive cold solder joint that failed to solder properly due to an obvious proximity to a larger metal bit soaking away all the heat. Jolly well done (and covered up), you VW fuckers...

    2. Gene Cash Silver badge

      Re: guesswork

      On this side of the pond, we call it "firing the parts cannon"

      I've been in a lot of discussions where I've gone "and if swapping the $x00 ECU doesn't fix it, I'm not being charged for that ECU, right?" AND GOTTEN IT IN WRITING because I've learned the hard way.

      1. heyrick Silver badge

        Re: guesswork

        Had a Citroën Saxo where when idling, the engine speed would ramp up and down.

        The place that sold it, and two different garages, said it could be the oxygen sensor or... something else (I forget), and suggested we start with the oxygen sensor (€600 inc. labour), and if that doesn't work do the other thing (€800 inc. labour).

        Took a third option, traded it in for a C1 that never gave any problems.

        1. heyrick Silver badge

          Re: guesswork

          "(€600 inc. labour)"

          Why the hell does replacing a little gizmo like that (~€35) cost so much?

          I currently drive an Aixam (I don't have a licence) and to change the rear pulley (CVT) it required a full engine removal to get to it. Hours of work. That cost me about €600.

          1. jake Silver badge

            Re: guesswork

            "Why the hell does replacing a little gizmo like that (~€35) cost so much?"

            Last century's Johnson/Evinrude small single cylinder outboards in the 4 to 6.5 hp class (sometimes called sailboat pusher motors) have a copper water line from the water pump down by the prop up to the power head. The connection between engine block and tube is sealed with a simple o-ring. When the engine is run, the o-ring gets wet. When the engine is shut off, the o-ring dries out. Leaving behind anything dissolved in the water. Especially salt, if the motor is run in the ocean.

            These deposits build up over time, gradually putting pressure on the o-ring. Either the iron block, or the soft copper has to give. The copper loses, of course. So eventually, the copper pipe is pinched off, and the engine no longer gets fresh water, and so it overheats.

            The fix is simple. Remove the copper pipe, heat it up, run a mandrel through it to pop out the pinch, replace the o-ring (a 19 cent part) & Bob's your Auntie.

            Unfortunately, R&Ring the pipe involves pulling the entire power head, which is just short of a complete engine overhaul. About a 4 hour job. At $165/hr shop rate. Plus any parts that may need to be replaced because they are too worn or corroded to be reassembled. Plus the cost of removing stuck bolts, of course. And you might as well put in a new water pump, replace the gearcase prop seal and change it's oil while you're in there.

            Bottom line: An honest shop might have to charge well over $1,000 to replace a 19 cent o-ring ... on a motor that is worth maybe $500 if it runs well and looks pretty.

  10. Anonymous Coward
    Anonymous Coward

    A colleague and I did overnight maintenence at a financial institution to add more memory to a Sun Ultra-60 server (one of Sun's first forays into using a standard PC chassis).

    Unplug server, take out of onto a non-static mat, open it up, and find we had to move the powersupply to get to the memory slots.

    do that, put in the extra memory simms, put the powersupply back, close her up and reconnect.

    Nope, new memory not seen.

    Rinse and repeat a few times, testing while the server is still unscrewed, PS still not back in place - all fine. Put it back together, and only half the memory could be seen.

    Eventually, thumbs now getting raw from the constant reseating of memory simms, we found that there was a obscured ribbon cable connecting the PS to something, that would flex the motherboard just enough to pop the one end of the newly added simms out of their slots when replacing and screwing the PS back in place. Some careful finagling later and it was all working as it should.

    One of the many bad decisions that signalled the beginning of the end of Sun Microsystems, was in adopting cheap PC chassis...

  11. Will Godfrey Silver badge

    Elfin Safety

    ... nearly closed down a factory with a safety barrier they insisted on. This was on a machine that that took in sheets of printed thin card, 'pastry' cut them into small opened out pill packets, then folded them and glued them - it was an absolute work of art. The numpty decided that the moving machinery was too dangerous for the operators to adjust all the tiny hooks and levers that did the folding while the machine was running.

    There was a slight problem with this, fine adjustments had to be made at speed to take into account things like air flow. The machine also must never be started with packets already in it. Nor must it be stopped while the glue line is running. I'm sure you can see where this is going! Medical supplies companies are extremely fussy about delivery times, so after the loss of a major company, the business decided to move the trouble maker to another unit that had 'greater' need of him.

    P.S. It was also controlled by a cough PLC with plug-in I/O modules the size of a large book. For a while I had a regular job replacing driver transistors :)

  12. BenDwire Silver badge
    Facepalm

    Almost de-ja vu

    I thought this story was going to end the same way mine did, but no. My ending was far more embarrassing:

    20 years ago I had a IP-Cop firewall looking after my home network. As such, I'd used the smallest PC case I could find and some old Pentium II motherboard I'd salvaged from somewhere. In those days the power switch really was a power switch, with 240V (yes, that long ago) running to the front panel.

    One Friday evening the internet went down, and after much diagnostic swearing I decided that the fizzing noise from the little machine must be the cause. It was obviously a fried PSU so I decided to swap it out with a spare I kept for such occasions ... except the output plugs didn’t match the obsolete mother board. No matter, I thought, I have a soldering iron and heatshrink so just get on with it. Some hours and bleeding fingers later (did I mention it was also a cheap case) I had the thing reassembled and ready to go. Switch on. Nothing ... apart from the same fizzing noise.

    Only then did I replace the mains lead and all was well ...

    1. Anonymous Coward
      Anonymous Coward

      Memories

      of those old AT era power supplies.

      Full mains to the power switch, and unkeyed connectors to the logic board with one correct combination of possible connections and 9 wrong ones that would absolutely blow the logic board if you flipped the switch on them.

      So many opportunities for "fun" and expensive mistakes.

      1. Anonymous Coward
        Anonymous Coward

        Re: Memories

        AT power connectors had keying. They also came in pairs that also were keyed to each other. I'm not saying you couldn't connect them wrong, just that it would take some effort.

        1. Anonymous Coward
          Anonymous Coward

          Re: Memories

          Iirc - there you two cables/connectors on those AT PSU. You just had to connect them with the black wires where in the centef

          1. Martin an gof Silver badge
            Facepalm

            Re: Memories

            Unless - if memory serves - it was a Dell, where the connectors were exactly the same but all the wires were in different slots.

            M.

        2. John Brown (no body) Silver badge

          Re: Memories

          "AT power connectors had keying. They also came in pairs that also were keyed to each other. I'm not saying you couldn't connect them wrong, just that it would take some effort."

          As a field engineer during that time, I can say with absolute confidence that they were NOT all keyed. It did eventually become a sort of standard, but it most certainly wasn't originally. I had case where a new PSU was to be fitted and the connectors on the leads had the key tags on them but the connectors on the system board did not or the key tags were in the wrong place A quick couple of snips with side cutters to remove the key tags and we were back in business, being careful to make sure P8 and P9 were the correct way around :-) (and yes, double and triple checking that it wasn't a special OEM PSU with the same connectors but different wiring before starting the replacement!)

  13. Boris the Cockroach Silver badge
    Unhappy

    The hell that

    is interlock switches....

    where I attend on a daily basis has them everywhere... safety doors, robot cages, manager's ofice door(no wait I'm fitting that one next week...)

    What really bugs me about them is that every machine/robot manufacturer goes out of their way to use a different switch on everything, so that when an interlock switch goes to the great cage in the sky, you have to get the exact one to replace it. even if the machinery is supplied by one company, they have to use different switches on the enclosures.

    Oh... and dont try running the machinery after bypassing the lock.... that ends......... badly

    On the plus side though it gives the janitor something different to mop up.....

    1. Unoriginal Handle

      Re: The hell that

      "On the plus side though it gives the janitor something different to mop up....."

      Indeed...

      Many years ago (early 80s) the gentleman at a bed manufacturer who, having presented flowers to the lady who jilted him a few days before proceeded to the factory floor, defeated the interlocks on the machine which turned coconut husks into matting for mattresses and put himself in.

      The largest piece of him which was found was about 3/4 of a hand...and ruined a large length of mattress component.

    2. Swarthy

      Re: The hell that

      One of my sprogs decided to try bypassing safety interlocks - she wanted to run the microwave with the door open (unsure why, ADHD in a 10 year-old girl: good idea, bad idea makes no difference - it's an idea.)

      Fortunately, the interlocks defeated her, but it was a pyrrhic victory. It cost a few hundred USD to replace the microwave, but the child was safe.

      1. DropBear

        Re: The hell that

        I am greatly puzzled - if the interlocks held, why was there anything to replace...?

        EDIT: for that matter, even if the interlocks FAILED, why was there anything to replace...?

  14. imtekcs
    FAIL

    I have worked Information Technology for over 30 years. I have had enough experience to know that there are many holes in this story. I have issue with this article being highlighted as "Kris" being under so much pressure and being a savior. Explanation forthcoming.

    I have issues with how this article puts down the repair technician. We all make mistakes but showing up "About five hours later the tech shows up smelling as if he lived in his car and slept in his ashtray". Most service companies have a 4 to 8 hour response window. A tech should show up looking professional but whether or not he smokes, has nothing to do with his ability or should it have been highlighted in this article. Just sounds like "Kris" is smug and nitpicking.

    Even though he got pleasure from seing the techs smug look. I am not defending the repair tech for his smoking or his look. Some of the best break fix techs do not care about outward appearance.

    Many of your big server companies like Dell and others do not always employ on site repair techs. Maybe in larger cities or for certain customers but most of the time you will get a 3rd party repair tech. Whether that tech is with a large or small local certified solution provider or sometimes these repair techs are contracted with tech "side hustle" apps. Most of the time these repair techs have insane work loads and are only given so much time per case based on the initial tech support phone call. It's not what you want to hear but this is a practice that has gone on for a long time.

    Where I have issue with "Kris" is where he blames the repair tech for a wrong diagnosis and the amount of down time. Granted the repair tech made a wrong diagnosis but "Kris" has not taken any blame or responsibility for the downtime.

    1. If this is a mission critical application, why didn't "Kris" have a fail over contingency plan? A disaster recovery plan? If this was so important, there should have been a multi tier fail over plan. No mention of a fail over server or cloud based solution to keep the application going. One single point of failure brought the entire application to a halt for 1 week.

    2. Most servers can get a 3 to 5 year service support plan to include hardware failures. Why wasn't there a service and support plan on the server? They usually guarantee a repair tech on site within 4 hours depending on the tier of support. So if this app is so critical, why wasn't the hardware protected to ensure maximum up time? Was the server too old to still have a support contract? If it was that old, then why wasn't a new server budgeted for? These are all valid questions. I know some companies do not like to spend money on IT but then it is up to the IT Manager to make his case and budget for these issues. Since this company has an R&D department, it sounds like the could spend more money on IT.

    3. Spare parts. Since they pay for each repair call, why doesn't "Kris" have some spare parts on hand. If I supported a server without some form of service agreement, I would keep spare parts on hand. Not a lot but at least a power supply, fans. RAM, etc.

    Where I have issue with the repair tech:

    It is indicated that the server has multiple power supplies. Multiple power supplies usually do not die all at once. It can happen but usually doesn't. That would have been my first indication to be looking for another cause first. Checking to make sure it was plugged in correctly. Hopefully the server was attached to a UPS and I would have checked to make sure the power outlets on the UPS still work.

    "Kris" could have easily done this too. You always check the easiest and most obvious points of failure.

    Conclusion:

    Without knowing more, to me it sounds like "Kris" is not qualified to do his job and wants to shift focus off his shortcomings. Most of the younger IT people I encounter are like this. They claim to be IT but only have one focus. Afraid to do and learn more, especially on the hardware side of things. If I had been in "Kris's" situation, I would have been terminated for not having a disaster recovery plan and not making more effort to find a solution much sooner. Granted "Kris" did find the interlock solution. But it makes me question what led to the failure? Poor server environment? Someone tinkering with the server chasing too much? Improper grounding? Or just a simple switch failure? These are issues that have to be considered.

    1. yetanotheraoc Silver badge

      employee vs employer

      It seems to me your three numbered issues with "Kris" are properly the responsibility of the business.

      1. Anonymous Coward
        Anonymous Coward

        I've been dispatched as the tech enough times that I disagree.

        I carried spares for common components, and especially power supplies. I also carried a multi-meter, and knew which pins I had to bridge to trick the PSU into firing up. I would do everything I could to resolve a server down call on the first visit, and If we couldn't get the server running, we would discuss with the client the impact of downtime, and if we could, swap the drives into a working server chassis to resolve the down time while we worked on the server chassis. I pulled a few all nighters over the years rebuilding something overnight and finishing up just as people were coming back to the office.

        We also billed for our time as a result, and our clients were usually happy to pay the bills.

        In contrast were the talking apes that some of the server manufacturers would dispatch for our "24x7" and "same business day" service contacts, which never ceased to amaze be. Though to be fair some of them were actually solid techs, but the company also refused to support them or issue them the tools and spares they needed to do their job properly. One vendor got so bad that we actually had to take a 50,000 volt prod to a sever to get them to replace an intermittently failing component after the 5th service call. But after years of decline in service quality and professionalism, they were sending out glorified geek squad kids, who had no qualms dragging a repair out for weeks and shutting a business down in the process.

        1. Rob Daglish

          Re: I've been dispatched as the tech enough times that I disagree.

          > they were sending out glorified geek squad kids, who had no qualms dragging a repair out for weeks and shutting a business down in the process.

          Well yes... but given the levels of pay some of these providers have, you can make more money stacking shelves in Tesco, driving a bus, or a hundred and one other things, so where are the smart people going to go? Not IT it seems!

    2. doublelayer Silver badge

      I think some of your critiques are making assumptions that aren't necessarily the case.

      "If this is a mission critical application, why didn't "Kris" have a fail over contingency plan?"

      We don't know how critical it was, just important enough that it being down for days led to some unhappy people. Since they weren't threatening to fire him, just putting pressure on, that suggests it was probably important to someone but not mission critical.

      "No mention of a fail over server or cloud based solution to keep the application going."

      We don't have an exact date on this, but it appears to predate the availability of cloud, so it would have to be another server or a different one with extra capacity. Another server isn't cheap. It's not automatically his fault if the request for that level of overprovisioning was declined.

      "Why wasn't there a service and support plan on the server?"

      You assume there wasn't. Someone got called out, didn't they? Why couldn't that have been as a result of a support plan? They could have one which guarantees a support person shows up (which happened) but doesn't automatically cover all expenses, hence why they would expect to pay afterward, especially as there were new parts brought in as a result of the failed attempts.

      "Since they pay for each repair call, why doesn't "Kris" have some spare parts on hand."

      You assume they pay for each call. Perhaps they only pay for parts, and their support plan requires that they only use authorized spares. Why buy a ton of those if the engineer is supposed to bring them? Also, how do you know they didn't have some of those things. Maybe they replaced a broken power supply with their spare a while back, hadn't received the replacement spare, and then the engineer decided power supplies had to be replaced (for no good reason as you've already stated) and made them get two new ones. You can assume any level of competence you like, and assuming the lack of it doesn't prove it any more than assuming they planned for everything.

      "Where I have issue with "Kris" is where he blames the repair tech for a wrong diagnosis and the amount of down time."

      Both being the tech's job. The tech should try to diagnose things rather than just swapping out parts every three days until it worked. Had the tech tested each component, leading to a day's downtime while he went through everything he could think of, I don't think there would be that much complaining as the downtime was necessary to identify the problem. However, the tech's approach didn't appear to check the hardware very much, so most of the downtime was waiting for stuff that wouldn't be needed if the tech tried diagnosing the cause instead of guessing.

      1. Rob Daglish

        Indeed. HP & Lenovo both emphasize in their warranty provider training (or did when I last re-did my exams) that you aren't there to randomly swap parts, but should diagnose the issue and be able to support your diagnosis before ordering any part...

        1. DropBear
          Devil

          Are there official training flowcharts for that...? Because if I have to use my multiple-decades-powered experience of thinking outside the box to find and fix the problem then "HP & Lenovo" can go eat shit and I'll fully claim any resulting success and any proceeds of it as my own.

  15. kmceject

    Pop Goes the Switch!

    Many moons ago when I was young I worked at a shop that the wild west would have had a name for how crazy it was. Most programmers had their machines denuded of all outer panels because 'they got too hot' and the like. One day i popped in on a particularly egregious offender who was messing with his tertiary pc (Yes, he programmed on two at the same time and the third was there for experimentation. Apparently from his exasperation the machine had been working the day before and at this time in the afternoon he had been working on getting it up since 6am. He hadn't even touched his breakfast as I recall.

    I watched over his shoulder as he plugged in yet another power supply (there was a tower of them next to him, along with power cords, power strips, his multimeter, and many tools. He once again let loose a stream of profanity as this one too failed to show life. It was about this point I reached over past him and pushed down the white plunger at the front of the case with a piece of loose, dirty scotch tape (cello tape I think you might know it as). With a whir and a beep the machine began to boot. He had taped over the switch when he pulled the cover off, but anyone who knows that type of tape knows it can have problems sticking to rough surfaces. The plunger was to the case interlock switch.

    I came back the next day to see a big strip of duct tape on all similar switches in his office!

    BTW- there was one PC he had in his office that had a pushbutton switch in the front of the case with a square, white plastic pushrod that led to the back where it pushed the button on the power supply. This piece was built as a gentle slope. One day at a hobby shop I noticed a package of HO scale skiers on sale so I grabbed them. I knew the PC was a loaner so when I snuck into his office I used scotch tape to adhere them to the slope! He loved it, but hated the tape so he crazy glued them in place instead. I used to wonder if anyone at the returns site ever noticed when they got it back...

  16. yetanotheraoc Silver badge

    interlock switches

    Good to know about those, even better to think of them. Mine is more who me than on call. My "new" car (2007) has power windows. I've driven plenty of cars with power windows but never owned one. Anyway, it was a bummer when three of the power windows stopped working. No problem, I'll check the fuses. Anything more complicated than that is over my pay grade, I would have to take it in for service. In logical land, this particular fallacy is called a false dichotomy. Of course it wasn't a fuse (which I already guessed since the driver's window was still working), so I dropped it off at the shop and waited for the estimate. Because of their incredulity they called me twice. And charged me zero for flicking the switch. I joked with them that they should have charged me, anybody so stupid deserves to pay. At least now I know.

    1. Anonymous Coward
      Anonymous Coward

      All too human.

      At least you are willing to own up to it. I can forgive almost any honest mistake, but I have had to deal with too many screamers that made similar mistakes and didn't want to own up to them. Thanks for making the world slightly better by not being one of them.

      1. John Brown (no body) Silver badge

        Re: All too human.

        The biggest problem with people who won't own up to their mistakes is that they are also the ones least likely to learn from them and repeat them again and again.

    2. Martin an gof Silver badge
      Boffin

      Re: interlock switches

      Sounds a bit like when we went to take delivery of a new car some years ago. After signing all the paperwork, and waiting while the "technician" put the correct (as specified in the paperwork) numberplates on this time, the dealer proceeded to demonstrate the workings of the car to us proud new owners.

      Except that he couldn't get the keyfob to unlock the doors. Press the button, only the driver's door unlocked. Press it again, same thing. Press the second button, the doors all locked. Press the third button, only the boot unlocked. Use the switch in the cabin, all doors lock and unlock as expected. Open the driver's door from inside while the doors are locked, all the doors unlock.

      Confused is putting it mildly. This particular manufacturer had a feature where a single press on the keyfob would just unlock the driver's door, but a second press within a second or two would unlock all the others. Some kind of personal safety thing, to stop people jumping into the car when you unlock it.

      We drove off in a car with "a problem" and the promise that the dealer would get in touch with HQ the following day and see if they had any ideas.

      That evening I actually read the manual, only to discover that on that particular model - and only on that particular model - you didn't open all the doors with a double-click (as it were); you had to hold the button down for about three seconds.

      M.

  17. MachDiamond Silver badge

    If you have one, you have none

    Anything that is truly critical needs to have a backup. The more money the company loses for every hour that thing is down, the more backups it needs. If you've outsourced something that will cause the company to fold up and die, you shouldn't be allowed past the door of the locked ward.

    Murphy's law tells you that not having a back guarantees that something bad will happen and it will happen at the worst possible time. There was a time when it was possible to send parts from airport counter to airport counter when early morning overnight shipments weren't good enough. That's gone now unless you have a super duper background check and certification and pay huge money. There are alternative means, but they aren't always easy to find and arrange when you are panicking. Better to have a spare sitting in a box close to hand that can be put in place in minutes.

    I dated a woman that worked for GM a long time ago. At the time, she told me a production line being stopped cost the company an estimated $125,000/hour. If 'all' it took was getting parts on a Lear jet for a mere $20,000, they'd do it without needing prior permission from corporate. That would be a lucky case since the parts were somewhere, just not at the plant. Having to hunt down some parts from somewhere with that cost clock ticking would be scary. The big auto plants have airstrips nearby for just such emergencies and they are also handy for the C-Level execs. I heard of one tech that was picked up and flown to a GM plant to sort out a really bad problem.

    1. Robert Carnegie Silver badge

      Re: If you have one, you have none

      I just read an article about companies planning to run small pilotless cargo aeroplanes - but they're not allowed to yet.

      https://www.bbc.co.uk/news/business-63595481

      Drones may be allowed on set routes, such as in this case to (or from?) the Isle of Wight, just offshore from England.

      https://www.bbc.co.uk/news/uk-england-hampshire-58672437

      1. MachDiamond Silver badge

        Re: If you have one, you have none

        "I just read an article about companies planning to run small pilotless cargo aeroplanes - but they're not allowed to yet."

        And never should be allowed.

  18. _garfield_
    Facepalm

    Faulty chassis intrusion switch = 3 new floppy drives, 2 new motherboards and a RAID riser.

    Once played a similar game with a Dell server a couple of decades ago. Noticed server had the hardware warning light flashing on the front, but nothing shown in the vendor tools installed on the server. Box was NT4 and Dell suggested booting it from a floppy image they supplied to dump the management controller logs to the floppy for analysis. Problem was it wouldn't boot off of any floppy.

    Field support sent out with a new floppy drive and I left them to it to change it after I shut things down. 20 minutes later they tell me the floppy was still broken, so they would return next day with a new motherboard.

    Next day a different technican arrives, and installs new system board. Machine still not booting from the floppy... and now the on board RAID wasn't detected. Field service tech makes phone calls and it turns out to be a known issue between firmware versions.on system board and RAID controller, and it should have been picked up previous day before parts sent out. Old system board reinstalled so machine was working again after an hour of downtime.

    Day 3 same technician turns up with new floppy, system board and RAID controller. As he's unpacking I point out the cable packed with the floppy is a straight cable, and the one currently in the server had a twist. Swapping just the drive cable brought the floppy back to life (first technician had put wrong style of cable in while swapping drive), and we were able to run the diagnostics.

    Logs reported a chassis intrusion. We were able to verify by disconnecting switch and replacing it with a jumper to clear the error.

    So finally on day 4 a technican arrives with a new switch for the front door of the server.

    1. MachDiamond Silver badge

      Re: Faulty chassis intrusion switch = 3 new floppy drives, 2 new motherboards and a RAID riser.

      "So finally on day 4 a technican arrives with a new switch for the front door of the server."

      This is the downside of field service. There's no way to have every part that might be needed even if the technician was driving an HGV. My recommendation would be if the first trip doesn't fix the problem, the device should probably go back to a depot for repair if possible where there are parts, a full set of manuals, more people that may know about a recurring fault and all of the test equipment that may be needed.

      Management that's any good will realize that having spares for critical infrastructure is important even if it eats into their bonus fund. The likelihood of something going wrong completely draining that bonus fund is a good indicator or the depth needed for backups.

      When the cable internet goes Tango Uniform in my town, at least 3/4 of the businesses can only take cash (for as long as cash is still left). A sizable enough population in the area are completely unemployable and enjoy sponsorship by the State so all of their shopping is done on plastic (expect for the weed store, which can only take cash so far). If the internet were down for several days, people would starve or have to do without energy drinks, snacks and booze. I'm surprised that at least some stores don't have some sort of backup even if it is slow.

  19. 196

    "Hack Heaven"

    Has anyone seen the movie Shattered Glass ( Film ) ?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like