back to article Bank's datacenter died after travelling back in time to 1970

The steady process of time means that The Register has once again arrived at Friday and the timeslot we reserve for On Call – our weekly reader-contributed tale of tech support trials and tribulations. This week, meet a reader we will Regomize as "Colin" who shared a story of his time working as a support engineer for a major …

  1. jmch Silver badge
    Facepalm

    Yearly tasks....

    The thing is, with daily tasks, they just get done automatically. Monthly, OK there's a month end process. Yearly tasks are only ever triggered if they are literally scheduled for Jan 1st (or 2nd if it's a manual task). One can so easily imagine the batteries being installed in June, someone putting a calendar entry for the following June and everybody forgetting about it. Then the first person leaves the company, the batteries continue on their merry way for 3 years, and before you know it, there isn't even anyone on the team who remembers what that box is and what it does.

    Similair applies to server certificates with multi-year validity. Shorter renewal cycle could be more of a PITA, but actually work better for reliability.

    1. Pascal Monett Silver badge

      Re: Yearly tasks....

      Completely agree. Yearly tasks are a pain to schedule, especially when the responsible person leaves the company (whatever reason).

      I'm thinking there should be a yearly calendar process for business. The business needs to know these things, and they cannot be tied to the knowledge of one person. How many times has Borkzilla been caught out because it forgot to renew its own certificates ? And it's far from being alone in this. If there was some sort of shared calendar showing yearly actions and who is responsible for them, then I think this kind of issue would be almost resolved.

      At a business level, that should be easy to organize. Create a user named Yearly Renewal, share his calendar with all upper management, and have HelpDesk insert the proper data and the names of people responsible. When one person leaves, check Yearly's calendar for all events under that person's name and figure out who to reassign them to. Then make sure everyone is checking Yearly's calendar at least once a month.

      That should work, no ?

      1. Ali Dodd

        Re: Yearly tasks....

        modern email/organisation systems (as they are more than just email if they contain a calendar system) should have a Shared Calendar facility so you don't even need a fake user to handle it, which could expire or be purged in "tidying up" by accident..

        1. collinsl Bronze badge

          Re: Yearly tasks....

          And you wouldn't have to pay for a license for the fake user

          1. C R Mudgeon Bronze badge

            Re: Yearly tasks....

            No, you'd pay ten times as much for the "enterprise" feature.

            1. Jou (Mxyzptlk) Silver badge

              Re: Yearly tasks....

              Actually no, such a shared calendar is, at leas in the Microsoft world, for free. Even if you (ab-)user a shared mailbox to do the job.

              Dunno about other solutions, I haven't been working with anything else for past >10 years in the business world. Before that a lot of various mail and calendar systems.

          2. JulieM Silver badge

            Re: Yearly tasks....

            What is this "paying for licences" of which you speak?

      2. Anonymous Coward
        Anonymous Coward

        Re: Yearly tasks....

        Before I retired, part of my work involved sorting out company management systems. A friend had developed a process mapping program that forced a simple structure (which, whilst initially difficult to work with for some processes, actually did simplify the documentation to an extent that everyone could actually follow it). The point for this thread is that every process had to have an owner and, when anyone changed roles (left, jointed, promoted, etc.) you could quickly print off a list of all the processes they owned. That list became part of the induction process for the new role holder.

        The system could also deliver a list of all the tasks anyone was responsible for, but that was far longer and many tasks were the responsibility of a shared post (such as "Project Engineer") and tended to be built into those role inductions.

      3. Anonymous Coward
        Anonymous Coward

        Re: Yearly tasks....

        For companies that use a CMMS [1], it's common to simply use that. I'm in manufacturing, and every site I've worked at has had such a system for maintenance on production equipment, HVAC, utilities, etc. Adding IT is a quick and easy solution. It doesn't have to be yearly, either; usually you can define any time period you want. It's great for reminders of user audits, server PMs, performance checks, backups...

        [1] A CMMS (Computerized Maintenance Management System) is a computer system that generates WOs (work orders) for routine scheduled work, allows manual WO generation for nonroutine work, can list work that is due soon or overdue, shows the history of work including tech's comments, etc.

        1. Sam not the Viking Silver badge
          Pint

          Re: Yearly tasks....

          Customer's site: 2-machines for 'normal' operation, 1-machine 'assist', 1- machine 'standby'. Spare: Offered, but not accepted.......

          The rules: 2-machines plus 1-standby must always be available.

          Attitude: "We've got a standby machine"

          Actuality: 1-machine under maintenance.

          Result: Non-compliance.

          Customer: Panic. "Must be the manufacturer's fault".

          In the end we built a 'spare' machine at our own cost to act as standby when one machine was under maintenance. Did the customer buy it? No. We charged a healthy premium for a 'quick turn-round' on repairs. The machine cost was recovered within a few months and after that it was embarrassingly lucrative.

          Result to us: ----->

        2. Stuart Castle Silver badge

          Re: Yearly tasks....

          The problem with introducing any system like that is cost. It's harder to persuade people to invest in something if it doesn't produce a result they can measure. After all, while maintenance is good, all it does is enable you to carry on doing what you have been doing. It won't necessarily help you do it better, so it doesn't excite people. It's the same with user audits and backups. After all, if they are done and good, your performance will be what it should be anyway. They may even ask why you can't just buy a wall chart and put reminders on that.

          Even with the best intentions of doing maintenance properly, purchases like this often get bumped because the company needs something more urgently

          If you have actual physical machines that need maintenance, you can sometimes see if there are any problems, and any missed maintenance will probably have a quantifiable effect. If you own a large factory, and one of the machines goes down, lost production time can be a significant cost, which would also motivate those looking at buying a system to manage their maintenance schedule.

          And yes, I have had to deal with users when their computer system failed, and they neglected proper maintenance including tested backups. It wasn't a good experience for them, and, as a result, wasn't a good experience for me.

          1. Alan Brown Silver badge

            Re: Yearly tasks....

            " It's harder to persuade people to invest in something if it doesn't produce a result they can measure."

            If you standardise your hardware then a spare machine can cover multiple clusters. This is easier to sell to manglement and has the advantage of keeping hardware like "lego" rather than being individually customised

      4. vcragain

        Re: Yearly tasks....

        No matter what you set up to force the checking, there is always the forgotten thing that happens because of something else - either everything is chugging along so smoothly that people get too comfortable, or too much else is going on & nobody does that one 'silly little check' that can cause the big mishap. Human nature every time - unless there's some systemic way to check - an alert if that fails etc,, then we have the 'thing that does the alert didn't work' issue - been there - regardless how hard you try there is always a potential screw up somewhere that will get you ! Checklists help, people checking other people helps too - every human makes mistakes ! Know exactly how you will recover if/when - ie backup, backup, backup !

      5. H in The Hague
        Pint

        Re: Yearly tasks....

        " Yearly tasks are a pain to schedule"

        Alas they are. I know of the voluntary group where the new secretary or treasurer didn't apply for a grant in time so the group was ineligible for it. That made a hole in their finances though fortunately they'll survive.

        So when I friend recently became chair of a similar organisation I advised them to make sure that any time-limited grant applications are a task of the secretary or treasurer, but that everyone on the committee should note those dates in their diaries and check if the applications have been filed. Better to learn from other people's mistakes :(

        That time of the week again, 'cheers' to all Commentards.

      6. Excused Boots Bronze badge

        Re: Yearly tasks....

        Yes it ‘should’ work, but meanwhile back in the real world……..

        The problem, where a task is everyone’s responsibility, it rapidly becomes nobody’s responsibility, “no I didn’t do it, I was told Jim was doing, no, can’t remember who told me but…”

        Your suggestion about when Fred leaves, and any jobs assigned to him, get assigned to someone else, is good, except, who is responsible for doing that? Nine times out of ten it will be the aforementioned Fred!

        I’m afraid it’s a bit like turtles, only in this case it’s more like, “I thought someone else is dealing with it”; all the way down!

        1. John Brown (no body) Silver badge

          Re: Yearly tasks....

          In our case, the task and job allocation system is our primary go to for 1st, 2nd and 3rd line support. Infrequent tasks are set to trigger with relevant amounts of notice and allocated to a team, not an individual. Said tasks will stay in the list until someone updates it as completed and never show red as overdue, at which point, hopefully it will auto-reset to the next interval, whether that be monthly, yearly or whatever. It seems to be clever enough to allocate always on a working week day, bringing it forward a day or two if required to avoid weekends and holidays. The potential flaw in that plan is that someone forgets to set it up in the first place, but that's managements problem :-)

        2. FIA Silver badge

          Re: Yearly tasks....

          Yes it ‘should’ work, but meanwhile back in the real world……..

          "...that thing sounds complicated so we best not try, it'll not work anyway."

          Yup, worked at places like that. Probably best avoided. Over time this attitude distils, until you get a workforce not only averse to risk, but to even risking doing their jobs with any degree of competence, as the warm comfort of gentle monotony is much much easier.

          I was going through some personal stuff which meant changing jobs wasn't on the cards. The main frustration about the whole thing is the one upcoming project that sounded difficult (and therefore interesting) was never ever started... as too much risk.... just dangled like some future development carrot.... whilst the existing creaking poor quality and hard to use system just plodded along.

          1. ZorgonsRevenge
            Mushroom

            Re: Yearly tasks....

            Well, I currently work in a "place like this".

            Here is what happens in our IT dept. We have critical systems that no one cares about because they were installed a "number of years ago" on a new and sparkly version of Centos5 ..then one day it stopped working due to a "certificate issue"

            The main problem was that the guy who installed it had left the company due to The rolling "1% annual pay rise regardless of performance" despite having to sit through the corporate wank session who told us all about the bumper profits. We were told that there was no money in the kitty due to a need to replace a bunch of laptops for the sales guys... that did not stop sales from getting fat bonuses, but more to the point..I don't work in sales so why tell me that???

            Like a lot of employees, it dawned on us that we could literally turn water into wine and feed the forty thousand and it would make no actual difference to our salaries.. so why bother? We all just do the absolute bare bum basic minimum and look for other jobs in support.

            Between the massive employee turnover and poor morale..there are a lot of "end-of-year functions" that are ignored.

            After all why make more work for yourself if it's never appreciated?

            Sorry for the rant!

            1. Jou (Mxyzptlk) Silver badge

              Re: Yearly tasks....

              > currently work in a "place like this".

              Make that past tense, the sooner the better. Just don't forget your "I told you so" CYB.

      7. Pixel Green

        Re: Yearly tasks....

        At this point in time with tech more important than ever, and only ever more complicated, it's a must for any business which loses more than a few £ per hour due to outages.

        For all those reasons, we've just gone through the process at our company of deploying calendars in Asana for each department, scheduling monthly/quarterly/yearly tasks that are easily forgotten and must be done. Then pulling all of them up into a company-level one overseen by the Ops Director through some small automated rules.

      8. Terry 6 Silver badge

        Re: Yearly tasks....

        specially when the responsible person leaves the company

        Or even more likely, has moved on to new project/office/department/building ( multiple layers of likely movement iow) and left their old tasks to the new person, who may not even realise, or have been told about, every single detail- because the outgoing person has a squillion small asks that popped up on their diary from time to time Especially if there has been more than one change. There is absolutely no way that a hand-over will contain every detail of the outgoing person's year, or indeed the significance of the ones that are included.

      9. Jou (Mxyzptlk) Silver badge

        Re: Yearly tasks....

        > Yearly tasks are a pain to schedule

        especially since the certificates don't need a yearly renewal, you do the renewal 10 days before. So it is a "nearly yearly" task. Several procedures are that way. THOSE are the problematic. Why is a certificate not simply 380 days, or a password expire time not simply 200 days so it fits nicely into an actually yearly (or twice yearly) schedule? This is one of those "why does IT shoot into its own foot" questions...

        1. Killfalcon Silver badge

          Re: Yearly tasks....

          Not my field, but can you not set the validity date on the new cert to start when the previous one expires?

      10. The Man Who Fell To Earth Silver badge
        Boffin

        AA quad AC adapter's have been around for decades

        In a 'battery operated" stationary critical piece of equipment like a battery powered radio time clock, you use an ac adapter that interfaces to the equipment through dummy batteries.

        I first used the 1970's version of this in, surprise, the 1970's. A modern version can be bought off Amazon for next to nothing.

        https://www.amazon.com/Lenink-Power-Supply-Battery-Replace/dp/B09YTVTZ1V

        1. F. Frederick Skitty Silver badge

          Re: AA quad AC adapter's have been around for decades

          A word of warning with those battery replacement gadgets, or worse still bodging a wall wart onto a battery powered device. The batteries will supply their rated voltage, which will gradually drop as they discharge. Those battery replacement things or a cheap wall wart will provide a wildly fluctuating voltage, often way more than the nominal rating. If the device you connect it to doesn't have the right circuitry to clamp the incoming voltage to what's expected then you could find it breaks pretty quickly. I'd recommend checking the output on any battery replacement gadget before use, or using a good quality regulated wall wart that has the circuitry to ensure a steady voltage.

          1. The Man Who Fell To Earth Silver badge

            Re: AA quad AC adapter's have been around for decades

            I agree entirely.

            I actually use one of these for 2 AAA batteries at home that, instead of using a USB output wall wart, uses a more standard wall wart. ( https://www.amazon.com/gp/product/B0873Y5MQ5/?th=1 ) Having said that, I would advise one to do their due diligence and make sure the power supply is quiet, stable and at the correct voltage.

    2. Vincent Ballard

      Re: Yearly tasks....

      The CAB have pretty much killed off certificates with multi-year validity, but since getting bitten by a certificate expiry I have a weekly task which sends me a report of certificates that are expiring in the next few weeks. Now the only way that certificates cause me support headaches are the users in third world countries who use 10-year-old Android phones whose root certificate lists need updating, and we only get one or two of those a year.

    3. Anonymous Coward Silver badge
      Holmes

      Re: Yearly tasks....

      Once you've realised that this thing needs batteries changing yearly or $things will fail, why not replace (or supplement) those batteries with an external power brick? Plugged into the UPS on the basis that if the UPS dies, that clock will be the least of your worries.

      1. This post has been deleted by its author

      2. Prst. V.Jeltz Silver badge

        Re: Yearly tasks....

        I guess the change request of :

        "I wont to cut the dc plug off a 9v answering machine psu and solder the wires to the battery terminals in the Network Clock time-getter"

        might seem a bit too Wallace & Gromit.

        1. keithpeter Silver badge

          Re: Yearly tasks....

          Smoke alarms start to peep when the battery gets low and then squeal when it gets really low. Perhaps a 'fail loud' modification to critical battery operated equipment?

          1. Zarno
            Facepalm

            Re: Yearly tasks....

            "Oh, that thing? We stuffed it in a bag with some towels till it stopped making noise."

            or "Oh, we replaced it with a nicer wall clock that didn't need batteries so often."

            I could list way too many theoretical situations...

            1. Doctor Syntax Silver badge

              Re: Yearly tasks....

              In this case replacing it with something different would be a "Yes, you" situation for whoever unplugged it and the systems went down.

            2. Yes Me Silver badge
              Facepalm

              Re: Yearly tasks....

              Sigh. Replace it with NTP for heaven's sake. That is rather what NTP was designed for. Needs no batteries. Not even a clock.

          2. ITMA Silver badge
            Devil

            Re: Yearly tasks....

            Virtually all intruder alarm "siren/strobe" units (or "bell boxes" as they used to be known) use a "hold off" voltage from the main control panel to stop them sounding.

            If the hold-off voltage disappears for any reason - such the alarm being triggered, the cabe being cut etc - the siren and strobe would start going.

            So buy an off the shelf siren/strob unit, power it off a 12V main adaptor (which will keep its internal backup battery going) and have the clock unit battery act as the "hold off" voltage.

            Battery dies, the siren/strobe starts going (and no doubt sevel people will need a change of underwear especially if very close to them and you don't tell them what it is for).

            It is that or use the same "hold off" principle to have voltage from the clock battery stop a relay contact from closing - if the battery fails, the relay contacts close and connect mains voltage to some bugger's chair.

            Or you could be boring and just run the clock off the mains with a battery backup.

        2. Bebu Silver badge
          Windows

          Re: Yearly tasks....

          "I wont to cut the dc plug off a 9v answering machine psu and solder the wires to the battery terminals in the Network Clock time-getter"

          6V surely. Quartet (4) x AA (1.5V).

          Even shoving 6V into 4 alkaline cells might be a little explosive.

          I would trust Gromit to get it pretty much right as he would probably use a 6V sealed lead acid motorcycle battery and a trickle charger. Wallace I shudder to think...

          Given AA cells were and are inexpensive, a shorter maintenance schedule (every 3 months) might have been safer.

          Say four roles that get an annual calendar reminder to replace the batteries and ensure the other three roles still exist and were not vacant.

          Seems like the systems in the story were synched directly from the radio time signal services. I would have thought even back in Novell Netware's heyday crystal controlled precision time sources would have been de rigeur and I imagine those precision sources would have been optionally synched from the wireless time services.

          1. collinsl Bronze badge

            Re: Yearly tasks....

            A 5V nominal system would make sense given the likelihood of AA cells to vary in voltage output over their lives and between each other.

            1. Anonymous Coward
              Anonymous Coward

              Re: Yearly tasks....

              At which point, if the current needed is low enough, a USB-A plug can be wired to the battery terminals. Connect to a cheap USB power brick.

              I did this with some 2xAAA lights on our mantle so I wasn't replacing the batteries weekly. Yes, that's "too high" voltage for them, but most LEDs can take it and most circuits have voltage regulators on them, and the current is really low.

              1. imanidiot Silver badge

                Re: Yearly tasks....

                I suspect this whole thing happened well before the invention and widespread implementation of USB.

                1. BartyFartsLast Bronze badge

                  Re: Yearly tasks....

                  GIven it was a Novell box and it was running on what sounds like a radio clock, Droitwich, MSF, WWV or whatever, then yeah, it was a *long* time ago.

                  1. TimMaher Silver badge
                    Windows

                    Re: Droitwich etc.

                    And the now absorbed Woolwich BS.

                  2. theblackhand
                    Facepalm

                    Re: Yearly tasks....

                    A long time ago? What are you talking about?

                    The 1990s was only 10 or so years ago....

                    1. katrinab Silver badge
                      Windows

                      Re: Yearly tasks....

                      30 years ago.

                      1980 is closer to the end of WW2 than it is to today.

                      1. Anonymous Coward
                        Anonymous Coward

                        Re: Yearly tasks....

                        Was that really necessary?

                        Now I feel old, and I'm at work, on a Saturday, and I'd far rather be in bed

                        1. ShortLegs

                          Re: Yearly tasks....

                          Thats two of us then...

                          ... you're not migrating services from one DC to another in "LON" are you?

                        2. herman Silver badge

                          Re: Yearly tasks....

                          I used to set my watch by the BOOM!!! of the Noon Gun on Signal Hill.

                  3. John Brown (no body) Silver badge

                    Re: Yearly tasks....

                    "GIven it was a Novell box and it was running on what sounds like a radio clock, Droitwich, MSF, WWV or whatever, then yeah, it was a *long* time ago."

                    And these clocks, to do the job as described, surely must've had some way to "talk" to the servers, most likely a wired connection of some sort, serial or ethernet (RJ45 or BNC). That doesn't sound like the sort of kit I'd expect to be running off batteries as standard, especially back then when electronics were more likely to be a bit more power hungry than modern kit.

                    1. Richard 12 Silver badge

                      Re: Yearly tasks....

                      This will have been long before NTP, so these will have been the Master Clocks for a wired electronically synced clock system.

                      Popular in schools and factories since 1900 or so.

                      There will have been some proprietary gateway in the servers, that most likely wasn't ever tested as to the behaviour if the clock stopped because "That can't happen".

                      1. Yes Me Silver badge
                        Headmaster

                        Re: Yearly tasks....

                        NTP dates back to RFC958 in Sept. 1985. Of course, it may have taken Novell and Microsoft a wee while to notice.

          2. Anonymous Coward
            Anonymous Coward

            Re: Yearly tasks....

            I’ve seen this done in very similar circumstances to the original story but with a multi voltage adapter. In that case the voltage switch was on the pulg side of the adapter. Two thick pieces of plastic were cut to length and then glued in place to prevent a change from the required output of 6v. The reason for doing this was one fateful day when some new batteries turned out to be deficient. They failed overnight and the first person in was confronted with a barage of alarms from the things that relied on the clock signal output.

          3. Martin an gof Silver badge

            Re: Yearly tasks....

            Seems like the systems in the story were synched directly from the radio time signal services. I would have thought even back in Novell Netware's heyday crystal controlled precision time sources would have been de rigeur and I imagine those precision sources would have been optionally synched from the wireless time services.

            In my radio station days the (analogue) clocks in the studios had fairly bog standard Quartz movements, but they were synchronised back to a laser-trimmed unit in the central racks, which was itself synchronised to MSF and optionally (I think) DCF. No NTP or GPS (for proles) back in those days. The racks unit had a Lead-acid backup battery kept floating, but the studio clocks just ran from bog standard AAs if I remember correctly. In constant use, a clock slowing due to loss of synch or stopping for a dead battery would have been noticed pretty quickly and reported in no uncertain terms to Engineering the first time a DJ with half a million listeners crashed the IRN bulletin at o'clock!

            As for TOA, it seems odd that if the servers stayed running, the external clock stopping would have caused their internal clocks to reset...

            M.

            1. Anonymous Coward
              Anonymous Coward

              Re: Yearly tasks....

              We’d the same set up except the clocks were powered from the mains and fed from a unit in the server room. Caused a problem when one studio was used for a photo shoot and they wanted a particular time on the clock. This was possible but the PR bloke didn’t change it back at the end of the shoot. Had to do a studio swap at short notice and the presenter almost read out the time before the 3pm news as being “almost 10am”. Swiftly fixed following the discovery.

              1. Martin an gof Silver badge

                Re: Yearly tasks....

                the presenter almost read out the time before the 3pm news as being “almost 10am”

                I think even our mostly quite dim and engrossed in their own thoughts presenters would have noticed the difference between 3pm and 10am! In our case, changing the time would have involved disconnecting the serial line from the clock (carried on standard audio-grade shielded twisted pair IIRC) and then adjusting the clock in the usual manner.

                It's a while back, but it's possible that the AA in the studio movement was only for backup and the power actually came up the serial line from the central unit too (I don't remember the clocks being plugged in to the mains, but my memory is pretty rubbish). Looking back I don't remember changing the AAs very often, but then given that we're talking about four or five clocks, me only being there for six years or so and the AAs in clocks at home often lasting two or three years, even if they were entirely battery powered at most I'd have done fewer than 20 battery swaps in my time there, assuming my boss didn't do any himself.

                M.

                1. doublelayer Silver badge

                  Re: Yearly tasks....

                  Almost certainly, they would recognize the difference, but when your brain is in reading mode, it has a very efficient pipeline for reading and speaking which is not always delayed enough for the rest of it to stop you. "It's just coming up to ten no wait a minute how could it be" is not really much better than stating the incorrect time.

                2. Anonymous Coward
                  Anonymous Coward

                  Re: Yearly tasks....

                  The bloke said he was looking at the clock as he was about to open the microphone fader and his gob. Then at the last moment something in his brain said the clock is wrong, don’t say a word. To be fair it was combined with a studio move during a live show which was basically done in the space of one song. Something went wrong in the original studio and he was told to move quickly.

                  1. Martin an gof Silver badge

                    Re: Yearly tasks....

                    Our on air talent absolutely hated studio moves. Mainly, I think, because of the need (in those days) to pack up CDs and carts and lug them to the room next door. Only in exceptional circumstances could we force a move and I often found myself working on a "live" mixer; channel strip removed while leaving the rest of it running. Even (as I think I've recounted here before) one famous occasion where a jock on his last show had (totally against all rules) taken a half a bottle of gifted cider into the studio and tipped it into one half (script space in the middle) of the brand new desk leaving him with (something like) one guest microphone, one CD player (of three), two (of four) cart machines, IRN and a telephone TBU. He didn't call me - it was the next guy who called the pager at 10pm. They could easily have done a studio handover between shows, but no!

                    The beauty of that desk was that it was "remote". The only things in the control surface were the Penny & Giles conductive plastic faders, a gain knob, PFL and (where appropriate) start or talkback buttons. All the electronics was in a rack off to the side. Cleaned up reasonably well, bloke left with wet trousers. Studio smelled a bit for a few days.

                    M.

            2. ShortLegs

              Re: Yearly tasks....

              It wasn't the Netware box that failed. It was the NT server.

              Netware was literally bullet proof.

              1. Doctor Syntax Silver badge

                Re: Yearly tasks....

                This is the Netware that shut down completely if what looked like a process (in Unix terms) was killed? Not even metaphorically bullet-proof.

              2. Ahosewithnoname

                Re: Yearly tasks....

                NetWare was truly a thing of wonder, but bulletproof it most certainly wasn't

          4. TimMaher Silver badge
            Coat

            Re: Wallace

            He would have used cheese.

            1. Paul Herber Silver badge

              Re: Wallace

              Cheese has cheese mites, and mite is an anagram of time!

              Sorted.

              1. David 132 Silver badge
                Pint

                Re: Wallace

                You just edam that anagram up on the spot didn't you?

                Still, gouda nuff for me. Cheers!

          5. Antron Argaiv Silver badge

            Re: Yearly tasks....

            Nowadays, all the cool kids are using GPS time (at least the ones who run the cell sites near me are)

          6. doublelayer Silver badge

            Re: Yearly tasks....

            "Say four roles that get an annual calendar reminder to replace the batteries and ensure the other three roles still exist and were not vacant."

            That will just lead to Alice replacing the batteries on Monday, Bob taking them out and putting in a new set on Tuesday, Carol removing the batteries but not finding a new set, so putting the old set on her desk to remind her to go get a new set, and David putting in the batteries that were dead in the first place because Bob put Alice's set somewhere else and Alice forgot to charge the old ones after taking them out. A proper procedure includes some way to verify that it has happened, but that introduces more overhead like finding a place to store that information and making sure everyone checks it.

            1. M.V. Lipvig Silver badge

              Re: Yearly tasks....

              No, person 1 replaces the batteries Jan 1, person 2 on April 1, person 3 on July 1 and person 4 on Oct 1. Then HR lays off all 4 people on Nov 1.

              1. MachDiamond Silver badge

                Re: Yearly tasks....

                "Then HR lays off all 4 people on Nov 1."

                The more modern example is where Elon fires the whole department that was, as part of their duties, tasked with changing batteries in things.

          7. keith_w

            Re: Yearly tasks....

            9 volt batteries: https://www.amazon.ca/AmazonBasics-Everyday-Alkaline-Batteries-8-Pack/dp/B00MH4QM1S/ref=asc_df_B00MH4QM1S&mcid=39eabb23539a35dab6150dbd0fcf0308?tag=bingshopdesk-20&linkCode=df0&hvadid=80470598951659&hvnetw=o&hvqmt=e&hvbmt=be&hvdev=c&hvlocint=&hvlocphy=&hvtargid=pla-4584070143719256&th=1

        3. Richard 12 Silver badge

          Re: Yearly tasks....

          In some cases it doesn't work, because a AA cell has a far lower noise floor than any power supply, especially one connected to a UPS.

          UPS are 'ing awful supplies, there are a lot of things that won't work properly when fed by the terrible "inverters" in most of them.

    4. Prst. V.Jeltz Silver badge

      Re: Yearly tasks....

      someone putting a calendar entry for the following June and everybody forgetting about it.

      Isnt the point of a calendar entry so that you *can* forget about things and be reminded of them when the time comes?

      1. Strahd Ivarius Silver badge
        Boffin

        Re: Yearly tasks....

        you have to remember to tick the box "send a reminder" on your calendar, otherwise you won't get notified of the due task.

        and when changing batteries, you should also have a task asking you to buy new ones BEFORE the due date, of course...

    5. Anonymous Coward
      Anonymous Coward

      Re: Yearly tasks....

      That's why I change my toothbrush on Jan 1 and Jul 1. No other way to remember...

      1. J.G.Harston Silver badge

        Re: Yearly tasks....

        I was a school governor and there was a yearly site check that was done in late July, which was simple because you've got the school year that imposes its schedule on you and prevents you forgetting.

        Similarly, I check all my smoke detectors when the gas appliances are checked annually. The plumbers know when their certificate runs out and pesters reminds me, so again a schedule that doesn't get forgotten.

        When I do my tax return I have a bunch of stuff I do at the same time - and while I may put it off for months, there is a hard deadline each year so over the long term the yearly tasks get done.

        I check my car tyres every time I get petrol. I check various other things at the annual MOT. External prompting ensures it gets done.

        But if it is something that is independent of anything prodding you to do, it will get forgotten about because that's how the human brain works. If the DVLA didn't tell me my car tax was about to expire, or the insurance company tell me the insurance was about to expire, I'd never remember to pay it.

        1. John Brown (no body) Silver badge

          Re: Yearly tasks....

          "If the DVLA didn't tell me my car tax was about to expire, or the insurance company tell me the insurance was about to expire, I'd never remember to pay it."

          Hopefully your MoT is due at the same time as one of those, or you'll forget it?

          1. Richard 12 Silver badge

            Re: Yearly tasks....

            MoT is at the same time as the car service - both are always due simultaneously unless you have an unreasonably high annual mileage.

            That way the car reminds you to do them.

            At least, mine currently says "service in 7 weeks"

            1. Doctor Syntax Silver badge

              Re: Yearly tasks....

              The garage then forgets to reset the timer so when you get it back the car's still demanding that you get it serviced.

      2. David 132 Silver badge
        Windows

        Re: Yearly tasks....

        I use the same methodology for scheduling my showers!

      3. Terry 6 Silver badge

        Re: Yearly tasks....

        Even quarterly. Our mattress is meant to be turned 4 times a year. So, solstice and equinox days.

    6. PRR Silver badge

      Re: Yearly tasks....

      > ...daily tasks, they just get done automatically. Monthly, OK there's a month end process. Yearly tasks are only ever triggered if they are literally scheduled for Jan 1st...

      Well, but we all have birthdays and anniversaries. Also car registration and property tax bills. Some may be very "dangerous" if forgotten.

      My friend RJ was in-charge of taxes at a major telephone company, and tickled himself for hundreds of state and local tax bills. (Maybe the town/state sends a memo, but if it is lost, it is still your problem.)

      Identify or hire a person in your organization who can do that (including pass-on when he leaves).

      1. MachDiamond Silver badge

        Re: Yearly tasks....

        "Well, but we all have birthdays and anniversaries. Also car registration and property tax bills. Some may be very "dangerous" if forgotten."

        For a company, it's better to have the reminder/task done based on a more common event such as a bank holiday. The clocks shift twice each year where I live so that's the day (a Sunday) when I change smoke detector batteries. Anybody else in the household will also know when they get changed regardless of whether they remember by birthday or not. I also write the date on the battery with a felt pen. The batteries I use are made for long life/low drain applications and are mostly white.

      2. Doctor Syntax Silver badge

        Re: Yearly tasks....

        Some may be very "dangerous" if forgotten.

        Wife's birthdays being the most dangerous.

        A good while ago I was on the phone to an insurance sompany and had to give my wife's birthday. The agent told me I had it wrong (they had day and month numbers swapped). I assume he must have been young and unmarried.

        1. MachDiamond Silver badge

          Re: Yearly tasks....

          "The agent told me I had it wrong"

          You also need to remember where you proposed, that date, where you first kissed and where, your wedding anniversary.... etc. This is why doing well on events and dates in history class when at school is good training for later in life. And you thought the class was useless.

          1. Doctor Syntax Silver badge

            Re: Yearly tasks....

            Having had trouble remembering the exact wedding date cal has been a great resource - just check which of the two alternatives was a Saturday.

    7. trindflo Silver badge
      Alert

      Re: Yearly tasks....

      At least in the story the batteries could just be replaced and hadn't turned into a mound of corrosive white powder. Mine needed something closer to a Dr. Who regeneration cycle...but faster. And yes, I'm sure my predecessor had a calendar entry for it.

  2. Anonymous Coward
    Anonymous Coward

    Priorities

    > "I was immediately summoned to explain what had happened"

    There is a crisis in progress and one person in the building who can address it, so naturally you call that person to your office to tell you what they would be doing about it if they weren't talking with you. Why come does it happen that way around so often? Would it hurt to go to where they are dealing with it and ask for a time estimate?

    1. Anonymous Coward
      Anonymous Coward

      Re: Priorities

      "Are you working on the current outage / issue?"

      "Yes"

      "Thank you, carry on, I will let everyone know not to disturb or interrupt you"

      That is all it needs.

      1. Anonymous Coward Silver badge
        Go

        Re: Priorities

        "Anything I can help with?"

        1. drand

          Re: Priorities

          As my dad was fond of saying - "you can help by staying out of the way".

          1. MachDiamond Silver badge

            Re: Priorities

            "As my dad was fond of saying - "you can help by staying out of the way"."

            There must be a manual they hand out at hospital to new dads. Mine worked from the same one as yours.

            1. Doctor Syntax Silver badge

              Re: Priorities

              They're simply repeating an instruction your mother gave him when you were born.

        2. Anonymous Coward
          Anonymous Coward

          Re: Priorities

          "Yes - keep other people from stopping me to ask about the status!"

          1. Anonymous Coward
            Anonymous Coward

            Re: Priorities

            SOP for dealing with major outages at the last workplace was:

            1. Lock the Mission Control doors.

            2. Identify the person least likely to be able to help fixing the problem. Promote them to Incident Communications Manager for the duration.

            3. Redirect all Mission Control phones to the ICM.

            4. Get on with fixing the problem, keeping the ICM informed as to progress.

            1. WanderingHaggis

              Re: Priorities

              We put a sign on the door WE KNOW and our big boss was smart enough to tell people to not bother us. There are a few good bosses.

      2. jmch Silver badge

        Re: Priorities

        "That is all it needs."

        One critical thing missing, which is an estimate to resolution (even if very finger-in-the-air estimate multiplied by pessimism factor), since based on that other procedures might need to be triggered.

        Also, as somebody else noted, asking"anything I can do to help?" is always appreciated even if the answer is almost invariably "just make sure no-one disturbs me while I fix this"

        1. Doctor Syntax Silver badge

          Re: Priorities

          "an estimate to resolution"

          The answer to that is the same as that to "What's wrong?": "If I knew that it wouldn't be broken."

        2. David 132 Silver badge
          Happy

          Re: Priorities

          One critical thing missing, which is an estimate to resolution (even if very finger-in-the-air estimate multiplied by pessimism factor)

          The Scotty approach. "It'll take six hours Captain. But as we dinnae have six hours, I'll have it ready in two."

          "Wait. Mr Scott, have you always multiplied your estimates by a factor of three?"

          "O'course, Captain. How else can I maintain my reputation as a miracle-worker?"

      3. Peter Gathercole Silver badge

        Re: Priorities

        I had such a situation. 03:30 in the morning, just made a breakthrough in getting the backup server restoring it's database after a major storage (and a bug in the restoration procedure) failure that had taken the office support environment and a couple of the test and development environments out for several days.

        One of the project managers is sitting in the next desk over keeping everyone else away (it was really surprising how many people were still around at that time in the morning) so I could get on with the problem, interrupts me quietly and asks "Coffee?"

        Answer, "Tea please" and I returned to proving that the backup server was working enough to start the restore of the rest of the servers in the environment.

        I have literally never been so grateful for a project manager sitting on my shoulder.

        1. MachDiamond Silver badge

          Re: Priorities

          "I have literally never been so grateful for a project manager sitting on my shoulder."

          I've had a total of one engineering manager that good. There should be a degree required with this sort of lesson included.

    2. Anonymous Coward
      Anonymous Coward

      Re: Priorities

      It begins at home. I've lost count of the number of times SWMBO has stopped me getting ready in a hurry because we are late, to tell me we are late and I need to hurry. SMH²

      Anon 'cos I'm attached to my testicles and I'd like it to stay that way.

      1. Potty Professor
        IT Angle

        Re: Priorities

        My SWMBO was always late for anything. As far as she was concerned, leaving the house at the time we were supposed to be at the appointment counted as "being on time", I solved this by telling her a fictitious appointment time that was sufficiently advanced that we would have time to actually be on time. I used to say that she would be late for her own funeral - but unfortunately she wasn't. RIP Rosamund, still love you.

        1. GlenP Silver badge

          Re: Priorities

          I solved this by telling her a fictitious appointment time

          We had a corporate manager in the Netherlands who really resented any travel in work time. He'd try and insist on us leaving the office at normal finishing time, driving to the airport, flying to Schiphol then taking a train right across the country, with the result you wouldn't reach the hotel until after midnight but would still have to be in the office for 08:00. For the return he'd make sure he delayed your departure from the office until the last possible moment for you to catch the last possible return train to get your homebound flight.

          Of course the end result was a lot of lying. "Sorry, the later flights were all full!", when they weren't. "The only return flight is the 14:30!", when actually flying out on the 15:30 flight.

          1. MachDiamond Silver badge

            Re: Priorities

            "Of course the end result was a lot of lying. "Sorry, the later flights were all full!", when they weren't. "The only return flight is the 14:30!", when actually flying out on the 15:30 flight."

            You could have tested the manager by saying you "could" leave later, but the only seats left were business/first class. Get a wet signature on the upgrade so accounting doesn't throw the cost back at you although I'd expect the manager, who's bonus will require containing costs, will suggest an earlier departure.

      2. Anonymous Coward
        Anonymous Coward

        Re: Priorities

        You might think you are, but no, SWMBO owns them, you just keep them warm.

        1. Anonymous Coward
          Anonymous Coward

          Re: Priorities

          "You might think you are [attached], but no, SWMBO owns them, you just keep them warm."

          Perhaps, but you are still actually attached to your testicles, as some of us might be all too painfully aware.

          (That's a Viz strip that I'm not sure that I particularly want to imagine or read, but unfortunately it's too late now…

          Buster Gonad's cousin, whose testicles are not quite so unfeasibly large, but whose pocket gobstoppers instead are very much on the outside and float around, generally orbiting his waist like some kind of binary planetary system, completely unattached to his body. Exactly how his man yoghurt gets from inside these juicy plums to the, uhh, pump delivery nozzle is one of the great unexplained mysteries of the universe… Hilarious adventures ensue…)

          (And, as for warm, if the silly things didn't need to be 2°C cooler, and were sensibly tucked away like their counterparts, life would be so much better for those of us of the male persuasion (and probably a whole lot less painful). A real, umm, design cock-up by whoever bodged that demo model - you were supposed to fix it in production!)

          [Different AC, staying AC so that I don't get locked up in a padded cell]

          1. Benegesserict Cumbersomberbatch Silver badge

            Re: Priorities

            if the silly things didn't need to be 2°C cooler, and were sensibly tucked away like their counterparts, life would be so much better for those of us of the male persuasion.

            Each to their own. There's some entertainment value in having them accessible.

            The equation is something like cooler gonads -> more efficient meiosis -> higher quality tadpoles -> more kids -> evolutionary selection pressure. And then there's the whole keeping them warm gives you cancer thing. And google "bell-clapper deformity" if you want to skip your next meal.

    3. Anonymous Coward
      Anonymous Coward

      Re: Priorities

      I once rather stunted my career prospects within an org because of this.

      There were two large (but separate) outages, affecting two large customer networks. It was during the later hours of our night shift, but around peak time for them. There were only 2 of us on (due to cost-cutting leading to understaffing).

      The two of us took a network each and started investigating.

      Except... the phone kept ringing, with our CEO demanding updates/ETA etc. This wasn't driven by customer demands - they were well aware we were working on it and that we'd give updates as/when they became available, it was just the CEO and his micro-managing ways.

      Around about the 5th time that the phone interrupted my train of thought, I answered only to hear our CEOs voice, again asking for updates and ETAs that couldn't yet be given. Rather than explaining again that we weren't yet in a position to provide these, I said "at least 8 hours".

      He was *beyond* dismayed and asked what could possibly have gone so badly wrong that it needed that kind of recovery time. So, I explained that the team were unable to effectively investigate and recover services because they kept being interrupted by pointless calls asking for status updates.

      He didn't phone back again, but did phone my boss later in the day (who backed me up).

      During the process for a later internal job-move (and promotion) though, the hiring manager needed approval from the CEO, who denied it.

      1. John Robson Silver badge

        Re: Priorities

        Should have got your boss in earlier...

        Had a self proclaimed paper pusher manager at a previous role, and they were brilliant.

        They had a great memory for "the team saw something maybe similar seven years ago, here's the first and last couple of emails - does it sound relevant" and was excellent at being a management shield when needed.

        I only had one "event" where I ended up on three calls first thing in the morning - the third was meant to be a progress report from the first - but the time between them had been taken up with the other call from a different set of management.

        Now both sets of management did need to be updated, and updated properly (and that really meant by me), and the incident wasn't getting any worse (as we were now running correctly and had backups of all the raw incoming data from the event) - it eventually took two full weeks to repair and reprocess the data (which had been sent in the wrong timezone by a handful of systems, there was a nice overlap of timestamps at one end of the data which took more effort to resolve than I care to remember). But the first thing I said on the third call was "no progress, and there never will be if I don't get off these calls now."

        1. Doctor Syntax Silver badge

          Re: Priorities

          Just don't answer the phone. You can explain this later by saying you were too busy fixing the problem. Once the problem is fixed it's difficult to argue against that. A variation would be that you were too short-handed to spare anyone to answer the phone, something that will need to be addressed in the post-incident review. Alternatively just unplug the phone and explain that the problem had taken the phones out as well.

    4. Anonymous Custard Silver badge
      Headmaster

      Re: Priorities

      Doubly so when they insist on PowerPoint presentations, pretty graphs and all the other nonsense required to prettify the data enough that their brains can parse it.

      Or perhaps not even then, and you have to spend extra time explaining the basics, and then when another manager joins half way through then doing it all again from the beginning.

      Leading to the oft repeated mantra of "Do you want me to do this update, or just waste my time actually fixing the issue?"

      1. HuBo
        Go

        Re: Priorities

        Ah! But MS Office's genAi CoPilots stand ready to solve that problem for one and all, in one short synthetic heartbeat. Complete with face-saving hallucination if need be ... that tech is a godsend for all On-Call Who-Me's (unless that's the very tech that titsupped itself of course!)!

        1. Anonymous Coward
          Anonymous Coward

          Re: Priorities

          Finally a use case for it! We have a pilot project to try it out, and for anything technical it seems utterly useless. But producing a content-free update for management? Yep, seems right up its alley.

          1. Antron Argaiv Silver badge

            Re: Priorities

            The company I work for is currently all-in on MS metrics. *everything* is measured (for some definition of "measured") and presented ina dashboard. Decisions are made based on the KPIs. AI can only make this more "interesting".

            I particularly enjoy the "liklihood of resignation" metric that my manager is required to estimate.

            1. Doctor Syntax Silver badge

              Re: Priorities

              A metric that becomes a target ceases to be useful as a metric. In fact, it becomes dangerous as everything else will get rearranged to optimise it.

        2. Doctor Syntax Silver badge

          Re: Priorities

          Just need text to voice for the output.

          I'm hoping that when our POTS is replaced by a non-resilient VOIP service one of the by-products will be some form of VOIP>text and text > VOIP drivers. That will then be coupled up to something like Eliza as an extension to which scammers, would-be smart meter installers etc can be directed. (I wouldn't trust ChatGPT or the like - they might hallucinate an agreement to something. I just want an arrangement in which the callers can wander around getting more and more frustrated.)

          1. Andy A

            Re: Priorities

            I know any number of organisations who have implemented exactly that. There's a feature labelled "Chat" which replaced the old FAQ section, with a perfectly equivalent lack of relevance to the real world.

          2. Anonymous Custard Silver badge
            Trollface

            Re: Priorities

            I just want an arrangement in which the callers can wander around getting more and more frustrated.

            And for bonus geek points, include reference to pale bulbous eyes watching them in there at random intervals.

            1. Uncle Slacky Silver badge

              Re: Priorities

              Then tell them they've been eaten by a grue.

              1. Doctor Syntax Silver badge

                Re: Priorities

                I was thinking of retaining some of the old Eliza dialogue around asking if the user was unhappy talking to a computer once they used that word. Given that the source would spoken there are opportunities for claiming to have problems with the user's accent or asking them to speak more clearly if an input line doesn't include any trigger words.

          3. Anonymous Coward
            Anonymous Coward

            Re: Priorities

            Meet Lenny:

            https://www.youtube.com/watch?v=XSoOrlh5i1k

            I have this installed on our VOIP phone system. Cold callers and scammers get forwarded to "him".

            At my previous employer, we actually had someone show up in person at our office saying that they were there for a meeting with Lenny.

            1. Doctor Syntax Silver badge

              Re: Priorities

              I originally thought of that but I'm not sure Lenny would pick up on key words and suck callers in in quite the same way. I also think call centres will eventually get to recognise him.

          4. John Brown (no body) Silver badge

            Re: Priorities

            "I just want an arrangement in which the callers can wander around getting more and more frustrated.)"

            <envisions a system as you describe, but attached to Colossal Cave> :-)

            1. Doctor Syntax Silver badge

              Re: Priorities

              Indeed.

    5. ColinPa

      good boss

      Having finished some work for a bank, I was due to fly home Sunday lunch time. Sunday 0600 the phone rang. It was the boss saying they had a problem. They had tried to fix it.but the backups didn't work. Please could I help. He said anything I need - let him know. I quietly worked on the phone with the customer's guys, getting it sorted. Every 10-15 mins I messaged my boss giving status. I said we needed a networking guy... and 10 minutes later their top networking person was on the phone.

      We got it fixed by lunch time - just in time for me to get my flight.

      Afterward my boss told me the extent of the outage. Half of the bank was down - it was on the national news. He said he fielded the calls from every one- including the head of the bank's disaster committee who wanted to talk to the people fixing it. My boss said to him, if they tell you the problem and the solution, you will not understand it - trust us.. He didn't tell us the scale of the problem (although we guessed) because he didn't want us to rush and mess it up.

      He was a good boss - he handled all the flak, and he trusted us to get on with it.

      1. Anonymous Coward
        Anonymous Coward

        Re: good boss

        We had a director who showed up at one site when there was a crisis, and fended off the crap while the techies fixed the issue. Someone asked why he was there, he said that the shit had hit the fan so he'd turned up to hold the umbrella. Good guy.

        1. M.V. Lipvig Silver badge

          Re: good boss

          Glad to hear it. I've worked for way too many directors who only showed up to direct the stream to wherever it would cause the most trouble.

        2. Doctor Syntax Silver badge

          Re: good boss

          Knowing that this is part of their job is one of those things that differentiates a manager from a mangler.

    6. Don Bannister

      Re: Priorities

      I used to work in the broadcast world, and came across a couple of such stories where the big boss wanted to know what was going on ....

      1. In person. "What are you doing to get this fixed ?" Reply: "Talking to you"

      2. On the phone, after rather tetchy reply from the fixer. "Do you know who I am ?" Reply: "Yes - do you know who I am ?" "No" Reply: "Good !"

      1. Anonymous Custard Silver badge
        Headmaster

        Re: Priorities

        1. In person. "What are you doing to get this fixed ?" Reply: "Talking to you"

        The better answer is: "What are you doing to get this fixed ?" Reply: "Currently nothing, because I'm having to talk to you instead"

        It semi-subtly gets the point across, without the aid of a clue-by-four to the cranium.

      2. Doctor Syntax Silver badge

        Re: Priorities

        "Do you know who I am ?"

        "No, sorry, can't help you there."

    7. mhoulden

      Re: Priorities

      One time someone asked me to call round to get rid of a virus on his home computer. He didn't have broadband (this was quite a long time ago) so I had a CD full of security patches and virus checkers. It would have been pretty straightforward if he wasn't looking over my shoulder all the time and sounding like Stan Laurel just after he'd pushed Oliver Hardy out of a window. In the end I sent him away by asking him to get me a cup of coffee and not to rush back. By the time he did return I'd removed the virus and was installing the latest service pack. I also had words with him about the importance of backups that you can use on another computer.

    8. Andy A
      Facepalm

      Re: Priorities

      It's amazing the number of times when, as the only person on site capable of addressing the situation when something important has gone TITSUP, some high-up DEMANDS that I attend a two-hour conference call to discuss why the fault has not been cleared.

      Naturally this both raises the blood pressure and forces a return to Step One of the fault-finding and repair process.

      It was doubly frustrating when I couldn't turn the mobile phone off because I needed its meagre light to see inside a server, the building's lights being on a timer and it being "after hours".

      1. Doctor Syntax Silver badge

        Re: Priorities

        "some high-up DEMANDS that I attend a two-hour conference call to discuss why the fault has not been cleared."

        The temptation to take it seriously might be irresistible.

      2. Anonymous Custard Silver badge
        Boffin

        Re: Priorities

        It was doubly frustrating when I couldn't turn the mobile phone off because I needed its meagre light to see inside a server, the building's lights being on a timer and it being "after hours".

        Flight mode is your friend and wingman in many such situations...

      3. Dante Alighieri
        Mushroom

        Re: Priorities

        Take the call - can't work as there are no lights - can you sort it?

        I'll phone back when they are on....

        Give them (at least one) job.

        And I have done the "longer since you interrupted me" to some martinet.

        1. TimMaher Silver badge
          Pint

          Re: Martinet.

          Fabulous! Haven’t heard that used in years, maybe even decades.

          Have a brew.

    9. doublelayer Silver badge

      Re: Priorities

      "Would it hurt to go to where they are dealing with it and ask for a time estimate?"

      Yes. I can't give you a time estimate until I have at least some idea of what went wrong. No, not the symptoms, the causes. It takes a while to figure out the causes, but sometimes after identifying the causes, the fix is thirty seconds and a reboot. The nontechnical always manage to assume that their statement of the problem is all the technician needs to know how hard fixing that problem is.

    10. M.V. Lipvig Silver badge

      Re: Priorities

      Occasionally in my job, I get asked to join a customer bridge. I always, ALWAYS refuse to do so. The last time I did, it was nothing but a bunch of C-suiters asking inane questions about the problem, a bunch of "I'm better than you" posturing, legal threats, and wailing and gnashing of teeth. You should have heard them when I told them their circuit wasn't being worked on because I was on their bridge and unable to work on their outage.

      These days there are people whose whole job is to join bridges and listen to customer babble, which is OK by me. They make more than I do, but better them than me!

    11. OzBob

      Re: Priorities

      Yes, I was working for a big multi-national car manufacturer and had been trying to solve a production outage for 20 mins, only to spend 15 mins of that explaining to 3 different users who had my number, why I was not progressing the resolution of said issue. In the end the 4th call came through and I picked up the phone and said "if you are not my boss or the servicedesk, you are not important enough to talk to me right now". Once I heard the voice and determined it was not either of those two, I hung up the phone and left it off the hook.

      My boss later called me aside to talk about my "ubiqituous telephone manner" to a very senior user but I stood my ground and said that when I am resolving an issue, his job is to get everyone off my back so I can get on with the work.

      1. Doctor Syntax Silver badge

        Re: Priorities

        See my comment above about the difference between a manager and a mangler. He failed the test.

    12. Ahosewithnoname

      Re: Priorities

      "How long will it take to fix?"...

      "10 minutes longer than it would have done if you'd not called me"

      Was always a favourite one of mine

  3. Roger Kynaston
    Pint

    radio ntp

    I remember those toys. I can't remember why the local authority I was working at couldn't just allow port ntp through the firewall but there was a "reason". It wasn't batteries that caused the outage though. Window cleaners had dislodged the network cable.

    icon because it is nearly that time

    1. Giles C Silver badge

      Re: radio ntp

      I used to manage a network with a couple of those (Galleon Ntp servers) one was fine but the unit at one office had problems.

      The first antenna used the time signal from rugby which worked until they moved it to Birmingham (I think) at which point a large row of trees blocked the antenna completely.

      After this we swapped to a gps unit mounted on the roof which worked until the enclosure sprung a leak and fried the electronics.

      The final time I heard about it was about 2 years after I left the company when some builders cut through the cable as it snakes under the floor and they called me to ask where the thing was routed to….

      When nobody was trying to sabotage it they were brilliant at their job and ran for about 15 years - they may still be working away…

      1. keithpeter Silver badge
        Windows

        Re: radio ntp

        Anthorn in Cumbria

        1. Giles C Silver badge

          Re: radio ntp

          Ah I remember it had moved but couldn’t remember where to.

      2. markdiss76

        Re: radio ntp

        This bring back memories, but of time flipping the other way.

        Back in the Day was part of a team administering a Novell Netware 4.x network - pre NTP days, so we had one of these https://web.archive.org/web/20010411015625/http://www.galsys.co.uk/galdata1.htm keeping time set on the Master Netware replica from the radio signal in Rugby. One day it picked up some kind of erroneous transmission and set the network time forward to the year 2032 or something similar. After resetting it and correcting the network time we had the dreaded "Synthetic network time" on the eDirectory replicas as we were stuck with future dated timestamps until we ran the command to declare a new "epoch".

        It was such a joy once NTP arrived with no single server source issues ready to blow things up.

        M

      3. Peter Gathercole Silver badge

        Re: radio ntp

        The MSF feed from Rugby and then Anthorn is broadcast on the longwave band at a low frequency of 60KHz.

        This frequency will travel through most things, and will also get bounced around the atmospheric layers quite easily, so I don't think that a row of trees, or even a whole forest would have stopped it. But when the change happened, you probably should have re-aligned the antenna (unless you were to the south-east of Rugby), it being quite directional at that frequency.

        The other thing you should have done was change the offset to cope with the change in distance to the transmitter. The standard code includes a nanosecond offset to cope with the distance from the transmitter, although for most purposes, this is irrelevant, given the speed of light and the small(ish) size of the UK.

        The clock I used to manage many years ago was the classic Radiocode Clock RCC8000 which was a 19" rack mounted unit that fed an xntp server via a serial link (another reason why nanosecond offsets were largely irrelevant). It was eventually replaced by a Meinberg time appliance that drove NTP directly, although it used the German DCF time signal transmitted from Mainflingen on 77.5 KHz.

        1. Anonymous Coward
          Anonymous Coward

          Re: radio ntp

          Dis like someone telling me how brick wall be built cause "I hung pichur once, put nale in wall wit poundy poundy"

  4. Jason Bloomberg Silver badge
    Trollface

    Meet a reader we will Regomize as "Colin"

    Is that a British "Colin"?

    Or the American "Colene"?

    1. KittenHuffer Silver badge

      Learn something new every day!

      I thought the question was ALWAYS African or European?

      1. LordZot

        Re: Learn something new every day!

        Laden or unladen?

      2. A.P. Veening Silver badge

        Re: Learn something new every day!

        I thought the question was ALWAYS African or European?

        And the answer is 42.

      3. Arthur the cat Silver badge
        Happy

        Re: Learn something new every day!

        I thought the question was ALWAYS African or European?

        "Spit or swallow?" is first. It's only after that you might question which sort of swallow.

        1. Graham Dawson Silver badge

          Re: Learn something new every day!

          But where do the coconuts fit in?

          1. Anonymous Coward
            Anonymous Coward

            Re: Learn something new every day!

            under the mast?

  5. GlenP Silver badge

    Time...

    Is a PITA!

    I was seriously considering an atomic clock setup for one of our servers as, for some reason we couldn't fathom, it was consistently 3 minutes slow despite regular NTP calls and irrespective of the time server used. Had I been sure it would have resolved the issue I'd probably have gone ahead but in the end our support company managed to fix it with an obscure patch.

    Did 3 minutes really matter? Whilst not as critical as the time for financial trading yes, it did! The factory time clock updated from the server weekly (and when the clocks changed), that three minutes was the difference between some of the operators catching their bus home or waiting an hour for the next one.

    Not work but we had an issue with linked electronics (it was a proprietary ring bus system) on a yacht where the time on the chart plotter (which didn't have it's own RTC) had to be manually reset at power up when it was supposed to take the time from the GPS. Having finally managed to track down a USB interface for the network we went into packet sniffing mode and it turned out the scenario was:

    Chart Plotter sends Time Request Good so far

    GPS responds to Request Still good

    Chart Plotter responds to Request Eh? Why is it responding to its own time request?

    Chart Plotter receives time from GPS and updates Correct time

    Chart Plotter receives time from itself and updates Overwriting the correct time with the default start up time

    We were able to update the chart plotter config to stop it responding to time requests but why the heck it was ever enabled to do so was beyond us.

    1. Korev Silver badge
      Facepalm

      Re: Time...

      I had to fix an outage a few years ago where the Acopia fronting the storage used the PKI servers for its clock and everything else used Active Directory and the PKI clock became inaccurate the Windows boxen wouldn't talk to the Acopia any more!

    2. Lee D Silver badge

      Re: Time...

      I like to operate an NTP server in any significantly-sized network.

      It doesn't need to be microsecond-accurate, it doesn't need an expensive GPS card in it, it just needs to be the "one true reference" for all time on your local network, in case Internet goes down.

      You can do that with just any device running an NTP server and letting it drift from real-time - it'll just drift back into sync when it's reconnected.

      I have used network router for this, I have used VMs for this (VMs don't hold time perfectly well on their own but in concert with an outside NTP upstream server they are fine), I have even exposed an internal NTP server as an NTP pool server for other people to use (because NTP pool has a lovely monitoring feature where it will alert you if your server is uncontactable or getting out of sync with everyone else's in the pool).

      You can get MSF modules to pull in an atomic clock radio signal and feed into the NTP server.

      You can get GPS modules to do the same.

      And you can sync with as many NTP servers as you like, it really doesn't matter.

      Then have all your clients refer to that AND other NTP servers (in case it goes down!). Windows allows as many as you like, but doesn't present the option in the GUI.

      I normally have all clients using 0.uk.pool.ntp.org and 1.uk.pool.ntp.org (guaranteed to be two different UK NTP Pool servers), time.windows.com AND whatever internal IP (hence not DNS-reliant) I'm running an NTP server on, and that NTP server syncs to a bunch of well-known NTP servers on the Internet.

    3. Dante Alighieri
      Facepalm

      Re: Time...

      Where I work, the parts of diagnostics on the vlan (10*) have no access to the hospital LAN or NTP or elsewhere. Running 5 minutes or more adrift (flagged x lots to IT) meant I managed to time travel such that I issued a report on an image that had yet to be taken. According to the time stamps.

      Love the NHS.

      And our CT scanners will only do NTP with Japan.

      I run my own NTP across the few computers at home. It just makes sense.

    4. Benegesserict Cumbersomberbatch Silver badge

      Re: Time...

      As Susan Sto Helit will tell you, it's always now everywhere.

      1. MachDiamond Silver badge

        Re: Time...

        "As Susan Sto Helit will tell you, it's always now everywhere."

        For that, you get a gold star.

    5. bazza Silver badge

      Re: Time...

      Time is important in a corporate setting. Log data is an important part of defending a company against malicious staff; it’s what can form the basis of evidence used to formulate a winning prosecution.

      Get it even slightly wrong, and the log data is inadmissible. Can’t prove that the timestamps are bang on? Hard luck. By “prove”, you need log data showing correction / updates of time from a trusted source - the Rugby radio clock MSF is a good one - taking place on a routine basis. Having two independent sources is a good idea too.

  6. Stuart Castle Silver badge

    Surely the clocks should have been mains powered, with at least a UPS to act as a backup?

  7. Stephen Wilkinson
    Joke

    Did Colin play bass for Bad News?

  8. Peter Prof Fox

    The easy and obvious method

    Is to have a list of things to do when changing to and from daylight saving time. Even a little bit of redundancy for an annual event.

  9. Lee D Silver badge

    Plug the damn thing in with a power supply, or failing that some AA eliminator/emulator batteries ( I can't think what they're called, but they're basically an empty battery shell in the same size, with a connector in the middle to attach positive/negative leads of a real PSU).

    And then put the power supply ON A UPS.

    1. Doctor Syntax Silver badge

      The cleaner will just unplug it from the UPS for their vacuum cleaner.

      1. Arthur the cat Silver badge

        The cleaner will just unplug it from the UPS for their vacuum cleaner.

        There's a lot to be said for using a fused spur if you can, but it's a bugger finding low voltage power supplies that aren't meant to plug into a standard mains socket. (If you know of any that can be wired into a mains spur, please let me know in a reply.)

        1. H in The Hague

          "If you know of any that can be wired into a mains spur, please let me know in a reply."

          You could just mount one of these in an appropriate box:

          https://uk.rs-online.com/web/c/?searchTerm=power+supply+din

          1. Anonymous Coward
            Anonymous Coward

            This is how door access systems are set up. Locked box, fused spur

  10. tip pc Silver badge
    Coat

    battery powered network time clock?

    Why have a battery powered network time clock when everything that needs that time is dependent on electricity to run?

    At worst the batteries could be for backup but you'd use rechargeables

    or

    just plug the time clock into the UPS and ensure it boots before everything else so its available to do its job the moment a client asks.

    lastly, every system i've ever used that uses a network time protocol like ntp has a local battery backup for its time plus employs a mechanism to slowly drift its time to a new time from ntp if its wildly different. i doubt all those systems would be on Jan 1st 1970 by 6am, even if the time source broke 2 days before (yes i know but is an example of how it works).

    1. Doctor Syntax Silver badge

      Re: battery powered network time clock?

      "Why have a battery powered network time clock when everything that needs that time is dependent on electricity to run?"

      People buy what's available for the given budget.

      1. DS999 Silver badge

        Re: battery powered network time clock?

        I would think even someone like me who has no sort of EE type degree could remedy that. Just buy a "wall wart" that outputs the correct voltage, strip its wires and solder it against the battery connectors in the radio.

        D cells for instance are typically 1.5v, if the four are connected in series (i.e. they alternate direction) when installed that means you need a wall wart that outputs 6v. You can use anything you find regardless of the wattage since if the batteries are lasting a year their nominal draw must be tiny.

      2. tip pc Silver badge

        Re: battery powered network time clock?

        People buy what's available for the given budget.

        that is very true, but given a time server is meant to be serving requests 24/7 over a network etc, i find it hard to believe a battery operated network time clock can operate for more than a week on 4x aa batteries before needing replacements.

        Time clocks are polled frequently and respond often.

        1. Doctor Syntax Silver badge

          Re: battery powered network time clock?

          That's tough seeing as it was the basis for the entire story.

  11. Boris the Cockroach Silver badge
    Facepalm

    Remember those batteries

    on motherboards.

    Just enough voltage to keep the BIOS memory alive

    Funnily enough we have the same system on the robots.... little batteries designed to keep the memory alive... should be swapped out every 2 years, or even maybe during the 3 months the voltage has dropped far enough to trigger the 'replace batteries' alarm but not far enough for the robot to lose memory.

    We have a procedue for servicing those now after someone had to spend 3 days typing all the robot parameters in by hand, then checking the motion, before loading in the control programs.

    As I tell the PFY, you can learn the easy way by listening to other people's experience, or you can learn the hard way.... we picked the hard way with those batteries....

    1. doublelayer Silver badge

      Re: Remember those batteries

      "As I tell the PFY, you can learn the easy way by listening to other people's experience, or you can learn the hard way.... we picked the hard way with those batteries...."

      I hope that, if you're building these robots, you learned that the price of flash is not that high and certainly beats any device where you need a battery to keep it from bricking itself. And if you don't build them, I hope you still learned that and will choose robots accordingly next time you have to buy some, because maybe that will convince manufacturers not to skimp on what's probably 0.002% of the manufacturing price. As I commented to a friend who was choosing how much room for expansion to put in a hardware design (that was intended to receive software updates going forward), it's much more expensive to try to convince customers to buy version 2 of the hardware when version 1 was clearly underpowered from the start than to make version 1 good enough that, by the time there is a version 2, the customers were happy with what they got.

      1. Richard 12 Silver badge

        Re: Remember those batteries

        The CMOS battery also keeps the RTC running.

        Generally the manufacturer burns the BIOS/UEFI settings into flash, but some customers make changes...

        1. doublelayer Silver badge

          Re: Remember those batteries

          Which is why you include a little more flash so the customer's config can be retained when the battery breaks, and if this thing is networked, you enable some ability to sync the time with NTP. I'm not saying that it has to run without a battery entirely, but that having the battery die should not cause a significant fault. There was a time when flash was too expensive to do that. It is no longer. If you need a particularly large amount of flash such that including it would be expensive, consider having a user-supplied storage method instead. Lots of OEMs have managed to provision their hardware with those chips.

          As I said in my previous comment, failing to do this is what makes customers think that this manufacturer is being too cheap to be a reliable supplier. For example, a supplier of equipment who told a friend of mine that the equipment they bought would not get the feature update they were releasing because it only had 8 MB of RAM and that really wasn't enough to run this software. While that was probably true, my friend was right to ask why the equipment they designed in 2017 only had 8 MB of RAM, as this made it look like they had planned to cut off updates quickly.

          1. Jou (Mxyzptlk) Silver badge

            Re: Remember those batteries

            > designed in 2017 only had 8 MB of RAM

            That is so true. Comparison: The smallest Ryzen 3 1200 from 2017, has 8 MB 3rd level cache (only embedd versions have less cache, and don't look at Intel i3 CPUs of that time). The very first Raspberry Pie from 2012 had 256 MB RAM.

            1. doublelayer Silver badge

              Re: Remember those batteries

              In fairness, this wasn't a full computer, but an embedded, low-power device. It didn't need a Raspberry Pi, and having one would have caused some problems with power management and OS overhead. We didn't begrudge them having that amount of RAM as much as having that amount of RAM and then writing software that needed, maybe, 12 MB to run. It is a problem that recurs at other levels as well. Manufacturers need to ask themselves whether some spec will cause a problem for their customers a few years from now because otherwise, the customers will probably try to find someone else to provide the equipment next time.

          2. Terry 6 Silver badge

            Re: Remember those batteries

            *It does seem sometimes that as a rule of thumb, any component that a manufacturer thinks won't be noticed will be designed* sub-par.

            Or redesigned. I had a film camera camera when I was a kid. Dad took it on holiday at if broke so he wanted to replace it. The original one had vanished from the shops, only version we could find was a rebadged one in Dixons. Absolutely identical to look at. Same price or thereabouts. Similar name. Apparently only the badge was different. 'Cept it never worked properly. The film would slip and frames overlap ( I didn't tell dad, he'd have been so upset).

            When I took it apart it was obvious why. The original toothed wheel had been shiny steel. This one had lots of nylon parts instead.

    2. Jou (Mxyzptlk) Silver badge

      Re: Remember those batteries

      So there is no backup of the robot configuration and parameters which you can easily restore? What kind of robots are those which don't offer a backup-restore procedure?

      1. Boris the Cockroach Silver badge
        Unhappy

        Re: Remember those batteries

        There is a backup/restore feature built in.

        However to configure the communication interface takes parameters

        Which have just been wiped by the battery fail.

        I wish I was making this up...

        <<starts screaming and runs off into the night like a banshee

    3. hayzoos

      Re: Remember those batteries

      I remember those "non-replaceable" ones well. At a job from decades ago one had died. For both budget and user sentiment for the computer which had been upgraded to Windows 95, I could not properly fix it. I implemented a new procedure. Do not turn off the "hard drive" (box under the "computer" (box with the screen on the front)). On power failures or other cause of powering off the "hard drive", I followed my part of the procedure. I set the time according to Timex (a handy-dandy device worn on the wrist) and re-entered BIOS parameters such as hard drive geometry according to notes I had previously made on an Ampad and a No. 2 stylus - actually from a backup made on a Xerox. The last step on those note was a reminder to fix the time on the coffee maker and the security camera VCR.

      This particular computer was also the one which caused me to realize that Windows 95 had a bug related to long uptimes. Windows update not yet being invented, let alone evolving to a reboot second Wednesday of the month. Windows 95 was actually a lot more stable in retrospect if you kept it off the internet.

  12. Matthew "The Worst Writer on the Internet" Saroff
    Pint

    I agree with not ratting out the help desk

    But they should now be responsible for everyone's first pint FOREVER.

  13. aerogems Silver badge

    I remember one time at an old job I tried to cover for someone, but they wouldn't accept it.

    Part of my job involved creating new part numbers in SAP, and some months prior one of the engineers had requested a specific part number and description. SOP for my department was to stash a copy of the email in an ECM header long text, and then use that ECM to create the new part number. Fast forward a few months, or maybe even years, the person is upset about something or other with the part number. I immediately look up the ECM and find the email where the same engineer had requested it be exactly that way, but since I generally liked this person as they would typically take some time trying to make sure descriptions fit within business rules, unlike the rest of the engineers, I tried to give them some hints in a reply that they had requested it that way and if they wanted to make some changes it was still possible because the part number was still sitting at "just created" status. Unfortunately, they wouldn't let it go, and so after about 3-4 back and forth exchanges with them, I finally just dropped a copy of their original email in a reply back to them. To their credit, upon seeing this, they immediately backed down and apologized. So, I didn't press the issue any further, and let them off as easily as I could.

    As a general rule, if I like someone I'll try to come up with some way to let them save face, even if we both know it's bullshit. Similar to the "loose cable" or similar excuse of our hero. And in the case of this week's hero, seems like the least they could do since the helpdesk was having to deal with all the irate traders.

  14. Herby

    Clocks & time...

    Sometimes they get a bit confused.

    I have this nice wall clock from Radio Shack which tries to sync to WWVB (60 kHz signal). Unfortunately it doesn't do this well for some reason, and just free-runs if it doesn't. This is fine for the most part, but a few weeks ago we went from PDT (daylight time/summer time) to PST (standard time/winter time). Now if the clock had been able to receive the time signal, it would have corrected things nicely, but alas, it appears that our government (which operates the time signal stations) probably has cut the budget so they reduced the power. Thus the signal isn't what it used to be, and the clock hasn't "fall back" an hour. Oh, well. I just replaced the batteries (nice reminder!) and set the time manually. Life goes on.

  15. Dafyd Colquhoun

    Bodgy time

    I worked at a substation that had a particularly peculiar NTP server. For some reason, known only to the clowns the "designed" the SCADA they made NTP report local time instead of UTC. Broke every standard, but somehow the numpties managed it in an RTU.

    All was "fine" until New Years Day (and yay, working a public holiday for no compensation, because of crappy industrial law in the country). Turns out the fudge code to apply UTC offset from localtime had a hard coded year.

    Standards - they are there for a reason!

  16. Free treacle
    Pint

    Colin, wherever you may currently be, you're a solid guy.

  17. Kevin McMurtrie Silver badge
    Holmes

    Ashtrays

    From the article title I thought it had something to do with cigarette smoke in the moving parts.

  18. Jsmirh939

    Probably would have told the truth.

    Might seem mean but this was a catastrophic event, and a post mortem should be used to determine the root cause and determine which processes need fixing.

    Can't fix what isn't shown to be faulty... and critically important.

  19. M.V. Lipvig Silver badge
    Trollface

    Shouldn't every on call tech

    be regomized as Colin, when they were called in to fix something?

  20. JavaJester

    Covering for the Helpdesk

    Good call. Nothing hath fury like the helldesk scorned.

  21. JimC

    |Solidarity

    In far off days before a good number of you were born, payroll for a large organisation (and this was a very large payroll, one of the biggest in SE England) was done by running a batch program which took some hours and generated a computer tape (yes, one of those big reels beloved of old school SF films). The tape was then couriered off to the bank, and the bank would process the tape which would make all the payments to the right accounts. Usually.

    This was not to be a good day. There was a problem with the output, which was thought resolved by editing the tape. Tape went off to the bank. There was a further problem and more editing, but eventually the tape was run. And it made all the payments. Into a single building society account. This was spotted, and the building society stopped the money actually appearing in the lucky person's account. However they refused to actually return the money until all the i's and t's had been dotted and crossed, which I suppose is not unreasonable with the sums involved.

    Trouble was everyone needed to be paid on time, so my late employer had to borrow the money for the entire payroll until they got the original money back. The interest payments on such a large sum were appreciable, and it was reckoned to have been the most expensive ICT cockup in the organisation's history. There was a big post mortem, but it allegedly failed to find out where the fault lay. I always believed that the technical staff on both sides very deliberately failed to find out where to pin the blame in case it was their side!

  22. Wzrd1 Silver badge

    The solution is simple

    Quarterly battery replacement. That way, they'll get replaced annually on a good year.

    And likely, management will complain about the budget and additional expense...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like