Bank's datacenter died after travelling back in time to 1970 The steady process of time means that The Register has once again arrived at Friday and the timeslot we reserve for On Call – our weekly reader-contributed tale of tech support trials and tribulations. This week, meet a reader we will Regomize as "Colin" who shared a story of his time working as a support engineer for a major …
The thing is, with daily tasks, they just get done automatically. Monthly, OK there's a month end process. Yearly tasks are only ever triggered if they are literally scheduled for Jan 1st (or 2nd if it's a manual task). One can so easily imagine the batteries being installed in June, someone putting a calendar entry for the following June and everybody forgetting about it. Then the first person leaves the company, the batteries continue on their merry way for 3 years, and before you know it, there isn't even anyone on the team who remembers what that box is and what it does.
Similair applies to server certificates with multi-year validity. Shorter renewal cycle could be more of a PITA, but actually work better for reliability.
Completely agree. Yearly tasks are a pain to schedule, especially when the responsible person leaves the company (whatever reason).
I'm thinking there should be a yearly calendar process for business. The business needs to know these things, and they cannot be tied to the knowledge of one person. How many times has Borkzilla been caught out because it forgot to renew its own certificates ? And it's far from being alone in this. If there was some sort of shared calendar showing yearly actions and who is responsible for them, then I think this kind of issue would be almost resolved.
At a business level, that should be easy to organize. Create a user named Yearly Renewal, share his calendar with all upper management, and have HelpDesk insert the proper data and the names of people responsible. When one person leaves, check Yearly's calendar for all events under that person's name and figure out who to reassign them to. Then make sure everyone is checking Yearly's calendar at least once a month.
That should work, no ?
Actually no, such a shared calendar is, at leas in the Microsoft world, for free. Even if you (ab-)user a shared mailbox to do the job.
Dunno about other solutions, I haven't been working with anything else for past >10 years in the business world. Before that a lot of various mail and calendar systems.
Before I retired, part of my work involved sorting out company management systems. A friend had developed a process mapping program that forced a simple structure (which, whilst initially difficult to work with for some processes, actually did simplify the documentation to an extent that everyone could actually follow it). The point for this thread is that every process had to have an owner and, when anyone changed roles (left, jointed, promoted, etc.) you could quickly print off a list of all the processes they owned. That list became part of the induction process for the new role holder.
The system could also deliver a list of all the tasks anyone was responsible for, but that was far longer and many tasks were the responsibility of a shared post (such as "Project Engineer") and tended to be built into those role inductions.
For companies that use a CMMS [1], it's common to simply use that. I'm in manufacturing, and every site I've worked at has had such a system for maintenance on production equipment, HVAC, utilities, etc. Adding IT is a quick and easy solution. It doesn't have to be yearly, either; usually you can define any time period you want. It's great for reminders of user audits, server PMs, performance checks, backups...
[1] A CMMS (Computerized Maintenance Management System) is a computer system that generates WOs (work orders) for routine scheduled work, allows manual WO generation for nonroutine work, can list work that is due soon or overdue, shows the history of work including tech's comments, etc.
Customer's site: 2-machines for 'normal' operation, 1-machine 'assist', 1- machine 'standby'. Spare: Offered, but not accepted.......
The rules: 2-machines plus 1-standby must always be available.
Attitude: "We've got a standby machine"
Actuality: 1-machine under maintenance.
Result: Non-compliance.
Customer: Panic. "Must be the manufacturer's fault".
In the end we built a 'spare' machine at our own cost to act as standby when one machine was under maintenance. Did the customer buy it? No. We charged a healthy premium for a 'quick turn-round' on repairs. The machine cost was recovered within a few months and after that it was embarrassingly lucrative.
Result to us: ----->
The problem with introducing any system like that is cost. It's harder to persuade people to invest in something if it doesn't produce a result they can measure. After all, while maintenance is good, all it does is enable you to carry on doing what you have been doing. It won't necessarily help you do it better, so it doesn't excite people. It's the same with user audits and backups. After all, if they are done and good, your performance will be what it should be anyway. They may even ask why you can't just buy a wall chart and put reminders on that.
Even with the best intentions of doing maintenance properly, purchases like this often get bumped because the company needs something more urgently
If you have actual physical machines that need maintenance, you can sometimes see if there are any problems, and any missed maintenance will probably have a quantifiable effect. If you own a large factory, and one of the machines goes down, lost production time can be a significant cost, which would also motivate those looking at buying a system to manage their maintenance schedule.
And yes, I have had to deal with users when their computer system failed, and they neglected proper maintenance including tested backups. It wasn't a good experience for them, and, as a result, wasn't a good experience for me.
" It's harder to persuade people to invest in something if it doesn't produce a result they can measure."
If you standardise your hardware then a spare machine can cover multiple clusters. This is easier to sell to manglement and has the advantage of keeping hardware like "lego" rather than being individually customised
No matter what you set up to force the checking, there is always the forgotten thing that happens because of something else - either everything is chugging along so smoothly that people get too comfortable, or too much else is going on & nobody does that one 'silly little check' that can cause the big mishap. Human nature every time - unless there's some systemic way to check - an alert if that fails etc,, then we have the 'thing that does the alert didn't work' issue - been there - regardless how hard you try there is always a potential screw up somewhere that will get you ! Checklists help, people checking other people helps too - every human makes mistakes ! Know exactly how you will recover if/when - ie backup, backup, backup !
" Yearly tasks are a pain to schedule"
Alas they are. I know of the voluntary group where the new secretary or treasurer didn't apply for a grant in time so the group was ineligible for it. That made a hole in their finances though fortunately they'll survive.
So when I friend recently became chair of a similar organisation I advised them to make sure that any time-limited grant applications are a task of the secretary or treasurer, but that everyone on the committee should note those dates in their diaries and check if the applications have been filed. Better to learn from other people's mistakes :(
That time of the week again, 'cheers' to all Commentards.
Yes it ‘should’ work, but meanwhile back in the real world……..
The problem, where a task is everyone’s responsibility, it rapidly becomes nobody’s responsibility, “no I didn’t do it, I was told Jim was doing, no, can’t remember who told me but…”
Your suggestion about when Fred leaves, and any jobs assigned to him, get assigned to someone else, is good, except, who is responsible for doing that? Nine times out of ten it will be the aforementioned Fred!
I’m afraid it’s a bit like turtles, only in this case it’s more like, “I thought someone else is dealing with it”; all the way down!
In our case, the task and job allocation system is our primary go to for 1st, 2nd and 3rd line support. Infrequent tasks are set to trigger with relevant amounts of notice and allocated to a team, not an individual. Said tasks will stay in the list until someone updates it as completed and never show red as overdue, at which point, hopefully it will auto-reset to the next interval, whether that be monthly, yearly or whatever. It seems to be clever enough to allocate always on a working week day, bringing it forward a day or two if required to avoid weekends and holidays. The potential flaw in that plan is that someone forgets to set it up in the first place, but that's managements problem :-)
Yes it ‘should’ work, but meanwhile back in the real world……..
"...that thing sounds complicated so we best not try, it'll not work anyway."
Yup, worked at places like that. Probably best avoided. Over time this attitude distils, until you get a workforce not only averse to risk, but to even risking doing their jobs with any degree of competence, as the warm comfort of gentle monotony is much much easier.
I was going through some personal stuff which meant changing jobs wasn't on the cards. The main frustration about the whole thing is the one upcoming project that sounded difficult (and therefore interesting) was never ever started... as too much risk.... just dangled like some future development carrot.... whilst the existing creaking poor quality and hard to use system just plodded along.
Well, I currently work in a "place like this".
Here is what happens in our IT dept. We have critical systems that no one cares about because they were installed a "number of years ago" on a new and sparkly version of Centos5 ..then one day it stopped working due to a "certificate issue"
The main problem was that the guy who installed it had left the company due to The rolling "1% annual pay rise regardless of performance" despite having to sit through the corporate wank session who told us all about the bumper profits. We were told that there was no money in the kitty due to a need to replace a bunch of laptops for the sales guys... that did not stop sales from getting fat bonuses, but more to the point..I don't work in sales so why tell me that???
Like a lot of employees, it dawned on us that we could literally turn water into wine and feed the forty thousand and it would make no actual difference to our salaries.. so why bother? We all just do the absolute bare bum basic minimum and look for other jobs in support.
Between the massive employee turnover and poor morale..there are a lot of "end-of-year functions" that are ignored.
After all why make more work for yourself if it's never appreciated?
Sorry for the rant!
At this point in time with tech more important than ever, and only ever more complicated, it's a must for any business which loses more than a few £ per hour due to outages.
For all those reasons, we've just gone through the process at our company of deploying calendars in Asana for each department, scheduling monthly/quarterly/yearly tasks that are easily forgotten and must be done. Then pulling all of them up into a company-level one overseen by the Ops Director through some small automated rules.
specially when the responsible person leaves the company
Or even more likely, has moved on to new project/office/department/building ( multiple layers of likely movement iow) and left their old tasks to the new person, who may not even realise, or have been told about, every single detail- because the outgoing person has a squillion small asks that popped up on their diary from time to time Especially if there has been more than one change. There is absolutely no way that a hand-over will contain every detail of the outgoing person's year, or indeed the significance of the ones that are included.
> Yearly tasks are a pain to schedule
especially since the certificates don't need a yearly renewal, you do the renewal 10 days before. So it is a "nearly yearly" task. Several procedures are that way. THOSE are the problematic. Why is a certificate not simply 380 days, or a password expire time not simply 200 days so it fits nicely into an actually yearly (or twice yearly) schedule? This is one of those "why does IT shoot into its own foot" questions...
In a 'battery operated" stationary critical piece of equipment like a battery powered radio time clock, you use an ac adapter that interfaces to the equipment through dummy batteries.
I first used the 1970's version of this in, surprise, the 1970's. A modern version can be bought off Amazon for next to nothing.
https://www.amazon.com/Lenink-Power-Supply-Battery-Replace/dp/B09YTVTZ1V
A word of warning with those battery replacement gadgets, or worse still bodging a wall wart onto a battery powered device. The batteries will supply their rated voltage, which will gradually drop as they discharge. Those battery replacement things or a cheap wall wart will provide a wildly fluctuating voltage, often way more than the nominal rating. If the device you connect it to doesn't have the right circuitry to clamp the incoming voltage to what's expected then you could find it breaks pretty quickly. I'd recommend checking the output on any battery replacement gadget before use, or using a good quality regulated wall wart that has the circuitry to ensure a steady voltage.
I agree entirely.
I actually use one of these for 2 AAA batteries at home that, instead of using a USB output wall wart, uses a more standard wall wart. ( https://www.amazon.com/gp/product/B0873Y5MQ5/?th=1 ) Having said that, I would advise one to do their due diligence and make sure the power supply is quiet, stable and at the correct voltage.
The CAB have pretty much killed off certificates with multi-year validity, but since getting bitten by a certificate expiry I have a weekly task which sends me a report of certificates that are expiring in the next few weeks. Now the only way that certificates cause me support headaches are the users in third world countries who use 10-year-old Android phones whose root certificate lists need updating, and we only get one or two of those a year.
Once you've realised that this thing needs batteries changing yearly or $things will fail, why not replace (or supplement) those batteries with an external power brick? Plugged into the UPS on the basis that if the UPS dies, that clock will be the least of your worries.
This post has been deleted by its author
Virtually all intruder alarm "siren/strobe" units (or "bell boxes" as they used to be known) use a "hold off" voltage from the main control panel to stop them sounding.
If the hold-off voltage disappears for any reason - such the alarm being triggered, the cabe being cut etc - the siren and strobe would start going.
So buy an off the shelf siren/strob unit, power it off a 12V main adaptor (which will keep its internal backup battery going) and have the clock unit battery act as the "hold off" voltage.
Battery dies, the siren/strobe starts going (and no doubt sevel people will need a change of underwear especially if very close to them and you don't tell them what it is for).
It is that or use the same "hold off" principle to have voltage from the clock battery stop a relay contact from closing - if the battery fails, the relay contacts close and connect mains voltage to some bugger's chair.
Or you could be boring and just run the clock off the mains with a battery backup.
"I wont to cut the dc plug off a 9v answering machine psu and solder the wires to the battery terminals in the Network Clock time-getter"
6V surely. Quartet (4) x AA (1.5V).
Even shoving 6V into 4 alkaline cells might be a little explosive.
I would trust Gromit to get it pretty much right as he would probably use a 6V sealed lead acid motorcycle battery and a trickle charger. Wallace I shudder to think...
Given AA cells were and are inexpensive, a shorter maintenance schedule (every 3 months) might have been safer.
Say four roles that get an annual calendar reminder to replace the batteries and ensure the other three roles still exist and were not vacant.
Seems like the systems in the story were synched directly from the radio time signal services. I would have thought even back in Novell Netware's heyday crystal controlled precision time sources would have been de rigeur and I imagine those precision sources would have been optionally synched from the wireless time services.
At which point, if the current needed is low enough, a USB-A plug can be wired to the battery terminals. Connect to a cheap USB power brick.
I did this with some 2xAAA lights on our mantle so I wasn't replacing the batteries weekly. Yes, that's "too high" voltage for them, but most LEDs can take it and most circuits have voltage regulators on them, and the current is really low.
"GIven it was a Novell box and it was running on what sounds like a radio clock, Droitwich, MSF, WWV or whatever, then yeah, it was a *long* time ago."
And these clocks, to do the job as described, surely must've had some way to "talk" to the servers, most likely a wired connection of some sort, serial or ethernet (RJ45 or BNC). That doesn't sound like the sort of kit I'd expect to be running off batteries as standard, especially back then when electronics were more likely to be a bit more power hungry than modern kit.
This will have been long before NTP, so these will have been the Master Clocks for a wired electronically synced clock system.
Popular in schools and factories since 1900 or so.
There will have been some proprietary gateway in the servers, that most likely wasn't ever tested as to the behaviour if the clock stopped because "That can't happen".
I’ve seen this done in very similar circumstances to the original story but with a multi voltage adapter. In that case the voltage switch was on the pulg side of the adapter. Two thick pieces of plastic were cut to length and then glued in place to prevent a change from the required output of 6v. The reason for doing this was one fateful day when some new batteries turned out to be deficient. They failed overnight and the first person in was confronted with a barage of alarms from the things that relied on the clock signal output.
Seems like the systems in the story were synched directly from the radio time signal services. I would have thought even back in Novell Netware's heyday crystal controlled precision time sources would have been de rigeur and I imagine those precision sources would have been optionally synched from the wireless time services.
In my radio station days the (analogue) clocks in the studios had fairly bog standard Quartz movements, but they were synchronised back to a laser-trimmed unit in the central racks, which was itself synchronised to MSF and optionally (I think) DCF. No NTP or GPS (for proles) back in those days. The racks unit had a Lead-acid backup battery kept floating, but the studio clocks just ran from bog standard AAs if I remember correctly. In constant use, a clock slowing due to loss of synch or stopping for a dead battery would have been noticed pretty quickly and reported in no uncertain terms to Engineering the first time a DJ with half a million listeners crashed the IRN bulletin at o'clock!
As for TOA, it seems odd that if the servers stayed running, the external clock stopping would have caused their internal clocks to reset...
M.
We’d the same set up except the clocks were powered from the mains and fed from a unit in the server room. Caused a problem when one studio was used for a photo shoot and they wanted a particular time on the clock. This was possible but the PR bloke didn’t change it back at the end of the shoot. Had to do a studio swap at short notice and the presenter almost read out the time before the 3pm news as being “almost 10am”. Swiftly fixed following the discovery.
the presenter almost read out the time before the 3pm news as being “almost 10am”
I think even our mostly quite dim and engrossed in their own thoughts presenters would have noticed the difference between 3pm and 10am! In our case, changing the time would have involved disconnecting the serial line from the clock (carried on standard audio-grade shielded twisted pair IIRC) and then adjusting the clock in the usual manner.
It's a while back, but it's possible that the AA in the studio movement was only for backup and the power actually came up the serial line from the central unit too (I don't remember the clocks being plugged in to the mains, but my memory is pretty rubbish). Looking back I don't remember changing the AAs very often, but then given that we're talking about four or five clocks, me only being there for six years or so and the AAs in clocks at home often lasting two or three years, even if they were entirely battery powered at most I'd have done fewer than 20 battery swaps in my time there, assuming my boss didn't do any himself.
M.
Almost certainly, they would recognize the difference, but when your brain is in reading mode, it has a very efficient pipeline for reading and speaking which is not always delayed enough for the rest of it to stop you. "It's just coming up to ten no wait a minute how could it be" is not really much better than stating the incorrect time.
The bloke said he was looking at the clock as he was about to open the microphone fader and his gob. Then at the last moment something in his brain said the clock is wrong, don’t say a word. To be fair it was combined with a studio move during a live show which was basically done in the space of one song. Something went wrong in the original studio and he was told to move quickly.
Our on air talent absolutely hated studio moves. Mainly, I think, because of the need (in those days) to pack up CDs and carts and lug them to the room next door. Only in exceptional circumstances could we force a move and I often found myself working on a "live" mixer; channel strip removed while leaving the rest of it running. Even (as I think I've recounted here before) one famous occasion where a jock on his last show had (totally against all rules) taken a half a bottle of gifted cider into the studio and tipped it into one half (script space in the middle) of the brand new desk leaving him with (something like) one guest microphone, one CD player (of three), two (of four) cart machines, IRN and a telephone TBU. He didn't call me - it was the next guy who called the pager at 10pm. They could easily have done a studio handover between shows, but no!
The beauty of that desk was that it was "remote". The only things in the control surface were the Penny & Giles conductive plastic faders, a gain knob, PFL and (where appropriate) start or talkback buttons. All the electronics was in a rack off to the side. Cleaned up reasonably well, bloke left with wet trousers. Studio smelled a bit for a few days.
M.
"Say four roles that get an annual calendar reminder to replace the batteries and ensure the other three roles still exist and were not vacant."
That will just lead to Alice replacing the batteries on Monday, Bob taking them out and putting in a new set on Tuesday, Carol removing the batteries but not finding a new set, so putting the old set on her desk to remind her to go get a new set, and David putting in the batteries that were dead in the first place because Bob put Alice's set somewhere else and Alice forgot to charge the old ones after taking them out. A proper procedure includes some way to verify that it has happened, but that introduces more overhead like finding a place to store that information and making sure everyone checks it.
9 volt batteries: https://www.amazon.ca/AmazonBasics-Everyday-Alkaline-Batteries-8-Pack/dp/B00MH4QM1S/ref=asc_df_B00MH4QM1S&mcid=39eabb23539a35dab6150dbd0fcf0308?tag=bingshopdesk-20&linkCode=df0&hvadid=80470598951659&hvnetw=o&hvqmt=e&hvbmt=be&hvdev=c&hvlocint=&hvlocphy=&hvtargid=pla-4584070143719256&th=1
I was a school governor and there was a yearly site check that was done in late July, which was simple because you've got the school year that imposes its schedule on you and prevents you forgetting.
Similarly, I check all my smoke detectors when the gas appliances are checked annually. The plumbers know when their certificate runs out and pesters reminds me, so again a schedule that doesn't get forgotten.
When I do my tax return I have a bunch of stuff I do at the same time - and while I may put it off for months, there is a hard deadline each year so over the long term the yearly tasks get done.
I check my car tyres every time I get petrol. I check various other things at the annual MOT. External prompting ensures it gets done.
But if it is something that is independent of anything prodding you to do, it will get forgotten about because that's how the human brain works. If the DVLA didn't tell me my car tax was about to expire, or the insurance company tell me the insurance was about to expire, I'd never remember to pay it.
> ...daily tasks, they just get done automatically. Monthly, OK there's a month end process. Yearly tasks are only ever triggered if they are literally scheduled for Jan 1st...
Well, but we all have birthdays and anniversaries. Also car registration and property tax bills. Some may be very "dangerous" if forgotten.
My friend RJ was in-charge of taxes at a major telephone company, and tickled himself for hundreds of state and local tax bills. (Maybe the town/state sends a memo, but if it is lost, it is still your problem.)
Identify or hire a person in your organization who can do that (including pass-on when he leaves).
"Well, but we all have birthdays and anniversaries. Also car registration and property tax bills. Some may be very "dangerous" if forgotten."
For a company, it's better to have the reminder/task done based on a more common event such as a bank holiday. The clocks shift twice each year where I live so that's the day (a Sunday) when I change smoke detector batteries. Anybody else in the household will also know when they get changed regardless of whether they remember by birthday or not. I also write the date on the battery with a felt pen. The batteries I use are made for long life/low drain applications and are mostly white.
Some may be very "dangerous" if forgotten.
Wife's birthdays being the most dangerous.
A good while ago I was on the phone to an insurance sompany and had to give my wife's birthday. The agent told me I had it wrong (they had day and month numbers swapped). I assume he must have been young and unmarried.
"The agent told me I had it wrong"
You also need to remember where you proposed, that date, where you first kissed and where, your wedding anniversary.... etc. This is why doing well on events and dates in history class when at school is good training for later in life. And you thought the class was useless.
> "I was immediately summoned to explain what had happened"
There is a crisis in progress and one person in the building who can address it, so naturally you call that person to your office to tell you what they would be doing about it if they weren't talking with you. Why come does it happen that way around so often? Would it hurt to go to where they are dealing with it and ask for a time estimate?
SOP for dealing with major outages at the last workplace was:
1. Lock the Mission Control doors.
2. Identify the person least likely to be able to help fixing the problem. Promote them to Incident Communications Manager for the duration.
3. Redirect all Mission Control phones to the ICM.
4. Get on with fixing the problem, keeping the ICM informed as to progress.
"That is all it needs."
One critical thing missing, which is an estimate to resolution (even if very finger-in-the-air estimate multiplied by pessimism factor), since based on that other procedures might need to be triggered.
Also, as somebody else noted, asking"anything I can do to help?" is always appreciated even if the answer is almost invariably "just make sure no-one disturbs me while I fix this"
One critical thing missing, which is an estimate to resolution (even if very finger-in-the-air estimate multiplied by pessimism factor)
The Scotty approach. "It'll take six hours Captain. But as we dinnae have six hours, I'll have it ready in two."
"Wait. Mr Scott, have you always multiplied your estimates by a factor of three?"
"O'course, Captain. How else can I maintain my reputation as a miracle-worker?"
I had such a situation. 03:30 in the morning, just made a breakthrough in getting the backup server restoring it's database after a major storage (and a bug in the restoration procedure) failure that had taken the office support environment and a couple of the test and development environments out for several days.
One of the project managers is sitting in the next desk over keeping everyone else away (it was really surprising how many people were still around at that time in the morning) so I could get on with the problem, interrupts me quietly and asks "Coffee?"
Answer, "Tea please" and I returned to proving that the backup server was working enough to start the restore of the rest of the servers in the environment.
I have literally never been so grateful for a project manager sitting on my shoulder.
My SWMBO was always late for anything. As far as she was concerned, leaving the house at the time we were supposed to be at the appointment counted as "being on time", I solved this by telling her a fictitious appointment time that was sufficiently advanced that we would have time to actually be on time. I used to say that she would be late for her own funeral - but unfortunately she wasn't. RIP Rosamund, still love you.
I solved this by telling her a fictitious appointment time
We had a corporate manager in the Netherlands who really resented any travel in work time. He'd try and insist on us leaving the office at normal finishing time, driving to the airport, flying to Schiphol then taking a train right across the country, with the result you wouldn't reach the hotel until after midnight but would still have to be in the office for 08:00. For the return he'd make sure he delayed your departure from the office until the last possible moment for you to catch the last possible return train to get your homebound flight.
Of course the end result was a lot of lying. "Sorry, the later flights were all full!", when they weren't. "The only return flight is the 14:30!", when actually flying out on the 15:30 flight.
"Of course the end result was a lot of lying. "Sorry, the later flights were all full!", when they weren't. "The only return flight is the 14:30!", when actually flying out on the 15:30 flight."
You could have tested the manager by saying you "could" leave later, but the only seats left were business/first class. Get a wet signature on the upgrade so accounting doesn't throw the cost back at you although I'd expect the manager, who's bonus will require containing costs, will suggest an earlier departure.
"You might think you are [attached], but no, SWMBO owns them, you just keep them warm."
Perhaps, but you are still actually attached to your testicles, as some of us might be all too painfully aware.
(That's a Viz strip that I'm not sure that I particularly want to imagine or read, but unfortunately it's too late now…
Buster Gonad's cousin, whose testicles are not quite so unfeasibly large, but whose pocket gobstoppers instead are very much on the outside and float around, generally orbiting his waist like some kind of binary planetary system, completely unattached to his body. Exactly how his man yoghurt gets from inside these juicy plums to the, uhh, pump delivery nozzle is one of the great unexplained mysteries of the universe… Hilarious adventures ensue…)
(And, as for warm, if the silly things didn't need to be 2°C cooler, and were sensibly tucked away like their counterparts, life would be so much better for those of us of the male persuasion (and probably a whole lot less painful). A real, umm, design cock-up by whoever bodged that demo model - you were supposed to fix it in production!)
[Different AC, staying AC so that I don't get locked up in a padded cell]
if the silly things didn't need to be 2°C cooler, and were sensibly tucked away like their counterparts, life would be so much better for those of us of the male persuasion.
Each to their own. There's some entertainment value in having them accessible.
The equation is something like cooler gonads -> more efficient meiosis -> higher quality tadpoles -> more kids -> evolutionary selection pressure. And then there's the whole keeping them warm gives you cancer thing. And google "bell-clapper deformity" if you want to skip your next meal.
I once rather stunted my career prospects within an org because of this.
There were two large (but separate) outages, affecting two large customer networks. It was during the later hours of our night shift, but around peak time for them. There were only 2 of us on (due to cost-cutting leading to understaffing).
The two of us took a network each and started investigating.
Except... the phone kept ringing, with our CEO demanding updates/ETA etc. This wasn't driven by customer demands - they were well aware we were working on it and that we'd give updates as/when they became available, it was just the CEO and his micro-managing ways.
Around about the 5th time that the phone interrupted my train of thought, I answered only to hear our CEOs voice, again asking for updates and ETAs that couldn't yet be given. Rather than explaining again that we weren't yet in a position to provide these, I said "at least 8 hours".
He was *beyond* dismayed and asked what could possibly have gone so badly wrong that it needed that kind of recovery time. So, I explained that the team were unable to effectively investigate and recover services because they kept being interrupted by pointless calls asking for status updates.
He didn't phone back again, but did phone my boss later in the day (who backed me up).
During the process for a later internal job-move (and promotion) though, the hiring manager needed approval from the CEO, who denied it.
Should have got your boss in earlier...
Had a self proclaimed paper pusher manager at a previous role, and they were brilliant.
They had a great memory for "the team saw something maybe similar seven years ago, here's the first and last couple of emails - does it sound relevant" and was excellent at being a management shield when needed.
I only had one "event" where I ended up on three calls first thing in the morning - the third was meant to be a progress report from the first - but the time between them had been taken up with the other call from a different set of management.
Now both sets of management did need to be updated, and updated properly (and that really meant by me), and the incident wasn't getting any worse (as we were now running correctly and had backups of all the raw incoming data from the event) - it eventually took two full weeks to repair and reprocess the data (which had been sent in the wrong timezone by a handful of systems, there was a nice overlap of timestamps at one end of the data which took more effort to resolve than I care to remember). But the first thing I said on the third call was "no progress, and there never will be if I don't get off these calls now."
Just don't answer the phone. You can explain this later by saying you were too busy fixing the problem. Once the problem is fixed it's difficult to argue against that. A variation would be that you were too short-handed to spare anyone to answer the phone, something that will need to be addressed in the post-incident review. Alternatively just unplug the phone and explain that the problem had taken the phones out as well.
Doubly so when they insist on PowerPoint presentations, pretty graphs and all the other nonsense required to prettify the data enough that their brains can parse it.
Or perhaps not even then, and you have to spend extra time explaining the basics, and then when another manager joins half way through then doing it all again from the beginning.
Leading to the oft repeated mantra of "Do you want me to do this update, or just waste my time actually fixing the issue?"
Ah! But MS Office's genAi CoPilots stand ready to solve that problem for one and all, in one short synthetic heartbeat. Complete with face-saving hallucination if need be ... that tech is a godsend for all On-Call Who-Me's (unless that's the very tech that titsupped itself of course!)!
The company I work for is currently all-in on MS metrics. *everything* is measured (for some definition of "measured") and presented ina dashboard. Decisions are made based on the KPIs. AI can only make this more "interesting".
I particularly enjoy the "liklihood of resignation" metric that my manager is required to estimate.
Just need text to voice for the output.
I'm hoping that when our POTS is replaced by a non-resilient VOIP service one of the by-products will be some form of VOIP>text and text > VOIP drivers. That will then be coupled up to something like Eliza as an extension to which scammers, would-be smart meter installers etc can be directed. (I wouldn't trust ChatGPT or the like - they might hallucinate an agreement to something. I just want an arrangement in which the callers can wander around getting more and more frustrated.)
I was thinking of retaining some of the old Eliza dialogue around asking if the user was unhappy talking to a computer once they used that word. Given that the source would spoken there are opportunities for claiming to have problems with the user's accent or asking them to speak more clearly if an input line doesn't include any trigger words.
Meet Lenny:
https://www.youtube.com/watch?v=XSoOrlh5i1k
I have this installed on our VOIP phone system. Cold callers and scammers get forwarded to "him".
At my previous employer, we actually had someone show up in person at our office saying that they were there for a meeting with Lenny.
Having finished some work for a bank, I was due to fly home Sunday lunch time. Sunday 0600 the phone rang. It was the boss saying they had a problem. They had tried to fix it.but the backups didn't work. Please could I help. He said anything I need - let him know. I quietly worked on the phone with the customer's guys, getting it sorted. Every 10-15 mins I messaged my boss giving status. I said we needed a networking guy... and 10 minutes later their top networking person was on the phone.
We got it fixed by lunch time - just in time for me to get my flight.
Afterward my boss told me the extent of the outage. Half of the bank was down - it was on the national news. He said he fielded the calls from every one- including the head of the bank's disaster committee who wanted to talk to the people fixing it. My boss said to him, if they tell you the problem and the solution, you will not understand it - trust us.. He didn't tell us the scale of the problem (although we guessed) because he didn't want us to rush and mess it up.
He was a good boss - he handled all the flak, and he trusted us to get on with it.
I used to work in the broadcast world, and came across a couple of such stories where the big boss wanted to know what was going on ....
1. In person. "What are you doing to get this fixed ?" Reply: "Talking to you"
2. On the phone, after rather tetchy reply from the fixer. "Do you know who I am ?" Reply: "Yes - do you know who I am ?" "No" Reply: "Good !"
1. In person. "What are you doing to get this fixed ?" Reply: "Talking to you"
The better answer is: "What are you doing to get this fixed ?" Reply: "Currently nothing, because I'm having to talk to you instead"
It semi-subtly gets the point across, without the aid of a clue-by-four to the cranium.
One time someone asked me to call round to get rid of a virus on his home computer. He didn't have broadband (this was quite a long time ago) so I had a CD full of security patches and virus checkers. It would have been pretty straightforward if he wasn't looking over my shoulder all the time and sounding like Stan Laurel just after he'd pushed Oliver Hardy out of a window. In the end I sent him away by asking him to get me a cup of coffee and not to rush back. By the time he did return I'd removed the virus and was installing the latest service pack. I also had words with him about the importance of backups that you can use on another computer.
It's amazing the number of times when, as the only person on site capable of addressing the situation when something important has gone TITSUP, some high-up DEMANDS that I attend a two-hour conference call to discuss why the fault has not been cleared.
Naturally this both raises the blood pressure and forces a return to Step One of the fault-finding and repair process.
It was doubly frustrating when I couldn't turn the mobile phone off because I needed its meagre light to see inside a server, the building's lights being on a timer and it being "after hours".
"Would it hurt to go to where they are dealing with it and ask for a time estimate?"
Yes. I can't give you a time estimate until I have at least some idea of what went wrong. No, not the symptoms, the causes. It takes a while to figure out the causes, but sometimes after identifying the causes, the fix is thirty seconds and a reboot. The nontechnical always manage to assume that their statement of the problem is all the technician needs to know how hard fixing that problem is.
Occasionally in my job, I get asked to join a customer bridge. I always, ALWAYS refuse to do so. The last time I did, it was nothing but a bunch of C-suiters asking inane questions about the problem, a bunch of "I'm better than you" posturing, legal threats, and wailing and gnashing of teeth. You should have heard them when I told them their circuit wasn't being worked on because I was on their bridge and unable to work on their outage.
These days there are people whose whole job is to join bridges and listen to customer babble, which is OK by me. They make more than I do, but better them than me!
Yes, I was working for a big multi-national car manufacturer and had been trying to solve a production outage for 20 mins, only to spend 15 mins of that explaining to 3 different users who had my number, why I was not progressing the resolution of said issue. In the end the 4th call came through and I picked up the phone and said "if you are not my boss or the servicedesk, you are not important enough to talk to me right now". Once I heard the voice and determined it was not either of those two, I hung up the phone and left it off the hook.
My boss later called me aside to talk about my "ubiqituous telephone manner" to a very senior user but I stood my ground and said that when I am resolving an issue, his job is to get everyone off my back so I can get on with the work.
I remember those toys. I can't remember why the local authority I was working at couldn't just allow port ntp through the firewall but there was a "reason". It wasn't batteries that caused the outage though. Window cleaners had dislodged the network cable.
icon because it is nearly that time
I used to manage a network with a couple of those (Galleon Ntp servers) one was fine but the unit at one office had problems.
The first antenna used the time signal from rugby which worked until they moved it to Birmingham (I think) at which point a large row of trees blocked the antenna completely.
After this we swapped to a gps unit mounted on the roof which worked until the enclosure sprung a leak and fried the electronics.
The final time I heard about it was about 2 years after I left the company when some builders cut through the cable as it snakes under the floor and they called me to ask where the thing was routed to….
When nobody was trying to sabotage it they were brilliant at their job and ran for about 15 years - they may still be working away…
This bring back memories, but of time flipping the other way.
Back in the Day was part of a team administering a Novell Netware 4.x network - pre NTP days, so we had one of these https://web.archive.org/web/20010411015625/http://www.galsys.co.uk/galdata1.htm keeping time set on the Master Netware replica from the radio signal in Rugby. One day it picked up some kind of erroneous transmission and set the network time forward to the year 2032 or something similar. After resetting it and correcting the network time we had the dreaded "Synthetic network time" on the eDirectory replicas as we were stuck with future dated timestamps until we ran the command to declare a new "epoch".
It was such a joy once NTP arrived with no single server source issues ready to blow things up.
M
The MSF feed from Rugby and then Anthorn is broadcast on the longwave band at a low frequency of 60KHz.
This frequency will travel through most things, and will also get bounced around the atmospheric layers quite easily, so I don't think that a row of trees, or even a whole forest would have stopped it. But when the change happened, you probably should have re-aligned the antenna (unless you were to the south-east of Rugby), it being quite directional at that frequency.
The other thing you should have done was change the offset to cope with the change in distance to the transmitter. The standard code includes a nanosecond offset to cope with the distance from the transmitter, although for most purposes, this is irrelevant, given the speed of light and the small(ish) size of the UK.
The clock I used to manage many years ago was the classic Radiocode Clock RCC8000 which was a 19" rack mounted unit that fed an xntp server via a serial link (another reason why nanosecond offsets were largely irrelevant). It was eventually replaced by a Meinberg time appliance that drove NTP directly, although it used the German DCF time signal transmitted from Mainflingen on 77.5 KHz.
Is a PITA!
I was seriously considering an atomic clock setup for one of our servers as, for some reason we couldn't fathom, it was consistently 3 minutes slow despite regular NTP calls and irrespective of the time server used. Had I been sure it would have resolved the issue I'd probably have gone ahead but in the end our support company managed to fix it with an obscure patch.
Did 3 minutes really matter? Whilst not as critical as the time for financial trading yes, it did! The factory time clock updated from the server weekly (and when the clocks changed), that three minutes was the difference between some of the operators catching their bus home or waiting an hour for the next one.
Not work but we had an issue with linked electronics (it was a proprietary ring bus system) on a yacht where the time on the chart plotter (which didn't have it's own RTC) had to be manually reset at power up when it was supposed to take the time from the GPS. Having finally managed to track down a USB interface for the network we went into packet sniffing mode and it turned out the scenario was:
Chart Plotter sends Time Request Good so far
GPS responds to Request Still good
Chart Plotter responds to Request Eh? Why is it responding to its own time request?
Chart Plotter receives time from GPS and updates Correct time
Chart Plotter receives time from itself and updates Overwriting the correct time with the default start up time
We were able to update the chart plotter config to stop it responding to time requests but why the heck it was ever enabled to do so was beyond us.
I like to operate an NTP server in any significantly-sized network.
It doesn't need to be microsecond-accurate, it doesn't need an expensive GPS card in it, it just needs to be the "one true reference" for all time on your local network, in case Internet goes down.
You can do that with just any device running an NTP server and letting it drift from real-time - it'll just drift back into sync when it's reconnected.
I have used network router for this, I have used VMs for this (VMs don't hold time perfectly well on their own but in concert with an outside NTP upstream server they are fine), I have even exposed an internal NTP server as an NTP pool server for other people to use (because NTP pool has a lovely monitoring feature where it will alert you if your server is uncontactable or getting out of sync with everyone else's in the pool).
You can get MSF modules to pull in an atomic clock radio signal and feed into the NTP server.
You can get GPS modules to do the same.
And you can sync with as many NTP servers as you like, it really doesn't matter.
Then have all your clients refer to that AND other NTP servers (in case it goes down!). Windows allows as many as you like, but doesn't present the option in the GUI.
I normally have all clients using 0.uk.pool.ntp.org and 1.uk.pool.ntp.org (guaranteed to be two different UK NTP Pool servers), time.windows.com AND whatever internal IP (hence not DNS-reliant) I'm running an NTP server on, and that NTP server syncs to a bunch of well-known NTP servers on the Internet.
Where I work, the parts of diagnostics on the vlan (10*) have no access to the hospital LAN or NTP or elsewhere. Running 5 minutes or more adrift (flagged x lots to IT) meant I managed to time travel such that I issued a report on an image that had yet to be taken. According to the time stamps.
Love the NHS.
And our CT scanners will only do NTP with Japan.
I run my own NTP across the few computers at home. It just makes sense.
Time is important in a corporate setting. Log data is an important part of defending a company against malicious staff; it’s what can form the basis of evidence used to formulate a winning prosecution.
Get it even slightly wrong, and the log data is inadmissible. Can’t prove that the timestamps are bang on? Hard luck. By “prove”, you need log data showing correction / updates of time from a trusted source - the Rugby radio clock MSF is a good one - taking place on a routine basis. Having two independent sources is a good idea too.
Plug the damn thing in with a power supply, or failing that some AA eliminator/emulator batteries ( I can't think what they're called, but they're basically an empty battery shell in the same size, with a connector in the middle to attach positive/negative leads of a real PSU).
And then put the power supply ON A UPS.
The cleaner will just unplug it from the UPS for their vacuum cleaner.
There's a lot to be said for using a fused spur if you can, but it's a bugger finding low voltage power supplies that aren't meant to plug into a standard mains socket. (If you know of any that can be wired into a mains spur, please let me know in a reply.)
Why have a battery powered network time clock when everything that needs that time is dependent on electricity to run?
At worst the batteries could be for backup but you'd use rechargeables
or
just plug the time clock into the UPS and ensure it boots before everything else so its available to do its job the moment a client asks.
lastly, every system i've ever used that uses a network time protocol like ntp has a local battery backup for its time plus employs a mechanism to slowly drift its time to a new time from ntp if its wildly different. i doubt all those systems would be on Jan 1st 1970 by 6am, even if the time source broke 2 days before (yes i know but is an example of how it works).
I would think even someone like me who has no sort of EE type degree could remedy that. Just buy a "wall wart" that outputs the correct voltage, strip its wires and solder it against the battery connectors in the radio.
D cells for instance are typically 1.5v, if the four are connected in series (i.e. they alternate direction) when installed that means you need a wall wart that outputs 6v. You can use anything you find regardless of the wattage since if the batteries are lasting a year their nominal draw must be tiny.
People buy what's available for the given budget.
that is very true, but given a time server is meant to be serving requests 24/7 over a network etc, i find it hard to believe a battery operated network time clock can operate for more than a week on 4x aa batteries before needing replacements.
Time clocks are polled frequently and respond often.
on motherboards.
Just enough voltage to keep the BIOS memory alive
Funnily enough we have the same system on the robots.... little batteries designed to keep the memory alive... should be swapped out every 2 years, or even maybe during the 3 months the voltage has dropped far enough to trigger the 'replace batteries' alarm but not far enough for the robot to lose memory.
We have a procedue for servicing those now after someone had to spend 3 days typing all the robot parameters in by hand, then checking the motion, before loading in the control programs.
As I tell the PFY, you can learn the easy way by listening to other people's experience, or you can learn the hard way.... we picked the hard way with those batteries....
"As I tell the PFY, you can learn the easy way by listening to other people's experience, or you can learn the hard way.... we picked the hard way with those batteries...."
I hope that, if you're building these robots, you learned that the price of flash is not that high and certainly beats any device where you need a battery to keep it from bricking itself. And if you don't build them, I hope you still learned that and will choose robots accordingly next time you have to buy some, because maybe that will convince manufacturers not to skimp on what's probably 0.002% of the manufacturing price. As I commented to a friend who was choosing how much room for expansion to put in a hardware design (that was intended to receive software updates going forward), it's much more expensive to try to convince customers to buy version 2 of the hardware when version 1 was clearly underpowered from the start than to make version 1 good enough that, by the time there is a version 2, the customers were happy with what they got.
Which is why you include a little more flash so the customer's config can be retained when the battery breaks, and if this thing is networked, you enable some ability to sync the time with NTP. I'm not saying that it has to run without a battery entirely, but that having the battery die should not cause a significant fault. There was a time when flash was too expensive to do that. It is no longer. If you need a particularly large amount of flash such that including it would be expensive, consider having a user-supplied storage method instead. Lots of OEMs have managed to provision their hardware with those chips.
As I said in my previous comment, failing to do this is what makes customers think that this manufacturer is being too cheap to be a reliable supplier. For example, a supplier of equipment who told a friend of mine that the equipment they bought would not get the feature update they were releasing because it only had 8 MB of RAM and that really wasn't enough to run this software. While that was probably true, my friend was right to ask why the equipment they designed in 2017 only had 8 MB of RAM, as this made it look like they had planned to cut off updates quickly.
> designed in 2017 only had 8 MB of RAM
That is so true. Comparison: The smallest Ryzen 3 1200 from 2017, has 8 MB 3rd level cache (only embedd versions have less cache, and don't look at Intel i3 CPUs of that time). The very first Raspberry Pie from 2012 had 256 MB RAM.
In fairness, this wasn't a full computer, but an embedded, low-power device. It didn't need a Raspberry Pi, and having one would have caused some problems with power management and OS overhead. We didn't begrudge them having that amount of RAM as much as having that amount of RAM and then writing software that needed, maybe, 12 MB to run. It is a problem that recurs at other levels as well. Manufacturers need to ask themselves whether some spec will cause a problem for their customers a few years from now because otherwise, the customers will probably try to find someone else to provide the equipment next time.
*It does seem sometimes that as a rule of thumb, any component that a manufacturer thinks won't be noticed will be designed* sub-par.
Or redesigned. I had a film camera camera when I was a kid. Dad took it on holiday at if broke so he wanted to replace it. The original one had vanished from the shops, only version we could find was a rebadged one in Dixons. Absolutely identical to look at. Same price or thereabouts. Similar name. Apparently only the badge was different. 'Cept it never worked properly. The film would slip and frames overlap ( I didn't tell dad, he'd have been so upset).
When I took it apart it was obvious why. The original toothed wheel had been shiny steel. This one had lots of nylon parts instead.
I remember those "non-replaceable" ones well. At a job from decades ago one had died. For both budget and user sentiment for the computer which had been upgraded to Windows 95, I could not properly fix it. I implemented a new procedure. Do not turn off the "hard drive" (box under the "computer" (box with the screen on the front)). On power failures or other cause of powering off the "hard drive", I followed my part of the procedure. I set the time according to Timex (a handy-dandy device worn on the wrist) and re-entered BIOS parameters such as hard drive geometry according to notes I had previously made on an Ampad and a No. 2 stylus - actually from a backup made on a Xerox. The last step on those note was a reminder to fix the time on the coffee maker and the security camera VCR.
This particular computer was also the one which caused me to realize that Windows 95 had a bug related to long uptimes. Windows update not yet being invented, let alone evolving to a reboot second Wednesday of the month. Windows 95 was actually a lot more stable in retrospect if you kept it off the internet.
I remember one time at an old job I tried to cover for someone, but they wouldn't accept it.
Part of my job involved creating new part numbers in SAP, and some months prior one of the engineers had requested a specific part number and description. SOP for my department was to stash a copy of the email in an ECM header long text, and then use that ECM to create the new part number. Fast forward a few months, or maybe even years, the person is upset about something or other with the part number. I immediately look up the ECM and find the email where the same engineer had requested it be exactly that way, but since I generally liked this person as they would typically take some time trying to make sure descriptions fit within business rules, unlike the rest of the engineers, I tried to give them some hints in a reply that they had requested it that way and if they wanted to make some changes it was still possible because the part number was still sitting at "just created" status. Unfortunately, they wouldn't let it go, and so after about 3-4 back and forth exchanges with them, I finally just dropped a copy of their original email in a reply back to them. To their credit, upon seeing this, they immediately backed down and apologized. So, I didn't press the issue any further, and let them off as easily as I could.
As a general rule, if I like someone I'll try to come up with some way to let them save face, even if we both know it's bullshit. Similar to the "loose cable" or similar excuse of our hero. And in the case of this week's hero, seems like the least they could do since the helpdesk was having to deal with all the irate traders.
Sometimes they get a bit confused.
I have this nice wall clock from Radio Shack which tries to sync to WWVB (60 kHz signal). Unfortunately it doesn't do this well for some reason, and just free-runs if it doesn't. This is fine for the most part, but a few weeks ago we went from PDT (daylight time/summer time) to PST (standard time/winter time). Now if the clock had been able to receive the time signal, it would have corrected things nicely, but alas, it appears that our government (which operates the time signal stations) probably has cut the budget so they reduced the power. Thus the signal isn't what it used to be, and the clock hasn't "fall back" an hour. Oh, well. I just replaced the batteries (nice reminder!) and set the time manually. Life goes on.
I worked at a substation that had a particularly peculiar NTP server. For some reason, known only to the clowns the "designed" the SCADA they made NTP report local time instead of UTC. Broke every standard, but somehow the numpties managed it in an RTU.
All was "fine" until New Years Day (and yay, working a public holiday for no compensation, because of crappy industrial law in the country). Turns out the fudge code to apply UTC offset from localtime had a hard coded year.
Standards - they are there for a reason!
In far off days before a good number of you were born, payroll for a large organisation (and this was a very large payroll, one of the biggest in SE England) was done by running a batch program which took some hours and generated a computer tape (yes, one of those big reels beloved of old school SF films). The tape was then couriered off to the bank, and the bank would process the tape which would make all the payments to the right accounts. Usually.
This was not to be a good day. There was a problem with the output, which was thought resolved by editing the tape. Tape went off to the bank. There was a further problem and more editing, but eventually the tape was run. And it made all the payments. Into a single building society account. This was spotted, and the building society stopped the money actually appearing in the lucky person's account. However they refused to actually return the money until all the i's and t's had been dotted and crossed, which I suppose is not unreasonable with the sums involved.
Trouble was everyone needed to be paid on time, so my late employer had to borrow the money for the entire payroll until they got the original money back. The interest payments on such a large sum were appreciable, and it was reckoned to have been the most expensive ICT cockup in the organisation's history. There was a big post mortem, but it allegedly failed to find out where the fault lay. I always believed that the technical staff on both sides very deliberately failed to find out where to pin the blame in case it was their side!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
