Curious tale of broken VPNs, the Year 2038, and certs that expired 100 years ago Back in late 2010, "Zimmie" was working in IT support for a vendor that made VPN devices and an associated operating system. He got a call on a Monday from a customer – a large specialty retailer in the US – about its VPN hardware that had stopped working over the weekend. After looking into the report, the problem appeared to …
I've only had a quick look for long-life certificates, but one of GlobalSign's root certificates expires on...
Tue, 19 Jan 2038 03:14:07 GMT
Spooky!
a simple problem can actually be bigger than you think
Minor by comparison but we had an issue with NTP and the servers consistently setting the time to about 2 1/2 minutes slow. It was difficult to track down and not really taken seriously, after all what's a couple of minutes here and there? Except the same servers are used to reset the time on the factory time clocks; 2 minutes late clocking out is, for several employees, the difference between catching one bus or having to wait 30 minutes for the next one.
We never really did pin down the root cause, patching VMware, BIOS updates and MS patches eventually cured it so the shop floor were happy, as was their supervisor who was fed up signing off early finishes!
NTP implementations are frequently bad. I ran into a nasty one in ~1998 when the highest bit of utime imcremented - all the routers and dialin terminal servers I had (Allied Telesyn) had a NTP implementation which used signed integers and they promptly went into crash loops
At that time Allied-Telesyn was one of the largest vendors selling into China and when I called it in they were in full blown panic mode as the Chinese academic network (amongst others) was out of action
They said they'd call me back, but I contacted them first when I realised that switching off the NTP module stopped the crash loops - something that enabled them to solve the issue and save a couple hundred million in sales they were watching evaporate
The story has given me to some bad flashbacks to On Call issues.
One case there were issues with NTP on a air gapped network without a proper NTP source. Took hours to persuade the get the pseudo NTP to trust the Sun box used as the NTP source. (Not our network but one we had to use for something because of customers contractual requirements).
More recently, the NTPs of the DC's got out of whack and ended up pointing at each other.
As a result messaging and integration of the hospital systems so no results popping through in the early hours.
At least for that one I didn't need to go into site and once the DCs were sorted I could correct the rest of the Servers with a script and PDQ.
If you're in a Windows domain, you should use Group Policy to ensure the current PDCE is configured with a reliable upstream NTP source, and a policy for the rest of the domain members to use the default domain hierarchy for Windows Time. That is, the non-PDCE domain controllers will sync with the PDCE and announce themselves as timesources, and the remaining domain clients will sync time with domain controller during auth.
Raspberry Pi, which doesn't have an onboard battery clock, failed to set the time on boot and didn't trusti the repository to download a better client because the certificate start date was apparently in the future.
Having - cough - clocked the problem it was easy enough to set the time by hand, of course.
If you haven't powered it up in that long, you are likely to see the problem as it will think it is still 2012.
If it is rebooted regularly Raspbian will fake a hardware clock by writing the current time to a file at 17 minutes past each hour, and use the value in that file when it next boots. That's why syslog always shows it booting at XX:17 and incrementing until NTP kicks in with the correct time.
This post has been deleted by its author
Many (many) years ago I was installing broadcast playout computers at a radio station. Think of these computers as the database of music, but also handled automation like playing the news on the hour (using a line in feed from a satellite audio output).
The setup involved a server with a serial GPS clock attached running an NTP server, and the clients maintaining their time to this server.
I had built all the computers, and used SuperMicro boards. They were a dying breed of motherboards without graphics, sound et al on board. Nowadays all that stuff is of a decent quality and easily disabled but this was a period of time in the early noughties that some of that onboard stuff could cause all kinds of conflicts.
Anyway, the broadcast software kept throwing all kinds of errors in the last moments of an hour - starting to play a song 3 seconds before the news for example. This station was in Cyprus and going back and forth to fix it wasn't really an option. So, diagnosis continued back at the office/factory.
Long story short, replacing the motherboards with an Intel model (again, when they used to badge their own motherboards) solved the problem. It turned out, it seems, that those SuperMicro motherboard clocks were unreliable and drifting to an extent that affected the software time calculations.
I once wrote an NTP server based on borrowing the PTP time counters in the network interface MAC of an STM32F407 microcontroller - discovered an amazing 63 bit high resolution timer hiding in there that could be clock rate tweaked up and down in steps of about 10^-9 seconds per second. I lied to it about the LAN interface using PTP and it fired up this timer.
It hooked into a GPS module that was specifically not a precision timing module - there I found a proprietary timing message that lied about the time shift between GPS time and the 1 pulse per second - it lied and produced a sawtooth value designed to confuse people who bought the $10 GPS rather than the $100 timing GPS.. so I ignored it.
I never trusted it 100% but it used to be able to tell me the CPU clock frequency was something like 167.999995 MHZ instead of 168MHz ..
I dont think it suffered that negative time glitch .. but who knows..
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
