back to article Curious tale of broken VPNs, the Year 2038, and certs that expired 100 years ago

Back in late 2010, "Zimmie" was working in IT support for a vendor that made VPN devices and an associated operating system. He got a call on a Monday from a customer – a large specialty retailer in the US – about its VPN hardware that had stopped working over the weekend. After looking into the report, the problem appeared to …

  1. Victor Ludorum

    That might explain why...

    I've only had a quick look for long-life certificates, but one of GlobalSign's root certificates expires on...

    Tue, 19 Jan 2038 03:14:07 GMT

    Spooky!

  2. John Robson Silver badge

    Diagnostics akin to the 500 mile email...

    Nice work - but that's a seriously broken NTP server and client.

  3. GlenP Silver badge

    Time Consequences

    a simple problem can actually be bigger than you think

    Minor by comparison but we had an issue with NTP and the servers consistently setting the time to about 2 1/2 minutes slow. It was difficult to track down and not really taken seriously, after all what's a couple of minutes here and there? Except the same servers are used to reset the time on the factory time clocks; 2 minutes late clocking out is, for several employees, the difference between catching one bus or having to wait 30 minutes for the next one.

    We never really did pin down the root cause, patching VMware, BIOS updates and MS patches eventually cured it so the shop floor were happy, as was their supervisor who was fed up signing off early finishes!

    1. Doctor Syntax Silver badge

      Re: Time Consequences

      30 minutes? Luxury!

      1. Anomalous Cowturd
        Unhappy

        Re: Time Consequences

        What's a bus?

  4. FrogsAndChips Silver badge
    Facepalm

    Subtitle

    I've read it as the classical "It's always DNS" and read the whole article wondering when and how the DNS issue would arise...

    1. J. Cook Silver badge
      Pint

      Re: Subtitle

      Came here to post the same thing, and was beaten to it. :D

      Oh look, the NTP server says it's Pub-o-clock in my area. ::snerk::

    2. Doctor Syntax Silver badge

      Re: Subtitle

      "It's always DNS"

      Except when it's the UPS.

      1. KarMann Silver badge

        Re: Subtitle

        Generalised rule: It's always one TLA or another.

  5. HuBo
    Joke

    Two's complement

    I sure hope the El Reg up/down vote counter works the same way: "treated [...] as an unsigned integer [...] the small negative number [turns] into a HUGE positive number"!

    1. HuBo
      Pint

      Re: Two's complement

      Wow, that really settles it nicely! Get more than 2³¹ - 1 upvotes (2,147,483,647), and bang(!), the extra ones get reported as downvotes (knew it all along!)!

  6. Alan Brown Silver badge

    NTP implementations are frequently bad. I ran into a nasty one in ~1998 when the highest bit of utime imcremented - all the routers and dialin terminal servers I had (Allied Telesyn) had a NTP implementation which used signed integers and they promptly went into crash loops

    At that time Allied-Telesyn was one of the largest vendors selling into China and when I called it in they were in full blown panic mode as the Chinese academic network (amongst others) was out of action

    They said they'd call me back, but I contacted them first when I realised that switching off the NTP module stopped the crash loops - something that enabled them to solve the issue and save a couple hundred million in sales they were watching evaporate

  7. tip pc Silver badge

    Was it Cisco who had an and issue a few years back?

    I think it was Cisco or one if it’s Borg who had a certificate issue on their sdn product a few years ago.

    Was a painful fix iirc, not the same issue as in this take but close enough

  8. Paul Hovnanian Silver badge

    Obligatory

    xkcd

    1. Anonymous Coward
      Anonymous Coward

      Re: Obligatory

      «xkcd»

      The vision of 32k sheep jumping backwards over a stile in one tick would probably reform even the most recalcitrant sheep shagger.

      1. Jamie Jones Silver badge
        Happy

        Re: Obligatory

        64K sheep, actually, but I'm Welsh, so even that's not a problem!

    2. Anonymous Coward
      Anonymous Coward

      Re: Obligatory

      Or this one

      https://xkcd.com/2867/

  9. Anne Hunny Mouse

    Bad NTP setup

    The story has given me to some bad flashbacks to On Call issues.

    One case there were issues with NTP on a air gapped network without a proper NTP source. Took hours to persuade the get the pseudo NTP to trust the Sun box used as the NTP source. (Not our network but one we had to use for something because of customers contractual requirements).

    More recently, the NTPs of the DC's got out of whack and ended up pointing at each other.

    As a result messaging and integration of the hospital systems so no results popping through in the early hours.

    At least for that one I didn't need to go into site and once the DCs were sorted I could correct the rest of the Servers with a script and PDQ.

    1. Trixr

      Re: Bad NTP setup

      If you're in a Windows domain, you should use Group Policy to ensure the current PDCE is configured with a reliable upstream NTP source, and a policy for the rest of the domain members to use the default domain hierarchy for Windows Time. That is, the non-PDCE domain controllers will sync with the PDCE and announce themselves as timesources, and the remaining domain clients will sync time with domain controller during auth.

  10. Doctor Syntax Silver badge

    Raspberry Pi, which doesn't have an onboard battery clock, failed to set the time on boot and didn't trusti the repository to download a better client because the certificate start date was apparently in the future.

    Having - cough - clocked the problem it was easy enough to set the time by hand, of course.

    1. Tim99 Silver badge

      Which OS? I missed some versions, but mine have been OK back to the RP 1 B (2012) which I still have - I must see if it will still boot...

      1. druck Silver badge

        If you haven't powered it up in that long, you are likely to see the problem as it will think it is still 2012.

        If it is rebooted regularly Raspbian will fake a hardware clock by writing the current time to a file at 17 minutes past each hour, and use the value in that file when it next boots. That's why syslog always shows it booting at XX:17 and incrementing until NTP kicks in with the correct time.

        1. NotJustAStorageDude

          Add rtc to pi

          I use a couple of pi 4s for pihole, Plex, print servers for things windoze no longer likes... usual Linux stuff. I added a seeed rtc module to use one as a local ntp server. (Pi5 has rtc built in?)

  11. This post has been deleted by its author

  12. Anonymous Coward
    Anonymous Coward

    Avoided Y2K - looking forward to avoiding 2038!

    I just started a new job in I think November 1999 so wasn’t put on any Y2K projects - and will be long retired by 2038!

  13. mikecoppicegreen

    What' time is it, Eccles?

    for those who like a bit of ancient cultural reference -

    https://www.hexmaster.com/goonscripts/what_time_is_it.html

    1. RockBurner

      Re: What' time is it, Eccles?

      Ah, great stuff....

      For what it's worth, there's a lot of Goon Shows now on Spotify.

  14. GWP

    When its not DNS, its NTP

    Nice troubleshooting!

    One of these days I'll tell you about the time a global booking system nearly got taken down by NTP. And how I saved the day by going back to bed :)

  15. Roland6 Silver badge

    So I presume the solution, was to replace the NTP server.

    Also to implement a non-certificated based remote access to the VPN devices, so they could be remotely reset…

  16. Anonymous Coward
    Anonymous Coward

    NTP vs wonky motherboard clock

    Many (many) years ago I was installing broadcast playout computers at a radio station. Think of these computers as the database of music, but also handled automation like playing the news on the hour (using a line in feed from a satellite audio output).

    The setup involved a server with a serial GPS clock attached running an NTP server, and the clients maintaining their time to this server.

    I had built all the computers, and used SuperMicro boards. They were a dying breed of motherboards without graphics, sound et al on board. Nowadays all that stuff is of a decent quality and easily disabled but this was a period of time in the early noughties that some of that onboard stuff could cause all kinds of conflicts.

    Anyway, the broadcast software kept throwing all kinds of errors in the last moments of an hour - starting to play a song 3 seconds before the news for example. This station was in Cyprus and going back and forth to fix it wasn't really an option. So, diagnosis continued back at the office/factory.

    Long story short, replacing the motherboards with an Intel model (again, when they used to badge their own motherboards) solved the problem. It turned out, it seems, that those SuperMicro motherboard clocks were unreliable and drifting to an extent that affected the software time calculations.

  17. Paul Hovnanian Silver badge

    Keep an eye ...

    ... on the system clock.

    If necessary, feed a clock and calendar display from system time/date. Have the admins break for lunch and receive pay envelops based on that. It'll stay synced up.

  18. ocelot

    I once wrote an NTP server based on borrowing the PTP time counters in the network interface MAC of an STM32F407 microcontroller - discovered an amazing 63 bit high resolution timer hiding in there that could be clock rate tweaked up and down in steps of about 10^-9 seconds per second. I lied to it about the LAN interface using PTP and it fired up this timer.

    It hooked into a GPS module that was specifically not a precision timing module - there I found a proprietary timing message that lied about the time shift between GPS time and the 1 pulse per second - it lied and produced a sawtooth value designed to confuse people who bought the $10 GPS rather than the $100 timing GPS.. so I ignored it.

    I never trusted it 100% but it used to be able to tell me the CPU clock frequency was something like 167.999995 MHZ instead of 168MHz ..

    I dont think it suffered that negative time glitch .. but who knows..

  19. danielsprouse

    03:14:07 UTC on January 19, 2038 is when Dennis Ritchie returns, riding a flaming PDP-11, to smite non-posix OSes.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like