The Register Home Page

back to article Techie fooled a panicked daemon and manipulated time itself to get servers in sync

Why, look at the time! 7:30 AM on Friday morning, the moment at which The Register regularly runs a fresh instalment of On Call, the reader-contributed column that shares your finest tech support stories. This week, meet a reader we'll Regomize as "Kim" who was asked to investigate an air-gapped lab in which time had become …

  1. Michael H.F. Wilkinson Silver badge
    Coat

    So the NTP server was essentially told the advice on the cover of the Hitchhikers' Guide to the Galaxy (in large, friendly letters).

    I'll get me coat. The one with the cassette tapes of the radio plays of the HHGTTG in the pocket, please.

    1. simonlb Silver badge
      Pint

      That was almost too Prefect. Have a Pan Galactic Gargle Blaster on me.

      1. UCAP Silver badge
        Joke

        Our current Mondeo has never given us a problem since we brought it 7 years ago. We often refer to it as the "Ford Perfect".

    2. Andy 68

      Number of upvotes checks out. (now)

    3. IceC0ld

      wanted to UP vote this SO bad, but its on 42 ffs, what's a geeky nerd supposed to do LOL

      appears that SOMEONE isn't of the HHGttG attuned variety, and 42 just went over there heads:o(

      1. TSM

        I was fortunate enough to encounter the comment with the upvote counter at 68. The next person can decide whether or not they want to spoil the result :)

      2. Yummy Dummy

        Under, lad! Like a member of Parliament.

    4. TooOldForThisSh*t

      and don't forget your towel

    5. jj_0

      Client vs Server

      Surely it iwas the client PC that wasntild not to panic.

      Which was a bad solution IMHO, if Kimi would have set the time once manually on the client it wpuld jave stopped panicking abiout the time difference with the NTP server.

      1. Yummy Dummy

        Re: Client vs Server

        That's a separate change for every client. This is one change on the server.

      2. This post has been deleted by its author

    6. Roland6 Silver badge

      > The one with the cassette tapes of the radio plays of the HHGTTG in the pocket, please.

      Obviously of the first series recorded from air ..

      [ Reports are that one episode failed to record and so had to be done again for the reruns. ]

  2. Korev Silver badge
    Pint

    It's time to take a second to congratulate Kim

    (it's early enough for a beer somewhere, including Vulture Towers)

  3. Gerhard den Hollander

    been there, done that ...

    Been in a similar situation.

    Rather than chaning the panic treshold, stopping the ntpd stuff, manually setting the clock to within a few seconds of the correct time, starting ntpd did fix the problem

    (be careful when doing this, if you set the clock forward this is usually fine, setting the clock backward will lead to all kinds of interesting and amusing phenomena

    1. Pascal Monett Silver badge
      Happy

      Re: amusing phenomena

      Nice euphemism.

    2. Prst. V.Jeltz Silver badge
      Coat

      Re: been there, done that ...

      Great ! a lesson in not messing with time from Mr. "I'm my own grandfather" !

      1. Sudosu Silver badge

        Re: been there, done that ...

        FIXITFIXITFIXITFIXIT!

      2. bemusedHorseman
        Pint

        Re: been there, done that ...

        *manually resets the system clock backwards in time*

        Choke on that, causality!

        1. KayJ

          Re: been there, done that ...

          *the Eschaton has entered the chat*

    3. Anonymous Coward Silver badge
      Boffin

      Re: been there, done that ...

      Every time I've encountered such a problem, I've just used the 'force update' parameter of the ntp client.

      Under normal circumstances if the clocks drift enough to be out of tolerance then that indicates that something needs to be corrected so I'd be reluctant to change the threshold.

      1. FIA Silver badge

        Re: been there, done that ...

        Yeah, this... I'd've ntpupdated with the force option and seen if the problem re-asserts itself.

        As you say, the option is there for a good reason, and lesson #43 in IT is 'Things are there for a reason'

      2. Anonymous Coward
        Anonymous Coward

        Re: been there, done that ...

        Agreed. Don't increase the threshold because kerberos will stop working when the clocks are 300 seconds apart.

      3. gnasher729 Silver badge

        Re: been there, done that ...

        I wrote some software that needed to stay within 5 minutes from a server time. So I gathered a dozen idevices and checked their drifting behaviour.

        My manager had an eight year old iphone with a drift of 5 seconds per day, so it would have failed after 60 days. (All other phones were within 0.2 seconds per day). I’m not sure, but I suppose mechanical force might damage the clock chip.

    4. Bill Gray Silver badge

      Re: been there, done that ...

      setting the clock backward will lead to all kinds of interesting and amusing phenomena

      Quite. I briefly thought I might be reading "Who, Me?", and was about to see some sort of horrific consequence of suddenly setting things back by five minutes.

      The ideal thing to do, I'd think, would be to somehow set the clock to run slow for a while. Say, by 10% for 50 minutes. This would resemble the 'leap second smear' technique used to avoid a mere one-second discontinuity on some systems.

      Though come to think of it, I'd not be surprised if some OSes are bright enough to do exactly that for most time adjustments. Which may explain why the story didn't go on to describe any horrific consequences.

      1. tfewster

        Re: been there, done that ...

        ntpd --slew

        1. Bill Gray Silver badge

          Re: been there, done that ...

          Ah, thank you. That led me to a discussion of the issues involved in slewing gradually vs. just changing the time. The former is done for small corrections. The problem is that the slew rate is limited to one part in 2000. So a five-minute adjustment would take almost a week.

          I'd think the workaround for that would be to have a way to tell ntpd, "make the adjustment but allow for a 10% slew rate, so it'll only take fifty minutes". But the fact that the current limit is a mere 0.05% slew rate gives me pause. It suggests that there's some problem (which I'm not seeing) with higher slew rates.

          1. Gordon Shumway

            Re: been there, done that ...

            If it'd been off for a year, surely a week for the safe fix (vs instant gratification Russian roulette) doesn't matter much?

          2. gnasher729 Silver badge

            Re: been there, done that ...

            Higher slew rates: You might have a music player that is unaware of time slew and that might cause trouble. Or there is a 100m world record attempt and you take the time with 10% slew.

            1. Jou (Mxyzptlk) Silver badge

              Re: been there, done that ...

              Your music player is like this? And a 100m record attempt of less than eight seconds by all ten participants is not good?

            2. Bill Gray Silver badge

              Re: been there, done that ...

              For both of those purposes, one should call clock_gettime() with the argument CLOCK_MONOTONIC_RAW. As described at the link,

              Similar to CLOCK_MONOTONIC, but provides access to a raw hardware-based time that is not subject to frequency adjustments.

              As the name suggests, this particular clock is 'monotonic' (won't be adjusted backward). It should represent, to the system's best ability, the time since a fixed, unspecified point in the past, such that the time between t2 and t1 really will be t2 minus t1.

              At least in Linux, you get a variety of clock types for various purposes : "wall clock time" (should match the current UTC time, but may be adjusted backward or have the rate change), various time-since-boot options which may or may not include time that the system is suspended, etc. There's even a TAI clock that should enable you to get around leap-second issues, though I gather there are some problems with that (not usually set up on most Linuxes to provide the appropriate TAI-UTC count).

              Whether your music player or track event timing program is set up to use the correct clock is, of course, another matter.

    5. Anonymous Coward
      Anonymous Coward

      Re: been there, done that ...

      Didn't some versions on Debian default to run ntpdate at startup? Way back when I was responsible for a decrepit collection of "servers" held together with hopes, dreams, bailing wire, and bubblegum, that helped ensure a machine with a flakey RTC was able to function after a reboot.

      1. Gerhard den Hollander

        Re: been there, done that ...

        Cannot remember whether it was the default, but when we setup ntp the Solaris boxes were doing ntpdate on boot up.

        We had one linux box where ( forgot the details) the bios was in the wrong timezone or something similarly silly, so the system time on boot up would be 2 hours ( and some ms ) off.

        That one also got the ntpdate on startup treatment.

    6. DS999 Silver badge
      Facepalm

      Re: been there, done that ...

      stopping the ntpd stuff, manually setting the clock

      Realized this the moment I read it, but I knew someone would have already said the same thing with either the first or second post. Rare that we have an On Call where someone uses precisely the wrong solution!

      The threshold exists for a reason - if for example the RTC in a server goes bad and starts running really slowly/quickly, or the CMOS battery dies and the system boots in 1970. You want to know that so it can be fixed, not covered up by NTP.

      1. Anonymous Coward
        Anonymous Coward

        Re: been there, done that ...

        "The threshold exists for a reason - if for example the RTC in a server goes bad and starts running really slowly/quickly, or the CMOS battery dies and the system boots in 1970. You want to know that so it can be fixed, not covered up by NTP."

        Though if the machine is a Raspberry PI it doesn't have an RTC (unless it's a RPI5 and even then you need the optional battery for it) and so on every boot the date is always 1st Jan 1970 until NTP (or an alternative fudge) kicks in.

        1. druck Silver badge

          Re: been there, done that ...

          No a Raspberry Pi has a fake hardware clock which just stores the current time once per hour, and uses this when you boot.

          If you look at the logs the first entry is always 17 minutes past the last hour it was up (unless you have changed the default settings).

          1. Anonymous Coward
            Anonymous Coward

            Re: been there, done that ...

            "No a Raspberry Pi has a fake hardware clock which just stores the current time once per hour, and uses this when you boot."

            No. You appear to be referring to functionality in (at least some) Linux distros that run on the Raspberry Pi where some file in the OS has its tmestamp updated either routinely whilst the OS is running or/and during OS shutdown and then upon subsequent boot this file's timestamp is used to "jump" the RPIs clock from its 1st Jan 1970 default value to the value of that file's timestamp so that the clock is closer to the real time.

            This means that if you were to reboot the RPI then the "restored" time would be close to real time but if you shutdown the RPI and then booted it one week later then the "restored" time would be wrong by 1 week (at least until NTP syncing then kicked in).

            A Raspberry Pi does NOT have a fake hardware clock, such above-mentioned distros running on a RPI implement a "fake" hardware clock.

            1. Jamie Jones Silver badge

              Re: been there, done that ...

              Yeah. When FreeBSD starts up, it sets it's time from the hardware clock. If a hardware clock doesn't exist, it sets it to the time of some recent file. (With a warning in the console log)

              This generic way of doing it means it will also be used on a machine whose hardware clock has failed.

            2. druck Silver badge

              Re: been there, done that ...

              Ok, should have said the Raspberry Pi running Raspberry Pi OS, which the vast majority do.

              If you are going to install another OS of course it may work differently.

        2. DS999 Silver badge

          Re: been there, done that ...

          Such hardware doesn't start up NTP with the current time == 0. If it runs NTP at all it queries a time server and sets the time based on that (man ntpdate) THEN starts the NTP client.

    7. Anonymous Coward
      Anonymous Coward

      Re: been there, done that ...

      "Rather than chaning the panic treshold, stopping the ntpd stuff, manually setting the clock to within a few seconds of the correct time, starting ntpd did fix the problem"

      On Linux machines you can use chrony as your NTP client (and server too if you need one) and it has a configuration setting "initstepslew" to control *on startup* whether it slews (many "small" changes which take time [not a pun]) or steps (one "big" jump) changes of the time. This is needed as if the difference between machine time and "real" time is too large then slewing the machine time would never catch-up with "real" time.

    8. Guy Geens

      Re: been there, done that ...

      setting the clock backward will lead to all kinds of interesting and amusing phenomena

      As a Y2k veteran, I can only agree.

    9. Rob Daglish

      Re: been there, done that ...

      This led to one of my favourite Netware messages - Synthetic Time is being issued on Partition...

      Always felt like I was playing with the laws of time and space when that one came up!

    10. Timo

      Re: been there, done that ...

      On the systems I've worked with it seemed that tripping the panic threshold would cause the ntpd to give up permanently, and the only way to get it right was the run ntpsync before starting ntpd. So this "change the time server and hope all the clients fall in line" would not have worked, would it?

      I guess I never had the ability to change the time source as it was synced to an internet pools.

      1. Robert Carnegie Silver badge

        Re: been there, done that ...

        I think it's ntpd on the client which "panics", perhaps slightly misnamed as apparently ntpd writes a goodbye note in the relevant log, then terminates itself.

        Apparently this is where ntpd says that it can't see an ntp server. Or it may be that the command line finds the client ntpd not running.

        The story seems to be confused about whether just one client machine was not taking time from the ntp server, or all of them. I don't trust vagueness, but I'm inclined to interpret it as two problems to solve: (1) each client have a different time from the server, and (2) ntpd on clients doesn't work. Manually rebooting one client with time already corrected will address (1) and also (2), but restarting client ntpd without setting time wont' work, because ntpd will just panic again.

  4. Korev Silver badge
    Alien

    I had a similar one. I once worked somewhere there was an Acopia ARX virtualising various Windows shares. The Acopia was setup to use the PKI infrastructure for time, the Windows clients Active Directory. I came in one Monday morning to find that the PKI clock had drifted sufficiently far from AD time (actually the correct one) that the clients would no longer talk to it.

    It was a very quick fix, but took a while to figure out what's wrong as you kind of assume PKI would be correct...

    1. Pascal Monett Silver badge
      Trollface

      You know what they say about "assume" . . .

      1. Prst. V.Jeltz Silver badge
        Headmaster

        That every time you say it someone will gleefully trot out that exact tired response , like polly wanting a cracker, regardless of context and whether assuming was strategically justified?

        Sometime you just have to assume, whether its because its 99.99% likely correct , or because its an intermediate measure while you diagnose more likely culprits.

        ( tbf the response may have been warranted in this case )

        1. Tony Gathercole ...

          Life is full of assumptions. The trick is to understand when and where you are relying on the assumption and what the consequences of the assumption being wrong might be.

          Spent ten years of my (working) life drilling the importance of fully documented and managed Risks, Assumptions, Issues and Dependency (RAID) registers into project teams for one of the (then) major outsourcing players. Some folks never really got it no matter how hard we worked.

          1. Doctor Syntax Silver badge

            "Some folks never really got it no matter how hard we worked."

            Not surprising if they were major outsources. It would have got in the way of doing things the cheapest way possible and then what happens to the profit.

        2. MonkeyJuice Silver badge

          Useless management idioms ass-ide, of course, fault isolation without assumptions? Pull the entire engine out and dismantle the gearbox before you check the battery if you like, but I want to actually complete the job and get to the pub.

      2. Anonymous Coward Silver badge
        Joke

        > You know what they say about "assume" . . .

        No, I don't actually. But you assuming that I do probably says something.

      3. Admiral Grace Hopper
      4. cheekybuddha
        Mushroom

        >> You know what they say about "assume" . . . <<

        Hi Mum!

      5. Prst. V.Jeltz Silver badge

        I see the troll icon now . Well played

      6. that one in the corner Silver badge

        Never understood what you as so bad about being an ass - hard working, robust, our pre-mechanised industrial heritage relied upon them. They'll trot up and down cliffsides that a horse will shy from (goats do it better, but you try loading your mining gear onto a goat). A bit of a reputation for being slow to pick up new concepts, but that is really just them being cautious and weighing the options instead of dashing for the shiny shiny like those noisy magpies.

        Now, being an arse, that is not good.

      7. Anonymous Coward
        Anonymous Coward

        No, I don't, yet here you are, confidently asserting something as fact despite any concrete evidence either supporting of contraindicating thus.

        ;-)

    2. l8gravely

      Wow, there's a name I haven't heard in a while, Acopia. We tried out their NFS appliance and promptly broke it with the sheer volume of files and directories and data we had in our environment. It was a really need idea to Virtualize NFS volumes and let you migrate data transparently. But it just didn't scale well. Got to visit them at their office in the old Wang buildings, on the left side of the Pond.

  5. Richard Gray 1

    Lotus Notes

    I had one as a contractor where people were intermittently unable to retrieve their mail.

    As I recall the clients were attached to a Novell Network, and the notes was on a different server (neither my responsibility) managed by different teams, and so the time had drifted between them.

    Once I let them know the Novell server was adjusted to be more correct and mail started working again..

    I don't know if they actually got NTP or anything like that working on it .. like I say not my job at that place

    1. Andy A

      Re: Lotus Notes

      Netware 4 had a comprehensive time system. It carefully dealt with different time zones and WAN links of different speeds. I once sat through a talk by one of the devs responsible. He was a good speaker too.

      1. Rob Daglish

        Re: Lotus Notes

        I attended a number of the Novell BrainShare events - there were many good speakers amongst the Dev team, and in some of the surrounding uni's, who could make the most dull topics interesting, and were great to have a pint with afterwards!

        I miss Novell...

  6. Jou (Mxyzptlk) Silver badge

    My NTP-AD default...

    Apart from the obvious, to actually configure ntp servers via GPO, I change "MaxAllowedPhaseOffset" to 30 seconds. The default of up to 15 minute tolerance before doing anything lead too often to the effect of being between five to ten minutes off over several days, which is irritating in many of todays environments. The rest "default stuff" is fine.

    Some special picky services get less than 1 second as tolerable offset, but that is rarely needed.

    1. kmorwath Silver badge

      Re: My NTP-AD default...

      AD joined machines do not need to have an NTP server set. They automatically sync with their DCs. Only the root DC, it needs an external time source.

      Since Kerberos tokens are time based, a domain will work properly as long as machines are in sync with DCs, even if the DCs time is wrong.

      Today with IAMs using time-based tokens, syncing time became far more important than log correlation, authentication may fail and services stop working.

      MaxAllowedPhaseOffset only controls if the time is adjusted immediately or gradually.

      https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-service-tools-and-settings

      1. Jou (Mxyzptlk) Silver badge

        Re: My NTP-AD default...

        Several things... You split the point, and you make way too many assumptions about details I left out to avoid a wall of text about obvious stuff in an IT-forum.

        Things I skipped:

        1. Only Domaincotrollers get ntp servers configured with given names and that setting mentioned. Most other AD-joined machines on the network (usually) don't need any additional time configuration, which if OBVOUS. Exceptions are, ususally, hyper-v or vmware hosts and databases, which need either the same 30 seconds max, or the "less than one second" I mentioned in my posting.

        2. I could have explained more where that 15 minutes "default tolerance" comes from - which is, since Server 2000, Kerberos with it tokens. The tolerance is simply 'cause not all network links in multinational cooperations are fast, and that is the "works stable" tolerance. Over the years with newer server various other tolerances were increased too, like how long can a domain controller be disconnected before AD assumes the "split brain" situation and acts accordingly.

        3. There is no "root DC". The PDC-role, which you call "root DC", is just an assumed role which can be switched easily since Server 2000. You configure NTP on all DCs, not just one. If you don't you have to be ready for split-brain situations regarding AD.

        4. You link is nice, but MaxAllowedPhaseOffset is needed since the "gradual correction" is too slow in many cases, which lead to the five to ten minute time difference over longer periods, and if you don't do it the "force now" way you will get more which will lead to...

        5. "Today with IAMs using time-based tokens, syncing time became far more important than log correlation, authentication may fail and services stop working." - which is exactly the reason why I configure it this way in first place and wrote the post.

        Thank you for forcing me too wall-of-text this forum with obvious things.

        1. kmorwath Silver badge

          Re: My NTP-AD default...

          No, in a forest you don't set NTP on all DCs, anyway the documentation I posted explain all.

          Guest OS under an hypervisor can have their time set by the hypervisor itself (which, of course, need to be synced itself), or use the standard methods. Why on your systems the time drifts so much while the Windows Time Service should keep it in-sync continuously I don't know. You should investigate it yourself.

          And what was so 'obvious' for you, may not be for others.

          1. The other JJ

            Re: My NTP-AD default...

            For shits and giggles, run your DCs as Hyper-V guests with host time sync enabled, and join your Hyper-V hosts to the domain.

            Funs fine. Until it doesn't. And then you have the problem of logging into machines that can't communicate with their DC because their clocks have been chasing each other all over the shop.

            1. theDeathOfRats
              Pint

              Re: My NTP-AD default...

              Loved the "Funs fine. Until it doesn't."

              I'm assuming guessing it's a typo, but I'm stealing it anyway. With your permission, I hope.

            2. Jou (Mxyzptlk) Silver badge

              Re: My NTP-AD default...

              > Funs fine.

              I can hear your "stop talking bullshit and come into the real world" frustration scream here, in south-west Germany (not Bavaria).

              1. Albert Coates
                Linux

                Re: My NTP-AD default...

                The weird thing about that corner of Germany is that they say moin moin all day long, even in the evening, including the unbayerische Wüste.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: My NTP-AD default...

                  Mahlzeit!

                  1. Albert Coates
                    Pint

                    Re: My NTP-AD default...

                    They even wake in the middle of the night, muttering moigen, mogen, realise it's 3:09 am, stretch out for that pre-prepared wursty/cheesy snack and fall asleep, knowing it's gonna be full-on Mahlzeit soon enough...

                    Bitte, ein Bit*... -->

                    * You heathen, you mutter, grabbing a cold Paulaner for comfort

                    1. Jou (Mxyzptlk) Silver badge

                      Re: My NTP-AD default...

                      Nono, today you need at least 32 Bit to work, and 64 Bit are commonly needed to work. Per day, of course.

                      But how many Maß are that?

                      1. Albert Coates
                        Pint

                        Re: My NTP-AD default...

                        Although I hate doing maths in public, I Bit = 33 cl, daher 32 Bits (bzw 4 Wörte) = 10,56 Maßkrüge,

                        Alles hat ein Ende, nur die Wurst hat zwei.

                        1. Jou (Mxyzptlk) Silver badge

                          Re: My NTP-AD default...

                          Ja, aber Windows 11 geht nur mit 64 Bit!

                          (For English speakers: Windows 11 only works with 64 bit! <- Follow that link to get the reference.)

                2. Jou (Mxyzptlk) Silver badge

                  Re: My NTP-AD default...

                  Wrong corner, "moin" is north-west - originally.

              2. MJB7

                Re: My NTP-AD default...

                > South-western Germany, (not Bavaria)

                Waves from Baden-Württemburg (specifically Südbaden - plenty of people here still bitter about the merger)

                1. Albert Coates
                  Unhappy

                  Re: My NTP-AD default...

                  Yeah, the Ossies got a raw deal, eh? (hmm, wrong merger, perhaps...)

          2. kmorwath Silver badge

            Re: My NTP-AD default...

            Now I understand why I have to fix so many botched AD deployments. It looks people can't read the documentation and probably rely on some youtube video found randomly.

            Windows is not Linux. Administer it as if it was, and you're run into troubles. Then complain that Windows does not work when the reality is you don't know how to use it.

      2. Roland6 Silver badge

        Re: My NTP-AD default...

        > Today with IAMs using time-based tokens, syncing time became far more important than log correlation

        Seems like we’ve created a single point of failure in our IT, Iwonder when weapons will appear to exploit it…

  7. Prst. V.Jeltz Silver badge

    This is bringing back vague memories of adjusting the time (date) in order to get software installed .

    Cant remember why , might have been something to do with trial periods

    1. Anonymous Coward Silver badge
      Pirate

      Norton used to do a 90 day trial which I regularly extended to about 10 years by setting the clock forward before installing it.

      Errmm, "allegedly"

      Those installs all expired long ago though.

      1. Blofeld's Cat

        Ah yes, I added code to our trial software to try and prevent that sort of thing.

        Fiddling with the local clock produced a "Error HGW1895: Temporal anomaly" message.

        1. Prst. V.Jeltz Silver badge

          Thats gotta be one of the coolest error messages ever!

          better the the nonsense SSIS/VS spits out of me .

          SSMS has quite a good / bitchy one along the lines of "Connection lost and I made no attempt to try again or reconnect"

          1. tfewster
            Facepalm

            StorageWorks Disk Array Controller Failure Code 07080000 "The other controller crashed, so this one must crash too"

    2. blu3b3rry Silver badge

      I remember it being a cheat on certain Xbox games (Fable titles iirc) where advancing the console system date by a few years would break the time tracking built into the game, showering you with money from any income you had set up.

    3. Anonymous Coward
      Anonymous Coward

      adjusting the time

      And back in the days before ./configure, I remember having to sometimes adjust the size of the time integer declared in code I'd downloaded, so it would match that of the os/things I was linking against, before it would compile...

      :-/

  8. tfewster

    Tim (AKA the British Telecom Speaking Clock)

    Tim (AKA the British Telecom Speaking Clock) might have been a better regomised name.

    I'm guessing that NTP using port 123 was chosen because 123 was the telephone number for the BT speaking clock.

    1. Jou (Mxyzptlk) Silver badge

      Re: Tim (AKA the British Telecom Speaking Clock)

      I've never read a more plausible explanation why it is port 123. Reminds me on "error 404" origin...

      1. TimMaher Silver badge
        Windows

        Re: “error 404”

        IBM mainframe terminals used to throw a 505 when they couldn’t talk to their controller.

        I always liked that.

    2. jake Silver badge

      Re: Tim (AKA the British Telecom Speaking Clock)

      I seriously doubt that Father Time even knew that Tim existed, much less what the number was. Having grown up in Oakland, California he probably knew POPCORN, though ... According to Jon Postal, 123 was picked early in the NTP development process, primarily for ease of typing[0], and that Dave was surprised it was still available when they were finalizing the RFC.

      [EDIT] Father Time was David L. Mills, the guy who wrote NTP.

      [0] There is a lot of that in the BSD world, especially early on.

  9. An_Old_Dog Silver badge

    Y2K Testing Time

    I recall watching one of our server guys doing the fiddly process of setting date/times forward and back on the NTP server, DCs, and Novell servers and getting them all synched up on our (physically-isolated!) Y2K testing network.

    It was interesting.

  10. Giles C Silver badge

    Synthetic time detected

    Not an error on Novell servers you ever wanted to see, all it took was a bad record in the database and until time caught up you would have that message every few minutes…

    1. markdiss76

      Re: Synthetic time detected

      They were the days.

      If it had gone really crazy (I recall a Radio Time Sync box getting an erroneous signal from Rugby that threw a Novell network to around the year 2033 briefly before reverting back to the late 1990's), they you ran some magic set parameter on the Novell Console to declare a new "Epoch" which reset the timestamps and fixed it.

  11. hammarbtyp

    Don't Panic

    The panic threshold was the 1st thing we had to nix when porting NTP to an embedded controller, since we had no idea what the time would be set to when the controller was added to the network (also it was on a ship, which had its own issues with movable time zones)

    Never really understood why it was there, and if so why it could not be disabled

    Another issue was an alarm server going into meltdown when the NTP caused the clock to go back one hour

    1. rcxb Silver badge

      Re: Don't Panic

      Never really understood why it was there, and if so why it could not be disabled

      It's easily turned-off... "tinker panic 0" at the TOP of your ntp.conf will do it. Seems it was added around v4.0, in the late 1990s. I'm not sure if NTPv3 even had a panic threshold, but in any case, the normal procedure was to run "ntpdate" against a trusted server, to jump the date/time, *before* you start ntpd. This configuration was common in startup scripts of the era.

      The panic threshold is there to protect against buggy or malicious NTP servers. Setting your system date backwards will break time-sensitive things like "make". Setting your date backwards also allows using expired TLS certificates that may have been compromised.

      Another issue was an alarm server going into meltdown when the NTP caused the clock to go back one hour

      Well, you've provided a reason of your own... NTP will "slew" the time (see: "man adjtime" ) instead of jumping it backwards, but that's only practical for smaller adjustments and would take an absurd amount of time to get in-sync.

  12. Jo 5

    NTP is so old school. I only do PTP.

    1. Arthur the cat
      Joke

      NTP is so old school. I only do PTP.

      I just use a long case clock as a PPS source.

      1. Albert Coates

        Long case clock

        Damn those pesky weights in my gf's time machine, needed winding once a week. Someone would usually fiddle with the little nut underneath the pendulum as well.

        1. Arthur the cat

          Damn those pesky weights in my gf's time machine, needed winding once a week. Someone would usually fiddle with the little nut underneath the pendulum as well.

          And now I'm thinking of the beginning of Tristram Shandy.

      2. I could be a dog really Silver badge

        I just use a long case clock as a PPS source

        Funny you should say that ...

        I recall listening to a talk some years ago, and it included some history of time in the UK - how we used to have local time in various places, but with the coming of the railways, they needed a more consistent time - otherwise you could (theoretically) have trains arriving before they departed, not to mention accidents caused by confusion as to what times opposing trains could use a line. And how we ended up with a network of cabled connections spreading out from Greenwich Observatory which sent a signal at 12 noon which would be used to synchronise clocks around the country.

        The anecdote went that when this network was being decommissioned, someone found a broken circuit and decided to work out where it went. They tracked it down to some small shop in a distant town, which still had the remote indicator where a ball would drop when the signal was received, and that could be used by the townsfolk to check that their clocks were still correct and indicated noon. And what's more, that indicator was still there, and dropped at noon each day - even without the electrical signal from Greenwich. So the proprietor was asked how it was still working - and he replied that at noon he'd look out the window and as the church clock ticked over to noon, he'd manually pull the lever that the solenoid should be pulling to release the ball.

        So off to the church to enquire how they kept the time. Simple, said the vicar, I look at the ball in the shop window, and check that the clock reads noon when the ball falls.

        Apparently, after several years without an external time reference, the church clock was still within some small error - I don't recall now whether it was stated to be a few minutes or seconds.

        1. Albert Coates

          "Funny you should say that ..."

          You may be thinking of Bradshaw's Guide, which preceded Railway Time.

  13. noemailta

    Magic CAM satellite modules

    Years ago I watched European wide satellite TV and each different service required a different conditional access module (CAM). The main ones were Irdeto, Seca, and Nagra. A company produced a slot in module called Magic CAM and any viewing card for these three encryptions would work. Canal Plus Scandinavia introduced a system called Conax on 1 degree west and I wanted to get a CAM for this, the satellite retailer said just buy a Fortec Star receiver with a Magic Cam built in and use the card reader on the front of the receiver for the card, so I did that. They said the module would need occasional updating via null modem which was no problem.

    What I didn't realise at the time and the shop didn't tell me was the receiver had a generic CAM chip inside not a Magic CAM module, and the makers of Magic CAM were really annoyed about this. They waited for Conax to change their validation system requiring any generic CAMs to be updated, then they put in a time bomb.

    It was something like the first of April in the mid 2000s, I think around 2004. The Magic CAM update went in a week before and worked fine, then on the first of April a kill flag rendered the inbuilt CAMs useless on anything but official Magic CAM firmware. The retailer replaced my receiver free of charge and apologised for selling me something that was badged as official but clearly wasn't, and the company that put the time bomb in their firmware came in for a lot of stick - I think they went out of business not long after this. Fair play to them for protecting their hardware but by bricking a lot of stuff people didn't trust them anymore.

    1. Gene Cash Silver badge

      Re: Magic CAM satellite modules

      > company that put the time bomb in their firmware came in for a lot of stick

      Didn't... let's see, it was FTDI that did this too. Oh here we go:

      https://hackaday.com/2016/02/01/ftdi-drivers-break-fake-chips-again/

      1. Anonymous Coward
        Anonymous Coward

        Re: Magic CAM satellite modules

        Ugh, yes. Got hit with that at two different employers. Still not convinced that the "counterfeit" ones were really counterfeit, or if they were just the older version.

        The solution, though, was simply to use the old version of the driver that didn't care.

        1. Roland6 Silver badge

          Re: Magic CAM satellite modules

          I was pleasantly surprised with a cheap Serial-to-USB converter (that had a chip labelled FTDI) to find it didn’t suffer from this problem and still plug-and-plays with W10/11.

          Because it is rarely used, but when needed, it is very important, this is a useful attribute.Obviusly, at some point we an expect either/both USB 2.0 to no longer be supported or OS’s to stop shipping with drivers for old devices/interfaces. (*)

          Just having a garage clear out: 10-BaseT adaptors (ISA, PCMCIA, PCI) and cabling for a few PCs (used when staying in a hotel in the late 1980s to enable the team to work in their rooms and access a server, our requirement was all rooms adjacent or directly above, so cables could be dropped out of windows). Also rediscovered several parallel port connected disk enclosures - state-of-the-art in their time.

  14. Boris the Cockroach Silver badge
    Pint

    Now all I

    need is a way of stopping "Boss Panic" when the customers go "we need 5000 widgets"

    Explaining we made 100 widgets per hour and run 24 hrs means we can make the delivery on thursday usually stops the panic... but you never know when it will set in again.......

    Anyway .... Set_Time(int beer)

  15. JQW

    Banyan VINES

    Banyan VINES - an obscure and long forgotten rival to Novel Netware, had some odd time sync quirks. Severs would sync with each other on the network - initially synching with the fastest clock of them all (what!?). The entire network time could also be changed by manually setting it from the console of any server. VINES networks could be huge, multinational affairs, with hundreds (or even thousands) of severs in some cases.

    This could cause problems as the date prompt was always in MM/DD/YY format, even for European installations. Furthermore, for any site outside North America, you had to configure your server's Daylight-Saving Time settings manually - this was also prone to errors as time zones ahead of GMT had to have a negative offset, and not a positive one.

    Consequentially setting the system time often went wrong, with spectacular results. Setting the clock forward and back would break things, as various daemons would be stuck, such as the ones that emptied mailboxes and print queues, until they were restarted. Not fun on an 800 server global network.

    Banyan eventually fixed things by added dedicated time sync lists, and support for NIST dial-out via modem. NTP itself was never an option.

    1. Gene Cash Silver badge

      Re: Banyan VINES

      I actually think this is one of the major reasons they're long forgotten. I remember hearing A LOT about headaches this caused, and as I was a Novell admin at the time, I said "use Novell... it sucks... but it sucks less."

    2. Apocalypso - a cheery end to the world Bronze badge

      Re: Banyan VINES

      > Severs would sync with each other on the network - initially synching with the fastest clock of them all (what!?).

      I've never used Vines but this seems like the least worst option as it means all clocks but one go forward rather than backwards.

      1. Richard 12 Silver badge
        Mushroom

        Re: Banyan VINES

        Except that it really means every clock goes backwards shortly after.

        1. Apocalypso - a cheery end to the world Bronze badge

          Re: Banyan VINES

          > Except that it really means every clock goes backwards shortly after.

          Ah, I see what you mean. But then that is the admin's problem, not Vines' :-)

    3. Alistair
      Windows

      Re: Banyan VINES

      Dear gussie, there are still a couple others about that remember Banyan. And the nightmare of haivng an idiot in a foreign branch set time and date in their LOCAL time format on the prompt.......

      CA/US/PI/IN/SA/TK/UK, I think we got hit at LEAST once a month.

      1. jake Silver badge

        Re: Banyan VINES

        The absolute best thing about VINES was that it financed my initial foray into consulting ... in the late '80s and early '90s I made a whole bunch of money upgrading companies running Banyan Systems proprietary software to a proper, more open un*x and unbastardized TCP/IP.

        Fun times.

  16. xyz123 Silver badge

    LAN based syncing is probably better for this.

    ONE machine syncs with a remote NTP. The others sync with that PC, so they're usually within a few ms of each other

    1. Gerhard den Hollander

      It's been a while,

      But you'd setup 3 ntpservers, that synced against a few trusted ntp servers outside ( for which you'd punch holes in the firewall to only those )

      And then you set up the rest of your machines to talk to those 3.

      And in choosing which machines you'd pick things like your nis/yp servers or one of the central file servers or the machine with the tapedrives that's the main backup server.

      Indeed you never let all the machines talk to outside.

      That's not how ntp is intended to be used.

      We did consider buying our own atomic clock thingy. Not because we needed the accuracy but just because we could claim we had an atomic clock.

      But we never could really justify this for ourselves.

  17. MatthewSt Silver badge

    Trains...

    I had the opposite problem at my first job. The domain controllers all ran about 2 minutes slow (they sync'd with each other rather than anything external). My smartphone (Windows CE!) talked to Exchange which meant it got the time from there rather than the network, and I kept missing my trains!

    A quick trip to the sysadmin and setting up an upstream NTP server was all it took!

    1. I could be a dog really Silver badge

      Re: Trains...

      In our house, Mrs ICBADR causes me to have to do a lot of mental arithmetic - and given one of the affected clocks in in the bathroom, that means early in the morning when the brain isn't yet awake. Her approach to not being late is to ... drumroll ... have clocks set fast. So if she thinks she'll be 5 minutes late, she's actually on-time because the clock is 5 minutes fast.

      My approach is to know what time I need to be somewhere, and have a clock set correctly so I can work out when I need to set off. But I have to deal with different clocks being in different timezones - to the point where as a joke, a family member bought a chrissy present of a clock with two mechanisms side by side, labelled with mine and Mrs ICBADR's names. The former is set correctly, the latter is 5 mins fast. The aforementioned one in the bathroom is 15 minutes fast - which is a p.i.t.a. when I forget and panic that I'm running late.

  18. J.G.Harston Silver badge
    Mushroom

    Are you deliberately trying to induce OCD by swapping between sync and synch (eeugh!) all the way through the article?

    1. Doctor Syntax Silver badge

      That's what happens when you get out of sync. You have to drop in a few synchs to make up the shortfall.

      1. Androgynous Cupboard Silver badge

        Doubly confusing if you’re in London. Dropped H’s I can ‘andle, but extra ones are most disconcerting.

    2. Anonymous Coward
      Anonymous Coward

      As long as you don't refer to NSYNC.

  19. Anonymous Coward
    Anonymous Coward

    I worked at a somewhat well known IT security company where I discovered that their NTP servers in their data centres on multiple continents all cross-synced off each other except for a single NTP server that also synced from one Stratum 2 server in either Germany or Russia. What a mess it was!

    1. I could be a dog really Silver badge

      I'd argue that having your internal network sync amongst itself is the correct way to do it - and only have a small number (I'd have more than 1, but YMMV as the world won't end if you lose your upstream stratum 2 for a short time) of internal servers sync externally. These days I don't think I'd include Russia as a time source, but not that long ago I don't think it was seen as much of a risk.

      In general, each NPT server will seek the lowest stratum - so generally your externally synced one will win, and the others will be 1 or 2 stratums lower so only be backup candidates.

      1. Anonymous Coward
        Anonymous Coward

        <Original Anonymous Coward here>

        My point was that despite running multiple NTP servers per-DC they only had a *single* NTP server worldwide that synced with any external source and there was only a *single* external source used.

        Then they had the rats nest of internal NTP servers all syncing off each other, i.e. server A would sync with server B which was syncing with server A etc.

        I'd expect a minimum of 3 external servers (of lower stratum) to be synced with or/and reference time sources such as GPS be used.

  20. Fruit and Nutcase Silver badge
    Alert

    Don't panic!

    temporal cops[Lance Corporal]

    For the benefit of the transatlantic cousins...

    https://en.wikipedia.org/wiki/Lance_Corporal_Jones

    https://www.youtube.com/watch?v=nR0lOtdvqyg

    1. Jou (Mxyzptlk) Silver badge

      Re: Don't panic!

      Aha, the next British TV series I have not heard about 'till today: Dad's Army. Finished watching Red Dwarf Q1 2025, now the next :D.

      1. Fruit and Nutcase Silver badge
        Thumb Up

        Re: Don't panic!

        There is an absolute cracker in "The Deadly Attachment", the first episode of the sixth series.

        Don't want to spoil it for you, so not linking to a clip on YouTube.

        ===========

        There was also a series on radio, with the original cast, adapted from the TV series

        This is probably geo-fenced...

        https://www.bbc.co.uk/sounds/play/b007k1s5

        (moot point - doesn't list any available to listen on-demand)

        Available in Jeff Bezos' tat bazaar

        https://www.amazon.com/Dads-Army-David-Croft-Jimmy-Perry-audiobook/dp/B00M9BFVHK

        - other tat bazaars are available...

        1. Jou (Mxyzptlk) Silver badge

          Re: Don't panic!

          I just watched the first episode. We will see how far I can bear it. Just like there are some Japanese anime which are "too Japanese" for me, this one could be "too British", or I am simply too young. But hey, it is just the first episode, so that is a bit too early to tell how it will hold up. Some get better after a few episodes, but I don't know until I watched.

          1. Fruit and Nutcase Silver badge

            Re: Don't panic!

            Yes, let the cast and production team get into their stride - I recently heard the first episode on the radio - don't just go on that. If you are wavering, then, checkout that first episode of the 6th series.

            There's also 'Allo 'Allo - this time writers are David Croft and Jeremy Lloyd. (Dad's Army was written by David Croft and Jimmy Perry).

            Jeremy Lloyd - you'll spot him in some 60's films and also in Rowan & Martin's Laugh-In>. IIRC, he appeared alongside Goldie Hawn in one of the series. He was also, very briefly, Mr Joanna Lumley...

            https://en.wikipedia.org/wiki/Jeremy_Lloyd

            fitting in between Dad's Army and 'Allo 'Allo, still set during WW2, but set in Burma, David Croft and Jimmy Perry wrote

            It Ain't Half Hot Mum

            https://en.wikipedia.org/wiki/It_Ain%27t_Half_Hot_Mum

            You won't find the BBC repeating the series - most probably due to Michael Bates playing the role of Rangi Ram with make-up, rather than being played by an asian actor. The script would never get past commissioning today. It is available on DVD.

            https://en.wikipedia.org/wiki/Michael_Bates_(actor)

            The series of course gave us Windsor Davies as Battery Sergeant Major Williams.

            I shall now "Shut Up!!", as the Sgt Major would at the end of the episodes.

            "Oh dear, how sad, never mind"

            p.s.

            If you've not come across it before, look up "To The Manor Born" - an altogether different scene - with Penelope Keith and Peter Bowles

            1. I could be a dog really Silver badge
              Pint

              Re: Don't panic!

              And I only have the one upvote to offer. All classics.

              As you say, It Ain't Half Hot Mum isn't likely to appear on the airwaves again - not just because of using an actor to act as a character, but also for the attitudes and language used. I think that's sad, other stuff gets shown with a "this programme reflects the language and attitudes of the time and is likely to be considered offensive now" warning at the start. And for similar (but wrong) reasons[1], Love Thy Neighbour isn't likely to get shown again.

              As for To The Manor Born, brilliant writing, and Penelope Keith as Audrey Fforbes-Hamilton plays the part to perfection. And to round off, I'd add The Good Life which is on iPlayer (we were watching it just the other day). I know some would say the characters and situations are a bit absurd, but they are caricatures - a deliberate embellishment which I think makes them such good entertainment.

              The sad part is realising how long ago they were made, and how few of the actors are still alive. I raise a virtual (see icon) to their memory.

              [1]it's usually considered racist - but in fact if you actually consider it properly it's written to expose racism by showing Eddie to be the dick that he was.

        2. Jou (Mxyzptlk) Silver badge

          Re: Don't panic!

          Now I've watched a few episodes (still first season), and all I can think of "Who do you think you are kidding Mr. Trumpy" with the arrows going slightly different directions than in that intro. Nice light-weight entertainment! Somewhat reminds me of MASH first season (when the comedy had way more priority), just at a different location and without the "war is hell" parts between the comedy parts.

  21. Dante Alighieri
    Facepalm

    meanwhile in the NHS

    The PACS and the hospital network (pacs VLAN) don't talk very well - they certainly don't talk to the modality (scanners, xray etc) for coordinated time.

    I had to flag up I had reported an xray before the system said it was done given a 5 minute time gap. Yes modality was in the future.

    And explain in words of one syllable why this was important. More than once.

    No consequences could be imagined - until spelled out for the timelines for the coroner or other investigations.

  22. trindflo

    NET TIME

    I recall using the 'net time' command to get things immediately synchronized.

    https://ss64.com/nt/net-time.html

  23. Sudosu Silver badge

    Proxmox Cat

    When I was first playing around with Proxmox clusters I did not have the NTP set up correctly or broke it while playing around.

    After the nodes get out of sync, in the time date page of the GUI, it would post oanoanoan. (can't remember exactly but its close to this) until I set the time manually.

    It is what what I call the "No No Cat message" named after the famous traumatized internet cat.

  24. Poink!

    This has me thinking

    I know a few companies that have NTP via GPS. What could you do with a GPS spoofer or jammer? You could cause a bunch of chaos and possibly compromise security doing this.

    1. I could be a dog really Silver badge

      Re: This has me thinking

      Actually, a jammer would do very little - the systems would just carry on - all mutually synchronised and only slowly drift out of sync with real time.

      Spoofing would have an effect, but again, (note comments above about why there's a limit to how much and how fast NTP will sync by default) you could only cause the system clocks to drift a bit faster.

      Only if the internal systems were really badly set up could you cause problems (such as getting some of them to jump to different times.)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like