NTP server that traveled back in time caused massive Aussie mobile outage

Australian telco Telstra has revealed the cause of the recent incident that caused widespread connectivity problems across its mobile networks, inculding outages to Australia's 000 emergency services line, plus outages to electronic payments services and transport networks. The carrier explained itself in a submission [PDF] to a …

[jgarbo]

So, you mean, "From now on, we'll do our job." Right.

[fnusnu]

Looking forward to the regomised report coming from 'Bruce'

[lauwersw]

I thought NTP clients had a panic threshold to prevent issues like this? If the time presented by the server is larger than this threshold it refuses to accept it. Clearly not all clients work like that or the threshold was disabled intentionally.

[Ferry Michael]

SNTP clients

Unfortunately there are many things which claim to have NTP because they can ask for the time and get a response.

That is basically SNTP which just syncs time without any checking.

Proper NTP synchronises clocks (not time) and rejects big time jumps and will also not sync to servers where the clocks are too jittery as they might be if provided by an SNTP server whose time might jump about

[SVD_NL]

Re: SNTP clients

Exactly. On a system this large and critical, i'd expect at least 5 NTP servers, that way you can survive 1 or 2 going insane (or going down).

On the other hand, a lot of telco equipment in use today has a marketing bulletpoint that says it's Y2K proof....

[Paul Crawford]

As a previous commentard has pointed out, a lot of stuff is SNTP with the 'simple' aspect being no checking multiple for sanity, etc. In our own system we use the routers as a NTP server for all SNTP stuff as it is always available (if not then time is the least of your concerns) and the routers in turn can be synchronised to multiple NTP sources to do the sanity checking for the SNTP devices.

If you are looking to deploy GPS-based time servers then get ones that have receivers for multiple systems (so combinations of the USA's GPS, Galileo, Glonass, BeiDou etc) and use multiple NTP servers, maybe multiple GPS based ones if accuracy is super-important, or back up one receiver with a few reliable Internet servers if cost-sensitive. Here is an example of good practice:

https://timetoolsltd.com/ntp/network-time-protocol-ntp-best-practices/

[Anonymous Coward]

Back to the Future: Time-Adjustment in Progress

It was interesting watching the server folk going through various gyrations to change the time foward and back on the NTP hosts, Novell servers, Unix hosts, and Windows servers of our separate, physically-isolated, Y2K test network.

[Anonymous Coward]

By default, they do. On the other hand, you can remove or change any of the settings in the configuration. One thing I'd like to know is why a single NTP server failure could cause a problem. A proper design will include 3+ NTP servers to prevent a server problem from causing a failure to maintain the correct time, either by having a single server go offline or by having a single server start providing incorrect time. When an NTP client starts, and before it makes any adjustments, it compares the time it currently thinks it is with the time as reported by all of its NTP servers (again, this behavior can be changed in the config). If all of the times are in sync, it picks a primary time source, and continues normal operation. If one of the time sources is an outlier, NTP won't use it. It will continue to track it, but not use it. If conditions are wrong and it can't determine what the actual time is, it won't sync at all, and will continue to rely on its own internal clock. If conditions are terrible, like in the article, and the NTP server is wrong, all of the clients will sync with the NTP server despite its incorrect time because its trusted.

One point the article didn't make is that ±300 seconds is the window outside of which everything fails. Your clocks don't have to be 20 years behind for there to be problems.

If you're on a company's network, and using Linux (Windows 11 will use time.windows.com or Active Directory), check /etc/ntp.conf or /etc/chrony.conf and make sure there are multiple 'server x.x.x.x' entries. If there aren't you should forward this article to your compliance or legal departments. What you can't always tell from the NTP client perspective is how the NTP servers two hops above are configured. Its possible, but dumb, for a company to have 5 internal NTP servers (which is good) but have all five of them configured to sync from a single pool.ntp.org ip address (which is bad.) The issue with this scenario is that only the NTP server administrators (and the network admins but we can see everything) can see that there is a problem and but only if they know to look.

[RT Harrison]

You can change the time server in Win 11 from time.windows.com going through the Settings app. Pick Time & Language, scroll down to additional settings. Expand the list (down arrow next to Sync now). Click Change.

I use uk.pool.ntp.org - full list at NTP Pool Project.

time.nist.gov is also a good alternative.

You can only have one NTP server as a time source. I think you could have up to 4 before Win 11.

[IGotOut]

@RT Harrison

The kit were talking about here isn't going to be Windows based. God help us if it was.

A lot of the kit I worked on had 2 or 3 options and if one went miles out, they would usually ignore it (same with Windows in many cases).

But the more "dumb" kit will often only have one.

[theblackhand]

As far as I am aware, Windows has supported multiple NTP servers from the command line since Windows 2000:

net stop w32time

w32tm /config /manualpeerlist:"0.pool.ntp.org,0x1 1.pool.ntp.org,0x1 time.google.com,0x1" /syncfromflags:manual /update

net start w32time

Alter your NTP servers as appropriate - try and use in-country pools for personal use.

[Anonymous Coward]

The article did state the failing NTP server was one of 3. So unless Telstra did something incredibly stupid like partitioning its network in 3 zones that fed from a single time source, the culprit is all the stuff that pretends to support NTP (continuous tracking of multiple time sources) but actually uses SNTP (single time source checked once in a while with no drift monitoring).

The electronic and windows people love the “simplicity” of SNTP and find full NTP wasteful and overengineered.

[Taliesinawen]

I thought NTP clients had a panic threshold

> I thought NTP clients had a panic threshold to prevent issues like this? If the time presented by the server is larger than this threshold it refuses to accept it. Clearly not all clients work like that or the threshold was disabled intentionally.

The bad time came from a "trusted" internal source. The GPS Week Number Rollover reset to ~2006. Enough clients accepted the bad time to break certificate validation across the network.

[DS999]

NTP servers with GPS source are stratum 0 sources and always trusted. Though perhaps they should not be.

[searler]

Stratum 0

The stratum number is merely a human readable indicator of the stated distance from an accurate clock (eg GPS). It has no bearing on the NTP algorithm for trusting a server

If there is more than one server and one of the servers falls outside of a plausible window defined by the others, then that server will be ignored, regardless of claimed stratum

I haven't looked deeply recently but I suppose if authenticated NTP is also used then that might modify the decision algorithm with respect to selection between authenticated and unauthenticated servers

[spmm]

Huge impact/downtime for a lot of small businesses

Australians use a lot of eftpos, tap and go is the norm in the cities and larger regional centres. So no morning coffees, taxis to the airport, or purchases in addition to the inability to make emergency calls.

Diesel interstate trains use 4G to sync trains on tracks and for sales in the buffet cars, so the Brisbane to Sydney train, already slow was stopped. Victorian VLine trains were off line so people had to commute by bus, if they could find one.

Telstra says people have to provide invoices from the previous Wednesday to support a multipage request for compensation. Don’t think we will be ditching cash anytime soon.

As a valued customer (cough) I was bombarded long after the event with meaningless SMS texts intimating that something was happening, nothing was of course, many hours before the all clear was sounded and any type of rational explanation provided. At least they didn’t blame the Chinese and Russians off the bat.

[Cincinnataroo]

If their NTP server system is not robust enough for this problem, what else is waiting to break.

I presume the clown show goes wide and deep.

How do we purge the incompetence?

[Andre Carneiro]

“How do we purge the incompetence?”

Who is John Galt?

[John Riddoch]

I'm guessing this is related to the GPS counter overflow which would result in a 19.6 year jump (so back to 2026). Surprising that it happened as the last overflow was 2019, so you'd have expected it to have been an issue for 7 years by now, although sounds like they fiddled with something at some point which broke it.

[Anonymous Coward]

I have a vague idea that some of the GPSs affected had a software patch that didn't survive a cold reboot and needed re-applying. I'd guess the post-it note fell off the box...

[Bebu sa Ware]

GPS card that … “did not operate as expected”

I wasn't far off the U-Blox USB dongle hanging off the back of an old whitebox running WinXP.

Not slagging of U-Blox as I use one as a time source after my own systems' NTP service suffered a "Telstra" moment and were a few years out - stuffs kerberos, license servers &c

One is still wondering why such a large telco didn't run something independent of GPS which has been screwed with before now… such as atomic clocks in a joint effort with other institutions such as CSIRO.

[STOP_FORTH]

Re: GPS card that … “did not operate as expected”

You can do things properly or cheaply. Telcos used to do things properly, especially when it came to clocks.

Sadly, I am old enough to remember those days.

[Paul Crawford]

Re: GPS card that … “did not operate as expected”

One is still wondering why such a large telco didn't run something independent of GPS which has been screwed with before now

Such services are available, they used to be LW broadcasts, still available but typically with accuracies in the few millisecond region, not the tens of nanoseconds GPS delivered. However, there are professional solutions such as this from the UK's NPL where over a fibre route with PTP support you get similar accuracy and no radio monkeying:

https://www.npl.co.uk/products-services/time-frequency/npltime

Or you just maintain and/or replace your NTP servers properly...

[david 12]

Re: GPS card that … “did not operate as expected”

Such services are available, they used to be LW broadcasts,

Not in Australia.

In the far North East, you can (or could) sometimes pick up Japan, but Australia never had broadcast time.

[Denarius]

Re: GPS card that … “did not operate as expected”

CSIRO stopped allowing external network NTP access years ago due to network traffic costs. I still wonder why server was powered up off the network or on test network first if its Level 0 to confirm time is good.

[John_Ericsson]

NTP server .... you had one job ...one job.

[breakfast]

This isn't my field, so I could be talking the crazy hat talk here, but... if I was restarting one of the three NTP servers that my organisation absolutely depended on to maintain a national telecoms network, I just think I might check it was passing out the correct time before re-connecting it to the network.

You know, just in case for some reason it wasn't.

[John_Ericsson]

“We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault [which] had not been properly documented”

Sounds like a bodge job was applied to get something working.

"not been properly documented". I would love to read the policy and procedures on their documentation on sever/service restart procedures (As if they have any).

[stiine]

I'd love to know what the original instructions were. Were they 'After restarting the ntp server compare the time on the server to the time on your company smartphone'... kinda makes you wonder, doesn't it.

I wonder what that technician's timesheet looked like... started at 10pm, stopped at 2am in 2020 for a total of -91250 hours, you owe us $3.65M, please report to the medical officer to relinquish 1/3 of your liver, 1 kidney, one lung, 20cm² of skin, and 4 toes.

[Winkypop]

At the third stroke it will be 2006

Precisely.

[Tron]

A retail tech problem too.

Some online services refuse to work if your PC clock is off. I always thought it a bit iffy that they would be able to check it, pointless, and potentially a real problem as industrial and IoT stuff may suffer a mobo battery fail and lose a connection as a result. The time on an end user system shouldn't matter.

They do now flag the need to check your PC time and date. Previously they just refused to work, which was a pain for users who didn't know that their PC had a battery in it or never bothered with the time and date on their system, and for those of us who had to fix a 'PC not working properly any more'.

[alain williams]

Re: A retail tech problem too.

If your clock is off then SSL certificates will not work. They have "Validity" which is "Not Before" and "Not After".

It might be a pain but this is vital for security.

[Tron]

Re: A retail tech problem too.

I get the idea, but I would have expected everything on your system to check the time and date via the network rather than to trust the local PC date. Rather as your phone can auto-correct its date and you can manually get your PC date to correct itself by checking an online timeserver.

Your system date should be merely for your convenience or for saving files you create locally and never relied upon for anything as important as security validation. Once you connect to a network, everything that has a security consequence should surely check against a trusted date source.

[Claptrap314]

Re: A retail tech problem too.

I've had fun with some internet games by changing the local clock to sometime in the past. When NTP says, "No, that's not the time", I get the benefit of a lot of time...

[The Central Scrutinizer]

Sadly, this is all par for the course for Telstra.

They have a long and ignoble history of screwing customers, either accidentally or on purpose.

[Anonymous IV]

"A Hazy Shade of Winter"

The Bangles already identified this problem in that Paul Simon song:

"Time, time, time

See what's become of me

While I looked around for my possibilities

I was so hard to please..."

[Patch Wombat]

Re: "A Hazy Shade of Winter"

To quote Bob Dylan, " The times they are a changing"

[Herby]

Tommrow...

Here in the pacific time zone, I regularly change the time zone to Apia in Samoa which is on the other side of the date line to do the next days tasks. Doesn't change much, just the application's idea. For games, this is particularly useful to do the next day's task. Fun & games...

[Phil Kingston]

"a faulty backup power feed (“Power A feed”) within the chassis"

I'm intrigued to know what hardware that was... anyone any insight?