No, the Linux leap second bug WON'T crash the web

There’s a reason space missions don’t launch on the day a leap second is added to international clocks. Scientists don’t want to run the risk that the computer systems running things might hiccup on the new time and then malfunction, sending their multi-million dollar lifetime’s investment into a fatal nose dive. The rest of …

[Phil O'Sophical]

Time lords in Paris

She's bigger on the inside than the outside?

[frank ly]

Re: Time lords in Paris

Alons y!

[Anonymous Coward]

Re: Time lords in Paris

Brilliant!

[Donkey Molestor X]

Re: Time lords in Paris

> She's bigger on the inside than the outside?

Have you ever thrown a hotdog down a hallway?

Ever park your VW Beetle in a zeppelin hangar?

[Steven Raith]

Re: Time lords in Paris

Ever flung a bratwurst down the Blackwall Tunnel?

(I'm fond of that one)

[Destroy All Monsters]

Re: Time lords in Paris

HOW UNCOUGH AND LOCKER-ROOMY

[Anomalous Cowturd]

Re: Time lords in Paris

Like dropping a pencil down a well.

Snigger.

[Anonymous Coward]

Re: Time lords in Paris

The answer to all these problems is to strap a 2x4 to your arse so as not to fall in

[James O'Shea]

Re: Time lords in Paris

"She's bigger on the inside than the outside?"

Well, she _is_ a TARDIS. Or at least a Tard.

[The Jon]

Time delay

"The second the clocks ticked past midnight GMT, the bug awoke and took down Altea. 135 airlines had implemented the Altea reservation system by 2012 but the Australians were the first to go down once the clocks hit midnight."

Err, shurely everyone hits Midnight GMT at the same time?

[Anonymous Coward]

Re: Time delay

"Err, shurely everyone hits Midnight GMT at the same time?"

Nope GMT is a time zone, so being a pedant, most of the world never experiences GMT.

The time standard you are looking for UTC

[Irony Deficient]

Re: Time delay

Lost all faith…, which “you” are you addressing — Gavin or The Jon? (No region of Australia follows GMT.)

[davidp231]

Re: Time delay

Either way, don't call him Shirley ;-)

[Sacioz]

Re: Time delay

Call him Bruce , then ...)))

[John Brown (no body)]

Re: Time delay

Shirley it's Sheila?

[yoganmahew]

Re: Time delay

Airline systems worldwide use GMT (aka UTC or Zulu) time as a standard reference time. Local adjustments are then applied. Not everyone does it, but almost all do; all type B messaging has a GMT timestamp and IIRC EDIFACT messages do too. My guess is that the problem arose when GMT and local were found to be out of synch, so Australia would be the first to experience this, but I don't work in *nix so I can't say for sure.

[the spectacularly refined chap]

Re: Time delay

Airline systems worldwide use GMT (aka UTC or Zulu) time as a standard reference time.

GMT and UTC are two distinct time zones...

[Anonymous Coward]

Re: Time delay

Can people please learn the difference:

GMT - Time zone, same as EST or CET

UTC - Time Standard. Yes it is the "same" as GMT, but they are NOT the same thing.

If we decide to change GMT for some reason, no matter how odd. It would NOT affect UTC

[mad physicist Fiona]

Re: Time delay

Can people please learn the difference:

GMT - Time zone, same as EST or CET

UTC - Time Standard. Yes it is the "same" as GMT, but they are NOT the same thing.

If you are going to be pedantic at least get it remotely correct. They are two distinct measurements: GMT is the "natural" time, i.e. as determined by the rotation of the Earth. UTC is governed by atomic clocks and needs periodic insertion of leap seconds to keep it roughly in sync with GMT which itself neither has nor needs leap seconds. GMT and UTC both approximate the time at Greenwich but for any given instant the time is slightly different when expressed in each system.

[P. Lee]

Re: Time delay

>No region of Australia follows GMT.

And no computer should be doing anything else. Yes, that includes you, you brain-dead MS clients.

The computer runs on GMT, your shell translates that to your local time, which is why you don't get similar failures when you switch between summer time and normal time (if you do).

And we'll have non of that "UTC" malarky round here please! Go wash your mouth out!

What happened to using NTP to slew the time rather than making sudden jumps?

[JohnMcL]

Re: Time delay

Australians were the first to go down ? Apart from the unfortunate choice of words, what happened in closer-to-the-dateline New Zealand? Does the journo think NZ is ten years behind everywhere else?

[Christian Berger]

Few systems propperly account for this

I mean next to no system accounts for leap seconds. In fact it's even impossible to predict when the next leap second will be introduced.

Perhaps the best way would be to leave computers running at a continuous timescale. Something based on "Atom Time" and use something like timezone files to correct for the "propper" wall clock time. This way you would have to update your timezone files anualy, but if you don't, the time your computer displays will be off by a second while the internal time will still be consistent with what the other computers think.

This could probably be retrofited to most computers as they already use pre-made C library functions to convert from "Unix epoch" to local time. Few people do this with their own code, since it's quite a bit of work.

[Anonymous Coward]

Re: Few systems propperly account for this

plus UNIX epoch time has it's own issues to deal with. 2038 will be as bad a 2000 i.e not that bad after all.

[P. Lee]

Re: Few systems propperly account for this

>2038 will be as bad a 2000 i.e not that bad after all.

But heart-stopping nonetheless. I went out to watch the fireworks and when I got back to the house, all the lights had gone out.

The actual problem was a short-out at some roadworks due to all the rain, but it had me going for a moment.

[Michael Wojcik]

Re: Few systems propperly account for this

2038 will be as bad a 2000 i.e not that bad after all.

This foolishness again.

Y2K was "not that bad" because a shitload of remediation was done before the century rollover (or whatever the point of failure was for a given issue). There are many descriptions of specific, severe Y2K bugs that were corrected ahead of time.

One of my favorites was one from RISKS about a dialysis machine that went into self-cleaning mode when an "impossible" date was entered. That'd spoil your day right quick.

But don't let me stop you from generalizing from your anecdotal experience.

[SkippyBing]

Re: Few systems propperly account for this

GPS already does this, the system runs without leap seconds so has slowly been going out of sync with UTC since 1980 when it started. Your receivers add the necessary fudge to correct to local time anyway so a leap second is a minor additional factor. This probably also helps when GPS timing is being used for timing on critical infrastructure as leap seconds won’t affect it.

[Anonymous Coward]

So, does that work with NTP?

GPS already does this, the system runs without leap seconds so has slowly been going out of sync with UTC since 1980 when it started. Your receivers add the necessary fudge to correct to local time anyway so a leap second is a minor additional factor.

Given that most systems are timed off atomic clock or GPS references, does that mean a system using NTP for system time will generally follow this fudge as well? Or, in another way, will have no problem whatsoever?

Enquiring minds want to know.

[Paul Crawford]

Re: So, does that work with NTP?

GPS broadcasts the linear atomic time and the offset as separate fields, and all internal calculations (other than UTC output) use the former. Some GPS receiver's firmware has had buggy handling of the GPS-UTC offset change, but again that ultimately comes down to not testing it. You can buy GPS simulators, so its not like a company can't test for it, just they did not think and/or bother doing so.

Similarly NTP broadcasts the leap second event for the day before it happens, and then tells the kernel to apply the step at the appropriate time. AFIK the NTP daemon can get the pending leap-second info from an attached GPS used as a stratum-0 source, so it ought not to require networking to other peers to get that information.

The main bug was not in NTP itself, but in how the Linux kernel handled the application of the 1 second jump the the time_t UTC counter, as it allowed a dead-lock situation to occur. A standard type of problem for any multi-threaded software, and again one that ought to have been better reviewed and tested.

I don't know the reason(s) for the Java bug, but most likely it was related the kernel deadlock while waiting for "sleep" timers to expire.

[Paul Crawford]

Bigger picture

Really, there is a bigger picture here. Systems get screwed up for all sorts of different reasons! While we debate the leap-second we also should remember faulty hardware and numerous other bugs in both the OS (and any OS) and the applications.

If you have a big critical system you really ought to have some sort of watchdog on your servers to spot the signs of kernel panic/lock-up or application faults and reboot it. While brutal, at least you would be coming back on-line in minutes rather than hours while support folks are called to investigate and find they can't SSH in, etc, so they have to debate and then use ILOMs to reboot possibly hundreds of machines.

[Greg J Preece]

Re: Few systems propperly account for this

Don't GPS satellites move through time at a slightly different speed to ground-based fleshlings anyway? I seem to remember that being a bigger problem to solve than leap seconds.

[Paul Crawford]

Re: Few systems propperly account for this

Yes, GPS are subject to time dilation, but that is accounted for in the numbers they provide. Its only a problem if you don't correct for it by design!

[yoganmahew]

Re: Few systems propperly account for this

Mainframe complexes (well IBM ones anyway) use an external timer to keep them in synch (sysplex or STP). Generally speaking, they run a master (TOD - time of day) clock that all other clocks are 'built' from, so there's only one clock authority. Is this enough? Well, we'll see!

[Martin-73]

Re: Few systems propperly account for this

Actually I believe I heard somewhere that the difference in gravity at the satellites' altitude is more than enough to compensate for the relativistic effects and the correction is in the opposite direction than 'caused by the speed' of the satellites would suggest.. [googles]...

Ah yes, Here, section 2 shows the speed based time dilation is actually outweighed about 5:1 by the gravitational effect.

Fascinating in either case :)

[Paul Crawford]

Re: Few systems propperly account for this

A lot of space systems already use variations on "ephemeris time" that has a linear atomic basis and a variable offset to get UTC, etc. That is not a new idea, and as pointed out exactly the same approach is used by the GPS satellites.

The problem is NOT the introduction of leap seconds, it is the simple fact that they don't test systems properly to deal with this known attribute of time keeping.

Instead of trying to get rid of leap seconds, perhaps they should always add/remove one each alternate month with the occasional add two months in a row?

That way people would be forced to test for this and not cry every 1-2 years when untested/patched code throw a wobbly.

[Anonymous Coward]

Re: Few systems propperly account for this

> Perhaps the best way would be to leave computers running at a continuous timescale.

Ouch! Let us not get into that discussion, shall we? I still have the flashblacks from last time. :-)

[Anonymous Coward]

The Independent tells me all I need to know...

Written in the style of a hysterical 12 year old and with the level of understanding of a sleepy puppy, the Independent has it covered:

"But last time it happened, in 2012, it took down much of the internet. Reddit, Foursquare, Yelp and LinkedIn all reported problems, and so did the Linux operating system and programmes (sic) using Java."

http://www.independent.co.uk/life-style/gadgets-and-tech/news/leap-second-computer-chaos-feared-as-scientists-let-world-catch-up-with-clocks-9961881.html

I love it that the reporter doesn't bother to find out when in 2012 this happened, and that Reddit, Foursquare, Yelp and LinkedIn are deemed 'much of the internet'. I assume Twitter is the 'rest of it'.

I'm sure we used to have educated journalists once, or am I mistaken?

[Captain TickTock]

Re: The Independent tells me all I need to know...

The reporter didn't look it up because the world ended in 2012, silly ;-)

[John Brown (no body)]

Re: The Independent tells me all I need to know...

"I'm sure we used to have educated journalists once, or am I mistaken?"

Yes, you are mistaken We had reporters, journalists and Gentlemen from The Times.

Now we have reporters, opinions scribblers, and El Reg Hacks :-)

[Dazed and Confused]

Re: The Independent tells me all I need to know...

> and that Reddit, Foursquare, Yelp and LinkedIn are deemed 'much of the internet'. I assume Twitter is the 'rest of it'.

Can someone please tell the EU, all the commision seem to think that Google is the Interweb.

[CAPS LOCK]

Well, I don't know what you lot are going to do, but i'm....

Having an extra second in bed. Bliss.

[Rich 11]

Re: Well, I don't know what you lot are going to do, but i'm....

I'm going to spend it in the pub. There's no time to waste.

[Scroticus Canis]

Re: Well, I don't know what you lot are going to do, but i'm....

If they added a second you actually lost it! Turned 00:01 at what would have been 00:00. Just saying.

[poopypants]

FFS

Please learn how to spell QANTAS. There is no "U", and it is all caps. It is an acronym for the "Queensland and Northern Territory Aerial Services".

[Groaning Ninny]

Re: FFS

If Qantas don't use all caps, why should we? I think they're trying to move on from being a regional airline....

http://www.qantas.com.au/travel/airlines/company/global/en

"Founded in the Queensland outback in 1920, Qantas has grown to be Australia's largest domestic and international airline. Registered originally as the Queensland and Northern Territory Aerial Services Limited (QANTAS), Qantas is widely regarded as the world's leading long distance airline and one of the strongest brands in Australia."

Not that I'm a pedant.

[poopypants]

Re: FFS

Point taken. That doesn't excuse the spelling.

[Fruit and Nutcase]

Re: FFS

ahem.

Ffs

[P. Lee]

Re: FFS

> Qantas is widely regarded as the world's leading long distance airline

because Australia is so......far......from.....everywhere.

[Anonymous Coward]

Re: FFS

Aerial services? They do antennae too?

:)

[Michael Wojcik]

Re: FFS

They do antennae too?

Well, while they're up there, they might as well. Saves someone else a trip.

[John 203]

Won't somebody please think of the Kerbals?!

Should I avoid playing Kerbal Space Program on the day in question or are Kerbals expendible enough to take the chance?

[Irongut]

Re: Won't somebody please think of the Kerbals?!

Jeb will be fine but I think Bill will be hiding under his bed for the day. :)

[Boris the Cockroach]

Re: Won't somebody please think of the Kerbals?!

Its whether its going to give Frontier developments and the elite:dangerous servers any problem.

After all its not going to be el reg without their daily anti elite:dangerous story ;)

[Anonymous Coward]

Easy fix

If you are not sure, turn of NTPD at 23.00. Turn it back on at 1.00. The NTPD server will gradually sync at the new time over a few hours (I am sure most hardware/kernels can keep decent time for a few hours). No 61 second minute then.

[Anonymous Coward]

Re: Easy fix

Replying to the person that deleted his/her post:

http://lkml.iu.edu/hypermail/linux/kernel/1203.1/04598.html

[Anonymous Coward]

Re: Easy fix

That might help, but by 23:00 the system may have already received some kind of notification of the impending leap second so it still might try to do something 'clever' either at midnight or whenever you turn ntpd back on.

(I can't seem to find out when the leap second gets announced through NTP. One web page says "months" in advance, one mentions the previous day, one says an hour beforehand, ... If it is an hour then you should presumably shut down ntpd at 22:55 and turn it back on at 00:05 if you want your system to remain blissfully ignorant.)

[Paul Crawford]

Re: Easy fix

Better fix - just use the working code.

It was working properly in Linux, and then a patch was applied that broke it. No one noticed its implications at the time, and no one tested it on a leap-second generator. Then it failed in real life.

The moral is simple and need repeating: Test every bloody change you make!

[Anonymous Coward]

Re: Easy fix

Good comment. And I suppose MS stuff never broke as it never has the right time anyway.

[Anonymous Coward]

Re: And I suppose MS stuff never broke

Still fanbois toys still have problems with summertime & wintertime

[Dazed and Confused]

Re: Easy fix

> It was working properly in Linux, and then ...

Done of my Linux boxes had any problems and all the ones that would have been running would have been on pretty vanilla releases, mostly CentOS.

[John Crisp]

Oh how boring. Clearly one second isn't enough to bring out the shills :-)

Can't you make it more disasterous ?

[Anonymous Coward]

Altea crashed? What a surprise

Speaking as someone who has worked on that code in the past its amazing it ever works in the first place. Hidiously over designed and almost unmaintainable sphaghetti code.

Posting anon for obvious reasons.

[Anonymous Coward]

Re: Altea crashed? What a surprise

As my mate from the States years ago said about ntp when we was setting up IRC server links:

"It's beard code"

So you have a beard?

[ScarabMonkey]

A second bug has been found in leap!

[Destroy All Monsters]

Antivaxers and Y2K deniers

One by-product of that was that they temporarily took down air travellers in Australia. The Altea reservation and departure system run by Amadeus, one of the largest computer travel reservation systems on the planet, couldn’t cope and crashed. For 48 minutes, passengers and staff at Qantas and Virgin Australia were thrown back into the 1990s world of manual check-in and delayed flights.

To be used the next time some snide arsehole come on to you and pretends the Y2K effort was all hype and smoke.

Then punch him in the face to set his clock straight.

[Ken Hagan]

Re: Antivaxers and Y2K deniers

Any departures and reservations system that couldn't cope with Y2K would have spent most of the final few months of 1999 increasingly unable to accept "new" bookings. The same goes for most other time-dependent software. If you are tracking time, you usually need to be able to handle the near-future as well as the present or the recent past. Y2K was never likely to result in a midnight shutdown and always likely to be a case of systems showing their inadequacy a (short) while before they became totally unusable.

In addition, the vast majority of genuine Y2K bugs could be easily tested for in advance, once it had occurred to you to do so, just as it is already possible to test systems for leap-second compliance or Y2038 compliance.

Y2K wasn't *all* hype and smoke, but Gartner's 11-digit dollar estimate for them to solve the problem most certainly was, and they weren't alone in brazenly trying to cash in.

[yoganmahew]

Re: Antivaxers and Y2K deniers

@Ken Hagan

"Any departures and reservations system that couldn't cope with Y2K would have spent most of the final few months of 1999 increasingly unable to accept "new" bookings."

Eh, that's why most of us spent 1998 doing our Y2K work and were complete with a system run up to the leap year in 2000 (which was where we had to put most of the fixes in) by the middle of 1998.

In any case, most airline systems then (and many now) used PARS dates, so booking would not have been affected. It was the rollover and the leap year that gave most pause for thought.

[Anonymous Coward]

Anybody can shed any light?

Now I'm left wondering how none of my systems crashed or visibly misbehaved back in 2012. When had that bug been introduced?

[Anonymous Coward]

« Prior to 2012, the last time a leap second was deployed was in 2005 – twice. »

Beg your pardon?

[Stevie]

Bah!

"No, the Linux leap second bug WON'T crash the web"

So Gates and Jobs wasted all that money in their secret joint effort "Project Global Spin-Down?"

[P. Lee]

Re: Bah!

>So Gates and Jobs wasted all that money in their secret joint effort "Project Global Spin-Down?"

I think they picked the wrong strategy if, "Linux has a problem, the internet is going down!" is the headline.

[Henry Wertz 1]

Thanks Paul Crawford!

Thanks for the info! I was going to ask, if there's a crash related to the clock jumping, why doesn't my system crash then whenever I power up a system, the clock's a bit off, and ntp resets the clock (or indeed any time one rolls the clock forward or back)? But, the info you give makes it clear -- this in-kernel leap second handler that NTP calls was presumably buggy in in 2.2.whatever to 3.3.

I have a MythTV system running 2.6.38-gentoo-r3 so presuambly June 30 it'll be reboot time! 8-)

[Steve Foster]

Bah

All these minor system concerns around airline booking systems, check-in desks and so on and so forth, and yet no-one has thought to ask whether the SPB have tested LOHAN's systems for leap second bugs. Let's face it - if NASA's boffins decide to batten down the hatches, the question is are SPB up for the challenge? (presumably NASA having a quiet lie down gives LOHAN a reasonable launch window!)

[Scroticus Canis]

"high-power telescopic rifle" - ¿Meep?

Where can I get a rifle that is telescopic? Barrel extends and collapses (for longer or shorter range I presume)? Breach also longer for shorter rounds? Hell of an interesting technology for high-powered rifle methinks.

[LesB]

More bad reporting

I'm sure "Computing" used to be a specialist IT publication[1], so I was a little alarmed to see their take on this subject:

FUD

[1] In the days when it was 60%[2] job adverts

[2] 83% of statistics are made up on the spot

[tr00per87]

the freeware didn’t recognise leap seconds

I don't know, what the author meant, but freeware is a software distribution model and has nothing to do with Linux kernel.