Internet's safe-keepers forced to postpone crucial DNSSEC root key signing ceremony – no, not a hacker attack, but because they can't open a safe

The organization that keeps the internet running behind-the-scenes was forced to delay an important update to the global network – because it was locked out of one of its own safes. “During routine administrative maintenance of our Key Management Facility on 11 February, we identified an equipment malfunction,” explained Kim …

[don't you hate it when you lose your account]

Ironic yes

But safer than the cloud by a long shot

[c1ue]

Re: Ironic yes

Iron-ic

[hplasm]

Re: Ironic yes

Cold Iron -keeps out the daemons.

(Or in, in this case...)

[Anonymous Coward]

Presumably the locksmith will enter by the tradesman's entrance round the back?

[Charles 9]

Usually they just cracking the safe open physically and they have to put in for a replacement safe. I can see why they have to wait for Friday, though, if the safe has a safecracker rating of more than a few hours.

[Anonymous Coward]

Yep.

As with cryptographic keys, you cannot make something impossible to break/crack with brute force. You can only make it take too long or be uneconomical.

As safes are generally down to preventing theft, I see most consumer ones are not "safe". They are often able to be cracked/broken/opened in a few seconds let alone minuets if you lose the key.

Real security safes though are down to, as you say above, how long it takes to drill or crack with skill.

[Charles 9]

Re: Yep.

AIUI, most home safes only have a safecracker rating of five minutes. The engineering needed to make a safe with a rating of more than hour puts it out of reach of most people.

[J. Cook]

Re: Yep.

Yep. we had a 'small business' grade safe that we had to hire a locksmith to drill open, because some chucklehead put his size 13's on the handle to try and open it, not realizing that it was double locked. Seems those have an intentional weak spot on the gear train to keep someone from brute-forcing the door open. It took longer for the 'smith to verify where to drill the hole with the manufacturer than to actually pop the thing open.

[Peter2]

Re: Yep.

AIUI, most home safes only have a safecracker rating of five minutes. The engineering needed to make a safe with a rating of more than hour puts it out of reach of most people.

Really?

Because firearms are stored in quite inexpensive boxes (~£75) which are designed to be impossible to gain entry to (even with heavy cutting equipment) in less than half an hour, and the locks are perfectly secure enough for the police to be happy with you storing firearms and ammunition in them which one assumes wouldn't be the case if you could sort of glance at it and find it opens.

If meanwhile a safemaker can for similar money only make something that's good for 5 minutes then something's wrong. I'd suspect that the safe makers are making good money out of their better safes for business use and simply don't want to sell a cheap home grade one that's as good as the more expensive ones to avoid losing money on businesses going for a home grade safe.

[hplasm]

Re: Yep.

"Really?....)

Most Firearm 'safes' can be opened in under 30 seconds, with a bit of string or a twig....

[Peter2]

Re: Yep.

Firearm cabinets made to British Standard BS7558/92 can be broken into in under 30 seconds?

If you can do it in 30 seconds with a twig then i'm sure that there are many laboratories that would love to either hire you, or buy whatever sort of twigs your using since they are obviously better at getting through steel plate than a 2kg sledgehammer applied to a chisel on weak spots.

BS7558/92 requires real world testing, and getting a pass requires multiple cabinets to survive enthusiastically applied attack for considerably longer than 30 seconds. The minimum failure IIRC would be only surviving ten minutes. When attacked with an industrial blowtorch.

[Charles 9]

Re: Yep.

"Firearm cabinets made to British Standard BS7558/92 can be broken into in under 30 seconds?"

Two words: thermic lance. Beating sheer physics would require some serious (say, space shuttle-level) engineering.

[richard?]

Re: Yep.

Obvious quote - "You're only supposed to blow the bloody doors off!"

Thermic lancing it would undoubtably work quickly, and almost certainly destroy the contents so kind of pointless.

As they said in the article, the safe contains "sensitive equipment" - not much point in beating up the safe if it destroys the hardware token in the process.

[jason_derp]

Re: Yep.

"Because firearms are stored in quite inexpensive boxes (~£75) which are designed to be impossible to gain entry to (even with heavy cutting equipment) in less than half an hour ..."

I'm pretty sure firearm safes are meant to deter children in the country I live in, and that's about it. We even have to have special ammo boxes because the safes aren't rated for their safe containment, so I doubt much work went into their locks either given the prices I see.

[phuzz]

Re: Yep.

"As with cryptographic keys, you cannot make something impossible to break/crack with brute force."

Yes, but the difference is, you can't make a safe that will take longer to break into than the entire age of the universe.

[katrinab]

Re: Yep.

If you had "the entire age of the universe" to crack an encryption key, you would spend your time developing faster / better hardware to do the brute force rather than try to do it on existing equipment.

If you had something that 20 years ago was going to take 50 years on an impossibly fast supercomputer to crack, how many milliseconds would it take a raspberry pi to do it today?

[KirkBresniker]

Re: Yep.

CDC7600 ~= 36 MFLOP/s @ >500kW, $5M

Raspberry Pi ~= 1400 MFlops @ 6W, $25

50 * (36/1400) ~= 1.3 years

So that's quite a few milliseconds, but you'd still be better off waiting for the hardware to catch up to the demands of the problem.

[Anonymous Coward]

Re: Yep.

Real security safes though are down to, as you say above, how long it takes to drill or crack with skill.

More or less any large safe can be opened without much skill using a thermic lance, as there aren't any tough materials that can handle a sustained temperature of 4500 °C. You can see it done at https://www.youtube.com/watch?v=UMKBOoAOR7I Small safes aren't vulnerable because the lance will quickly destroy the contents of the safe, along with the safe itself, in the process.

If you need a good safe, you also need a good burglar alarm.

[Tom Chiverton 1]

Re: Yep.

Down side of a lance is everything in the safe being on fire...

[hplasm]

Obligatory-

"the tradesman's entrance round the back"

Oo-er, Missus!!

[Anonymous Coward]

"during what was apparently a check"

So someone can access and open the safes without any oversight by those who should have the only access?

[Prst. V.Jeltz]

Re: "during what was apparently a check"

But during what was apparently a check on the system on Tuesday night – the day before the ceremony planned for 1300 PST (2100 UTC) Wednesday – IANA staff discovered that they couldn’t open one of the two safes. One of the locking mechanisms wouldn’t retract and so the safe stayed stubbornly shut.

Sounds like the right people to me

[diodesign]

Re: "during what was apparently a check"

I think it was more checking they were operational.

FWIW, ICANN has the ability to override protections and literally drill its way into accessing the KSK HSM but it's rather obvious if that were to happen.

The point being that IANA/ICANN staff can check security systems but there are tamper-proof protections and other layers to prevent actual access outside of a ceremony, unless you brute force your way in, which is, shall we say, detectable.

C.

[Anonymous Coward]

Re: "during what was apparently a check"

I understand using a drill in an emergency situation. Even banks are forced to use drills if a safe deposit box can't be accessed for some reason, but that evidently leaves some evidences. Still, no one at the bank can "check" a safe box because the other key is not there.

Don't know what kind of tamper-proof bags they use, and how easy would be to replace them - since they do replace them after the ceremony.

It looks to me there's some security theatre, and the actual security is lower than they try to show.

[John Robson]

Re: "during what was apparently a check"

No, but they can test access to the room security boxes are kept in...

[John 104]

Re: "during what was apparently a check"

@LDS

You're right. They should just put the key pair on a file share and lock it down with ACLs.

[Anonymous Coward]

Re: "during what was apparently a check"

> So someone can access and open the safes without any oversight by those who should have the only access?

How else is the cleaner to get in and dust before the dignitaries arrive?

[Charles 9]

Re: "during what was apparently a check"

You're thinking a vault. Safes by definition ate smaller than that. And good ones are nigh airtight--no way for dust to get in.

[Anonymous Coward]

Re: "during what was apparently a check"

The comment you're replying to was done in sarcasm. ;)

[Charles 9]

Re: "during what was apparently a check"

The problem with sarcasm is that some people will believe you. And due to my network settings, I didn't see the icon then.

[Anonymous Coward]

Re: "during what was apparently a check"

Reminds of my time in KSA back in the early 80's when Khamis Mushayt PRX-205 telephone exchange used to reboot Weds afternoons. This was remotely monitored from Riyadh and we couldn't out what was going on, so all aboard the Phillips Falcon 20 jet to suss it out. At 2pm the cleaner entered and proceeded to open the equipment rear doors and mop the cabling/pins. Quick exit to the local OASIS club - trebles all round!

[Anonymous Coward]

official Safes

bad, bad memories there. One mis-set by ID10T who set numbers by leaning over to extreme side so the number seen was NOT the number in the middle of dial. Strangely enough, no-one could open it using numbers provided. Cue locksmith. Another where two tumblers locked and two unlocked. At least that time the new combination was tested with door open so repairs were possible. Safe was in a remote area so no chance of locksmith but it was extremely beneficial to have a large unit to stash items of high toxicity and other portable attractives.

Lastly, the mechanisms in safes are solidly constructed. Failure to retract suggests it was not unlocked. More likely, one set of tumblers combination was wrong, or worse, key and lock damaged.

[Jellied Eel]

Re: official Safes

Failure to retract suggests it was not unlocked. More likely, one set of tumblers combination was wrong, or worse, key and lock damaged.

Or worse, the correct combination, but for the wrong safe. Which they may discover on opening and finding nothing but a stale fortune cookie. Mind you, safe-swapping may be harder than safe-cracking, assuming the environment has decent security.

[JimboSmith]

Re: official Safes

There's a story about a bloke at the bbc in the 80's who reached a fairly senior position. During one of the government exercises for the transition to war (wintex/fallex) he received a call from the UKWMO (UK Warning and Monitoring Organisation) with the correct codeword "Falsetto". He said he was told to retrieve and prepare the attack warning (all clear, regional warning) cassettes in case they were needed. "Okay" he says "where are they???" They're in the safe in your office is the reply. He asks for the combination and is told he's supposed to know it! However he doesn't so calls his predecessor who will obviously have it. Sadly he says he doesn't know it either but not to worry just look at the picture on the wall behind the desk. "The combination is written on the back of that"

Only problem is that the new occupant had had that picture removed when he moved in as he found it awful.

[Anonymous Coward]

This sounds like something out of Mission Impossible!

Ethan Hawk has to break into IANA to allow his disapproved mates to spoof a master criminal's domain name and steel a bunch of BTC.

... hmm, now that IR35 is here and its pointless being a contractor, I might take up screen writing!

[Anonymous Coward]

The way the word ceremony is used this reminds me of one of those nature specials on birds. You know the ones where the male collects his little offering and shows off his colors in dance. But this time his little offering has no entry, so she must wait :-(

[Anonymous Coward]

Perhaps it might make more sense to call it a ritual rather than a ceremony. A ritual implies a specific procedure that is to be followed as exactly as possible.

[Anonymous Coward]

@AC ritual

indeed. Having done these procedures for a revenue raising gov department back in the day, ritual is an excellent description. Substituting suits and serious faces instead of mangled Gregorian chants and monk habits describes the ritual unlocking of the safe containing the "special" laptop, in front of witnesses from PHB delegate, application owner PM to IT security manager. Then the powering up, the selection of cert type, etc and the solitary creation of the magic passphrase, instantly written down and sealed inside two sealed and signed envelopes while everyone else kept their vision averted and selves across room. Then the key generation, copied to dedicated USB. Once all keys done, laptop off and back into safe, envelopes to another safe elsewhere behind lots of doors few could get past. All understandable and for once in security theater, necessary and sensible. No doubt my private phone and emails were monitored for a while afterward.

[Diogenes]

The way the word ceremony is used this reminds me

of the "ceremony" in The Handmaids Tale

[DJV]

They needed...

...the LockPickingLawyer!

[Paul Eagles]

Re: They needed...

He'd probably get the pick that Bosnian Bill and he made, and get picking.

[Claptrap314]

Re: They needed...

Well...that was a well-spent hour...

[Mike 125]

How long?

>Once the ceremony is complete – which takes a few hours

A few hours? Jeez. Given a choice, I'd poop that party and hit the nuke launch code testing ceremony instead.

[Antron Argaiv]

Re: How long?

Have they changed the code from "00000000" yet?

// Spoiler: Yes, it's now "12345678"

[PM from Hell]

This sounds like a lost opportunity for a DR test

Why didn't they move the ceremony to the back up location on Thursday? This seems exactly the situation that the alternative site was built for.

[Antron Argaiv]

One of the locking mechanisms wouldn’t retract

Let this be a lesson to you young pups:

WD-40 is not a lubricant!

[MudFever]

Re: One of the locking mechanisms wouldn’t retract

But KY Jelly is!

[Anonymous Coward]

"WD-40 is not a lubricant!"

Tell them....

https://www.wd40.com/myths-legends-fun-facts/

<G>

[Antron Argaiv]

Re: "WD-40 is not a lubricant!"

I made the mistake of using it to lubricate a door latch. Which now sticks when it gets cold. Had to flush out the waxy deposit with penetrating oil before it would loosen up.

Now, that being said, WD-40 IS an excellent product for keeping water out of places it doesn't belong, and that waxy deposit it leaves behind seems to be the reason.

I should modify my original statement, to say that WD-40 may be an excellent lubricant in the short term, but over the long term, it leaves deposits which may inhibit the free motion of the parts you were trying to lubricate. I have found it an excellent substance to use when attempting to remove car mufflers from those rubber donuts they mount them with. Loosens up all the dirt and rust and such, and the pin slides right out.

// Yeah, that's a can of WD-40 in the pocket

[Mike 16]

Re: "WD-40 is not a lubricant!"

---

Loosens up all the dirt and rust and such, and the pin slides right out.

---

I have found that "starter fluid" (light oil propelled by ether) is the best aid to rusted bolts etc. Far more effective than WD40, IMHO. Got that trick from a retired locomotive maintenance guy, and been using it ever since, although I haven't been in a steam locomotive cab for over 40 years.

[Scott Wheeler]

Re: "WD-40 is not a lubricant!"

A 50/50 mix of acetone and ATF (automatic transmission fluid) is the best I've used. You need to make a new batch every so often, as the acetone evaporates even from a chemical wash-bottle.

For soaking components, diesel is good as it's cheap and a pretty good penetrating oil.

[Charles 9]

Re: "WD-40 is not a lubricant!"

"I made the mistake of using it to lubricate a door latch. Which now sticks when it gets cold. Had to flush out the waxy deposit with penetrating oil before it would loosen up."

I believe that's why they say to use graphite powder when dealing with door mechanisms. Graphite is a dry lubricant.

[IGotOut]

Re: "WD-40 is not a lubricant!"

#Now, that being said, WD-40 IS an excellent product for keeping water out of places it doesn't belong, and that waxy deposit it leaves behind seems to be the reason."

It almost as if it's a Water Displacement which they got right on the 40th attempt

[Denarius]

Re: "WD-40 is not a lubricant!"

Nice, got a coffee spray. Nice to see someone up on product stories. It is recommended for cleaning glider tow releases,

[Andytug]

Re: One of the locking mechanisms wouldn’t retract

As you find out if you try to use it on the valves of brass musical instruments when you've run out of proper valve oil.

Works brilliantly for 30 seconds, then gums everything up due to the water present in the player's breath......

[SVV]

root key signing ceremony

This is all a little disappointing to me. Where are the bits about the Elders of the Internet in their hooded robes solemnly chanting mystical incantations in binary in order to open the door to the secret inner chamber where terrible secrets lurk, and he who must never be spoken of performs dreadful rites from the Book of the Face in order to prevent the global apocalypse of the untrusted site meltdown?

[Antron Argaiv]

Re: root key signing ceremony

Also one of my first thoughts.

Disappointed the ceremony is not held at the top of Big Ben.

[The First Dave]

Re: root key signing ceremony

_I'm_ disappointed that they don't calculate the keys manually.

[Antron Argaiv]

Re: root key signing ceremony

By flipping coins to create binary values?

[Glen 1]

Re: root key signing ceremony

"Disappointed the ceremony is not held at the top of Big Ben."

That's where it gets the best signal

[John Brown (no body)]

Re: root key signing ceremony

"Disappointed the ceremony is not held at the top of Big Ben."

The Illuminati booked it first. Gotta be quick if you want the best places.

[FBee]

"identical set of equipment on the other coast" ??!

Wait a minute...is this some sort of quantum security?

[Glen 1]

Re: "identical set of equipment on the other coast" ??!

It is and it isnt

[awoze]

Re: "identical set of equipment on the other coast" ??!

That's just putting spin on it.

[Chris Evans]

How long before things would stop working properly?

If this 'signing' has to be done regularly and it doesn't happen for what ever reason what happens?

What if both safes were inaccessible?

The 'override protections' seem to be ways to access the vaults.

I'm surprised there isn't at least a third safe!

With backups, father grandfather son is standard for normal data.

[Mike 16]

Re: How long before things would stop working properly?

---

I'm surprised there isn't at least a third safe!

---

With the current ICANN, I would not be surprised to hear that there was a third safe, and it was in the undersea lair of some fabulously wealthy Bond Villain.

(I also question the premise that "things" are currently working "properly", with the plain-English definition of the word, but that's another thread.)

[Dave 32]

Re: How long before things would stop working properly?

Yeah, it ought to be located on the other side of the world, in case some calamity happens to this side. And, it should be hosted by a country which knows how to implement physical security. Hmm, seems like North Korea would be the ideal location for it.

[Anonymous Coward]

Re: How long before things would stop working properly?

"I'm surprised there isn't at least a third safe!"

There is. A HSM sits in a secure warehouse somewhere, containing an encrypted copy of the KSK with "Recovery Key Share Holders" around the world possessing smartcards (shards) to decrypt it.

If both key management facilities fall into the ocean, 5-of-7 RKSH smartcards and an encrypted KSK smartcard can reconstitute KSK in a new HSM.

[Anonymous Coward]

beware assumptions

The ceremony sees several trusted internet engineers (a minimum of three and up to seven) from across the world ...

Assumes that US CBP will let them in.

[Starace]

Hmm

I just hope they didn't do what matey in the photo was trying and think they can open the door by pulling on the side with the hinges on it.

[Blofeld's Cat]

Er ...

“During routine administrative maintenance of our Key Management Facility on 11 February, we identified an equipment malfunction,”

Translation: The Post-It note on the bottom of a desk drawer came off and disappeared into the nozzle of a vacuum cleaner before anyone noticed.

"We understand, however, that following an emergency meeting on Wednesday, the issue should be fixed by Friday, and the ceremony has now been moved to Saturday."

Translation: There are only two dozen more bin bags and the compactor to search.

[Pete 2]

How about a race?

> the Key Signing Key (KSK): this is a public-private key pair, with the private portion kept locked away by IANA.

Between the NSA and the Chinese. Who can brute-force it quickest. Assuming they both haven't done so, already.

[John 104]

Interesting

Interesting story. Always fun to see how certain elements of securing things are implemented.

And, sigh. I remember El Reg back in the day. If you didn't know what DNS was, they certainly weren't going to explain it to you.

[katrinab]

Re: Interesting

I believe it is "RTFM", whatever that is.

[Anonymous Coward]

DNS != "Internet".

More of a parasitic centralization headcrab on the Internet.

[Anonymous Coward]

Re: DNS != "Internet".

Oh my. The author of that site seems to have far deeper problems than DNS. According to all the thoughty-trainy footnotes, IPv6 is not good, poor people should not have access to the Internet and he beats his "retarded girlfriend" "black and blue" regularly.

I wouldn't trust that man's opinion on anything.

[tip pc]

A site on each us coast, seems like a disaster waiting to happen.

What happens if there is another global catastrophe that stops air travel or destroys both those locations, we are so reliant on dns to make the internet work, dns not working will just compound any disaster and probably cause much More dissent.

Any proper DR would have these systems spread across multiple continents not just 2 sides of 1 continent.

[Charles 9]

Re: A site on each us coast, seems like a disaster waiting to happen.

Um, if both sites on both coasts are destroyed, it's pretty safe to assume something much bigger (like Works War III) is already on and that all bets are off.

[Anonymous Coward]

A hundred years after recovering from the apocalypse...

The governments finally admitted that the Internet was still there.

But all the certificates had expired.

[anonymous boring coward]

Did they try turning the safe off, and then on again?

[John Robson]

Why don’t we take this much care

Over SSL certificates.

Signing an SSL cert with your own CA (or just self signing) and publishing it through dnssec secured DNS really ought to be a thing by now.

[Drone Pilot]

Die Hard 2020 movie

Coming soon to a big screen near you.

Spoiler alert: Plot foiled by the steps which John McClane couldn't get his mobility scooter down.

[Conundrum1885]

Safe fail

I know of someone here who has a safe left by the previous deceased owner.

Needless to say getting into it even with part of the key isn't feasible.

[brotherelf]

I wonder if it's the safe that was already hard to open in KSK39.

Because of course you can watch the videos of the ceremony. It's only mildly more exciting than watching grass grow, but makes a good sleeping aid.