Leave your admin interface's TLS cert and private key in your router firmware in 2020? Just Netgear things

Netgear left in its router firmware key ingredients needed to intercept and tamper with secure connections to its equipment's web-based admin interfaces. Specifically, valid, signed TLS certificates with private keys were embedded in the software, which was available to download for free by anyone, and also shipped with …

[Paul Crawford]

Difficult position

Sure you ought to have a responsible disclosure approach, but when big companies have shit security and a bug process that allows them to prohibit disclosure of how shit, what do they do?

It sucks for folk with said routers, and maybe it would make a dent in business longer term, but what is needed is some legal process to punish companies producing shit software so they have to fix things and not just suppress reporting because of an NDA clause in their bounty program.

Flip side is its getting really hard to buy anything that is not a complete shit-show, so consumer choice to punish the guilty is a bit of a Hobson's.

[Nate Amsden]

confusing article

Been doing internet stuff for about 25 years now and this article is quite confusing. It sort of implies that Netgear is shipping a certificate authority and private key that is trusted by the browsers in their firmware which would be bad(I think that is unlikely).. but it also sort of implies it is simply shipping a regular (signed by a real CA not a self signed cert)SSL cert+key(because without the key the cert can't be used to serve data) that is valid for this "routerlogin" domain in their firmware. It's probably not the best idea but to me it's far from something to freak out about. I'd assume this cert is only valid for that "routerlogin" domain so in order to do something bad with it you'd need to trick your target into thinking your IP is for that domain somehow, in which case well you can use just about any domain or cert that's valid. I can certainly understand why they did it this way vs using a self signed cert to make it easier for the users.

I learned just a couple months ago that Chrome is even more picky about SSL. I have been self signing internal certs for 15 years now(I don't use chrome) and saw some screenshots from users recently showing Chrome flagging secure sites as "not secure", after investigating determined that Chrome wanted to see some extra fields populated (authorityKeyIdentifier, basicConstraints keyUsage, subjectAltName) in order for it to think it is "secure". Whatever.

Browsers are entirely too extreme with that kind of thing.

(for reference my home "router" is a PC Engines apu2 running OpenBSD. I have a Motorola cable modem in bridging mode, and my local wifi is provided by an Asus AP in "AP" mode hanging off one of the ports of my firewall I guess you could call it "DMZ" as it has no direct access to my internal network).

[Anonymous Coward]

Re: confusing article

"I learned just a couple months ago that Chrome is even more picky about SSL"

Depends on what you are looking at with it. In terms of it checking it's validity, chrome doesn't give a crap, it doesn't check revocation lists, slows things down.

But in regards to the problem, I don't see really the problem either. What are they going to do with it?

[talk_is_cheap]

Just move to Origin Broadband

It is going to depend on what devices this hole is found on. The one they document seems to be a high end Smart WiFi Router. As the hole would allow the spoofing of the router you could grab the admin details. With these you could change setting on the router - so wifi access rights, firewall setting and VPN settings could open up wider security issues within any environment. Beyond that you have all the NETGEAR tools that turn the router into a file store and backup solution.

So to answer your question - with these certs what can't they do to a device and the network it is controlling and defending.

[Anonymous Coward]

Re: Just move to Origin Broadband

But they could do that without the cert. For this to work, you need to be on the WiFi or physical connection. So you need to be able to login. If it's a public connection, or shared business, admin access wouldn't be on the WiFi connection, it would be on the physical port, so would need to be in the building. Then you need to get them to route though you. Which again in a business should be monitoring mac changes / port jumping and ip conflicts.

If it's at home then how did they get on your network, that's a little more problematic.

If it's over the Internet, that a bigger problem, admin access should be denied there.

If it's another access point, that can be done without extracting the cert to get the credentials.

[A random security guy]

Re: confusing article

The article IS confusing. My inference is that this is what happened:

1. The TLS cert and its private key are embedded in the firmware.

2. The certificate+private key is used to authenticate a Netgear router to a browser that it is indeed a Netgear device you are talking to.

3. The connection is secure.

What the developers seem to have forgotten was:

a. The cert+private key can be extracted from the firmware

b. Anyone can now masquerade as a Netgear router and perform a MITM.

c. The same cert+private key are used by ALL their devices (or a rather large set).

Netgear could have chosen to do the following:

1. Use unique certs+private keys

2. Protect private-keys in a TPM or in non-readable OTP on the chip.

God only knows how many ARM chips have Secure areas but never used.

[Anonymous Coward]

"wanted to see some extra fields populated"

Well, it's years the CN alone is no longer enough for a proper certificate.

"Browsers are entirely too extreme with that kind of thing."

It's the contrary - for too long they just accepted everything - and the implied risks.

For example without some of the extra fields above, a certificate can be used to sign other certificates - even when it's not explicitly a CA. Or it can be used where it shouldn't be - i.e. a mail signing certificate used for TLS. When those fields are set, an application can check the certificate not only is "valid" - but is valid for the specific purpose it's being used.

One problem is that OpenSSL is not simple to use and most examples around are just old, simple examples that generates simple, outdated certificates.

[CheesyTheClown]

Re: "wanted to see some extra fields populated"

I’m not sure I agree. I keep a simple bash script on my PC I’ve hacked together which reads a few fields from a JSON file and generates a certificate that makes Chrome happy. It also informs me of all my current certificates that are about to expire. I think I got all the OpenSSL commands within the first page of links on a google search.

I think the problem is that certificates are difficult for pretty much anyone to understand since there are no good X.509 primers our there these days. I still see a lot of enterprise certificates signed by the root CA for the domain. Who the heck actually even has a root private key? I actually wish Chrome would block supporting certificates signed by root CAs. Make a root cert, sign a few subordinates and delete the root CA altogether.

That said, Let’s Encrypt has destroyed the entire PKI since a trusted certificate doesn’t mean anything anymore. A little lock on the browser just means some script kiddy registered a domain.

[Anonymous Coward]

Re: "wanted to see some extra fields populated"

Without looking at your scripts it's impossible to say how good (or bad) they are. Maybe you made the right search, but others may have not or chose easier routes. It is right that X.509 too often looks an arcane art full of strange terms and without good explanations. Too often is just 'do this and that' without an explanation.

You can't ever delete a root CA without making the whole PKI useless! You need it for example to revoke any compromised or no longer used subordinate, signing the CRL. Or to renew them. What you have to do it's keeping it offline an in a safe place.

Setting up a well built PKI requires knowledge and planning. The single CA is the simpler option, hence it's what you see most of the time, especially when it was an afterthought.

[Blazde]

Re: confusing article

It's not quite the same insecurity as using self-signed certs. If a self-signed cert were generated once per-device and private keys actually kept private as they should be (clue's in the name guys), inside each router, then you could blindly trust a cert once on first connection and add an exception but detect shenanigans in future.

With the global private key out in the open you have no confidence what you're connecting to each and every time you initiate a session, and it's right that browsers warn you about that.

[Bronek Kozicki]

"per-device private-public keys"

That should have been the *only* option, really. Generating certificates is not that difficult; the router would have to occasionally use ACME protocol to have the certificate signed (with its own unique name), but that's all there is to it. If the signing authority is Netgear, they might have bent the protocol a bit to allow for non-verified (non-unique and non-verifiable) alt name "routerlogin.net" shared by all routers.

My internal network devices do not have certificates and that's fine - I control what names they appear under on the internal network. Also, I control who connects to the network, so I do not mind the lack of encryption.

[Anonymous Coward]

Re: "per-device private-public keys"

Well, yes, if every router has the same private key for its https interface, it has to be embedded in the firmware so reverse engineering the firmware gives EVERY users' key! A router should always have a unique private key.

[Bronek Kozicki]

Re: "per-device private-public keys"

Actually come to think of it, if they used ACME encryption to create a valid certificate and then "bend the protocol a little" to attach a non-verifiable "routerlogin.net" alt name to it, then anyone could get one. Which is the same end result exactly as in the article.

[brotherelf]

.com is for comfiguration?

Well, there is nothing that stops browsers from implementing relaxed rules for certain TLDs, for example the ones reserved for documentation and internal purposes, in particular when they resolve to RFC1918 IPs. (Chrome has a flag for that for localhost, IIRC.) Then you'd just need the use the domains you're supposed to use for this kind of thing, and you'd be set.

[John Robson]

Re: .com is for comfiguration?

Router.localhost

Really not hard to come up with a non public domain (hint: rfc2606)

[dotdavid]

Re: .com is for comfiguration?

RFC2606 concerns itself with .localhost, .example, .invalid and .test. Perhaps you mean RFC6762?

[John Robson]

Re: .com is for comfiguration?

You'll note that I used localhost...

And referenced 6762 in a later post

Frankly I'd rather they used .invalid to emphasise that what they were doing was not on 'the common internet' to a layperson. 6762 requires mDNS - bleh, you can rely on .local because of it, and therefore use that anyway.

[Rich 2]

Re: .com is for comfiguration?

Despite what the article suggests, I find 19.168.1.1 easier to remember. Why they decided to (badly) bodge a domain name into the thing is beyond me

[Giles C]

Re: .com is for comfiguration?

I think you mean 192.168.1.2

The address you quoted belongs to Ford Motor Company

[Persona]

Re: .com is for comfiguration?

You deserve an upvote for the sarcasm in your comment ....... I just hope it was deliberate.

[mj.jam]

Secure Bootstrap is hard

The problem here is that it is hard to produce a secure bootstrap that is simple enough for a home user.

You want a dedicated certificate/key pair that is provisioned in the factory per unit and signed by the CA.

However this is tied to a specific domain name. So rather than say "connect to 'routerlogin.com' " you have to say, "connect to the address printed on the label" which will be something like 'SERIAL.routerlogin.com' which (assuming the router is acting as the DNS server) resolves to the IP address of the router.

But then the user has to type in that longer address, they will no doubt get it wrong, or fail to understand the instruction.

You can do it as a QR code or similar but then they need to scan that somehow. Waving their router in front of their laptop webcam isn't easy!

[John Robson]

Re: Secure Bootstrap is hard

RFC2606 is a good starting point, or 6762 - then you can just use router.local

[mj.jam]

Re: Secure Bootstrap is hard

mDNS helps a bit, but that only solves the DNS part. You still have to put the domain name in, and the domain still needs to match the certificate if you want to avoid warnings.

Two problems with mDNS.

1. Dodgy support on Android. So for example Chrome on Android doesn't work.

2. It is valid to do both mDNS and DNS for .local domains. You might think this isn't really an issue, but if you have an ISP that changes NXDOMAIN to their ad-filled landing page's IP address, your session to the .local can end up pointing at the wrong place.

[John Robson]

Re: Secure Bootstrap is hard

6762 expressly allows `.local` though - which gives a slightly better domain to use rather than a `.com` which is just wrong.

[theblackhand]

Re: Secure Bootstrap is hard

My reading of what Netgear are doing is trying to get a valid hostname/certificate that allows connections from a browser without security warnings for invalid certificates etc. The .com/.net domains allow Netgear to register valid domain names and get trusted certificates to avoid browser warnings around insecure connections or untrusted certificates.

I'm not sure a different hostname or self-signed certificate where the private keys reside locally on the router is the issue - it puts you back to the point where you need users to click through security warnings to begin router configuration.

I can imagine a process for getting valid certificates generated per device IF you have a working Internet connection (i.e. connect to trusted host using a public key embedded in firmware, register unique DNS name via trusted host and trusted host issues valid certificate, redirect from http://router.local to https://my device name>) HOWEVER:

- what happens if there is no working Internet connection yet?

- will this still cause browser warnings when you connect the first time you use the device?

Given the target market (consumers), I'm not convinced the risk of embedded certificate keys outweighs the advantages of convenience for the user, particularly when the troubleshooting steps would like be accepting untrusted connections which establishes bad habits. Using short certificate lifetimes, separate certificates per device model and auto-updates for firmware would significantly reduce the risk AND allow Netgear to respond quickly to any threats that became public.

Or you ship a setup application that allows you to bypass browser security warnings and is only used if the automated setup fails.

Unless I'm missing an obvious solution...

[Warm Braw]

Re: Secure Bootstrap is hard

It's only hard because https conflates privacy and identity. There are lots of situations on local networks where you would ideally like an encrypted connection but aren't really bothered about the identity of the target being authenticated by a certification authority. Imagine if ssh only supported certificate-based connections: it would be extremely inconvenient in most practical cases.

I'm firmly in the camp that there should be some form of encryption without authentication for http, but there is obviously a strong argument that this would confuse people as to what level of security they were actually getting. I think you could resolve that by using a different protocol name and restricting its use to connections on the local network, especially given that TLS is routinely MITM'd by corporate firewalls and there's no way of notifying that to users. Alternatively, browsers could tone down the security warnings for http if the target was on the local network.

[John Robson]

Re: Secure Bootstrap is hard

http3/ipv6 wil help...

But what's the point in encryption if you don't know if you are talking to Bob or Eve?

There really ought to be a "This is the first time you've seen this self signed certificate, if you are setting up a local device for the first time then check this number against the code on the label [Match][Does not match]"

[Rich 2]

Re: Secure Bootstrap is hard

There really ought to be a "This is the first time you've seen this self signed certificate...."

Like SSH does? I like that idea :-)

[Charles 9]

Re: Secure Bootstrap is hard

STILL too confusing for Joe Stupid, who probably would also lose or misread what's on the label.

[John Robson]

Re: Secure Bootstrap is hard

Joe Stupid wouldn't read the label, they'd just clink on "matches", which would basically always be the right thing to do (you need a local actor on a network which you are in the process of setting up... the window for intercept is tiny).

How often do you actively check that the SSH host key is the one you expected when you first hit a system? I very rarely do.

[Warm Braw]

Re: Secure Bootstrap is hard

But what's the point in encryption if you don't know if you are talking to Bob or Eve?

In the case of a third-party router it doesn't really help to know if you're talking to Bob or Eve since you don't actually know that it isn't passing all the information you provide it back to the mothership. In the event that it isn't, there is some value to ensuring someone else on the local network can't observe your admin password.

All TLS can tell you in that situation is that some authority signed a certificate for the identity claimed by the device. It doesn't tell you that the identity is that of the genuine manufacturer of the device or anything else that you might usefully employ to assess its trustworthiness. If you know how Bob styles himself on his certificate, you can check the certificate to see you're really talking to Bob. If the name on the router certificate matches the label on the box, that's all you know, you don't know who that name belongs to, so if you think that's a form of identification, you're at some risk.

[John Robson]

Re: Secure Bootstrap is hard

Router's talking back to $feared_third_party isn't really the remit of encryption between that device and the end user.

The point was that https without authentication isn't actually doing much...

What's the benefit in encrypting traffic to Eve, over sending the plain text to Bob?

SSH does certificate validation, but it also has a mechanism to say "this is the first time you've spoken to the device, check the fingerprint" (actual checking is optional).

I assert that *that* is what is missing. A mechanism in the browser to say (for self signed certs) "This is the first time you've visited this device, and it has this fingerprint". Ideally the fingerprint would be printed on the SSID label (the little ASCII grid image that SSH kicks out would look nice next to the easy-setup QR code IMHO).

You could even limit it to devices in the RFC1918 address space - anything outside that should probably be using let's encrypt as a minimum standard (other cert authorities exist).

Then each device can generate a cert seeded from its MAC/serial and that fingerprint could easily be printed onto the label - everyone has their own cert, and although it might be technically predictable you could add a model specific random element to the seed to make it challenging to find. You could even insist that the cert is replaceable, so that those who care can replace it when it is eventually compromised (i.e. when the seed inputs and process are discovered/leaked).

[Tom Wood]

Netgear are between a rock and a hard place

I am sure this was not some unknown vulnerability, rather a deliberate decision made by Netgear as the least-worst option.

Even if they did have per-device keys, it would make no difference, unless they also had per-device domain names with associated certificates (e.g. they asked you to visit abcdef123456.routerlogin.net, which is somewhat less user friendly). Having a bunch of different keys and certificates all for the same domain wouldn't give any more security as an attacker could use any one of them in a MITM attack.

I think Netgear probably knew the risks and took the decision to basically allow the key to be compromised as still being preferable to having to tell users to click through browser warnings (neither is secure anyway).

[Anonymous Coward]

Re: Netgear are between a rock and a hard place

Well, they didn't have to use a domain at all. 192.168.1.1 works fine for everyone else...

[Tom Wood]

Re: Netgear are between a rock and a hard place

Firstly, that would also trigger the "insecure" warnings in the user's browser, since nobody would issue a TLS certificate for that IP address.

Secondly, while an IP address might work fine for you, most less savvy users might be rather confused by it.

[Korev]

Re: Netgear are between a rock and a hard place

I imagine that anyone who needs (wants?) to access their router's configuration is already pretty techy - most "normal" people just use whatever their ISP gives them.

[Drone Pilot]

My mom, and millions like her, don't care.

Sadly, they don't.

While they, the non-Reg'-reading-humans don't care about this, the companies won't do anything about it.

Security costs money. They're selling them as low as they can go.

When there is a fine for Stupid, it might change :(

Question: Who buys a router these days anyway. My mom and friends use the one provided by MySafe ISP, el-reg readers, I suspect, use something more.

[theblackhand]

Re: My mom, and millions like her, don't care.

This is basically evolution in the home router space.

To make installing a new router, Netgear (and others) initially directed users to access the router via the IP address. Which worked as long as users didn't have to change the IP address. A DNS name was deemed friendlier and browser makers were discouraging the use of IP addresses for security reasons. Then came the push to move to secure communications, so Netgear used self-signed certificates. And then the move to ensure certificates were valid so Netgear obtained valid certificates. And now the move to avoid distributing public certificate keys.

The bit missing here is how do you opt out of security policies that provide little additional security (in most cases - I'm assuming internal access to a device you own with a default policy of no external access to the device) without encouraging user behaviour that is undesirable (ignoring browser security warnings)?

[Anonymous Coward]

'Who buys a router these days anyway'

Whoever needs something more secure and/or performing, and doesn't want to pay the extortion price some ISPs apply (usually a monthly payment used as a lock-in, as you have to pay the whole sum if you close the contract earlier) for quite low-end models and often with dumbed down firmware.

In Italy it took a ruling by AgCom (the telecommunications authority) to ban ISPs forcing customers to pay and use ISP routers. And telcos fought the rule hard before losing on all fronts.

Evidently the knew not a small number of customers would have bought their own routers. After all companies like AVM and others do sell a lot of models.

[Anonymous Coward]

Quote: "the public should know about these..in order to adequately protect themselves"

That is completely bogus. They publicise because they want to show off and raise their standing in the hacking community, plain and simple.

Users simply do not upgrade their devices when they happen to spot a bug/flaw/vulnerability in the press and then recognise their product name (with the exception of very serious high profile problems) - that would be a very inefficient and unrealible method, to just hope you spot them all. Most users either have a regular upgrade plan, respond to vendor notices or they never do it (patch firmware).

Sadly, most users are of the latter which means by these people exposing the flaw, they immediately make the majority of users of the product vulnerable to attack, and the black hats immediately add the flaw to their databases. The argument that "blackmailing the vendors with a timescale makes them react otherwise they wouldn't issue patches..." is bogus too - again, the risk is far more towards exposing users. Vendors should issue patches, but also shouldn't give details of any vuln fixed because, again, they're giving useful data to black hats on how to hack users who don't upgrade. Blackmail is never justified, especially when the victim is an innocent user, not the slow-reacting vendor.

[Charles 9]

Isn't the common view that the black hats will know the exploit already, regardless of the stance of the vendor (because it's often the black hats that figure out the exploits in the first place), meaning blackmail or legal threat is the only way to get the vendors to do something before word spreads among the black hats?

[John Robson]

And that releasing a patch already identifies the hole, because you look at what has changed.

[IGotOut]

Am I missing something.

My (Plusnet) router uses http and has never warned about an insecure connection (Firefox and just tried Chrome)

Is it because I connect to 192..... rather than a name?

[Tom Wood]

Re: Am I missing something.

You must be using an old browser. I just tried in Firefox and you see a crossed-through padlock icon when visiting 192.168.1.1, hovering over this the tooltip reads "Connection is not secure".

[TeeCee]

I think the blame ought to go to the browser lads for making their products go: "OMFG IS END OF WORLD PANIC PANIC PANIC PANIC" on sight of anything going on over plain old HTTP (even admin shit on your internal LAN), forcing Netgear and their ilk to play silly buggers to get around it.

It's either that or go batshit insane listening to Joe Public wingeing at them '"cos Crom sez iz danger and Crom is right 'cos Google haz kitten pix.".

[Anonymous Coward]

It may be they're forced to do it to keep the lawyers at bay.