There's Huawei too many vulns in Chinese giant's firmware: Bug hunters slam pisspoor code Huawei, the Chinese manufacturing giant targeted by the Trump administration as a national security threat, has some of the least secure networking products in the industry, according to Finite State. The US-based IoT-security outfit said it analyzed more than 1.5 million files associated with 9,936 firmware images linked to …
"We take our users security seriously" could actually be a selling point if even one manufacture actually did that for real. "Yes we charge more, but we actively look for vulns and provide real fixes for them for the lifetime of the device".
The problem, of course, is do the customers actually care enough about security to pay the extra for such hardware? Commercial operations answering to shareholders who want profits at any cost and Government buys who are more or less forced to always go with the lowest bidder don't help.
If other news reports are accurate, the CN gov has some pretty good hacking resources. Leaving CVEs unpatched is a sure way to call attention to your products. CVE by definition means they're documented, and thus can be found fairly easily.
Therefore, it seems to be a liability for said state, not an opportunity. Unless it's a decoy to hide the actual backdoors said smart hackers implemented.
If the firmware images go back to about 14 years this rises a question how Finite State chooses which firmware image to test.
Did the test against the latest version of a firmware for a device or did they test the 14 year old version as well?
Did the test firmware of devices which are out of support (i.e. 7+ years old)?
Did they differ between devices which are on the perimeter or which are, normally, only used within LANs?
Finite State could go ahead and check all 3com firmware images for security flaws, especially for devices which have the 3com logo. But you know, since they were officially bought in 2010 by HP, they shouldn't have any flaw since HP fixed every code plunder...
Exactly, the whole thing sounds very unprofessional.
And if a security company can't differentiate between a security vulnerability (that could be exploited to gain remote access) and a backdoor (deliberately placed in the code to provide convenient access that bypasses the security), I would respectfully suggest they are in the wrong business.
You yourself say the difference between a backdoor and a security vulnerability is the intention of the person who wrote the code?
How can you know the intention of the author just by looking at code?
You might make your backdoor a hard coded password and you might say something like that is obviously backdoor rather than a simple mistake. However I can see a lot of devs putting something like a hard coded password in "just for testing" and forgetting to remove it later. Yes that code is a deliberate backdoor but it was never supposed to make it into production. Maybe the original author doesn't even remember the password!
On the other hand you might make your backdoor an out of date library you know you can exploit later if you need to, making a simple "security vulnerability" your backdoor.
Any "deliberate" backdoor like a hard coded password is going to be rather easy to spot while making your backdoor hide among one of hundreds of other genuine coding mistakes is much more effective and gives you plenty of plausible deniability... "We're not malicious, just incompetent."
A vulnerability that you know about in software, even if it isn't your software, can be treated as a backdoor. But I'd say that the true "backdoor" is one placed by the programmer on behalf of a specific community, for instance themselves, and usually with its own key, although maybe a hardcoded password or whatever. Including vulnerable third-party code means that anyone can break in, and you don't want a back door that everyone can use.
"Including vulnerable third-party code means that anyone can break in, and you don't want a back door that everyone can use."
The 3rd party code was just an example but I'd say a backdoor that everyone can use is exactly what you want. Otherwise its bleeding obvious who has been doing the hacking. Also if your sloppy code backdoor is ever discovered you can patch it and move on to another one while looking like your fixing things.
If your hardcoded password or key is discovered you're going to have a lot of explaining to do and your going to have to release a patch that removes it. How are you going to keep hacking into your kit now? It's not likely you'll get away with removing one hardcoded password while inserting another at the same time.
NB I'm not accusing Huawei of including backdoors through deliberate sloppy coding. Just pointing out that differentiating between a security vulnerability and a deliberate backdoor isn't straightforward especially if the code is riddled with security vulnerabilities.
Also where it says they didn't port security fixes back to older binaries. I mean do they actually mean that older firmware isn't updated to include new fixes? That doesn't make sense. Unless the firmware is branched then only the latest supported firmware will contain the fixes, not many people fix old versions of firmware.
@Jou (Mxyzptlk): "If the firmware images go back to about 14 years this rises a question how Finite State chooses which firmware image to test"
[Disclaimer: I have no axe to grind, I have no incentive to protect Huawei or Finite State, I have not verified the methodology properly. I have skimmed the actual report, not only El Reg's summary.]
Finite State do say they focused on the most recent firmware versions of the devices they analysed. Methodologically, I do not see statistics on, say, how much better or worse the recent firmware versions fare than 14 year old ones - and maybe Finite State could do such an analysis. There are, however, examples of pairwise comparisons (that look appropriate) of the most recent vs. older firmware for a particular Huawei device. There is also an example of a comparison between Huawei, Arista, and Juniper switches (that should be relevant for the "and Cisco aren't any better" argument).
Finite State conclude that (in the example study at least) the most recent Huawei firmware is actually worse than the older version, while Juniper are better than Arista who are better than Huawei (and there are whole categories of security problems where Huawei score very poorly while Juniper and Arista look very well indeed). I am a bit baffled by Finite State's way of presenting the results (they use percentages compared to the worst cases they have ever seen), but I suppose it may be OK for relative assessments.
Another big point is the finding that Huawei apparently use outdated components and outdated SW versions known to be vulnerable. It is not clear to me how that affects the "past 14 years" detail that seems to generate a knee-jerk reaction in this forum. I am not making any argument, mind you, except pointing out that "14 years ago is irrelevant because things have changed" is an assumption, not a fact, and the report seems to indicate big flaws in that assumption.
In any case, I have not done any serious analysis myself, but I wouldn't be too quick in dismissing the report based on a mention of "going back 14 years" in a summary article.
"Across our entire Huawei firmware dataset, the average age of thirdparty components in the latest firmware versions was 5.36 years."
There are several references in the Finite report that states latest firmware with old unpatched source... read the finite article before making any statements.
With our IoT future this is of great concern and should be taken as such.
Unsupported kit can still be running the latest firmware, however it's unsupported so no longer being maintained.
I have an 15 year old switch from Cisco that is unsupported and the latest firmware available was available in 2006. I'm pretty sure there are some vulnerabilities in that "latest piece of firmware" that aren't fixed.
I would like to see this type of thorough analysis performed on other manufacturers and their products. Not that I doubt the findings here, but problems like this are critical and something needs to be done about them wherever they may be. No manufacturer should be left out of this investigation; it matters little if a bug was introduced deliberately or accidentally if it is used by a malicious party. Whatever your view on Huawei and the American government, this situation is very bad.
"I would like to see this type of thorough analysis performed on other manufacturers and their products."
Indeed. The claim is that Huawei is worse than everyone else, but according to the article here the dataset consisted only of Huawei files. It seems difficult to make a sensible comparison between vendors if you only look at one of them.
That is not true. There were points where three manufacturers were compared, and Huawei lost in that comparison somewhat badly. I am more than comfortable assigning the "bad practices used" label to Huawei from this report. My concern is that other manufacturers may also deserve this label, and I'd like to see it assigned out to all who deserve it.
No hard and fast comparison with other manufacturers, and analysing a whole back catalogue of devices. Don’t really need to read further to form my opinion on this “research”.
Such results would be entirely consistent with a company that had developed world-class security standards over the last few years, for example.
Maybe so, but at least this time it is based on actual data - whether you like the data or not, it is not just an article accusing without any basis.
Bloomberg did the FUD and never backed it up. Here is proof - of something. There are obviously many people here that can analyze this way better than me, but I'm just glad that someone has made an actual study based on actual data, and not just ran screeching through the streets that Huawei is dangerous.
I sort of agree, but the methodology, or lack there of, and the seeming lack of understanding of the difference between a security vulnerability and a backdoor do little to give the report any semblance of credibility.
The final part of the article is pretty damning, but the first part of the article just opens Finite up to ridicule.
If you know what your looking at (at the bottom of the finite article) then you know that it is pretty damning as for their methodology I am very interested. What they claim they have done and the time frames mentioned are extremely impressive and I would understand why they are not sharing too much (other than the output of their system). I don't know of any system out there that you can purchase that can do what they claim they can in an automated way.
If true the saying "light years ahead" would be what I describe their automated system is.... I've used many in the past free/online, purchased proprietary ones and there are none that I now of that do what they claim their system does in an automated way. So although it is hard to know for sure their methodology but I can say with confidence that is it definitely within the realms of possibility and I am jealous.
To paraphrase the article:
OMG we tested 14 year old firmware and found that it used (gasp) 15 year old libraries!!! In the intervening years, those libraries have been found to have contained bugs which have since been fixed but the firmware build 14 years ago didn't include the bug fixes from last year!!!!!!!!!!11!
Yeah, no shit.
"at least this time it is based on actual data"
The bugs probably are. But the "conclusions" drawn from them are not. How are the bugs distributed in time? Are the more recent versions any better than the early? How does it compare with the bug rate from other suppliers? Without that data they are just fud on a par with "5G kills trees".
A previous employer decided to add low end Huawei kit to his catalogue. A week later I had a L3 switch sent back that would drop connectivity on trunked ports after a being up for around a day. To their credit a firmware patch was ready the day after talking to their support, but how they shipped firmware that bad in the first place is beyond me.
I highly doubt anything malicious is going on at all and any 'backdoors' are unintentional.
Oh well ...
https://threatpost.com/cisco-warns-of-critical-flaws-in-data-center-network-manager/146050/
"Cisco Warns of Critical Flaws in Data Center Network Manager"
"This error means that an unauthenticated, remote attacker would be able to send specially crafted data to a specific web servlet that is available on affected devices, thus creating arbitrary files on the underlying DCNM filesystem. ... A successful exploit could allow the attacker to write arbitrary files on the filesystem and execute code with root privileges on the affected device.”
...
"The other critical vulnerability (CVE-2019-1619) is an authentication bypass flaw in the DCNM management interface, that could allow an unauthenticated, remote attacker to “bypass authentication and execute arbitrary actions with administrative privileges on an affected device,”
...
Because without that benchmark they could be better than the rest.
It's not good in an absolute sense.
Do any other mfgs come off better? Worse? About the same?
When I see stuff like this now I say "Caveat emptor."
Who bankrolled this? Why?
If you read the report, they did also compare it to a Juniper and another device, but only 2 devices were mentioned, and it wasn't stated whether they also had 14 years worth of code to plough through for those manufacturers...
Does it really matter if they are better or not than others, the Brits have more than one report on the matter and it iterates several times about engineering practices and things aren't getting better. Does anyone understand what a "smart" future means regarding these type of vulnerabilities... green lights in ever direction at intersections, Baltimore (need I say more, https://en.wikipedia.org/wiki/2019_Baltimore_ransomware_attack), banking systems taken out over night, power grids, trains, plains and automobiles.... never mind someone hacking your facebook type stuff is of negative concern. I don't care who makes it, even the guy sitting beside me, all critical infrastructure technology must be audited by 3rd party so these tech companies better be ready to hand over source or no critical infrastructure contract at all, that is my opinion and full disclosure I do white/black box auditing but having seen some of the stuff I have seen is why I feel this way.
Imagine buying a car without a crash test rating, black/grey/white box is the technology equivalent of crash test and if the auto industry is forced to do this before they can even advertise a model then all critical infrastructure appliances/IoT/computer/digital/electronic/telecoms.. must be audited.
had at least one potential backdoor, which is to say a serious or critical CVE-assigned flaw that an attacker may be able to exploit.
Sorry, a CVE-assigned flaw is not a backdoor! Exploitation of a vulnerability might gain access to the system, but it is not a backdoor, which is a deliberately coded way of bypassing security to gain access to the system.
I could believe that the code contains vulnerabilities and the coding quality is poor - and at least this is evidence that there are problems with Huawei firmware, but conflating a security vulnerability with a back-door is just fuelling the fake news cycle coming out of the US Government over this issue and actually makes the story sound more of a plant than a real revelation.
If they had stuck to facts and vulnerabilities, it would be much more believable, but calling all security vulnerabilities backdoors removes all credibility from the Finite State report. If a security company can't tell the difference between a backdoor and a security vulnerability, they really should look for a new business model!
So, are these real backdoors or are these security vulnerabilities that could be used to gain remote access to the kit? The wording is so wishy-washy that it doesn't sound like it is coming from a serious security company.
Edit: Also, they looked at code for images for the last 14 years. If they are looking at code for the last 14 years, of course they are going to find vulnerabilities that aren't patched! They should be looking at the current (supported) versions of the code to see what is still unpatched.
Or they should be using that last 14 years' worth of code to understand how quickly and reliably (or not) Huawei actively fix those problems.
This sounds like very shoddy work, somehow, unless they have a better statement for their methodology.
You see...
"These vulnerabilities manifested in the form of hard-coded, default user accounts and passwords, and several types of embedded cryptographic keys."
That sounds more like it. So why conflate that with "normal" security vulnerabilities?
But given that they allegedly "stole" the code from Cisco and Cisco has spent the last 18 months removing dozens of backdoors from its code, it probably isn't a great surprise. It would be interesting to know if those backdoors use the same credentials as Cisco's code did...
The rest of the report is pretty damning on the quality of the code, so why make yourself look ridiculous and diminish your credibility with the above?
contained one or more default credentials, with 227 having a default password for the root user.
And then they start obfuscating again. Default password? Where is the problem? Hard coded is a problem, providing a default and making/letting the user change it is something different - and a large telco or data center that doesn't change the default password on a device deserves all the ridicule it can get.
The problem is the devices are not supported if you change certain credentials, so they are left at defaults to stay in support/cut down on the need for the magical mystery tour that is tracking down how to change it in all the matching api's and interfaces also hard coded and change it there.
What is the point of examining ancient firmware of just one company?
Now if they did some real research they would examine the work of a range of similar products from different manufacturers, comparing their initial quality, current quality and most importantly, code development trends.
However, if they did that they probably would annoy their paymaster.
Never once did Finite state that they were using 14 year old firmware, they stated:
"It includes firmware images for legacy and current products spanning 14 years."
But regarding old source in new things:
" In some cases, engineers chose to use 20-year-old versions of software libraries rather than current, secure alternatives."
"Huawei practices abysmal software configuration management as demonstrated by their use of 79 distinct versions of the OpenSSL library across their most recent firmware releases. In some cases, Huawei used 10-year-old versions of libraries containing dozens of vulnerabilities rather than selecting newer, more secure options."
"Across our entire Huawei firmware dataset, the average age of thirdparty components in the latest firmware versions was 5.36 years."
"There were thousands of instances of components that are more than 10 years old."
"We investigated the latest version of firmware available for the Huawei AR1200 series enterprise
router, V200R007C00SPCc00, and found an average component age of 12.8 years across the tested components at the time the firmware was released. The AR1200 firmware used in this example was released April 13, 2017."
At least this was a quick search on this matter. Please read the Finite report before commenting.
"It includes firmware images for legacy and current products spanning 14 years." So they did state 14 years then, just not specifically firmware, products or both.
It does not state if the 20 year old version had vulnerabilities, just that it's old. "engineers chose to use 20-year-old versions" How do IS know that the engineers 'chose' to use old versions? What if they are looking at something from 2018 that had no alternative until 2019 and the engineers had no choice or had a low risk vulnerability in a component that controlled how long a red led light blinks?
"10-year-old versions of libraries containing dozens of vulnerabilities" but they do not state if those libraries were used in old legacy firmware, products or new. Old can be superseded by new firmware, software released at a later date due to.... shock horror.. old versions of libraries in their legacy firmware and hardware.
"component age of 12.8 years" They don't state what component. CPU? USB Port? resistor? Some components can be old.
Report is basically, We tested some things and found old stuff, old stuff = bad, therefore what we tested = bad.
Every bit of complex software will have vulnerabilities, show me one major Networking brand that hasn't. The rest of their kit just has vulnerabilities that haven't been discovered yet. That's the nature of modern software engineering .
You've never worked on a DO-178C project then!
As always it is a balance between provably good software, time and cost. In the commercial world, cost and time are king, whilst in the aerospace world, provably good is king.
Cisco, Juniper, et al, have found that public opinion is pushing quality more and they are fixing issues which is a good thing (note that I didn't say improving quality, as that can only be designed in from the start of the project.)
I led the production of code for a network switch used on aircraft (and spacecraft) which was subjected to 10's of man-years of reviews and penetration testing (plus years of automated, high volume traffic testing). But the cost of that meant that it was only ever going to be a niche product, whilst the rest of the product range (without such extensive work and mainly open source software) was much cheaper.
While Finance Departments do not see networking infrastructure as critical, they will never pay for the top quality product and the manufacturers will not offer it!
Yes everybody is fixing issues. The reason is that no kit is bug free, including your network switch used on aircraft.
I'm sure there was lots of testing and penetration attempts, however new attack vectors come out all the time. It just takes some CPU predictive branching issue to realise that you can force look ahead code into the execution of the current process and bam you have a vulnerability.
If you think that your switch was 100% bug and vulnerability free then you are not a suitable lead for that project as that is far too naive. You can make sure that something isn't vulnerable to known attack vectors and you can harden against that, but to think you have produced a piece of kit that it is impossible to have a bug or vulnerability in, then you are sorely deluded. You signed off on software version 1.0 and never needed to revisit it again? Really?
Even products that never have a vulnerability found means nothing, it just means that they haven't been discovered yet or that they are so niche no one is regularly trying.
In many ways this goes against the idea that Huawei is some sort of nefarious organisation in league with the Chinese state. Any backdoor that can be discovered through simple scans, is not a backdoor, its a sign of poor security standards and lackluster programming control.
That is not to say that there could not be true backdoors hidden in the code, however its hard to to balance the idea that somehow Huawei are these super programmers capable of stealing all of the Wests IP traffic with basically pretty awful coding methods.
Also the vulnerabilities (not backdoors) are only potential issues. It does not actually go to the issue of what someone could do with the vulnerability. It could alolow root control, but juts as easily it could just allow access to low level log files
I don't think these are backdoors, the Chinese military is better than that, but let's look at a few possibilities in general.
If I want to embed a backdoor into something but not get caught, I have a few options. I could do the standard hard-coded credential backdoor. This has to go unnoticed by the public. If it is seen, it can be tracked to me depending on how much the company wants to protect me. A patch will be demanded to remove the credential, and after that's installed, I'm stuck. I might instead choose to use some libraries I know I can break into. I'd use the latest version with the vulnerability I want, and I'd probably leave a few different ones open. I'd make the access mechanism complex so people can't easily stumble on the way in, but this mechanism lets me have deniability because I can play the "incompetence and not malice" card. It also lets me patch one of my vulnerabilities and maybe get away with leaving another one open. It does take more programming skill to implement this well.
That's how backdoors work. The reason I don't think these are deliberate is because coding standards are so bad. If they were in the middle, I'd have some suspicion, but nobody needs openSSL from 1999 to get a backdoor and that's just calling attention to problems. However, there's one more thing to consider.
If you were the Chinese government, and you wanted a backdoor in Huawei equipment, and the company didn't already have one for you and wasn't planning one for you, what would you do? This would be my plan: I'd get a PLA programmer employed at Huawei. The person I chose would be very skilled and knowledgeable about the type of equipment. If possible, I'd train them on Huawei source code, to which I assume the Chinese government has easy access from a government contract or having broken Huawei's corporate security. This person would then insert some carefully crafted vulnerabilities into the code for the devices. Nobody will notice internally; they're letting obviously insecure libraries through. When libraries are updated, this code can remain for quite a while, being disguised by the unintentional vulns left in by poor coding. This would also be harder to detect because so much focus is being placed on understanding all the rest of the codebase that my relatively small addition can last a while without being questioned.
can't upvote this enough
The investigation by GCHQ already revealed that they thought the Huawei code was not very well written, so this report is hardly news, and I trust GCHQs opinion more than I do of this security consultants firm testing firmware images from over 10 years ago
basically they are not being used as spies by the PRC as they dont have the skills, they are just shoddy coders, and the PRC may be using tknowledge of that to access, but so could the good old NS of A and the HQ of GC
The Australian Signals Directorate came to the same conclusion as GCHQ after doing red team testing of Huawei gear. I really wonder if the luddite politicians just seized on terms in those reports. eg the report may have said "Software has multiple known vulnerabilities that could be used as a backdoor" and the pollies just concluded "Huawei's software has got back doors in it, must be the PRC governments fault!"
They carried out their analysis using 'Finite States automated system' US company selling their service and product in the US and I assume their target market will be the US Gov. Very convenient timing for the release of their document, no idea who they were and now we're all talking about them, their marketing team should be expecting a hefty pay rise this year.
Yeah without a comparitor all this tells us is that over a period of years one manufacturer has poor development practices. We can't tell if it is in fact still better than all the others because Huawei might be bad but they are worse. Could be the complete opposite.
In terms of buying stuff this is of little help.
It would be fantastic if these mega corps could provide the code quality of OpenBSD. Maybe they should be paying OpenBSD devs to optimize for their devices? I know no software is error free but come on mega corps this is what you do, your hardware is only as good as the drivers and software that enable its functionality.
One expects these shoddy shortcomings from profit-oriented companies. But from idealistic communist enterprises surely not? Actually, more likely than not.
These findings seem to support earlier findings from the British monitoring of Huawei, of shoddy software. Perhaps all the good programmers get to work for elite hacking teams, and only the rubbish have to find "commercial" jobs.
Mueller report has 400+ pages. For Zaphod, purpose of whole thing is to conclude "There was no collusion." ...Although it does not say that, and conclusion of whole report is - quite strange, at least.
Now we know with what paper will Zaphod wave at G20 summit; he knows better then all of us what is all really about.
Short version: OK, you've got your evidence of "new weapon of mass destruction", go start war now with China, win your 2020 election.
We are just collaterals.
Sometimes it feels like a really cheesy reality show. Finite state in deed.
The timing of its release, two days before the G20 summit coming up this weekend. Although the technical aspects of this report do look damning and having domain knowledge in this deep down the rabbit hole domain I can say that it is indeed very interesting this happens before a world economic summit... the timing couldn't be better/worst (depending on what side you sit on).
Previously worked with a Chinese OEM, not Huawei, but not a small player either.
The official SDK supplied to use was mostly cracked software, some of it already infected with viruses, just installing the SDK sent our AV off on one, and further investigation showed it wasn't even false positives. Standard library code that was linked to by the compiler was clearly hacked too. Even the remote debugger software once installed wasn't just transmitting debug stuff from the PC to device but every single keypress, including credentials across the network.
We ended up hacking up our own toolchain instead.
This was going into medical equipment.
Was all under NDA so can't say anything else, but this is only the tip of the iceberg.
On the one hand, I was really impressed that my 2 year old Asus router was getting firmware updates for security issues every 2 or 3 months.On the other hand, how crappy was the original firmware? Now I'm just worried because they haven't updated the firmware in over 5 months - is it really good to go or have they just stopped supporting my model?
They report hard coded credentials as a vulnerability, but then later in the article it repeats the information with the word "default" in front of it and suggests that they are classifying a default password is a back door.
It suggests that there might be examples where there is an unchangeable password that will always allow access, however given that they lump that category in as being the same thing as a default configuration they aren't very clear on if they have actual examples of this (Worth adding: we know Cisco have done this...).
Is a router which by default has no password and just drops you straight in to a terminal more secure than one with a default password that can easily be googled? I would say that these are the same... If you are talking about default configurations, what else are you going to do? We are talking about enterprise kit, so the idea of every device shipping with a random default password printed on the label isn't really going to be something their customers appreciate...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
