Bad news: Your Cisco switch is a fake and an update borked it. Good news: It wasn't designed to spy on you Two types of fake Cisco switches – discovered after a software upgrade hobbled counterfeit gear at an unidentified IT firm – appear to have been designed for profit rather than espionage. F-Secure Consulting's hardware security team disassembled the unauthorized Cisco Catalyst 2960-X series switches at the IT company's request …
A commentard here coupla years ago recounted how he had personally audited some Huawei routers etc onsite in their factory, a few years previously.
He said they were all entirely running stolen Dell or Cisco code. (Can't remember which, right now)
They bailed on their procurement assessment at that point though; didn't bother going deeper to check the hardware/chips.
A commentard here coupla years ago recounted how he had personally audited some Huawei routers etc onsite in their factory, a few years previously.
He said they were all entirely running stolen Dell or Cisco code.
Would have been even more interesting if they had taken the case off the routers and looked at the boards etc.
Back in the days when they were running licensed Cisco and Dell code?
or 3com - The big presentation on "Huawei vulnerabilities" about 7 years ago was entirely holes in 3ware, which Huawei were running at that point (and all holes were present in 3com kit, even after Huawei ceased using them and dissolved the H3C partnership - some are still present in HP kit
I had a Chinese client ten years ago who were vociferously annoyed about some bent Cisco kit they'd had palmed off on them. It turned up with all its holographic stickers, supposedly from an approved channel vendor, but missing half its innards, so it's not just non-Middle Kingdom firms affected by this.
a few years ago we were supplying some Cisco 4451X routers to a gov dept and at some point someone had decided some of the ports are to be fibre-presented. No problem, we thought, we have some spare unused old-new-stock SFPs in a box and we duly sent some off. Then the manure hit the muck spreader as half of the SFPs were being reported as 'not genuine' and gov security came down hard.
They were all genuine Cisco parts, supplied straight from Cisco, and it turned out that the router's IOS had a 'compatibility' table that was a little out of date. The SFPs were manufactured last week in Dec and 1st week Jan and the Dec ones were flagged as genuine and the ones from a week later were rejected.
There were 2 options... a hidden command to tell the router they were not genuine parts (potentially putting them out of warranty/support) or, as we ended up doing, swap every one we could get our hands on to get enough 'workers' to complete the job.
AFAIK the 'not genuine' ones are working to this day in whatever switch/router we swapped them to
You wrote:
Well, at least it's not Chinese.
But you need to ask yourself where Cisco gear is manufactured...
IIRC, it is China.
per the article...
Trafficking in sham Cisco gear has been an issue for years. The California-based networking giant insists that maintaining the integrity and quality of its products is a top priority
Yes, this is a known and well publicized fact. I want to say at least 20years or longer if memory serves.
There were stories about how a factory would produce the kit, but out the back door you saw fakes being produced...
I'd say that they got better at counterfeiting and had a bit of inside help.
I'm not sure whether to be impressed or concerned when reading this. Someone put in a LOT of effort to discover vulnerabilities, design, test, manufacture and distribute a knockoff product, one with an inherent level of sophistication and complexity. It's amazing that doing this at all is worth the investment of both time and funds, in terms of payback - that's why I don't know how to feel about this.
This has to be the tip of a much larger iceberg, as the cost of developing a knockoff for a single product can't possible pay back that much money.
> Someone put in a LOT of effort to discover vulnerabilities, design, test, manufacture and distribute a knockoff product, one with an inherent level of sophistication and complexity.
I can count on some of the fingers of just one hand the number of countries that would be interested, and willing, to do this.
All that effort is just a further education programme for a country's engineers. Whichever big organisation buys some shiny new gear also budgets for some good engineering analysis of said shiny product and publication of the information obtained.
If you were running a developing country with big ambitions you would likely have introduced a similar system. And after all, you bought the shiny products so you can do what you like with them,
As far back as the early 80's, certain people invested huge amounts of time reversing the security protections on arcade game pcbs with only the use of fairly primitive electronic tools, creating replacements/workarounds for custom ICs etc. To then redesign and manufacture them in a few short months.
I have to admire their skillset, it would have required some very clever people indeed.
Reminds me of all those knockoff Street Fighter II: Champion Edition arcade cabs (they were colloquially known as "Rainbow Edition"). Capcom's fightback plan was to release Street Fighter II: Hyper Fighting for two reasons:
1. they released it as an upgrade PCB for the proper "Champion Edition" PCB, so anyone who bought a knock off couldn't upgrade (it made them have to get a legal copy in order to upgrade)
2. it instantly made "Champion Edition" old hat, so no one would want to buy knock-off "Rainbow Edition" when something new is out there
I'm imagining a certain country would not be happy with probably leaky Cisco gear at the heart of their comms networks. To develop a 'safe' local switch would be imperitive no matter what the cost. Hence this rather clever gear.
Whether that was delivered locally in fake boxes or not - the greedy manufacturer/subcontractor probably skimmed off a load of boards and shipped them out into the world market where it was all nearly profit. That state is probably a bit upset they didn't put a back-door in!
If you look at many "good" knock off products they are made at the same factory as the legitimate parts.
Run the product line for an extra hour each day and sell the extra parts via grey market.
Cisco had significant issues with this with Catalyst 29XX switch's and Cisco 2600/2800 routers (possibly others as well but personally have found grey market equipment including ~10 out of 30 2960Xs that couldn't be upgraded direct from the reseller when applying newer software images. As of IOS15 (released 2012) more hardware and software checks were introduced and started flagging up these parts and I haven't seen grey market Cisco kit in new shipments for at least 5 years.
The surprise is that these switches were running an old enough release to not be detected before 2020.
I am not surprised one bit.
Rather a lot of companies either don't want to go to the hassle and expense of keeping products under support, or can't be arsed to plan the downtime to update the software, or the failsafe excuse of "well, it's working fine, why should be update the firmware on it?"
We had a few devices that we didn't want to update, because they had a very specific and fiddly configuration on them that would break if we updated the firmware. (We also ran into firmware updates breaking actual OEM purchased switch port blades because the blade revision wasn't compatible with the newer microkernel, AND we couldn't run two different revs of the board in the same chassis.)
Fortunately, we ended up replacing them outright with newer units entirely.
THIS in spades
I'm going to call out BT Inet on this: They tried to sell us Cisco kit "at an amaaaazing 85% discount over list" - which was still more expensive than buying it retail from Insight
When we pointed that out, they just repeated the spiel about their discount being amazing and unbeatable
They didn't get the sale. Huawei did - and a large part of that was because Cisco's sales technique consisted of senior sales managers turning up and saying "We're Cisco, you WILL buy our product" - with some implied menace
The complete Huawei cost for more capable kit and 5 year support was significantly less than the Cisco support contract alone
It's all in the numbers - if they made enough of them, then the effort is worth it. If there's no concern about the consequences of dud kit failing at some point, then you don't have the overhead of customer support and quality control to worry about. You need to change your thinking from a Western 'quality-product' / 'brand-loyalty' perspective to a 'we're going to make some money and screw the customer and their business'. I remember reading an article that China were manufacturing fake boiled eggs.
https://www.youtube.com/watch?v=EVnhRDuXGPs
If that doesnt alarm you, look up China Gutter Oil - China has embraced capitalism in its rawest form - time to be afraid.
..if someone's capable to build their own Cisco switches I wonder what else is out there?
A lot. I went to a seminar on this some years ago thinking fakes were mostly about dodgy Rolex and handbags.. Then HMRC & Trading Standards types pointed out they'd also been seizing fake car and aero parts, medicines and anything that could make the makers a quick buck. And detecting those fakes generally relied on information from the legit brand owners given it's not easy to tell a palette of dodgy Cats from real ones. But once detected, then tracing the supply chain back to figure out where it originated. In this case, seems like the user bought the fakes from a reseller, so Cisco will be wanting to know where that reseller sourced that kit.
It was a fascinating and alarming seminar though, and a huge problem for everyone but the fake makers.
Yes, the sheer extent of routine counterfeiting in China is astounding. It is endemic, cultural, and often quite dangerous. Google the counterfeit _milk_ via adding melamine.
Right down to routinely bullshitting about the quality of raw materials. "Auditors" in the physical commodities sector are built like brick shithouses and their auditing tool is a 10foot Pole with a scoop. And they spend their days clambering over ore trains and grain warehouses smashing this pole down into the middle and bottom to get samples of what's _actually_ underneath the concealing top layer or outer layers. Don't bother slitting a grain sack to check what's in it; that sucker has to be torn right open since they routinely fill just the middles with garbage.
Bad news, I'm afraid. LIBOR-frigging (always vanishingly rare because v. difficult; now eliminated) only ever moved intrabank profits around between banks.
No change in the total, no big cultural implications, sorry. And billions? Hoooooo BOY, that must have been a big day on the market.
Reminds me of when I was teaching at uni and we sprung a whole bunch of people cheating on an assignment. Turned out they'd impersonated the senior lecturer to the textbook publisher and got a bunch of instructor guides sent out to a PO Box (paid for by cash). We'd got lazy and set the assignment off a worked answer; they all copied it out. 0% all round, guys.
We'd been marking off student numbers, decided to look up their names.
A-aaaaand 100% of them were Chinese.
And 100% of them came in to protest.
Many got loud, some got abusive, couple attempted physical intimidation (of my female tutors). Fortunately re the latters, I walked in on them mid threats.
But ALL of them were angry.
And stop and think about the attitude and dedication necessarily underpinning the weaselling of the instructor guides via impersonation....
Right down to routinely bullshitting about the quality of raw materials. "Auditors" in the physical commodities sector are built like brick shithouses and their auditing tool is a 10foot Pole with a scoop.
Yup. Problem isn't limited to China either. I had a fascinating client once which did inspections, in which I learned a little about that. Their challenge was getting inspectors onto ships, getting samples, testing, sharing the results with their clients and then deciding to accept or reject the shipment. Which provided several challenges, one being a narrow time window for consignments FOB because once unloaded, it became a much bigger problem for the customer. Main challenge for us was trying to get wayleaves to run fibre into ports, and for extra fun, bonded warehouses.
And that client was the one I think that made me want a spectral analysis of a Marmite sample. Easy to get for other addictive substances like THC, but a challenge still on my bucket list. I keep checking the price of gadgets like Raman and IR spectrometers, but a) they're still expensive and b) I'd probably gunk it up with Marmite.. :)
I think that was one of the examples given for the risks of fake product. Also there was the recent example of an inspector at a Japanese steel company faking test certificates. But refurbs or stuff that failed QA was one of the sources of grey/black market products.. And unless the customer could test, they'd be none the wiser until they failed. Pressure to cut costs also doesn't help.
"Inadequately refurbished used parts are another problem."
A friend of mine bought rotor blades for his Huey from the USA - after a couple hours on the machine they started looking/feeling odd so he pulled them off and had them reinspected
When the paint was removed it was discovered they'd been shot full of holes at close range with a 12-gauge plus folded and straightened - further tracing revealed the blades were an old set of lifetime expired ones which had been scrapped. Someone at the aviation scrapyard had taken the blades, bogged the holes, flattened and cleaned them up, selling them as new
These parts are $40k a pair - and it all happened in the USA
As a result of this discovery, written off helicoptor rotors are routinely put into industrial shredders or cut into small segments to prevent repeats - Helicoptors whose blades fold up mid-flight are colloquially known as "rocks" (and it HAS happened)
You know your shit is too expensive when people go to this much trouble to make bootlegs, and still make a profit.
Funny how that works. Since they're not writing all the IOS code, they can charge a lower price and and still make a profit. Unless you think this admittedly interesting hack was harder to write than maintaining all of IOS (and the corporate yacht).
Cisco can amortize the cost of their software across a huge number of devices, and several generations of their products. Counterfeiters have a much smaller pool of sales to work with. And it's not all that difficult to write software for networking gear. Many networking devices are just computers running Linux these days... Cisco's ASAs for instance.
"And it's not all that difficult to write software for networking gear."
Broadcom and Nvidia have done almost all the heavy lifting - virtyually everything at 100Gb/s and below is using commodity switch chipsets with a very small shim to give a frontend
"Many networking devices are just computers running Linux these days..."
See above. They're small linux systems controlling commodity chipsets
The thing that really irks about Cisco is that they charge extra for stuff which is BUILT IN to the chipsets and enabled by default (You can buy whitebox kit using the same chips and run whatever flavour of routeros you want) whilst making a big song and dance about "R&D" - that may have been true in the past but Since Broadcom came along with the Trident series 8 years ago, they're mostly just another box shifter
Let's not also forget that Cisco GOT to be dominant by shipping cheap unencumbered kit that undercut the existing Telco-oriented behemoths whilst providing a "useful" set of features.
It's the Microsoft model - "perfect is the enemy of Good Enough" - and once dominance is achieved in a market, break out the thumbscrews (Embrace, extend, Extinguish - remember the Hallooween memos)
The difference this time is that the USA government is joining in the industrial warfare and demonising cheaper kit from other countries instead of letting Cisco (and others) be forced to improve their product - we've seen this before - it's what happened to the USA car industry in the 1960s-80s when 25% import tax was imposed on light trucks and vans, creating a captive market (It's also happening in the Aviation sector - Comac is the current villain de jure, after Airbus proved impossible to take down)
I am assuming the term excludes the genuine product's built in Lawful Intercept; to quote Cisco "Lawful intercept is a process that enables a Law Enforcement Agency (LEA) to perform electronic surveillance on an individual (a target)".
More info and (example device) how to configure it here https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/lawful/intercept/book/65LIch1.html
10-min Network Engineering Survey (Cisco DNAC)
Cisco wants feedback from DNAC operators and want to know where they went wrong with DNAC.
Reminder: In space, no one can hear you scream.
Cisco pushed an update to my kit which destroyed it - because it "allegedly" copied some CISCO IP
Did they break any laws ?
If PORSCHE decided that the "my other car is a porsche" sticker on my Fiesta violated their copyright - they aren't allowed to come round and crush it
So that's a government agency after presumably getting a precedent set in a court case.
Was this CISCO deliberately bricking gear by overwriting a hack, or was the boot process badly implemented?
Microsoft were sending out DCMA demands to sites hosting LibreOffice, they claimed it was a mistake - but would they be allowed to brick my PC if a Windows update detected some non-Microsoft software called xxxOffice?
The article told us what happened. The counterfeiters wrote a bootloader so it would bypass some protection code. Cisco's update had a new bootloader. Cisco's update knew how to install the bootloader and that it would work on their gear. The counterfeit device didn't think it through and installed the new bootloader, wiping out their custom one. Their custom one being required, that didn't end well.
On a legal basis, it's not Cisco's responsibility. If they knew of counterfeit goods, it would have been easier for them to just call law enforcement. But they are not under any responsibility to ensure their updates work on equipment they didn't license the software to run on. Sadly, they often aren't required to make sure their software works correctly on the devices they do build either, though you can sue them for lost productivity if that happens.
In some ways this is on par with the pirate Sky boxes of years ago - remember the software update that bricked the priate boxes and have them display "Game Over" ?
If I found someone selling knockoffs of my kit (as opposed to genuine competitors), I'd be tempted to go down the same route (Disclosure, back in ISP days, I discovered an entire ISP in another country leeching off my DNS servers and started giving them special treatment rather than simply blocking the queries. I'm sure the customers loved being directed to goatse.cx
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
