back to article Bad news: Your Cisco switch is a fake and an update borked it. Good news: It wasn't designed to spy on you

Two types of fake Cisco switches – discovered after a software upgrade hobbled counterfeit gear at an unidentified IT firm – appear to have been designed for profit rather than espionage. F-Secure Consulting's hardware security team disassembled the unauthorized Cisco Catalyst 2960-X series switches at the IT company's request …

  1. Claverhouse Silver badge
    Thumb Up

    Hearing and Obeying, Mr. Trump

    Well, at least it's not Chinese.

    1. W.S.Gosset Silver badge

      Re: Hearing and Obeying, Mr. Trump

      A commentard here coupla years ago recounted how he had personally audited some Huawei routers etc onsite in their factory, a few years previously.

      He said they were all entirely running stolen Dell or Cisco code. (Can't remember which, right now)

      They bailed on their procurement assessment at that point though; didn't bother going deeper to check the hardware/chips.

      1. Roland6 Silver badge

        Re: Hearing and Obeying, Mr. Trump

        A commentard here coupla years ago recounted how he had personally audited some Huawei routers etc onsite in their factory, a few years previously.

        He said they were all entirely running stolen Dell or Cisco code.

        Would have been even more interesting if they had taken the case off the routers and looked at the boards etc.

    2. Outski

      Re: Hearing and Obeying, Mr. Trump

      I had a Chinese client ten years ago who were vociferously annoyed about some bent Cisco kit they'd had palmed off on them. It turned up with all its holographic stickers, supposedly from an approved channel vendor, but missing half its innards, so it's not just non-Middle Kingdom firms affected by this.

      1. Anonymous Coward
        Anonymous Coward

        Re: Hearing and Obeying, Mr. Trump

        a few years ago we were supplying some Cisco 4451X routers to a gov dept and at some point someone had decided some of the ports are to be fibre-presented. No problem, we thought, we have some spare unused old-new-stock SFPs in a box and we duly sent some off. Then the manure hit the muck spreader as half of the SFPs were being reported as 'not genuine' and gov security came down hard.

        They were all genuine Cisco parts, supplied straight from Cisco, and it turned out that the router's IOS had a 'compatibility' table that was a little out of date. The SFPs were manufactured last week in Dec and 1st week Jan and the Dec ones were flagged as genuine and the ones from a week later were rejected.

        There were 2 options... a hidden command to tell the router they were not genuine parts (potentially putting them out of warranty/support) or, as we ended up doing, swap every one we could get our hands on to get enough 'workers' to complete the job.

        AFAIK the 'not genuine' ones are working to this day in whatever switch/router we swapped them to

    3. Mike the FlyingRat
      Boffin

      @Claverhoue Re: Hearing and Obeying, Mr. Trump

      You wrote:

      Well, at least it's not Chinese.

      But you need to ask yourself where Cisco gear is manufactured...

      IIRC, it is China.

      per the article...

      Trafficking in sham Cisco gear has been an issue for years. The California-based networking giant insists that maintaining the integrity and quality of its products is a top priority

      Yes, this is a known and well publicized fact. I want to say at least 20years or longer if memory serves.

      There were stories about how a factory would produce the kit, but out the back door you saw fakes being produced...

      I'd say that they got better at counterfeiting and had a bit of inside help.

  2. Snake Silver badge

    Impressed?

    I'm not sure whether to be impressed or concerned when reading this. Someone put in a LOT of effort to discover vulnerabilities, design, test, manufacture and distribute a knockoff product, one with an inherent level of sophistication and complexity. It's amazing that doing this at all is worth the investment of both time and funds, in terms of payback - that's why I don't know how to feel about this.

    This has to be the tip of a much larger iceberg, as the cost of developing a knockoff for a single product can't possible pay back that much money.

    1. ST Silver badge

      Re: Impressed?

      > Someone put in a LOT of effort to discover vulnerabilities, design, test, manufacture and distribute a knockoff product, one with an inherent level of sophistication and complexity.

      I can count on some of the fingers of just one hand the number of countries that would be interested, and willing, to do this.

      1. Uffish
        Trollface

        Re: "a LOT of effort"

        All that effort is just a further education programme for a country's engineers. Whichever big organisation buys some shiny new gear also budgets for some good engineering analysis of said shiny product and publication of the information obtained.

        If you were running a developing country with big ambitions you would likely have introduced a similar system. And after all, you bought the shiny products so you can do what you like with them,

    2. elaar

      Re: Impressed?

      As far back as the early 80's, certain people invested huge amounts of time reversing the security protections on arcade game pcbs with only the use of fairly primitive electronic tools, creating replacements/workarounds for custom ICs etc. To then redesign and manufacture them in a few short months.

      I have to admire their skillset, it would have required some very clever people indeed.

      1. Test Man

        Re: Impressed?

        Reminds me of all those knockoff Street Fighter II: Champion Edition arcade cabs (they were colloquially known as "Rainbow Edition"). Capcom's fightback plan was to release Street Fighter II: Hyper Fighting for two reasons:

        1. they released it as an upgrade PCB for the proper "Champion Edition" PCB, so anyone who bought a knock off couldn't upgrade (it made them have to get a legal copy in order to upgrade)

        2. it instantly made "Champion Edition" old hat, so no one would want to buy knock-off "Rainbow Edition" when something new is out there

    3. Lon24

      Re: Impressed?

      I'm imagining a certain country would not be happy with probably leaky Cisco gear at the heart of their comms networks. To develop a 'safe' local switch would be imperitive no matter what the cost. Hence this rather clever gear.

      Whether that was delivered locally in fake boxes or not - the greedy manufacturer/subcontractor probably skimmed off a load of boards and shipped them out into the world market where it was all nearly profit. That state is probably a bit upset they didn't put a back-door in!

      1. Anonymous Coward
        Anonymous Coward

        Re: Impressed?

        If you look at many "good" knock off products they are made at the same factory as the legitimate parts.

        Run the product line for an extra hour each day and sell the extra parts via grey market.

        Cisco had significant issues with this with Catalyst 29XX switch's and Cisco 2600/2800 routers (possibly others as well but personally have found grey market equipment including ~10 out of 30 2960Xs that couldn't be upgraded direct from the reseller when applying newer software images. As of IOS15 (released 2012) more hardware and software checks were introduced and started flagging up these parts and I haven't seen grey market Cisco kit in new shipments for at least 5 years.

        The surprise is that these switches were running an old enough release to not be detected before 2020.

        1. J. Cook Silver badge

          Re: Impressed?

          I am not surprised one bit.

          Rather a lot of companies either don't want to go to the hassle and expense of keeping products under support, or can't be arsed to plan the downtime to update the software, or the failsafe excuse of "well, it's working fine, why should be update the firmware on it?"

          We had a few devices that we didn't want to update, because they had a very specific and fiddly configuration on them that would break if we updated the firmware. (We also ran into firmware updates breaking actual OEM purchased switch port blades because the blade revision wasn't compatible with the newer microkernel, AND we couldn't run two different revs of the board in the same chassis.)

          Fortunately, we ended up replacing them outright with newer units entirely.

    4. rcxb Silver badge

      Re: Impressed?

      It's amazing that doing this at all is worth the investment of both time and funds

      You apparently haven't seen the mark-up on Cisco gear.

    5. SmartAlec

      Re: Impressed?

      Counterfeit or compatible right?

      It'd be the latter if it updated itself.

      Anyway morel of the story: if it's not broke, don't fix it?

      Sort of...

    6. enormous c word

      Re: Impressed?

      It's all in the numbers - if they made enough of them, then the effort is worth it. If there's no concern about the consequences of dud kit failing at some point, then you don't have the overhead of customer support and quality control to worry about. You need to change your thinking from a Western 'quality-product' / 'brand-loyalty' perspective to a 'we're going to make some money and screw the customer and their business'. I remember reading an article that China were manufacturing fake boiled eggs.

      https://www.youtube.com/watch?v=EVnhRDuXGPs

      If that doesnt alarm you, look up China Gutter Oil - China has embraced capitalism in its rawest form - time to be afraid.

  3. Version 1.0 Silver badge

    Follow the money

    Shop for them via a google search, it looks like they cost about $2000 but you can find some for $800 ... if someone's capable to build their own Cisco switches I wonder what else is out there?

    1. Jellied Eel Silver badge

      Re: Follow the money

      ..if someone's capable to build their own Cisco switches I wonder what else is out there?

      A lot. I went to a seminar on this some years ago thinking fakes were mostly about dodgy Rolex and handbags.. Then HMRC & Trading Standards types pointed out they'd also been seizing fake car and aero parts, medicines and anything that could make the makers a quick buck. And detecting those fakes generally relied on information from the legit brand owners given it's not easy to tell a palette of dodgy Cats from real ones. But once detected, then tracing the supply chain back to figure out where it originated. In this case, seems like the user bought the fakes from a reseller, so Cisco will be wanting to know where that reseller sourced that kit.

      It was a fascinating and alarming seminar though, and a huge problem for everyone but the fake makers.

      1. W.S.Gosset Silver badge

        Re: Follow the money

        Yes, the sheer extent of routine counterfeiting in China is astounding. It is endemic, cultural, and often quite dangerous. Google the counterfeit _milk_ via adding melamine.

        Right down to routinely bullshitting about the quality of raw materials. "Auditors" in the physical commodities sector are built like brick shithouses and their auditing tool is a 10foot Pole with a scoop. And they spend their days clambering over ore trains and grain warehouses smashing this pole down into the middle and bottom to get samples of what's _actually_ underneath the concealing top layer or outer layers. Don't bother slitting a grain sack to check what's in it; that sucker has to be torn right open since they routinely fill just the middles with garbage.

        1. Yet Another Anonymous coward Silver badge

          Re: Follow the money

          How primitive, counterfeiting rice.

          Civilised countries make $Bn from counterfeiting inter-bank interest rates

          1. W.S.Gosset Silver badge

            Re: Follow the money

            Bad news, I'm afraid. LIBOR-frigging (always vanishingly rare because v. difficult; now eliminated) only ever moved intrabank profits around between banks.

            No change in the total, no big cultural implications, sorry. And billions? Hoooooo BOY, that must have been a big day on the market.

        2. W.S.Gosset Silver badge
          1. W.S.Gosset Silver badge

            Re: Follow the money

            Reminds me of when I was teaching at uni and we sprung a whole bunch of people cheating on an assignment. Turned out they'd impersonated the senior lecturer to the textbook publisher and got a bunch of instructor guides sent out to a PO Box (paid for by cash). We'd got lazy and set the assignment off a worked answer; they all copied it out. 0% all round, guys.

            We'd been marking off student numbers, decided to look up their names.

            A-aaaaand 100% of them were Chinese.

            And 100% of them came in to protest.

            Many got loud, some got abusive, couple attempted physical intimidation (of my female tutors). Fortunately re the latters, I walked in on them mid threats.

            But ALL of them were angry.

            And stop and think about the attitude and dedication necessarily underpinning the weaselling of the instructor guides via impersonation....

        3. Jellied Eel Silver badge

          Re: Follow the money

          Right down to routinely bullshitting about the quality of raw materials. "Auditors" in the physical commodities sector are built like brick shithouses and their auditing tool is a 10foot Pole with a scoop.

          Yup. Problem isn't limited to China either. I had a fascinating client once which did inspections, in which I learned a little about that. Their challenge was getting inspectors onto ships, getting samples, testing, sharing the results with their clients and then deciding to accept or reject the shipment. Which provided several challenges, one being a narrow time window for consignments FOB because once unloaded, it became a much bigger problem for the customer. Main challenge for us was trying to get wayleaves to run fibre into ports, and for extra fun, bonded warehouses.

          And that client was the one I think that made me want a spectral analysis of a Marmite sample. Easy to get for other addictive substances like THC, but a challenge still on my bucket list. I keep checking the price of gadgets like Raman and IR spectrometers, but a) they're still expensive and b) I'd probably gunk it up with Marmite.. :)

      2. Orv

        Re: Follow the money

        Fake aircraft parts have been a big concern for many years. Partnair Flight 394 went down in 1989 because three of the four bolts holding the vertical tail on were fakes made of metal that was too soft. Inadequately refurbished used parts are another problem.

        1. Jellied Eel Silver badge

          Re: Follow the money

          I think that was one of the examples given for the risks of fake product. Also there was the recent example of an inspector at a Japanese steel company faking test certificates. But refurbs or stuff that failed QA was one of the sources of grey/black market products.. And unless the customer could test, they'd be none the wiser until they failed. Pressure to cut costs also doesn't help.

  4. TheInstigator

    This has got to be due to the Chinese/Iranians/Russians/Syrians (delete as appropriate)

    Let's nuke them all just to make sure !

    (sarcasm - in case you didn't know!) ;)

    1. Charles 9 Silver badge

      And if they turn out to be Andromeda Strains or Feral Ghouls?

      1. TheInstigator

        Kill them too!

        1. Charles 9 Silver badge

          Andromeda Strains and Feral Ghouls are notoriously tough to kill (last I checked, you can't kill an Andromeda Strain). Worse, nukes just make them stronger.

          1. TheInstigator

            Let's spend billions of <insert currency here> and find a way of killing them - it's what humans do best

  5. Gene Cash Silver badge

    Too expensive

    You know your shit is too expensive when people go to this much trouble to make bootlegs, and still make a profit.

    1. MattPi

      Re: Too expensive

      You know your shit is too expensive when people go to this much trouble to make bootlegs, and still make a profit.

      Funny how that works. Since they're not writing all the IOS code, they can charge a lower price and and still make a profit. Unless you think this admittedly interesting hack was harder to write than maintaining all of IOS (and the corporate yacht).

      1. rcxb Silver badge

        Re: Too expensive

        Cisco can amortize the cost of their software across a huge number of devices, and several generations of their products. Counterfeiters have a much smaller pool of sales to work with. And it's not all that difficult to write software for networking gear. Many networking devices are just computers running Linux these days... Cisco's ASAs for instance.

  6. spold Silver badge

    Backdoors... who needs 'em

    I am assuming the term excludes the genuine product's built in Lawful Intercept; to quote Cisco "Lawful intercept is a process that enables a Law Enforcement Agency (LEA) to perform electronic surveillance on an individual (a target)".

    More info and (example device) how to configure it here https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/lawful/intercept/book/65LIch1.html

  7. HildyJ Silver badge
    Facepalm

    Still Cisco

    It might be good that they weren't designed to spy on you but it should be noted that the kit was able to work for so long because it used "previously unknown vulnerability in a security component, to defeat this Secure Boot process." Cisco being Cisco.

    1. Anonymous Coward
      Anonymous Coward

      Re: Still Cisco

      Probably 'hard coded root user credentials in critical admin module'. They love scattering those about all over the place.

  8. cjcox

    Things I also learned.

    I learned that people will purchase genuine product from you if you don't turn the customer upside down while extracting all money that falls to the ground while you perpetually whack them in the groin.

    1. seven of five Silver badge

      Re: Things I also learned.

      Well, it works for Oracle, doesn t it? Though even they use a bat with less nails these days - so good to us, they are.

      1. MyffyW Silver badge

        Re: Things I also learned.

        A bat you say? With nails. Luxury! My enterprise software "partner" used to actually kill us, spit on our graves and then disinter our corpses. Still we were happy back then. At least they didn't make us run eBusiness Suite.

        1. seven of five Silver badge

          Re: Things I also learned.

          > A bat you say? With nails. Luxury!

          Sure, Oracle. You get what you pay for, and all that.

          1. Omgwtfbbqtime
            Trollface

            Re: Things I also learned.

            "Sure, Oracle. You get what you pay for, and all that."

            You get Larry's yacht?

            1. seven of five Silver badge

              Re: Things I also learned.

              We get to look at it. (And also: which one? He´s got so many)

      2. Outski
        Headmaster

        Re: Things I also learned.

        FEWER nails, damn your eyes, FEWER!!!

        1. seven of five Silver badge

          Re: Things I also learned.

          Sorry, non native speaker. Will try to remember.

  9. P. Lee Silver badge

    Things I learned

    Look at sdn. Cisco’s wan sdn even.

    Features and speed don’t matter. Cheap administration matters.

  10. chivo243 Silver badge
    Trollface

    What's that old saying?

    Nobody ever got fired for buying Cisco?

    1. TRT Silver badge

      Re: What's that old saying?

      If you're going to buy from Cisco... be sure to wear some flowers in your hair.

      1. MyffyW Silver badge

        Re: What's that old saying?

        I was going to buy some flowers for my hair but found my Cisco switch was blowin' in the wind

        1. mutt13y

          Re: What's that old saying?

          Mumma take this switch off of me, I don't need it anymore

          1. velo101

            Re: What's that old saying?

            California reaming?

  11. Screwed

    In reading the subhead, I briefly mistook Autopsy and read Autospy. Quite appropriate.

  12. Anonymous Coward
    Anonymous Coward

    Where can I get the ripoff kit?

    It seems to be just as good as the Cisco version

  13. sanmigueelbeer Silver badge

    Attention DNAC operators: Cisco wants to know what you really, really think of DNAC

    10-min Network Engineering Survey (Cisco DNAC)

    Cisco wants feedback from DNAC operators and want to know where they went wrong with DNAC.

    Reminder: In space, no one can hear you scream.

  14. james 68

    Where can I get Some of these I wonder? They work just like the real thing but without backdoors? Sign me up.

  15. Anonymous Coward
    Anonymous Coward

    So they knew these were fake copies because they weren't a security threat ?

    ( ah, beaten to it in the 'obvious comment' race )

  16. Yet Another Anonymous coward Silver badge

    Computer misuse act

    Cisco pushed an update to my kit which destroyed it - because it "allegedly" copied some CISCO IP

    Did they break any laws ?

    If PORSCHE decided that the "my other car is a porsche" sticker on my Fiesta violated their copyright - they aren't allowed to come round and crush it

    1. nijam

      Re: Computer misuse act

      ... they aren't allowed to come round and crush it

      No, but if you take it through French customs, they might crush it, and (if you're lucky) let you get your stuff (and yourself) out of the car first.

      1. Yet Another Anonymous coward Silver badge

        Re: Computer misuse act

        So that's a government agency after presumably getting a precedent set in a court case.

        Was this CISCO deliberately bricking gear by overwriting a hack, or was the boot process badly implemented?

        Microsoft were sending out DCMA demands to sites hosting LibreOffice, they claimed it was a mistake - but would they be allowed to brick my PC if a Windows update detected some non-Microsoft software called xxxOffice?

        1. doublelayer Silver badge

          Re: Computer misuse act

          The article told us what happened. The counterfeiters wrote a bootloader so it would bypass some protection code. Cisco's update had a new bootloader. Cisco's update knew how to install the bootloader and that it would work on their gear. The counterfeit device didn't think it through and installed the new bootloader, wiping out their custom one. Their custom one being required, that didn't end well.

          On a legal basis, it's not Cisco's responsibility. If they knew of counterfeit goods, it would have been easier for them to just call law enforcement. But they are not under any responsibility to ensure their updates work on equipment they didn't license the software to run on. Sadly, they often aren't required to make sure their software works correctly on the devices they do build either, though you can sue them for lost productivity if that happens.

  17. cantankerous swineherd Silver badge

    doesn't the real thing have the NSA mucking about with it en route to the customer?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020