back to article FYI: Ticking time-bomb fault will brick Cisco gear after 18 months

Cisco has issued a warning that an electronic component used in versions of its routing, optical networking, security and switch products prior to November 16, 2016 is unreliable – and may fail in the next year and a half, rendering affected hardware permanently inoperable. "Although the Cisco products with this component are …

  1. Anonymous Coward
    Joke

    Why does Cisco remind me of HAL 9000...

    "I've just picked up a fault in the AE-35 unit. It is going to go 100 percent failure within 72 hours."...

    1. thewizard75

      Intel Atom issue seems likely

      After all, atom c2000 erratum avr54 is it ceasing to function due to the low pin count bus clock outputs stopping. We have a winner

      http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf

  2. Wedge

    Planned obsolescence

    When it's too obvious you are doing it blame it on a faulty part and do a recall....

    1. Pascal Monett Silver badge
      Coat

      Re: Planned obsolescence

      Generally that plan is set for five years. 18 months seems a bit ambitious to me, and some people just might notice. Oh, wait . . .

      1. Known Hero

        Re: Planned obsolescence

        @pascal monett

        That is only with the assumption that this all wasn't caused by a typo in their timer script :/

        Either way, would you feel comfortable buying older cisco gear after reading this ??

        1. Anonymous Coward
          Anonymous Coward

          Re: would you feel comfortable buying older cisco gear

          "would you feel comfortable buying older cisco gear after reading this"

          Why not, it's only their C2000-based stuff that's affected (so far), and in due course they'll all be repalced (?).

          Other vendors who have used the Intel chips in question will be affected too in due course, unless there's something vendor-specific going on.

  3. Anonymous Coward
    Anonymous Coward

    MTBF, meet unavoidable entropy

    For most all electronics manufacturing they run tests to see how the product will fair over the long term, and simulate some aspects of that since you just can't turn the clocks forward and see what happens. What that means is, say for a disk drive maker, they will build a lab and set many of the same disk products on a mission to exercise and die with variation on how and what constitutes death for that product. A good lab will have ovens and perhaps some freezer units to test the hardware at extremes. I got to visit a lab such as this back in the early 1990s at Apple. It basically had many versions of the computers being designed and readied for production, unless these, or other, pre-build tests fail. That's why you can look on the box it shipped with and it will tell you useful operating temperature data. Along with other "windows of variance" like power supply inputs, etc.

    With enough data you can piece together a profile of how a certain product will behave over the years, and provide your end consumers with a nice MTBF number indicating the Mean Time Between Failures. However, like with this clock signal component, you might not be able to tease out the failures of this device before 18 months have passed. It's a tricky situation and it makes sense for Cisco to replace h/w, as this does not look like something fixable in software.

    1. Anonymous IV

      Re: MTBF, meet unavoidable entropy

      MTBF?

      That initialism surely means Maximum Time Before Failure...

      1. Martin an gof Silver badge
        Boffin

        Re: MTBF, meet unavoidable entropy

        That initialism surely means Maximum Time Before Failure...

        The classic example where I work was the original-fit projectors. These were recommended by the company contracted for the fit-out and according to the "manufacturer" data (actually a badge-manufacturer) had MTBFs of 28,000 hours. In our use this would equate to about 10 years. The company (I didn't work for them back then) had nobody who really knew about these things, and I have seen one single email from one person who essentially said "there isn't a projector in the world that would last that long". As you might expect, he was ignored.

        When I started here, a large proportion of the projectors (there are over 30) were showing obvious signs of LCD and colour filter failure. This after around 5,000 hours run time. When I finally tracked down the original manufacturer (the badge manufacturers were still claiming 28,000 hours), they admitted that the LCD module had an expected lifetime of just 4,500 hours.

        Of course nobody had even begun to think about a capital budget to replace the projectors, and at £5,000 for a complete "optical block", repair was out of the question.

        It subsequently transpired that the power supplies started failing at between 7,000 and 8,000 hours, though this was due to dodgy capacitors and relatively easily fixed. The BM quoted me €1,000 for a new PSU.

        Long story short, we replaced the LCD projectors with DLP units from a different manufacturer which had (and have achieved) expected lifespans of 20,000 hours, cost about as much to buy new as the cost of an optical block for the originals and used lamps that cost less and lasted twice or three times as long. DLP has its downsides, but in our case it has worked extremely well.

        That said, I still have a couple of computers "out there" with original-fit Maxtor SCSI and SATA discs, now about 12 years old :-)

        (it's ok, I'm doing it as a sort of experiment and there are spares ready-and-waiting)

        M.

  4. Anonymous Coward
    Anonymous Coward

    Consumer Law

    "For customers with affected products under warranty or covered by service contracts through November 16, 2016, Cisco intends to provide replacement products."

    Under Oz laws at least you are entitled to a replacement from the vendor (and the vendor cannot simply point at Cisco) irrespective of the stated warranty so long as the item cost under $40k (which, being Cisco, probably counts out most of the items unfortunately!)

    1. Anonymous Coward
      Anonymous Coward

      Re: Consumer Law

      The Cisco gear mentioned in the article is not consumer gear, so that's a moot point.

      1. Jamie Jones Silver badge

        Re: Consumer Law

        I was wondering the same thing. We have similar in the UK.

        Does the "sold with pre-existing fault" rule (or however it;s phrased) not apply to businesses?

        1. Sandtitz Silver badge

          Re: Consumer Law @Jamie

          IANAL, but typically businesses buy equipment "as is". The warranty (or service contract) is there just to assure the buyer that they will get some use out of the equipment for at least the warranty/service period.

          If Cisco had sold these products knowing they were destined to fail soon the customer could take them to court for fraud.

      2. Anonymous Coward
        Anonymous Coward

        Re: Consumer Law

        Intended purpose is irrelevant if it's under $40k. Over for $40k it's not covered if it's for business use.

        1. Anonymous Coward
          Anonymous Coward

          Re: Consumer Law

          That's good to know because my ASA5506 is marketed not for Enterprise but SMB/SOHO and that's how I use one. And I'd be mightily pissed if I fell in between some sort of crack - not fish (enterprise kit) not flesh (consumer kit).

          1. Lord Elpuss Silver badge

            Re: Consumer Law

            Genuinely interested; why do you refer to enterprise kit as fish and consumer kit as flesh? Never heard this before.

            Presumably you're not talking about these fine chaps -> www.enterprisefishco.com/

          2. Jamie Jones Silver badge
            Thumb Up

            Re: Consumer Law

            Sandtitz and TheVogon - cheers for the reply

        2. TheVogon

          Re: Consumer Law

          There is no financial limit for consumers - upper or lower. The goods have to be fit for the intended purpose. You have up to 6 years to claim.

      3. sanmigueelbeer

        Re: Consumer Law

        Do NOT contact TAC (yet).

        Instead, get SmartNet (SNT) on the appliance. 24-hours after the SNT confirmation comes through, RMA the appliance and tell TAC the FN.

  5. Drew 11

    Anyone have one of these and can open it up and tell us the brand/model # of the crystal oscillator?

    1. Anonymous Coward
      Anonymous Coward

      Hope it is not a repeat of the bad caps

      I sure hope it is only used in enterprise gear, and not consumer gear like wireless routers and laptops. Otherwise this will be as ugly as the bad capacitors that made it into so much gear in the early 2000s...

      1. Alan Brown Silver badge

        Re: Hope it is not a repeat of the bad caps

        Badcaps are _still_ with us.

        No, seriously. I'm seeing kit less than a year old expiring with bulgy caps. The lure of being 20c cheaper in overall manufacturing cost on something worth $500+ is too much for some contractors.

  6. Magani
    Linux

    Not a canard

    Cisco insists this isn't a recall; rather it's a proactive replacement.

    If it looks like a duck and swims like a duck and sounds like a duck, it's probably a recall regardless of Cisco's spin doctoring.

    1. TRT

      Re: Not a canard

      It's an alternative recall.

      1. ecofeco Silver badge

        Re: Not a canard

        Beat me to it.

      2. Michael Thibault

        Re: Not a canard

        It's subscription hardware obsolescence. Replacements? The first one's free.

      3. Nolveys
        Headmaster

        Re: Not a canard

        It's an alternative recall.

        So we're not talking about Arnold Schwarzenegger, but Colin Farrell instead? Is Samantha Morton involved? I'm so confused.

  7. lawndart

    Probably a dodgy batch of redstone. Someone's mixed in some nether quartz dyed red with poppies.

    1. Linker3000
      Alert

      "Prize Plum"

      This issue was picked up by /r/networking on Reddit, and one Redditor suggests which component is the culprit - it would be useful for the electronics industry at large if the part was officially identified to help other manufacturers and users plan scheduled maintenance for this issue before it gets worse.

      https://www.reddit.com/r/networking/comments/5rmsw0/major_cisco_hardware_clock_issue_affecting/

      1. Gert Leboski
        Trollface

        Intel perhaps?

        From what I read, it could centre around the Intel Atom C2000 series. My understanding is that this includes the C2718 / Avotons which I understand from other research into a low power hypervisor host, would burn out/slow down/fail after around 12-18 months use.

        If it is around Intel kit, that would explain the obvious NDA that Cisco is under. Who else has the clout to impose such a thing on NetZilla?

        1. Anonymous Coward
          Anonymous Coward

          Re: Intel perhaps?

          Hadn't heard about this but Intel did mention it on their earnings call and set aside a reserve for it, so it is obviously something they believe will be a real problem (setting aside reserves is typically only done for major issues that will cost a lot of money, like Samsung's exploding phones or the Xbox red rings of death)

          If so, this is really good news for most of us, as only enterprise equipment will be affected. No one has an Atom C2000 in their home wireless router or smartphone.

          1. Scott 29

            Re: Intel perhaps?

            I have one C2750 in my freeNAS-mini though. At home.

            1. Alan Brown Silver badge

              Re: Intel perhaps?

              "I have one C2750 in my freeNAS-mini though. At home."

              This is something that worries me too. C2000s are widespread in systems and they're not usually socketed.

              I wonder what Supermicro and friends have to say?

            2. Spotswood

              Re: Intel perhaps?

              I have a C2538 in my Synology DS1518!

              The thought of losing all that data is freaking me out....

  8. JulieM Silver badge
    Black Helicopters

    Hmmmm

    This wouldn't at all be a sneaky way of sabotaging second-hand kit, now, would it?

    1. Down not across

      Re: Hmmmm

      This wouldn't at all be a sneaky way of sabotaging second-hand kit, now, would it?

      Highly unlikely. The healthy market in second hand gear is of great benefit to Cisco. There would be a lot fewer people with Cisco skills if there wasn't an abundance of old used Cisco kit on fleabay.

      I also wouldn't be surprised if some older, but not yet quite EOL, kit didn't end up used in smaller, less affluent companies possibly even with SmartNet on top.

  9. thewizard75

    Intel is likely

    Their January 2017 errata update for the atom c2000 series has issue avr54, system may stop responding or fail to boot due to problems with the low pin count bus clock outputs. Sounds likely to me

    1. diodesign (Written by Reg staff) Silver badge

      Re: Intel is likely

      Thanks - we'll look into it!

      C.

  10. theblackhand

    Having been through this before...

    Last time it was a memory issue:

    http://www.cisco.com/c/en/us/about/supplier-sustainability/memory.html

    I vaguely recall it being a supplier issue (supplier initially provided components to Cisco's spec but at some point the spec changed and wasn't picked up during QA).

    Or there's the long running capacitor plague - https://en.wikipedia.org/wiki/Capacitor_plague

    I know it sucks having to replace newish equipment, but there's not much more that a vendor can do (from memory Cisco replaced equipment with faulty memory as advance spares and return replaced components within two weeks for equipment covered by Smartnet and a tighter return window for non-Smartnet equipment although that might have just been because we were a large customer...)

    1. Jamie Jones Silver badge
      Coat

      Re: Having been through this before...

      Last time it was a memory issue:

      http://www.cisco.com/c/en/us/about/supplier-sustainability/memory.html

      I vaguely recall it being a......

      Yep, sounds like a memory issue there!

      /gets coat

      1. Version 1.0 Silver badge

        Re: Having been through this before...

        A long time ago, the company that I worked for, started have Z80 CPUs go tits-up after a while in the field - this was traced to an improper clock driver. I don't know the details but until they redesigned the boards (a commercial hi-speed EKG analysis system) we were running around with a tube of mil-spec Z80's which lasted a lot longer than the commercial version.

  11. Anonymous Coward
    Anonymous Coward

    Intel's first ... "3D" or FinFET-style transistor structure.

    "Intel produces this SoC on a custom-tuned variant of its 22-nm fabrication process, which has some of the finest geometries in the industry and is the first process to adopt a "3D" or FinFET-style transistor structure.

    We've already seen quite a few bigger cores manufactured at 22-nm, but the benefits of this process are arguably most notable for low-power chips like Avoton. Intel is taking full advantage of its celebrated manufacturing advantage here."

    Those words from

    http://techreport.com/review/25311/inside-intel-atom-c2000-series-avoton-processors

    (and similar reports elsewhere).

    Elsewhere, some chap called Prickett-Morgan says these are also aimed at microservers (HP were mentioned, both blades and standalone).

    "Intel's celebrated manufacturing advantage", eh? 'Course, it generally works out better if the manufacturing people and the design people are in close contact early in the design. Does that still happen early enough when you buy in the FinFET technology from outside (GlobalFoundries? Samsung?), at quite an advanced stage, and maybe roll it out without a full understanding of the implications?

    Anybody out there familiar with the potential medium term reliability issues of bleeding-edge ultrafine geometry semiconductors, topics like migration effects etc (especially the interaction with thermal effects in real hardware, maybe even with realistic workloads)? There's possibly a consultancy opportunity for you at Intel. Might be good money if you've got the right answers.

    Assuming that there's some relevance in e.g.

    https://www.semiwiki.com/forum/content/5031-electromigration-analysis-finfet-self-heating.html

    It could obviously also be something entirely different. We'll find out one day.

    For the rest of the world: if you have a product based on one of the affected chips, your vendor is waiting to hear from you.

  12. Alan Brown Silver badge

    probably related:

    http://www.tomshardware.com/news/intel-cpu-failure-atom-processor,33538.html

    I wonder what the situation in the UK is?

    (it'll be complicated, SOGA covers business as well as personal use and expected lifetime clauses are difficult at the best of times)

  13. Chris Stephens

    I made this correctly branded logo

    Make sure to post this on forums and such

    http://www.xymox1.com/Misc/modem/DeadInside.jpg

  14. Chris Stephens

    I rewrote the Cisco press release..

    ______________________________________

    Cisco strives to deliver technologies and services that work. However recently, Cisco became aware that Intel planted a timed obsolescence feature into its Atom 2000 that affects a large number of expensive Cisco products. In all units, we have seen the Intel Atom CPU degrade over time. Although the Cisco products with this Intel CPU are currently performing normally, we expect product failures to increase over the years as Intel built this in to sell more chips, beginning after the unit has been in operation for approximately 18 months. Once the Intel Atom has timed out, the system will become a brick, will not boot, and is not recoverable. This requires the end user to buy a new product. The Intel Atom is also used by a huge number of other vendors on a large number of products.

    We have identified all Cisco products that have Intel Inside and tried to work with Intel to quickly get a chip that works however they keep asking us to provide examples of the failure. All products shipping currently do not have this issue as far as we know. To support our customers and partners, Cisco will flog Intel and recall all units under warranty or covered by any valid services contract dated as of November 16, 2016, which have Intel Inside and shove them up Intel's ass. Due to the age-based nature of the failure and the crap ton of replacements, not to mention the cost, we will be prioritizing orders based on the products’ time in operation.

    Q: When did you become aware of this issue?

    Cisco learned about the timed failure and potential customer impacts due to this feature in late November 2016. Cisco and Intel have been working as quickly as possible to hide the impact and scope of the issue, create and test PR releases, and put in place a plan to hide Intel from lawyers without causing undue panic or effect Intel's stock price.

  15. Anonymous Coward
    Anonymous Coward

    Are these the chips

    that go into the androids chased by the Bladerunner...

    Not so far in the future now...

    I'll get my coat...

  16. Mike Shepherd
    Meh

    "see this rather worrying errata (sic)"

    Were was this man brung up?

  17. the future is back!

    New Supermicro blades going to brick yard big-time?

    Just mentioning, the Supermicro rack gear checked out here in Reg seems to feature blades with the Atom chips. https://www.theregister.co.uk/2017/02/07/supermicro_gets_super_server_win/

  18. Anonymous Coward
    Anonymous Coward

    for the record

    See also:

    https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/

    "Intel's Atom C2000 chips are bricking products – and it's not just Cisco hit

    Chipzilla and Switchzilla won't confirm connection but the writing is on the wall"

    This story could run for a while.

    Unlike the affected hardware.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like