back to article Cisco UCS servers slugged by 'This SSD will self-destruct in 40,000 hours' firmware farrago

A Friday challenge for netadmins: solve a silkscreen ID bug in a Cisco Firepower appliance. What? You haven’t heard of that error before? That’s not surprising, because the bug is purely cosmetic. As Cisco explains in a new field notice: “The rear panel of some Cisco Firepower 1120 and 1140 security appliances might be …

  1. Anonymous Coward
    Anonymous Coward

    Obsolescence by design?

    "Due to a firmware index bug, a drive that operates for 40,000 hours will experience an invalid index and will cease to function."

    Like those IPhone batteries slowing down perhaps, so as to 'encourage' a new purchase?

    1. bazza Silver badge

      Re: Obsolescence by design?

      In this case it's not by design, it's by Total and Utter Bollix-Up. At least, I'm inferring that it was unintended by the fact that there's a firmware fix...

      The other bug can be fixed in the time-honoured way with Tippex.

      1. Anonymous Coward

        Re: Obsolescence by design?

        I suspect it's worse than that. SSDs degrade based on the number of times you write to a location, eventually making that location unusable. SSDs can handle this with spare locations they can"swap in" electronically. But once these are used up, a full SSD can go titsup.

    2. Mark192

      Re: Obsolescence by design?

      AC said "Like those IPhone batteries slowing down perhaps, so as to 'encourage' a new purchase?"

      This was to lengthen the product life by avoiding the phone shutting down.

      A phone with old batteries will show, say, 12% remaining but when it tries to draw a lot of power the battery will not be able to meet the demand and the phone will shut down.

      Limiting the power that the phone can draw (by underclocking) stops these unplanned shutdowns (or at least delays them).

      My old Samsung Galaxy S3 would shut down, solved by a new battery. iPhones didn't have user-replaceable batteries so they went for a different option.

      1. vir

        Re: Obsolescence by design?

        This is my biggest problem with Apple (I am the mostly-satisfied owner of iThings): by keeping things at a higher level of "abstraction", they overall save some headaches but cause a few really infuriating ones. For example with this battery issue (putting aside the lack of user replaceable battery, etc), the way they went about it was almost the worst possible choice. If they had come right out and said "your phone could unexpectedly shut down at low battery levels; we've introduced a new setting you can activate to incur a slight performance penalty to reduce the chances of an unplanned power-down", they could have avoided a lot of the user wrath.

        1. Mark192

          Re: Obsolescence by design?

          Yeah, be open and upfront about it. The multiple thumbs down on my explanation above shows that people prefer the evil explanation over the real one.

          The lack of openness and transparency cost Apple several hundreds of millions. That said, their implementation in earlier phones was crap (and charges for replacing the battery fairly high). The focus the lawsuit brought forced some welcome changes.

  2. redpawn

    Kind of Hertz

    What does this do to the mean time before failure for this product?

    1. Kevin Johnston

      Re: Kind of Hertz

      Does it not just change from bathtub to square wave?

      1. bazza Silver badge

        Re: Kind of Hertz

        Or one of those bath tubs nicely sloped for one's comfort and reclinement, the other end is a steep slope upto the taps. The only question then is, mixer or separate taps?

        1. Martin an gof Silver badge

          Re: Kind of Hertz

          Separate taps. Mixer taps are an abomination on so many levels (mainly to do with their ridiculously complex innards which fail easily), but in the UK where we traditionally have utterly daft plumbing (high pressure cold, low pressure hot) they are even worse. We even have one at work which does not allow you to vary the flow - turn it beyond "off" and you get cold at full blast followed by a progression to hot at full blast. Not good in a small handwashing sink with both supplies pumped, the only way to avoid splashing everywhere was to crank the isolation valves underneath almost closed.

          Just rejected a free sink from a neighbour for our refurbishment because it was single tap-hole. To be fair, he was of the same mind, which is why it was languishing in his garage :-)

          As for SSDs, if this is a common problem to HPE and Cisco, presumably it's a reasonably standard device, not one manufactured specifically with custom firmware. In other words, what are the chances that similar drives are out there in commercial channels? I have HDDs with well over 40,000 hours runtime and as things gradually move to SSD it'd be nice to be able to avoid sudden loss of data.


          1. John Robson Silver badge

            Re: Kind of Hertz

            Meh, hot water here is at mains pressure, as it is at my parents house.

            No need to have a tank with an open header any more

            1. John Robson Silver badge

              Re: Kind of Hertz

              Should also point out that mine is heated on demand, theirs is tank fed.

              1. Martin an gof Silver badge

                Re: Kind of Hertz

                Heating water on demand does not deliver full mains pressure to the tap, particularly if it's done by an asthmatic "combi" boiler which will reduce the flow rate in order to maintain the set temperature. I should know - I've lived with plenty of the wastes-of-space over the years. Might be ok for handwashing, but filling a bath? Taking a shower while someone else is trying to do the washing up?

                Of course it is possible to get mains pressure hot water from a cylinder, as fast as you like (until it runs out), but having a pressurised cylinder leads to slight complications with regard to inspection and testing making it a bit difficult to D-I-Y.


          2. Richard Jones 1

            Re: Kind of Hertz

            I have a single-hole tap with separate flow managers. They operate either as singletons or together to produce a mixed flow. Other sinks also allow either hot, cold or mixed with one handle and no stupid twirling of the controls.

            I agree that the isolation valves are worth their weight in anything negotiable for a whole range of reasons. The type of mixer taps you wrote about were no doubt specified as they 'looked good' but were utter crap for the users to 'enjoy'

            1. Martin an gof Silver badge

              Re: Kind of Hertz

              Nah, the new mixer was specified as "the one the plumber actually has on the van" when the old one failed. The proper answer to too much pressure is not a cracked-open isolating valve, but a pressure regulating valve on each pipe, but a tap like yours will do it too... albeit at (I'd imagine) considerably more cost than a pair of basic quarter-turn taps.


  3. Anonymous South African Coward

    Cisco has not disclosed who made its dodgy drives, but HPE named Western Digital as the source of its not-so-hot hot sand

    WD again... they'll lose market share bigtime...

    1. BenM 29 Silver badge

      WD have finally published a list of drives that are affected by the SMR/CMR fiasco -

      WD are basically saying 'meh you caught us...' - particularly great is the admission that WD Blue destop drives could be either. WD Reds which are, ironically, NAS rated have only the RED PROs completely free of contaigion.

      1. Martin an gof Silver badge

        Thanks for that link - very informative.

        The only Reds I've used have been 2.5" 1TB which are apparently all CMR and I've found them slightly "meh", not so much on performance, which has been fine, but on life where I've had one or two fail earlier than I would have hoped. Oh, no, actually I just realised we have some 3.5" 2TB Reds in a low-use NAS. SMR might explain why it's a very slow target for Time Machine :-/

        We stopped using Blues at work in our public-facing machines when we realised that they go aggressively to sleep (after just eight seconds of inactivity) and take 2 or 3 seconds to wake up, meaning that a display can be working perfectly well on the "idle" screen, someone comes along and taps the screen and everything freezes for a couple of seconds while the drive spins up. Those machines don't need vast amounts of storage (Windows and a single application) so even small SSDs are more than sufficient, and cheap enough to be a no-brainer for replacing the Blues.

        The biggest surprise for me though was seeing that 2.5" 1TB Blacks are shingled. Blacks have always been marketed as "performance" drives (7,200rpm, larger caches) and have had a slight price premium as a result. I even used a couple of the smaller ones in a NAS some years ago and found that they worked very well - those same drives are now in my main machine mirrored as a "scratch" working pair I use to hold the source files when editing home videos.


        1. Anonymous Coward
          Anonymous Coward

          "Oh, no, actually I just realised we have some 3.5" 2TB Reds in a low-use NAS. SMR might explain why it's a very slow target for Time Machine :-/"

          Can't say I've ever had a performance problem with any of my 3.5" 2/3TB Reds :)

    2. rcxb Silver badge

      WD again... they'll lose market share bigtime...

      It was actually SanDisk, before WD purchased them.

      I doubt WD is worried. They've got no effective competition in the spinning hard drive market (60% market share), and the enterprise SSD market is a 3-way race between WD, Samsung and Intel at this point. WD has long since managed to buy-out their competitors, like HGST and SanDisk.

      1. jelabarre59

        I doubt WD is worried. They've got no effective competition in the spinning hard drive market (60% market share),

        Which is what makes it such a pain in the ass to buy spinning disk these days, especially at your fiendly local WorstBuy store. Staples sells other brands, and on extremely rare occasions you might find some ion their stores (probably somebody slipped up and decided to actually send product to the store).

    3. jelabarre59

      Yet another defective WD drive, colour me NOT surprised.

  4. AndrueC Silver badge
    Thumb Up

    Had a similar issue with the first SSD I bought (think it was a Kingston unit though). After three months it would shutdown. After power cycling it would run for an hour then shut down again if I remember correctly. So I went to the manufacturer's website and discovered that a firmware fix had been made available the day before. Apparently some counter was rolling over and originally only had enough bits to count three months worth of hours.

    On the Cisco front I remember a few years back when I was working as part of a small team in a small office. The office was actually a converted barn but because we were a satellite office of a large multi-national we had quite a decent server room. Still - for the most part we just got on with some programming and a bit of hardware poking and ignored anything what happened in the server room.

    Then one day a small packet arrived. In itself not unusual - we often had personal stuff delivered and even occasionally work-related hardware gubbins. But on opening it we discovered a small fan. No-one knew anything about it. So we checked the paperwork that came with it and discovered that it had been 'ordered' by one of our Cisco routers. It turned out that one of its fans had failed and apparently instead of just letting itself get all hot and bothered (we'd never have noticed until it finally went into melt down) it ordered a replacement part. I have to admit to being somewhat impressed by that.

    1. Gene Cash Silver badge

      'ordered' by one of our Cisco routers

      OK, not being familiar with Cisco iron, how the hell did it know its postal delivery address?

      1. Anonymous Coward
        Anonymous Coward

        Re: 'ordered' by one of our Cisco routers

        Duh, it asked SkyNet of course...

      2. AndrueC Silver badge

        Re: 'ordered' by one of our Cisco routers

        how the hell did it know its postal delivery address?

        I assume that it was registered with Cisco by our corporate IT department when they installed it. All that then needs to happen is that it notifies Cisco HQ of the failure and their support systems look its serial number up in a database and arrange for the fan to be sent out to the registered address.

        It's technically possible Cisco offers monitoring services that will raise the alarm. I just can't immediately find any specific service that includes ordering spare parts. So maybe it was our IT department sending it out but usually they came and visited us (anything to get out of London for the day, lol).

        Edit: I have found this document.

        "Devices equipped with Smart Call Home technology can be enabled to continuously monitor their own health. Once enabled, this feature can notify you of potential issues using a secure, personalized web portal that contains messages, detailed diagnostics, and recommendations. If a serious problem arises, Smart Call Home can automatically generate a service request with Cisco TAC that is routed to the right team for your particular problem"

        It doesn't specifically mention sending out spare parts but it shows how they have the technology to know that 'Router 3 in Bucknell has a failed fan" from there to "Send a replacement out" isn't far :)

  5. Anonymous Coward
    Anonymous Coward

    Just wondering......

    .....confused...........why do hard drives NEED an internal clock? Even if it's not tracking YYYY:MM:DD:HH:MM:SS.....there still needs to be a clock in there to count the hours!

    1. Jon 37

      Re: Just wondering......

      If it gets sent back under warranty, because it broke, it's useful to know how much it's been used. Taking that information, and aggregating across many drives, can give you useful information about why drives are failing. (For example, if you discover SSDs don't normally fail before X TB written or Y years, then you should be increasing or reducing your warranty cover to those thresholds).

      Similarly, that information may be exposed via SMART monitoring, which allows the owner to monitor how much their drive has been used and arrange for it to be replaced at an appropriate time.

      All hard drives and SSDs have a small processor in them, and all processors have some kind of timer that can be used to increment a field once per second, so there's no extra hardware for this.

  6. Anonymous Coward
    Anonymous Coward

    I've not seen this problem before

    Yes, I have seen some issues with hard drives having data read errors but it's always been easy to pull the RL02 out of the drive and dust it, pop it back in and it has always worked, it's been running about 256320 hours now.

    Sure, an SSD is faster but it dies faster too, cell phones are lightweight, fit in your pocket, and if they don't fall out and die then after a couple of years you need to buy a new one anyway. My old POTS phone's still working too although I can't take a picture with it but it will keep working if I drop a picture frame on it.

    I wonder what the work would look like today if the first steam engine had died and had to be replaced after 40000 hours use? I think we'd have just said, "Junk it, let's go back to a horse and cart"

    1. stiine Silver badge

      Re: I've not seen this problem before

      True, but the wheel is only guaranteed for 3000 revolutions, and the axle for 4000.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like