back to article Cisco loses customer data in Meraki cloud muckup

Cisco has admitted to a cloud configuration cockup that erased customer data. The networking giant explained: “On August 3rd, 2017, our engineering team made a configuration change that applied an erroneous policy to our North American object storage service and caused certain data uploaded prior to 11:20AM Pacific time on …

  1. Anonymous Coward
    Anonymous Coward

    Well, I don't like to say...

    I told you so... oh, go on then... I told you so!

    1. Anonymous Coward
      Anonymous Coward

      Re: Well, I don't like to say...

      Not a big deal.

      It was some stuff on your personalized log in page that some have set up.

      Got a shit ton of emails from Cisco, and found it to really be a meh moment.

      1. Richard 12 Silver badge

        Re: Well, I don't like to say...

        For you, This time, Perhaps.

        For others, and next time for you, maybe more.

        To err is human, but to accidentally destroy multiple businesses, you need a Cloud.

        1. Mage Silver badge
          Facepalm

          Re: Well, I don't like to say...

          Only use cloud for copies of stuff for the public (to have capacity), or for temporary loads.

          Have a tested backup and disaster recovery plan, with off site storage (even if for small companies it's only a USB HDD the boss takes home). You backup the Cloud, never TO the Cloud as it's a 3rd party with unknown security, privacy and reliability. If you believe Cloud Marketing specs, with no independent audit, I have some bridges for sale, really cheap.

  2. a_yank_lurker Silver badge

    Forecast

    Cloudy with a 20% chance of TITSUP.

    1. TRT Silver badge

      Re: Forecast

      Cloudy with a chance of meatbag balls ups.

  3. Anonymous Coward
    Anonymous Coward

    What's the issue?

    It's not Meraki is any kind of firewall so what the worst that could happen? apply someone else's policy to your firewall or something?

    This is the cloud, everything is perfect, and good value while supported.

    /sarcasm

  4. Harry the Bastard

    not happy at all

    leaving aside the incident and clear inability to recover from backup, this isn't a trivial thing for some

    we're a meraki managed service provider, every single managed organization (i.e. customer or demo instance) has lost data, some of these are global companies with many sites

    the floor plans are gone, as are mounting photos, these aren't just eye candy, they are needed to help people find the correct device out of possibly hundreds

    floor plans can be recovered and uploaded, that still leaves scaling and positioning every one of the buggers, that'll be many days of effort, multifloor sites are especially fiddly

    but mounting photos are sent direct from the meraki app to the cloud, to replace them means someone going to every device at every site in every country and retaking them

    this is the first incident on such a scale they've had, mistakes are made, that's life

    but Cisco not having backup is not a mistake, especially as they give us no way to make our own

    1. Anonymous Coward
      Anonymous Coward

      Re: not happy at all

      "but Cisco not having backup is not a mistake, especially as they give us no way to make our own"

      I was going to chastise you for not having your own backups, but really, you can't make your own?

      We ruled them out a long time ago. We din't like the protection racket model.

      "Nice firewall you have there, be a shame should something happen to it if you don't agree to our price hikes."

      1. Harry the Bastard

        Re: not happy at all

        yep, they give no way to make our own backup, snapshot state, or, ideally, pull a complete copy of an instance offline to save

        our policy is to keep records of everything practical, we manually document a lot even though it adds overhead, but as they claim to have our data replicated multiple times and to back it up nightly, it seemed like it really ought to be recoverable in the event of mistake!

        yeah, it's not right for all situations, aside from the licensing model the configuration constraints sometimes rule it out, but if you're a corporate the overall tco is fine, and they are generally good at things like homologation and logistics requirements for global deployment - some of the alternatives are pure pain once you get outside usa/eu

        1. TRT Silver badge

          Re: not happy at all

          The fact that the units keep running with the same configuration file if they can't reach the CnC servers is a good feature. That you can't keep your own copy of that configuration file and point the unit to it as a failover is a PITA.

  5. Anonymous Coward
    Anonymous Coward

    The Cloud...

    Other peoples computers you have no control over.

  6. TRT Silver badge

    That's a shame...

    Meraki made some really nice kit. Vastly overpriced of course, on the "network-as-a-service" side of things, but nice kit all the same.

    1. Tom Paine

      Re: That's a shame...

      Meraki made some really nice kit. Vastly overpriced of course, on the "network-as-a-service" side of things, but nice kit all the same.

      So do Aston Martin, and I'm not running one of those, either.

      1. TRT Silver badge

        Re: That's a shame...

        I had one of their free trial AP units with 3 years licensing. Stopped working as soon as the 3 year ran out. £350+ to renew each year. They can go suck a fat one. And they expect me to make purchasing decisions based on that price??!!

        Now if Aston Martin gave me a Vanquish free for three years, I'd take them up on the offer too. I wouldn't recommend them to the fleet the company.

        1. Harry the Bastard

          Re: That's a shame...

          "I had one of their free trial AP units with 3 years licensing. Stopped working as soon as the 3 year ran out. £350+ to renew each year. "

          £350??? who told you that?

          even if you buy just one year, per-AP is $150 rack rate for license and hardware replacement service (LIC-ENT-1YR), it drops fast if you go multiyear... $300 for 3-year, i.e. $100/year

          1. TRT Silver badge

            Re: That's a shame...

            That's the price I was quoted by one of their salespeople. It was very hard to find any other source for their rates. But even $150 is a lot.

            Or maybe I mistook their quote, the line was bad. Could have been for 3 years.

  7. Allonymous Coward

    I can't believe I got in first with the obligatory XKCD

    And here it is:

    https://xkcd.com/908/

  8. Anonymous Coward
    Anonymous Coward

    Bad weekend for Cisco

    The Meraki issue is minor compared to the global Spark outage this AM. 10:58-16:38 UTC.

    https://status.ciscospark.com/

    1. Anonymous Coward
      Anonymous Coward

      Re: Bad weekend for Cisco

      "upstream service provider issue" - I wonder if that is cloud based, oh no cloud doubting me again there.

      Upstream sounds more bucolic than some other term like "another cloud failure" so don't worry, babbling stream, nice thoughts, relax, keep up the payments.

  9. Anonymous Coward
    Anonymous Coward

    Outages not the same as lost data

    Your comparisions to AWS and Google are not relevant.

    (But this reminds of the time we had two similar locked boxes in the mailroom, one for off-site storage, the other for secure recycling. I remarked on the similarity to a colleague, who replied: it only matters when you need to restore a backup.)

  10. Oh Homer
    Coat

    Meerkats did WHAT?

    Oh wait...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021