back to article I can't believe you've done this: Cisco.com asks visitors to explain to IT why they have broken the website

Cisco has suffered an odd series of outages that briefly KO'd its website and corporate blogs. The Register was first alerted yesterday to the blog outages, which replaced reams of corporate content with the default WordPress post-installation admin config page. Downtime started at around 2pm GMT and lasted for roughly an hour …

  1. dnicholas

    Like rain on your wedding day

    It's like when you try to phone a telecommunications company and there's no one answering the phone.

    It's not ironic, it's moronic

    1. Aladdin Sane

      Re: Like rain on your wedding day

      What we've got here is failure to communicate,

      1. Brian Miller

        Re: Like rain on your wedding day

        Not even Cisco can configure their switches properly!

        1. deevee

          Re: Like rain on your wedding day

          Its what happens when you start defining your network in software!

      2. Sandtitz Silver badge
        Thumb Up

        Re: Like rain on your wedding day

        "...some men you just can't reach"

      3. The Pi Man

        Re: Like rain on your wedding day

        Dontcha think?

    2. TRT Silver badge

      Re: Like rain on your wedding day

      I wonder what precipitated?

  2. sanmigueelbeer
    Thumb Up

    The Executive Summary will be fun reading ...

    I can't wait for the release of the Executive Summary.

    1. Anonymous Coward
      Anonymous Coward

      Re: The Executive Summary will be fun reading ...

      I managed to find a leaked copy...

      "After taking a helicopter view of this issue, we have discovered that the people responsible for maintaining the relevant systems have been let go some months earlier in-spite of these requirements being clearly communicated last month.

      Management are disappointed that the responsible parties failed to communicate potential issues, take ownership of these issues or execute their core competencies and will look at firing anyone in the near vicinity to ensure that a clear example is set.

      We will also provide additional funding to HR to purchase motivational posters for the walls to gain traction on staff morale and improve buy-in. Just like it says in my MBA text book."

      I know, not enough buzzwords, so probably fake...

      1. macjules

        Re: The Executive Summary will be fun reading ...

        Just ran that through the PRSPB (PR SpokesPerson Bullshit) app and it generated:

        "After a detailed review we have concluded that those to blame were a number of disgruntled former employees that were let go in our recent round of employee opportunity enhancements. At Cisco we pride ourselves on our core values of 100% dedication to all our customers so we will be offering a $0.01 refund to any affected customers.

        Federal authorities are co-operating with us to identify the former employees who took down our website and we are hopeful that successful prosecutions will ensue as a result."

        1. Bronek Kozicki
          Meh

          Re: The Executive Summary will be fun reading ...

          That's way too realistic, hence not funny.

        2. Aus Tech

          Re: The Executive Summary will be fun reading ...

          Well, if they still had access, that is wholly Cisco's fault, for leaving live accounts on their servers, after the services of said employees were terminated. I'm only a lowly sysadmin, but I know better than leaving defunct accounts on active servers. Ideally, they should have been removed before the people were told that they were no longer required. That way they can't go back to their desk, and leave assorted nastiness that will cause outages at some indefinite time in the future. It's not as if it takes long to get rid of the accounts.

      2. sanmigueelbeer
        Joke

        Re: The Executive Summary will be fun reading ...

        After taking a helicopter view of this issue, we have discovered that the people responsible for maintaining the relevant systems have been let go some months earlier in-spite of these requirements being clearly communicated last month.

        You sound like Kim Jong Un. Is that the reason for AC?

        1. Anonymous Coward
          Anonymous Coward

          Re: The Executive Summary will be fun reading ...

          They won't let me bring field artillery or anti aircraft guns into work so I'm only Kim Jong Un at weekends.

          I may need to look for a new employer who is more interested in efficiency.

  3. LeahroyNake

    Ummm

    Why the flippn heck is a company the size of Cisco using WordPress?

    I'm going to take a few guesses so bear with me.

    1. User accounts, it's really hard obviously.

    2. Updating and creating WordPress pages is easy for marketing and sales people that think HTML is some kind of illness.

    3. They couldn't manage to build their own CMS.

    4. It was free... Ish.

    5. Someone in management though it was cool.

    6. It's much easier than a cli... It comes with a gui :0

    7. OOO pretty plugins that are totally safe.

    This is the point that someone points out that 'everyone' uses WordPress or Joomla lol

    1. Ben Tasker

      Re: Ummm

      > 3. They couldn't manage to build their own CMS.

      To be fair, advocating building your own over using off-the-shelf is insanity unless you've a usecase which actually demands it.

      I'm not a particular fan of Wordpress, but for every dodgy marketing inspired wordpress install you can generally fimd a creaky breaky old self-built CMS that was someone's pet project until they got bored, died or left.

      The difference being that you can find a plethora of docs to tell you how you screwed up your wordpress install.

      Not Invented Here syndrome can be actively harmful. Particularly when there are plenty of choices outside those you've listed, especially if you've a budget to work with.

      Though the answer is probably that they outsourced the design/build to someone at some point. Something that's also a lot cheaper/easier to do if you haven't built your own and are using something vaguely standard

    2. Anonymous Coward
      Anonymous Coward

      Re: Ummm

      the website is now maintained by the receptionist, so it needs to be simple!

  4. Anonymous Coward
    Anonymous Coward

    Not just any multibillion-dollar global megacorp here

    We're not talking about a mega-grocery store here. It's one of the giants of networking hardware and technologies who can't keep its house in order.

    I agree with people posting here that it is very likely experienced people were let go to be replaced by young hipsters. Just looking at the new fancy Cisco.com website gives you nausea.

  5. JohnFen

    Sadly, this wording is still common

    "Please contact the server administrator, it-webmasters.cisco.com and inform them of the time the error occurred, and anything you might have done that may have caused the error."

    While I understand what it's trying to get at, this is one of the more stupid ways to words this for a number of reasons.

    First, contacting the webmaster to inform them that the error happened and what time is pointless -- that information is already in the webserver logs.

    Second, 500 is an internal server error. The odds are vanishingly tiny that it was triggered by anything a user has done. To put the onus on users to report anything about this is stupid.

    That's even leaving out the fact that El Reg already mentioned: the wording of the message can easily be interpreted by users as meaning "you have done something wrong that caused this, tell us what it was".

    1. Doctor Syntax Silver badge

      Re: Sadly, this wording is still common

      "First, contacting the webmaster to inform them that the error happened and what time is pointless -- that information is already in the webserver logs."

      Having the information in the logs doesn't mean it doesn't need to be pointed out to the webmaster.

      1. Gerardo McFitzpatrick-O'Toole

        Re: Sadly, this wording is still common

        > Having the information in the logs doesn't mean it doesn't need to be pointed out to the webmaster.

        It *doesn't* [mean that it isn't necessary for a member of the public to send an administrator an email], but it *should* do, seeing as it's the public website of a big company that ought to be able to give the impression that they know what they're doing with software on networked computers, with logging and alerting/reliability engineering and such.

      2. JohnFen

        Re: Sadly, this wording is still common

        Surely, important websites routinely check their logs and have automated alerts get sent when serious errors (such as those that cause 500 HTML codes) get logged.

        OK, I almost made it through that without laughing. In any case, they should. It isn't acceptable (or reliable) to put the responsibility for this sort of monitoring on the users.

    2. The Pi Man

      Re: Sadly, this wording is still common

      I’m more concerned that you need to contact it-webmasters.cisco.com, not it-webmasters@cisco.com ?

    3. dgeb

      Re: Sadly, this wording is still common

      > First, contacting the webmaster to inform them that the error happened and what time is pointless -- that information is already in the webserver logs.

      It isn't necessarily pointless - if the person reporting it does include context about what they were doing/trying to do, the time at which it occurred is definitely useful to correlate that report to the logs about what happened server-side.

      Agree that just reporting that an error occurred and the time is completely pointless, however.

      >Second, 500 is an internal server error. The odds are vanishingly tiny that it was triggered by anything a user has done.

      For a web application of any complexity, I would say that it is actually pretty likely that it was triggered by something specific to the user's action. Some particular edge-case or combination of events that no-one thought to create a test for. If you get a 500 in response to *any request at all* then almost certainly not.

      None of that is to say that it isn't a pretty terribly written error message, of course.

  6. sanmigueelbeer

    Back to the stone age

    This was so bad TAC had gone back to basics: Pen & paper

    This outage is another blamed on "internal system change" (first one was the WebEx outage of September 2018).

  7. Danny 2

    US - Chinese tariff negotiations continue apace

    The Cisco Kid is strong and tough and only the best is good enough.

    The creamiest filch, the whitest shit , the goodness that’s in Cisco kit

    [Huawei is the laugh of an inscrutable evil genius as your chin hits the concrete]

  8. Swampie

    Of course it is the customers fault

    The Beatings will Continue until Morale Improves!!!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like