back to article Cloudflare's outage was human error. There's a way to make tech divinely forgive

Edge is terribly trendy. Move cloudy workloads as close to the user as possible, the thinking goes, and latency goes down, as do core network and data center pressures. It's true  – until the routing sleight-of-hand breaks that diverts user requests from the site they think they're getting to the copies in the edge server.  If …

  1. Pascal Monett Silver badge

    "a hangover from the days when storage and CPU were too expensive"

    Indeed, they are no longer expensive - at least not when compared to their cost fourty years ago.

    Unfortunately, in that time we also have learned to use up all the CPU with new OS versions (eh, Borkzilla ?), to no longer really care about our RAM usage and, since disk space is a penny a GB, well we clutter our things with lots of useless data.

    All this edge stuff is being implemented willy-nilly, and we are collectively learning the consequences of the absence of planning.

    Cloudflare is going to have to revisit its update procedures and its engineers are going to have to revisit their panic intervention process.

    Hey, everybody's learning something, so it's all good.

    1. Neil Barnes Silver badge

      Re: "a hangover from the days when storage and CPU were too expensive"

      absence of planning is planning for absence?

    2. Missing Semicolon Silver badge

      Re: "a hangover from the days when storage and CPU were too expensive"

      They still are expensive.

      If you look at standing up internet-scale systems, with clustering and orchestration and such, even moderate-sized stuff runs to thousands a month.

      Why to you think AWS is so rich?

  2. Flocke Kroes Silver badge

    beckups

    Did anyone else stop at that point then read the rest of the article with a posh accent?

    1. Anonymous Coward
      Joke

      Re: beckups

      > Did anyone else stop at that point then read the rest of the article with a posh accent?

      I got as far as edge lords of large chunks of web content and started to wonder if there was some Glastonbury-related inspiration to his writing?

      1. Anonymous Coward
        Anonymous Coward

        Re: beckups

        When it mentioned edge, I thought it was about MS Edge.....

    2. elsergiovolador Silver badge

      Re: beckups

      I read it in Toast of London voice...

    3. ragnar

      Re: beckups

      A New Zealand accent!

  3. OhForF' Silver badge

    "Move cloudy workloads as close to the user as possible"

    First they say we get all kind of benefits by moving all workloads to the cloud then they push it back ever closer to the user.

    If the trend continues we'll end up with a PC (Personal Cloud).

    Will the cycle from centralized computing on a server to personal computing on a local machine and back to the server ever end?

    1. Mayday Silver badge
      Paris Hilton

      Fog and dew computing

      Are “things”

      See if you can manage to look them up with scoffing or choking.

      Before you ask - I don’t think I could make such wanky terms up.

    2. Peter2 Silver badge

      I've had a Personal Cloud for decades in the form of my web hosting package which provides cloud email (aka webmail) as well as imap or pop3 access, in addition to my own personal version of OneDrive in the form of mapped FTP drives.

      My personal feeling is that the cycle from centralised computing back to personal computing will probably never end.

      You'll have a monthly plan for a centralised service which will get steadily more expensive until somebody dusts off the concept of storing it on your machine and a NAS box and releases a new bit of software which they sell.

      Except that then they have to keep patching it, and the only way of getting paid is to release new versions, even if they don't really add anything useful. And them somebody will turn around and produce a "cheap" monthly subscription to a centralised service to replace this.

      I suspect the cycle will continue ad infinitum.

    3. Anonymous Coward
      Anonymous Coward

      Yes, the new Cloud(TM) super fast, resilient to Internet outages future ....

      "Please plug this server sized box into your network"

    4. Doctor Syntax Silver badge

      "Will the cycle from centralized computing on a server to personal computing on a local machine and back to the server ever end?"

      No. Once a particular state of the cycle has become more or less universal there's not much more money to be made out of it so profits demand pushing it along once more.

      1. Numen

        Not only that

        But when you've used a certain paradigm long enough, you'll find there are problems that are hard to solve. So there's a stampede to a "new" paradigm that doesn't have those problems. Then, when you discover that approach has problems, head over to the "newer" one.

        Rinse and repeat.

        Centralize, decentralize. Cloud, on-prem. Big systems, groups of smaller systems.

        Seems to be about 3 paradigms that run in a circle. It's fun to watch something from the 60's or 70's be rediscovered. (Server class memory is one example.)

        The new approach is the solution to all problems, until it isn't.

    5. Anonymous Coward
      Anonymous Coward

      Hmm. Personal cloud. Like a box you can set on or under your desk that contains all the data and programs you need, right there - can't be disrupted by network issues, CDN problems, etc. Being so local, you could even hook a keyboard, monitor, and mouse directly to it.

      Wait, I think I had one of these in the early 90s...

  4. Anonymous Coward
    Anonymous Coward

    edge lords

    I saw what you did there…

    Let's just hope that CloudFlare have a sense of humour, and The Register does not suddenly find its ejaculations falling into the darkness, «clicketty-click»… ;-P

  5. DougMac

    Rollback...

    Apparently the author didn't read the indepth Cloudflare analysis.

    But, they already _had_ a known good config fallback.

    The biggest problem came about when different engineers kept falling back to different known good config states.

    1. Doctor Syntax Silver badge

      Re: Rollback...

      You've answered one question I was asking myself: how do you know which state to restore? The answer seems to be "you don't".

  6. Crypto Monad Silver badge

    "Yet if it happens, and keeps happening, why aren't systems more resilient to this sort of problem?"

    Conversely you could also ask: given the ludicrous complexity of these loosely-coupled mega systems we're deploying, why don't these sort of problems happen more often?

    "Is it actually universally true that state can be retrieved?"

    Yes: turn it off and on again.

    This is a widely deployed solution, and sadly often necessary when state gets tied up in knots.

  7. Surrey Veteran

    BGP’s are the sort of things that you only remember that exist once that is broken. Specially those pretty old Linux boxes on comm rooms , I remember horror histories.

    Anyway more than algorithms, what you really need is a sensible approach to change management, I.e. make sure that you backup config files, etc and specially regression testing,

  8. hashBang

    Tonight - enhance the network with the new hyper availability always up changes.

    Tomorrow - fix the outage caused by changes made the previous night.

  9. Anonymous Coward
    Anonymous Coward

    Quibble!!

    Quote: "...files, static collections of bits, are nothing but state. Make a copy of this, store it with a label and a timestamp, and you're done...."

    Where are these "files, static collections of bits"? This quote is true for static (and not open) files on persistent storage. But.....some of my code uses persistent files in memory (you know.....mmap to speed things up locally). So......there are some arrangements where "files....are state"......but not the state you want to backup!!!!

  10. Toni the terrible Bronze badge
    WTF?

    Wonderful for Cloudfare

    Fixed everything?

    No, Cloudfare is still blocking me and others from using websites like Just Eat, Crunchyroll etc as a paying customer; mostly when I use Firefox but also Chrome to a lesser extent and other browsers too. I dont even get to prove I'm human - though CAPTCHA really doesn't do that. I normally just move onto a rival

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like