back to article AWS US East region endures eight-hour wobble thanks to 'Stuck IO' in Elastic Block Store

Amazon Web Services' largest region yesterday experienced an eight-hour disruption with the Elastic Block Store (EBS) service that impacted several notable web sites and services. The lack of fun started at 8:11pm PDT on Sunday, when EBS experienced "degraded performance" in one availability zone (USE1-AZ2) in the US-EAST-1 …

  1. Anonymous Coward
    Anonymous Coward

    'Stuck IO' in Elastic Block Store

    Does that mean it broke?

    1. Anonymous Coward
      Anonymous Coward

      Re: 'Stuck IO' in Elastic Block Store

      Yes. Apparently it was in only one of six availability zones in that region, meaning the other five were working just fine.

      The takeaway, if you didn't already know this, is to distribute your workloads between AZs - or preferably between regions.

      It's no different if you self-host. The chance of your data centre being down for 6 hours due to a power failure or fire suppression event is definitely non-zero. So you need your application to be running in at least one other data centre - or at least able to be brought up quickly there.

  2. Pascal Monett Silver badge
    Flame

    "Sometimes it's hard to find the silver lining"

    Well said.

    That is not the problem of the manager who championed it, then left to go champion it again somewhere else.

    I'm starting to hate cloud. Did you know that tower cases now come without any place to use an optical drive, or any ability to plug in a USB key on the front panel ?

    I recently upgraded my PC which had been chugging along since 2010 and thought hey, while I'm at it, why not change tower ? Well today's towers expect you to throw all your data to the cloud.

    I wonder how they expect people to reinstall Windows ?

    Because that does happen, you know.

    Oh, silly me, you bring it to a repair shop to pay a PFY to hook up an external USB optical drive and do the install you can't do anymore.

    Obviously.

    1. cawfee
      Joke

      Re: "Sometimes it's hard to find the silver lining"

      Windows ships on USB's now. If you need to re-install might I suggest rotating your case about 180 degrees?

      1. Pascal Monett Silver badge
        Thumb Down

        Re: might I suggest rotating your case about 180 degrees?

        Oh sure, I'm sure that won't be a problem what with the screens, PSU, keyboard, sound and network cables, among other things, that are attached in the back.

        And yes, I obviously want to squat under the desk every time I have to interact physically with the tower.

        Brilliant idea !

        1. nematoad
          Thumb Up

          Re: might I suggest rotating your case about 180 degrees?

          Yes, scrabbling about at the back of a tower is both a pain in the arse and in the back.

          That's why I was so pleased to see when I got my last case that it had USB ports at the front.

          At my age I am no longer suited for the gymnastics need to fiddle with ports at the back once they are set up.

          1. Anonymous Coward
            Anonymous Coward

            Re: might I suggest rotating your case about 180 degrees?

            I'm not sure that hosting your application on a tower PC under your desk really compares with hosting it in a cloud data centre.

            Then again: xkcd 908.

          2. ronkee

            Re: might I suggest rotating your case about 180 degrees?

            Get a USB hub and you wouldn't even need to go under your desk.

            1. Anonymous Coward
              Anonymous Coward

              Re: might I suggest rotating your case about 180 degrees?

              USE LINUZ!!!!111

    2. Plest Silver badge

      Re: "Sometimes it's hard to find the silver lining"

      Really?!

      Do you really install an O/S off a DVD these days? The speeds are bloody awful, get a £5 USB fob off Amazon and dump the ISO into that, you can have Windows installed clean in less than 15 mins, heck I installed OSX ( 8GB O/S ) in under 25 mins from USB stick.

      You do know that USB-C can also be used to drive HDMI video right? USB is not for crappy little keyfobs for a few Word docs. My wife runs triple-A games from Steam off an attached USB drive, so don't use the missing optical drive as an excuse to have a pop at "the cloud"!

    3. This post has been deleted by its author

    4. max allan

      Re: "Sometimes it's hard to find the silver lining"

      The silver lining of cloud is easy to find. You have 5 other zones in us east. Several other regions in the us with at least 3 more zones each. Many more regions in the rest of the world.

      If your design is so stupid that it relies on a single zone, you haven't done cloud. You've taken all your existing problems and moved them to cloud. And you deserve all the downtime you get.

      And if someone let a manager move on without having run some DR scenarios on the deployment, then the senior managers are to blame for their incompetence. Don't try to push the blame for management incompetence onto the tools they are getting people to misuse.

      If a building site had a box of hammers and a manager said "you can only use one hammer" and that hammer broke, would you be complaining about how having lots of hammers was not helping the build. Or would you be calling the manager a total fudging incompetent moron who needs to get the hello off my site without expecting any more salary to be paid?

      1. hoola Silver badge

        Re: "Sometimes it's hard to find the silver lining"

        Whilst I agree with you on the design side, as with everything to do with cloud, the more you put into the design the more the price goes up.

        Increasingly costs for subscription and cloud services are just getting "nodded through" as there is a budgetary figure then the real figure. The second only because apparent when auditors start poking or someone in Finance asks why you are paying £1 million to company XYZ.

        It is just like the mobile phone and car personal leasing, all that matters is the monthly costs, the total costs are just ignored.

      2. Anonymous Coward
        Anonymous Coward

        Re: "Sometimes it's hard to find the silver lining"

        "If your design is so stupid that it relies on a single zone, you haven't done cloud."

        Right so what's the name of the AWS service that allows the EBS volume of my server's C drive to be located in two AZ's then? Thought so.

        1. Anonymous Coward
          Anonymous Coward

          Re: "Sometimes it's hard to find the silver lining"

          There’s a few ways to cut this spending on what you’re running on you “server”- but generally you’d use several copies of an EC2, each with its own EBS and replication across them that is application specific.

          If it was a database, using a managed database service makes that easier. If you want you can also take snapshot backups that gets stored on S3 (replicated for you). It’s really not that hard, it just needs someone to plan for it.

        2. Anonymous Coward
          Anonymous Coward

          Re: "Sometimes it's hard to find the silver lining"

          It's called Elastic Load Balancer. Run two or more instances of your app, in different AZs, with their own local storage for OS disk.

          For the application data there are a ton of options: EFS (which is multi-zone replicated by default), S3, RDS, etc.

          1. Ragarath

            Re: "Sometimes it's hard to find the silver lining"

            It's called Elastic Load Balancer. Run two or more instances of your app, in different AZs, with their own local storage for OS disk.

            For only 3 times the cost!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like