back to article AWS wobbles in US East region causing widespread outages

Technical errors with the US-EAST-1 region of Amazon Web Services have caused widespread woes for customers, including difficulty accessing the management console and some other service problems. The issues appear to be centred on the US-EAST-1 region, which is the oldest AWS region and located in Northern Virginia. This can …

  1. fredesmite2
    Mushroom

    Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

    Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

    using someone else's computer system .. thinking they care about it as much as you do.

    1. Gene Cash Silver badge

      Re: Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

      Don't forget, cloud computing started as Amazon selling the excess data center capacity it only used during the Christmas rush...

      So I'm not surprised it wobbles during December.

      1. gurrman

        Re: Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

        Im not sure if serious but thats not how AWS got started. A proposal doc was written, it got approved and Amazon built the first service.

        I used to believe the "spare capacity" thing too until I spoke so someone who knew and then thought about it and it didn't really make sense at all - were they just gonna shut down all their customers every Christmas?

        1. Anonymous Coward
          Joke

          Re: Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

          > were they just gonna shut down all their customers every Christmas?

          Well they've just done it now!

          <badum-tish>

    2. Phil Kingston

      Re: Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

      There's a bit more to it

    3. Anonymous Coward
      Anonymous Coward

      Re: Remember - CLOUD COMPUTING is NOTHING MORE THAN ..

      "CLOUD COMPUTING is NOTHING MORE THAN ..using someone else's computer system "

      It's also someone else's data centres, power supplies, HVAC, physical site security, patching, backups, trained personnel, etc.

      Excellent, means I don't have to do all that stuff thats commodity and focus on value add stuff on top.

      It's someone else's sewage works...

  2. Must contain letters
    FAIL

    I’d love to comment but…

    .. the bloody internets broken again. Remind me why the bean counters thought this OAAS* was a good move?

    Outage as a service. (tm)

    1. bombastic bob Silver badge
      Unhappy

      Re: I’d love to comment but…

      I've been seeing load performance issues with things relying on googleapis all week, and youtube videos that just sit there not playing for several minutes within the last hour.

      More may be broken than people are willing to admit...

      (then again this is my anecdotal experience out here on the left coast of the USA, so YMMV)

      1. CrazyOldCatMan Silver badge

        Re: I’d love to comment but…

        and youtube videos that just sit there not playing for several minutes

        Judging by the quality of much of Yootuube that'll be a benefit..

  3. Jason Hindle

    Strange. I can't log into my AWS account at the moment

    But I can ssh to one of my EC2 instances.

    1. Androgynous Cow Herd

      Re: Strange. I can't log into my AWS account at the moment

      by IP, or by FQDN?

      By IP...I'm not surprised.

      DNS...it's ALWAYS DNS

      1. Jason Hindle

        Re: Strange. I can't log into my AWS account at the moment

        Just checked my .ssh/config and it is actually the fqdn I'm using. Surprising - I do agree it is usually DNS.

  4. Dunstan Vavasour

    Everything seems to depend on us-east-1

    Unable to create a S3 bucket in eu-west-1

    Make multi-region redundancy seem a little less than convincing.

    1. Peter-Waterman1

      Re: Everything seems to depend on us-east-1

      I can create a bucket in that region without issue. There are no services that span multiple regions. Every service is tied to a specific region, - s3 in Ireland is independent of s3 in Frankfurt. Even the global console is pinned to a single region, the effected region, all other region consoles are available.

      1. Kevin McMurtrie Silver badge

        Re: Everything seems to depend on us-east-1

        Everything seemed to depend on us-east-1 this morning. It looks like Amazon is shuffling internals as fast as they can.

  5. michaelvirks

    Big learning: never just have a root account on AWS, even if you are am occasional user one-man-show.

    Apparently individual IAM user accounts can log in through region-specific management consoles like eu-west-1.console.aws.amazon.com, eu-central-1.console.aws.amazon.com etc

    While all root account login requests get redirected to us-east-1, the only region that handles root login requests, which is also the region affected by the outage.

  6. Anonymous Coward
    Joke

    A developer tweeted

    > One developer said "AWS goes down and I spend 2 hour trying to debug why my code is not working," illustrating the extent to which public cloud services are assumed to be up and running.

    Error checking. We've heard of it.

    1. Howard Sway Silver badge

      Re: A developer tweeted

      We've heard of it. But those who have been convinced to run stuff in the cloud because they think it's always available are the same sort of people who think they don't need to check for connection failures - because it's "always available".

      1. myootnt

        Re: A developer tweeted

        Troubleshooting 101: Take the OSI model, add a layer 0 and a layer 8.

        Layer 0: Is it plugged in to an active power source?

        Layer 8: Does the user know what they are doing?

        Work your way from either direction through the layers. Results guaranteed.

    2. buiv

      Re: A developer tweeted

      Good Point

  7. Merrill

    "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable"

    Leslie Lamport, 1987

  8. PandyH

    Although Amazon are saying to use a regional console to access the management console, you still cannot login.

    Accessing the eu-west-2 console gets you a login prompt, followed by an "Internal Error" - looks like at least the authentication all hinges on us-east-1.

  9. Anonymous Coward
    Anonymous Coward

    Cloud is shit

    Using cloud for ANYTHING business critical is wrong. Completely wrong. Own your infrastructure.

    1. Michael Hoffmann Silver badge
      Facepalm

      Re: Cloud is shit

      Because we NEVER had business critical outages until the cloud came along, amirite?

    2. Anonymous Coward
      Anonymous Coward

      Re: Cloud is shit

      Own your own data center?

      Own your own power station?

      Own your own ISP?

      Or just own the things you have a few years of experience administering?

    3. Plest Silver badge
      Facepalm

      Re: Cloud is shit

      No, use cloud if you like but plan like you would your own datacentre. So number 1 concern on the very first planning meeting....what do we do when it all goes TITSUP?

      Then work on from there for every other concern!

      1. Strahd Ivarius Silver badge
        Devil

        Re: Cloud is shit

        what do we do when it all goes TITSUP?

        Blame the IT team of course!

  10. Marty McFly Silver badge
    Thumb Up

    The value of the cloud!

    Can we accept the premise that 'crap happens'? Regardless of whether I have an on-prem or cloud infrastructure, let us assume that 'something will always go wrong at some point'. Nothing can be done about it, a bad day will happen eventually.

    With on-prem, I scramble to sort out the problem, go 24x7, buy take-out food for the grunts, and pray I get it solved before the powers-that-be bring me a platter upon which to place my head.

    With the cloud, I can sit back and watch someone else own the headache. And when it is all done, the powers-that-be can take a bite out of a cloud vendor's backside.

    Once we get past the denial of imperfection and accept that the inevitable outage will eventually happen, then we can see the value of letting someone else eat the crap-sandwich.

    1. This post has been deleted by its author

    2. Anonymous Coward
      Anonymous Coward

      Re: The value of the cloud!

      That's all very well but for a lot of us clearing up the crap was feeding our families.

      1. The Basis of everything is...

        Re: The value of the cloud!

        Hmm. Maybe designing and operating things so crap didn't occur in the first place was a much better way of feeding your family?

        Tech evolves. Get used to finding a new way to feed your family every few years. And as somebody who has spent so much time cleaning crap I suspect you might have some clue on how things should be done. So you get more food for your family, and time to sit and eat with them. Win.

        1. Flywheel
          Facepalm

          Re: The value of the cloud!

          Maybe designing and operating things so crap didn't occur in the first place

          I believe that's the normal way of doing things, but then there are bean-counters and budget reviews to f*ck things up. That's the reality.

      2. Peter-Waterman1

        Re: The value of the cloud!

        I think this is the biggest myth to cloud that people have when they don’t understand. I have been working in the cloud industry for 5 yrs, and I can say I have not once seen a company that’s moved to cloud suddenly be able to get rid of people. Your network admin becomes the cloud network admin, your security team become the cloud security team. Your widows admins becomes the cloud admins and so on. The only roles that I can think of that are in danger are the security guards abs people who manage the racking of servers.

        1. Plest Silver badge

          Re: The value of the cloud!

          "and I can say I have not once seen a company that’s moved to cloud suddenly be able to get rid of people"

          Absolutely! 'Cos software is built by people ergo it WILL fail, not IF but WHEN it fails the Ops teams are all ready with our IT equivalent of a mop and bucket!

        2. JohnSheeran

          Re: The value of the cloud!

          You get a downvote. Your comment is analogous to "racism isn't real because I haven't personally seen it".

          Cloud provider's have earned a reputation of "fire all of your infrastructure people because DevOps".

          Cloud technologies offer a lot of very good things but the real resistance comes from the fact that a lot of companies have cut loose a lot of their key people because they won't need them in the cloud. We all know that's not true but that's exactly what providers like AWS sell to companies.

    3. Anonymous Coward Silver badge
      Big Brother

      Re: The value of the cloud!

      "And when it is all done, the powers-that-be can take a bite out of a cloud vendor's backside"

      hahahahahahahhahahahhahahahhaha

      That's not how it works. They still take a bite out of your backside because you were the one who recommended/implemented/in some way touched the cloud system. The only difference is that you have someone to shout at as well. It's the chain of screaming.

    4. Anonymous Coward
      Anonymous Coward

      Re: The value of the cloud!

      The problem with what you’re saying is that you don’t get to prioritise what gets fixed. And from experience, AWS don’t give a shit about anyone other than a few large customers. Everyone else can fuck off and wait in line with no means of escalation.

  11. Anonymous Coward
    Anonymous Coward

    For crying out loud!

    I only quite that place less than a day ago and already they can't keep the lights on?!

    1. Don Casey

      Re: For crying out loud!

      Speaking of keeping the lights on... woke up this morning (West Coast USA) to discover Alexa refused to turn on any of my lights (which are 50% on some combo of WEMO, FEIT, Belkin, etc...). Fortunately the phone-based app and/or the Samsung hub all worked just fine.

      1. Korev Silver badge
        Joke

        Re: For crying out loud!

        I woke up this morning and the lights weren’t working, so I walked over to the wall and pushed this plastic thing and then everything went bright.

        I don’t think this technology will ever catch on…

        1. Plest Silver badge

          Re: For crying out loud!

          Funny, mine is a closed circuit in house light management system 'cos I'm tech savvy, not a troglogyte and know that connecting something vital like heating and lighting control to a server 5,000 miles away is bloody stupid. So my central control unit is 6ft away under the stairs and worst case I too can just override the management system with a wall switcht to bathe my rooms in glorious Swan(*) designed electric light!

          (*) Joseph Swan, a Geordie, invented the electric light bulb, no that two-bit, self promoting yank Edison!

          1. Don Casey

            Re: For crying out loud!

            Yes; hence the phone apps and hub... what Alexa brings to the mix is voice control.

            1. Strahd Ivarius Silver badge
              Joke

              Re: For crying out loud!

              That is why I named my daughter Alexa so I have a fallback if Bezos' one doesn't wake up

  12. GlockandRoll

    Everything woke turns to shit...

  13. DrM
    FAIL

    The baskets keep getting bigger

    Put all your eggs in one basket, save big bucks!

    1. onefastskater

      Re: The baskets keep getting bigger

      I tend to believe Mark Twain hit it on the head when he said, “Put all of your eggs in one basket and WATCH THAT BASKET.

  14. ronkee

    It's interesting to see the wider single points of failure that crop up during an outage. We see support site and status pages going down. That's not well architected.

    This also makes an argument for treating cloud as a utility.

    1. disgruntled yank

      Utility

      Like the electrical grid in Texas?

  15. disgruntled yank

    Cloud

    A reminder from two years ago: https://twitter.com/pragmaticandy/status/1168916144121634818

  16. Anonymous Coward
    Anonymous Coward

    AWS Cloud just gives me access to more and better servers, storage, switches, bandwidth and scripts than we could afford or have the time to develop ourselves. It's still going to go down but at least they give me the option to run on multiple regions . . . oh wait, my public sector customers only want their data in the UK, even encrypted and held on encrypted disks and transmitted encrypted and even double encrypted backups . . . oh, ok, so that's London then, so it's just like my old co-lo space but bigger . . . right :)

    1. Anonymous Coward
      Anonymous Coward

      Could you tell me about those ‘better’ servers, and that ‘better’ storage? I’d genuinely like to know because I’ve never actually seen evidence of that. Makes, models, specs etc…

  17. Anonymous Coward
    Anonymous Coward

    remember folks, dont be a bare metalson! - https://www.youtube.com/watch?v=AXxr0pghWS0

  18. Kimo

    Ohio State University spent years getting faculty to put everything into the online course management system, and then moving it from their hardware to AWS and Microsoft clouds. Yesterday was the second to last day of semester, with lots of projects and quizzes due.

  19. Anonymous Coward
    Anonymous Coward

    AWS is best! Russia all for all USA wins!

  20. martinusher Silver badge

    I thought I'd broken it....

    Yesterday I dragged out the (artificial) Christmas tree, got it hooked up to a smart plug and set up the "Christmas" skill in the Echo to operate it. On testing it each switch worked OK, then the skill......and then everything stopped working. "Damn!", I thought, looking forward to a fun time troubleshooting this because there's zero diagnostics in those IoT things. After thinking about it for a bit I reckoned that since those smart plugs had disappeared and the central heating thermostats had also disappeared that the problem was probably in the link between AWS and the manufacturers' websites. I had to go out for a couple of hours so I left it and sure enough when I returned everything was working as intended. Obviously someone had unplugged AWS and plugged it in again.

    I don't mind using "the cloud" for my Christmas lights, garden lights or my Lava lamp. I can't think why anyone would be insane enough to use this technology for anything mission critical. There are just too many points of potential failure. There's zero diagnostics -- its a mess. (Incidentally, my Christmas lights are on the Pacific coast, the AWS outage was in the east, we're told, and Heaven only knows where the systems that drive my plugs and thermostats are based. A ton of technology to substitute for a simple PLC that just sets a control bit......its insane. Obviously vendor lock in is more important than functionality but then I know that, having spent years pushing back against marketing and its so called "bright ideas". Still, its given me a new project -- hack the protocol and get all this cloud crap out of the loop.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like