back to article Major AWS outage across US-East region breaks half the internet

A major outage is affecting Amazon Web Services (AWS), with even Amazon's own web page reported to be offline and dozens of other online services and websites affected, including disruption in the UK. AWS reported an issue to its Health Dashboard at 12:11 AM PDT (7:11 UTC) of increased error rates and latencies for multiple …

  1. Korev Silver badge
    FAIL

    Too much in us-east-1

    Too much of the planet relies on us-east-1, society should start removing reliance on this point of failure...

    1. breakfast Silver badge
      Facepalm

      Re: Too much in us-east-1

      There's really no excuse for British companies to be reliant on that region in most cases - yes some services only operate out of us-east-1 but I bet a whole bunch of these are just places where technical teams used the default region because it didn't occur to anybody to change it.

      1. lordminty

        Re: Too much in us-east-1

        I think "technical teams" is doing some heavy lifting there!

        1. JWLong Silver badge

          Re: Too much in us-east-1

          "I think "technical teams" is doing some heavy lifting there!"

          Means the guy with the abacus is working on it!

          1. Yet Another Anonymous coward Silver badge

            Re: Too much in us-east-1

            "I think "technical teams" is doing some heavy lifting there!"

            Who else is going to turn it off and on again ?

          2. Apocalypso - a cheery end to the world Bronze badge

            Re: Too much in us-east-1

            > Means the guy with the abacus is working on it!

            Oi! It takes skill to work an abacus :-)

            1. MyffyW Silver badge
              Coat

              Re: Too much in us-east-1

              Now, don’t underestimate the abacus. In skilled hands it’s a very sophisticated calculating device. Furthermore it requires no power, can be made with any materials you have to hand, and never goes bing in the middle of an important piece of work.

              So an electric one would be particularly pointless...

      2. elsergiovolador Silver badge

        Re: Too much in us-east-1

        eu-west-2 is still under Cloud Act and profit unlikely stays in the UK.

        Ergo British companies shouldn't use it at all.

        1. Macka

          Re: Too much in us-east-1

          If they have their data in US-EAST-1 already, they aren't going to care about that.

        2. Yet Another Anonymous coward Silver badge

          Re: Too much in us-east-1

          We only use West-Airstrip-1

      3. Mike 137 Silver badge

        Re: Too much in us-east-1

        "There's really no excuse for British companies to be reliant on that region in most cases"

        An analysis on the BBC lunchtime news suggested that a technical sub-component (e.g. DNS) based in US-east-1 may be sued by the hosting services based in UK/EU. So the customer is not either directly or knowingly using US-based services at any level.

        1. Anonymous Coward
          Anonymous Coward

          Re: Too much in us-east-1

          "a technical sub-component (e.g. DNS) based in US-east-1 may be sued by the hosting services based in UK/EU"

          I know that people can be sued for almost any reason in the USA, but suing "DNS", rather than a person/organisation, in the USA seems to be taking things a bit far! lol

          1. ariels-again

            Re: Too much in us-east-1

            It's always DNS. AAAA, I say sue them no matter what.

            1. MyffyW Silver badge

              Re: Too much in us-east-1

              80% of problems occur at the physical layer. 80% of everything else it's name resolution...

      4. Anonymous Coward
        Anonymous Coward

        Re: Too much in us-east-1

        Not sure if this is representative (anon obv.) but some software I use seems to connect to US East too eagerly.

        Some software I have to use for work (in UK) is from a US company.

        If even the slightest issues on AWS UK, it does not sensibly seem to fall back to something "close" e.g. Ireland, or another European AWS zone, instead it invariably ends up on USA East zone.

        Once it is on US East (even if issues on that AWS zone) it seems loath to leave (even though UK is set as default AWS zone)

        Whether that's a software issue or an AWS issue I do not know

        1. RegGuy1
          Joke

          Re: Too much in us-east-1

          it does not sensibly seem to fall back to something "close" e.g. Ireland, or another European AWS zone

          Did someone tell the AI code generator to create brexit friendly code?

      5. Anonymous Coward
        Anonymous Coward

        Re: Too much in us-east-1

        Doubtful. There are so many vestigial links to us-east-1 (being the first AWS region) that you can host whatever you want outside of the region...you're still exposed to internal AWS process and control planes operated there.

        Everyone wants distributed cloud and a hyperscaler breakup, yet no-one seems to be willing to pay the tens of billions required. So here we are.

      6. Craig100

        Re: Too much in us-east-1

        It's worse than British companies being reliant on it, as the GovUK OneLogin system went down. It appears private and personal data is going off-shore! So that'll make all your personal data subject to a DOGE/CIA data grab if they so wish. To me, this is shocking. Why aren't the UK Gov't system on sovereign IT estate. It's not like they don't have the hardware and expertise available. Passing business to Tony Blair's mates I guess.

        1. Excused Boots Silver badge

          Re: Too much in us-east-1

          "Why aren't the UK Gov't system on sovereign IT estate”

          That’ll be because there is no sovereign IT estate. And it’s not just ‘build a datacenter or two’ it’s all the infrastructure stack behind it.

          Think of it like this, suppose the government were to now work towards and pay for a completely UK-baed system, effectively replicating AWS / Azure infrastructure. However the sheer cost of doing this would require the closure of 25 major hospitals, a 10% reduction in NHS funding and 15% reduction in pension and other benefits - but, possibly, in ten years the UK would be ‘independent’.

          Or the tabloid press run with ‘Government spaffs all this money on bespoke UK system, pensioners starve, while a commercial setup is already there and would cost a fraction of the price’. It's a tough call, yes of course, with the wisdom of 20/20 hindsight really should have gone for a homegrown solution a decade or more ago. But we didn’t and neither did anyone else.

          Why would they?

          No this incident doesn’t necessarily mean that private data is being shipped offshore (well maybe it is but that’s another matter), what it does mean is that AWS is not quite as resilient as you might think or hope.

          Anyone who reads the 'On-Call’, or ‘Who Me’ sections of here will be aware of the dangers inherent of the mysterious legacy ‘box’ in the corner which everyone is afraid to touch!

          1. Roland6 Silver badge

            Re: Too much in us-east-1

            The UK government had more than enough datacentres to build out its own cloud prior to 2012, but through the buy-not-build mindset throw money at the US cloud providers rather than grow UK capability.

            That is a lot of money that mostly exits the UK domestic economy, leaving the government unable to pay pensioners...

            Yes hindsight is telling us the Tories didn't really have a clue as to how to run a country, not saying Labour are going to be any better and to do a fair comparison we really need to have them in office for the vast majority of the next 45 years...

            However, with the Whitehouse demonstrating it will resort to extortion, a government that wishes to remain "sovereign" is going to have to bite the bullet...

      7. Anonymous Coward
        Anonymous Coward

        Re: Too much in us-east-1

        the problem is - as i understand it that the issue isn't nessicarily that people designed stuff in the way that meant they where reliant on US-EAST-1, what happened is they are reliant on a service in their local region i.e London, but it's services (probably unbeknown to them) are reliant on US-EAST-1, dynamodb global tables and also IAM are two key that are reliant on it.

        The other thing is US-EAST-1 used to be critical for AWS for billing, so that will likely mean lots of services have a reliance on that.....

        In a nutshell, most people will have followed AWS best practice and ensured that their platform could handle the failure of a zone/region etc, but Amazon haven't ensured that their regions be able to stand up to the failure of US-EAST-1.

      8. Anonymous Coward
        Anonymous Coward

        Re: Too much in us-east-1

        I disagree. Plenty of us work for the British arm of a nominally US company, and so a lot of our infrastructure is logically in AWS East1.

        Also, the servers in AWS London for us at least had capacity constraints. At a higher price.

    2. a_foley

      Re: Too much in us-east-1

      Good luck removing reliance on US-east without catching the ire of the King of America, because he wants all that juicy data for himself!

      1. FIA Silver badge

        Re: Too much in us-east-1

        It doesn’t matter where the data is. If it’s controlled by a US company they can be compelled to hand it over.

        1. a_foley

          Re: Too much in us-east-1

          True. But it's easier if it's on US soil.

          1. Anonymous Coward
            Anonymous Coward

            Re: Too much in us-east-1

            All that matters is that the person who can authorise that access is on US soil.

        2. Anonymous Coward
          Anonymous Coward

          Re: Too much in us-east-1

          I think that that statement is a little unclear. It may mean that for some types of data being stored in European regions, Amazon has a choice of either breaking US law, or breaking the law in whichever the region the data is being held in.

          I'm pretty sure that I know which would win, but if, for example, large amounts of personal data as defined by GDPR-UK relating to UK individuals deliberately being located within, say, EU-WEST-2 (as I've been told is the main UK region) was being demanded by US agencies, this may trigger a diplomatic spat between the UK government and the US, with Amazon being caught in the middle. In theory, making a decision either way could cause serious fallout and fines for Amazon.

          Of course, the US laws demanding that data should be handed over almost certainly contain a gag order, to prevent the company hosting the data from letting anyone in the UK (or anywhere else) know it has happened.

          After the passing of the US Cloud Act, every government outside of the US should have done a risk review of their cloud usage. Did they? I doubt it.

          Since that act was passed, the only sensible option for governments other than the US was to stop using US owned cloud services.

          I am not an AWS architect, but it came as a bit of a shock to me that even though you opt to keep data and processing local to a region in your geopolitical domain, that US-EAST-1 could be s SPOF. That alone would make me think twice about even using AWS as a service, but that is my opinion, and I'm not in any way able to influence any decisions over this. Would this also affect local AWS private instances where you're using Amazon services to maintain local resources? (Please bear with me, I'm not that AWS knowledgeable).

          1. phuzz Silver badge

            Re: Too much in us-east-1

            Of course, if data on UK citizens made it's way to the US, then the British intelligence services could then get access to it thought their sharing agreements, without worrying about any of those pesky warrants or anything.

            1. Tron Silver badge

              Re: Too much in us-east-1

              That's the basis of the 'five eyes' agreement. You spy for others, they spy for you. No domestic laws broken.

          2. BristolBachelor Gold badge

            Re: Too much in us-east-1

            All our sensitive data is encrypted before uploading to US owned cloud services, even though there a contact that says it's only saved in Europe, never exported to US.

            1. Peter Gathercole Silver badge

              Re: Too much in us-east-1

              OK. You've encrypted it. How do you process it in the Cloud without storing the keys in the Cloud?

              You can decrypt it locally as you bring it down for local processing, but if you use it in the Cloud, Cloud based processes need to decrypt it, and if the keys are in the Cloud, they're just as vulnerable to being snaffled by the US as the data they're trying to protect!

              I've never understood the argument of encryption being the solution. Maybe someone can enlighten me as to what I've missed.

              1. Anonymous Coward
                Anonymous Coward

                Re: Too much in us-east-1

                I assume/hope! he's talking about offline backup storage

              2. sweh

                Re: Too much in us-east-1

                Never use the "server side encryption" (SSE) function of things like S3 because they are totally dependent on the Cloud Service Provider's (CSPs) tooling and the data is processed unencrypted on the CSP infrastructure. I made it a hard requirement for storing sensitive data in the cloud; S3 SSE was necessary for all class of data, but insufficient for sensitive data.

                For that we also use "client side encryption".

                In this, the keys need not be stored "in the clear"; they can be stored in HSMs (even on-prem HSMs), and pulled into compute memory for use in encryption/decryption. Good software even fragments that key throughout memory so it's not contiguous and need not even be in the same locations during an execution.

                Is this impossible to break? No, but it would require the CSP to take a memory snapshot of your machine and then try to find where the fragments exist and reconstruct the key. Or perhaps try and find the credentials to the HSM and spoof network traffic with the necessary identity to get the HSM to release the keys.

                This is an active attack and a LOT different to the CSP handing over of keys they possess to the authorities.

                1. druck Silver badge

                  Re: Too much in us-east-1

                  What you have described is a hardware key store.

                  A HSM doesn't just store keys to be brought in to memory, the keys should never leave HSM* and the HSM does all the encryption and decryption.

                  * Except when they are exported, encrypted by a HSM key, for importing to another HSM.

                  1. Peter Gathercole Silver badge

                    Re: Too much in us-east-1

                    If you were to be storing the keys in an HSM, you would require to always process your data on the systems with such a primed HSM. As I understand it, this completely negates the flexibility that Cloud provide. No scaling beyond the systems with the keys in the HSM, no moving regions.

                    If you give a facility to move the keys around, then the keys could be stolen through the same route, and if the keys are in locked keystores, the key for the keystore also has to be available.

                    If you want automated processing in the cloud, the cloud has to have some way of accessing the data.

                    1. druck Silver badge

                      Re: Too much in us-east-1

                      As I wrote in the small print you can export keys to other HSMs wrapped by a key only known to the HSM, which again is never present memory of the computer so can't be stolen through the same route.

              3. Martin M

                Re: Too much in us-east-1

                At some point, homomorphic encryption might help, at least for some use cases. Computing on untrusted hardware, without decrypting data at any point. Wildly impractical for almost everything right now though.

          3. retiredFool

            Re: Too much in us-east-1

            If you have any doubt about what amazon would do, simply look at the latest king don temper tantrum with Zelenskyy. He didn't do exactly what don wanted, so the response was predictable. Cut up Ukraine into pieces. Imagine what would happen to Bezos if amazon did not comply with a trumper order. Its pretty clear, the us is now a dicktatorship.

          4. POFaced

            Re: Too much in us-east-1

            So...you're not wrong, but there's some nuance to this. You can store all your PII or whatever data entirely outside of the impacted region, in this case us-east-1, and still be impacted by an outage.

            Why? Because this isn't an issue with where data is held, it's an issue with services (assuming EC2 based on the latest updates) AWS uses *internally* to power a crapton of other services. Losing access to the metadata store in DynamoDB, which seems to be something either that happened simultaneously or was a big cascading fault from the EC2 issue, causes problems across all regions because that's where AWS stores information about state and configuration for its services, as I understand it.

            AWS has been on a regionalising spree recently which has mitigated some of these issues, but there are still some key services powered by critical infra in us-east-1 (like Lambda, to give another example) which could logically lead to what looks to be a SPOF issue. Clearly there is tight coupling for some services in us-east-1 which AWS should have addressed yesterday (or maybe 5 years ago).

    3. CorwinX Silver badge

      Re: Too much in us-east-1

      Exactly.

      Single Point of Failure

      Your infrastructure can be the greatest on the planet but if your DNS servers take a nap...

      1. Sudosu Silver badge

        Re: Too much in us-east-1

        The two main things that almost everyone gets wrong (and that I have made a lot of money fixing):

        DNS

        Security

        1. Excused Boots Silver badge
          FAIL

          Re: Too much in us-east-1

          Honestly you could probably just focus on the first of these and still make a lot of money!

    4. the Jim bloke

      FTFY

      Too much of the planet relies on us-east-1, society should start removing reliance on this point of failure...

  2. DarkwavePunk Silver badge

    DNS

    It's always DNS...

    1. AlexGreyhead
      Coat

      Re: DNS

      I just posted same before I saw you'd beaten me to it.

      Coat --> leaving...

    2. Jellied Eel Silver badge

      Re: DNS

      It's always DNS...

      The BGP process sits in the corner, quietly crying to itself. Apropos of not much, I did wonder why Kindle was being more useless than normal when I was trying to buy.. I mean rent some books earlier.

      1. AMBxx Silver badge

        Re: DNS

        ** Borrow **

        Buy book.

        Download to Kindle.

        Disconnect Kindle from wi-fi before opening book.

        Request refund.

        Enjoy 'borrowed' book. Just make sure you leave wi-fi off until you've finished the book.

        1. phuzz Silver badge

          Re: DNS

          Or buy (so the author gets at least a tiny bit of money), and then strip the DRM and save a copy as an epub.

          Or better yet, don't buy from Amazon. (Kobo seem to have most books I want to buy)

          1. Phil O'Sophical Silver badge

            Re: DNS

            And ebooks.com often seems to match Amazon on price and discounts.

            1. Lon24 Silver badge

              Re: DNS

              eBay sellers are usually half the price and you get a real book. It's also good for recycling and most of the profit probably stays here.

              Plus you can always put it back on when read to further reduce cost.

        2. Anonymous Coward
          Anonymous Coward

          Re: DNS

          Or, in a shop, get a mate to distract the shopkeeper, then slip a book into your bag.

          Sigh. Is this really the sort of "tip" you're proud of?

          1. Anonymous Coward
            Anonymous Coward

            Re: DNS

            No, that's stealing.

            Copying is not stealing. Buying a book and returning it is not stealing.

            Don't be a dumbass.

            1. Anonymous Coward
              Anonymous Coward

              Re: DNS

              > Buying a book and returning it is not stealing.

              If you're wondering where your car is, don't worry, you'll get it back next month.

              You don't have any problem with that, do you?

              1. rg287 Silver badge

                Re: DNS

                If you're wondering where your car is, don't worry, you'll get it back next month.

                In the UK, that's actually not stealing. The criminal offence of Theft is defined as taking property with the intent to permanently deprive the owner of it.

                This made it functionally impossible to prosecute car thieves/joyriders because they could always say they just fancied a razz in that natty GTi and were going to take it back afterwards. Honest! The intent therefore couldn't be proved.

                As a result, we have a specific offence of "Taking without the owner's consent" (TWOC) for cars to cover theft and/or joyriding.

                Of course, removing a book from a bookshop without payment is theft. It's up to the bookshop whether they wish to allow refunds on a once-read book. The "refund an unread ebook" dodge on Kindle is iffy, since the marginal cost to Amazon of delivering that eBook is basically zero. It's more along the lines of fraud than theft - obtaining goods or services by deception.

                It's morally wrong, though I have no sympathy for Amazon. I'd be more concerned whether the author still gets their royalty (probably not on a refunded ebook - whereas a theft from a physical bookshop is on the retailer. The publisher & author have had their cut).

                1. Jellied Eel Silver badge

                  Re: Kindling fires

                  ..since the marginal cost to Amazon of delivering that eBook is basically zero.

                  It should be, but it isn't-

                  https://kdp.amazon.com/en_US/help/topic/G200644210

                  UK Delivery Costs = £0.10/MB

                  If you opt for the 70% royalty rate. If you let Amazon keep 65% of your sale price, delivery is then 'free'. But 10p/MB is just a tad extortionate for delivery, especially when they're the publisher, and part of the traditional publishers job is getting books into stores. So should really come out of Amazon's 30%, but that wouldn't have helped Bezos buy a superyacht or three.

                  I'd be more concerned whether the author still gets their royalty (probably not on a refunded ebook

                  Nope, refunds are clawed back from royalties. Except of course Amazon keeps the delivery fee. But the strangest part is this-

                  https://kdp.amazon.com/en_US/help/topic/G201541130

                  Authors are able to earn a maximum of 3,000 Kindle Edition Normalized Pages (KENP) Read per title per customer. This means that each time your Kindle eBook is borrowed and read, you can receive credit for up to 3,000 pages. We believe this results in an equitable distribution of the KDP Select Global Fund.

                  For royalties from books enrolled in Kindle Unlimited.. Which gets a bit weird, or interesting because it gives you an idea of how many people actually finished the book. I think some authors try to game KENP with the dreaded 'page turners' where pages might only be a couple of paragraphs.

            2. Anonymous Coward
              Anonymous Coward

              Re: DNS

              What do you mean, stealing? I was going to return the book after I read it, so by your argument, that's ok!

            3. MSArm

              Re: DNS

              "Copying is not stealing. "

              Course it is, by copying you are depriving the author of royalties for that book.

              Who is being the dumbass now?

              1. Anonymous Coward
                Anonymous Coward

                Re: DNS

                So what if you make and sell a table, a physical object? You can only sell it once. But because technology lets you copy something digitally for free, you demand multiple payments? Hmm. That sounds like abuse to me.

          2. Sudosu Silver badge

            Re: DNS

            "You wouldn't steal a car..."

            1. Anonymous Coward
              Anonymous Coward

              Re: DNS

              But stealing a font is just fine...

    3. Phil O'Sophical Silver badge

      Re: DNS

      And Amazon has indeed admitted that it is a DNS issue...

      1. vtcodger Silver badge

        Re: DNS

        Perhaps I misunderstand, but isn't DNS at its heart a highly specialized data base lookup and retrieval system? "You give it a name e.g."upyours.com" and it gives you back a set of structured information provided by the operators of upyours.com telling you how to send traffic to upyours.com. While DNS does screw up at times, isn't it more likely in this case that Amazon provided them with a faulty record?

        1. Anonymous Coward
          Anonymous Coward

          Re: DNS

          I think in this case "them" is Amazon themselves, they broke their own DNS system.

    4. Anonymous Coward
      Anonymous Coward

      Re: DNS

      ... unless it's BGP. Then it's BGP.

      But in this case, yes, it was indeed DNS. *Groan*

      1. Sudosu Silver badge

        Re: DNS

        I wonder if they will now start using hosts file like so many did in the past when they were unable to correctly configure their DNS.

        I have actually seen environments with scripts or GPO's that updated the hosts files.

        Alternately WINS was often used to plaster over DNS holes as well, at least in Windows environments.

    5. elsergiovolador Silver badge

      Re: DNS

      It seems like DNS is often treated as too simple and not glamourous enough to give it attention.

      1. Korev Silver badge
        Coat

        Re: DNS

        > It seems like DNS is often treated as too simple and not glamourous enough to give it attention.

        Yeah, it's such a Bind

        1. jsdc

          Re: DNS

          Brilliant - cheered me up.

    6. The Original Steve

      Re: DNS

      Was going to upvote but the current count is 53, so I can't really increase it... :D

    7. Joe Gurman Silver badge

      Re: DNS

      Except when it’s BGP.

      1. Blue Shirt Guy

        Re: DNS

        DNS or BGP?

        Vodafone say "why not both". :-)

  3. Eye Know

    365 Apps

    We also saw 365 apps crashing across a number of client companies. Possibly due to integrations, still investigating but it appears to be getting better in the UK.

    1. Insert sadsack pun here

      Re: 365 Apps

      Reddit is offline. My productivity has increased by 732%

    2. Anonymous Coward
      Anonymous Coward

      Re: 365 Apps

      No, that's just SNAFU for Office 365…

    3. Anonymous Coward
      Anonymous Coward

      Re: 365 Apps

      I'm surprised you had time to count them.

  4. Steve Button

    Took your time

    How come it took El Reg a couple of hours to pick this up?

    1. Anonymous Coward
      Anonymous Coward

      Re: Took your time

      They were busy firing up non-AWS servers...

    2. Excellentsword (Written by Reg staff)

      Re: Took your time

      Well, we had to wake up, have breakfast, get dressed, take the kids to school, make a coffee...

      1. Steve Button

        Re: Took your time

        I thought the CEO of every company gets up at 04:30 and has time to meditate, exercise and write some affirmations before even starting work at 6.

        Do you even have a CEO?

        Slackers.

      2. Paul Crawford Silver badge

        Re: Took your time

        Clearly El Reg has declined in standards a lot! It should read '...kids to school, make a splendid cup of tea'

        English breakfast or Early Grey? The would be an ecumenical matter...

        1. Roland6 Silver badge

          Re: Took your time

          I think there is something wrong with the sequencing, everyone knows you make.a cup of tea directly after waking up. Coffee is something you make after taking the kids to school.

          1. Lon24 Silver badge

            Re: Took your time

            And there was me thinking WFH made dressing and washing even more optional.

        2. TimMaher Silver badge
          Windows

          Re: “breakfast or Earl Grey?”

          I’d opt for gunpowder.

          1. lordminty

            Re: “breakfast or Earl Grey?”

            Real IT Professionals in the UK only drink single estate Assam in the morning.

            1. TRT Silver badge

              Re: “breakfast or Earl Grey?”

              Real IT people don't have a SPOF like that.

              Hybrid teas. That's more dependable.

              1. The man with a spanner Silver badge

                Re: “breakfast or Earl Grey?”

                "Hybrid teas"

                You drink roses?

            2. Anonymous Coward
              Anonymous Coward

              Re: “breakfast or Earl Grey?”

              Real IT professional everywhere only drink coffee. Quadruple espresso, lots of sugar, cream (half and half if heavy cream isn't available), other flavorings optional.

        3. Dizzy Dwarf

          Re: Took your time

          Doesn't sound like it's safe for work, but it is : Cup of Brown Joy

          1. Vikingforties

            Re: Took your time

            More of a Hotpot's fan myself. C'mon Mek us a Brew!

            https://m.youtube.com/watch?v=aM1jQ8cIFuU&pp=ygUWTGFuY2FzaGlyZSBob3Rwb3RzIHRlYQ%3D%3D

        4. Raphael

          Re: Took your time

          I'm pretty sure the late Lester Haines would have had a splendid pint of beer

      3. John Brown (no body) Silver badge

        Re: Took your time

        ...and the US staff were all still tucked up in bed too.

      4. J.G.Harston Silver badge

        Re: Took your time

        IT people have children???? How on earth did that happen?

        1. TRT Silver badge

          Re: Took your time

          Single point of failure again. This time in the implementation of Latex.

    3. Roland6 Silver badge

      Re: Took your time

      Alexa stopped working?

      So they were unable to get into their desktop: “Alexa open my desktop”…

      The laugh is if AI takes off in the way the pundits dream about, a cloud outage is going to be even more debilitating.

      1. John Brown (no body) Silver badge

        Re: Took your time

        Better yet, Ring doorbells were affected world wide. If that's not a wake up call to "the man on the Clapham omnibus", I don't know what is. I mean, a server in the US goes titsup and something as "simple" as your doorbell in another country stops working. FFS!

        1. Lon24 Silver badge

          Re: Took your time

          But spare a thought for all the delivery folks as they post the 'called but no one answered' excuse.

      2. RMclan

        Re: Took your time

        A colleague complained he had to actually go round his house and physically turn off lights when he left for work today instead of Alexa doing it as he went out of the door.

        He didn't get much sympathy, another colleague asked if he needed a plaster for his overworked finger.

        1. Roland6 Silver badge

          Re: Took your time

          Surprised he still knew what a light switch was, what it looked like and were they are placed...

          Suspect he needed a sit down and a hot cup of tea after all that effort.

  5. a_foley
    FAIL

    Thanks, Bezos!

    Can't work on my codebase now that npm's also hit by this!

    Hopefully, now all the people using AWS as an infrastructure rethinks their choice. But honestly, I don't think GCP or Azure is much better (can't trust M$ for any uptime at all!)...

    1. Anonymous Coward
      Anonymous Coward

      Re: Thanks, Bezos!

      Just use your local cache of npm-hosted modules. You do have a local cache, don't you? You're not relying on redownloading every. Single. Page load.

      Sure, you won't be able to pull in yet another new module, so time to work on the backlog in your own code.

      1. a_foley
        Unhappy

        Re: Thanks, Bezos!

        Yes, I do actually have a tonne of the stuff I'm downloading cached, but sadly there is a single package which I don't have cached, thus breaking the whole install...

    2. Greybearded old scrote
      Joke

      Re: Thanks, Bezos!

      "Hopefully, now all the people using AWS as an infrastructure rethinks their choice."

      Such optimism. You can't possibly have been paying attention.

    3. Anonymous Coward
      Anonymous Coward

      Re: Thanks, Bezos!

      My company doesn't use AWS, but was impacted regardless. It appears that Infoblox DNS management for our internal systems takes a feed from AWS, and the failure of that feed buggered Infoblox and our internal DNS

      (-‸ლ)

  6. Anonymous Coward
    Anonymous Coward

    No, no, no, you have this all wrong.

    It's merely a collaborative attempt to prevent the jewelry that was stolen from the Louvre on Sunday from being sold online.

    No, can't go into PR, I still have a soul..

    :)

  7. that one in the corner Silver badge

    HMRC taken down

    By problems in the US. Hmm.

    Time for reminders about "it is just someone else's computer", if it is in another country it's really not under your control (especially when "you" are a branch of government) and it is just hosting, not some specialised ability that isn't available elsewhere.

    1. Falmor

      Re: HMRC taken down

      Wonder if the "features that rely on US-EAST-1 endpoints such as IAM (Identity and Access Management)" means that even if you don't host your systems in the US, you are still reliant on US-EAST-1 for login and access.

      1. richardcox13

        Re: HMRC taken down

        I strongly suspect this is the case. Some aspect of DynamoDB (directly) or indirectly (other AWS services that use DynamoDB) depends on something in US-East-1.

        Much like the Azure Central-US-1 (IIRC) region having a wobble breaks Azure world wide (I should note MS have said they are working to reduce this dependency).

        Eliminating all single points of failure in a complex system is hard.

        1. PCScreenOnly Silver badge

          Re: HMRC taken down

          Should have been designed for it from day 1

        2. vogon00

          Re: HMRC taken down

          "Eliminating all single points of failure in a complex system is hard.

          Yes and no:-)

          The real issue is that reliability and the removal of SPOFs etc. is expensive....which is why it hasn't happened yet.

          How many times have you architected a properly resilient system, only to find it's been decimated by people counting beans:-)

          1. Albert Coates
            Pint

            Re: HMRC taken down

            "The real issue is that reliability and the removal of SPOFs etc. is expensive...."

            Aargh, my brain automagically substitutes BOFH each time I see it. I hereby curse you to Heck*, El Reg.

            * © Phil, Prince of Insufficient Darkness.

      2. FrogsAndChips

        Re: HMRC taken down

        us-east-1 is where the control plane for IAM is located. If it falls downs, your applications in other regions will still work, users will still be able to login, but you won't be able to make changes like creating new users, editing access policies...

    2. Steve K

      Re: HMRC taken down

      You don't know that "it is just hosting"

      It could be, but it could also be a whole load of serverless functionality also (Lambda etc)

      1. John Brown (no body) Silver badge

        Re: HMRC taken down

        "serverless functionality"

        So, how doe that work? Or does "serverless" simply mean "you" don't own the server, it's just someone else’s server in another country?

        1. Anonymous Coward
          Anonymous Coward

          Re: HMRC taken down

          Yes, serverless is nonsensical marketing speak for "you don't own the server or the app", but you can configure an instance of the app.

        2. FrogsAndChips

          Re: HMRC taken down

          It just means you don't have to provision/configure/maintain the servers on which your services/functions/apps will run. You can still choose the region where the servers will be hosted.

          1. that one in the corner Silver badge

            Re: HMRC taken down

            In other words, it is - just hosting. Running some software on somebody else's hardware, when you could choose to run the same functionality on yet another person's chunk of hardware.

    3. jdiebdhidbsusbvwbsidnsoskebid Silver badge

      Re: HMRC taken down

      I get your point about sovereignty, but if UK gov stuff was all hosted on UK based servers, that wouldn't protect against this sort of failure.

  8. AlexGreyhead
    Coat

    DNS

    It's always DNS.

    El Reg taught me that wayy back in the day...

    :-/

  9. lordminty

    Who knew that putting all your eggs...

    ...in Amazon's basket might result in this omnishambles.

    Why are bank websites/services and HMRC even using non-Europe Region AWS?

    And why are banks even using AWS for stuff that their sites absolutely need to work?

    Does nobody do DR trials and simulations any more?

    Just imagine if the UK had Digital ID hosted on AWS.

    1. Doctor Syntax Silver badge

      Re: Who knew that putting all your eggs...

      "Just imagine if the UK had Digital ID hosted on AWS."

      You won't need to imagine. It very likely will be.

      1. Sudosu Silver badge
        Joke

        Re: Who knew that putting all your eggs...

        "UK Government overthrown in violent coup by citizens unable access pron (sic) due to digital ID outage"

    2. David Hicklin Silver badge

      Re: Who knew that putting all your eggs...

      But it is the same with any Cloud based service, as others keep on pointing out is that it is a server on somebody else’s infrastructure that you have no control over - and no idea what other services it relies on to keep it running.

      It only takes one domino somewhere to fall over and the who lot goes down.

      So far they have bodged there way to getting back on line fairly quickly - I am still waiting in expectation of the multi-day outage that will come one day. Only then will companies directors/boards get the message that it is a bad idea.

      Just imagine a ransomware attack getting into the M$, Google/Amazon/Azure/AWS world, and look at JLR, M&S and how long they took to recover

      1. PCScreenOnly Silver badge

        Re: Who knew that putting all your eggs...

        2 issues beyond there is clearly no internal failover / resilience inside AWS

        1, Bean counters do not like the idea of parallel clouds, so having AWS + Azure and being able to fail over = costs

        2. From what I have read (so I do not know for sure), making some apps work across multiple cloud providers is not easy/possible *

        *see recent story of Us being stuck in Azure

      2. Outski
        Coat

        Re: Who knew that putting all your eggs...

        It only takes one domino somewhere to fall over and the who lot goes down.

        Oh I dunno, Domino's pretty resilient if configured properly...

        1. that one in the corner Silver badge

          Re: Who knew that putting all your eggs...

          "Resilient". Apt, but not really the sort of word I want to associate with a pizza.

          Unless it was Italian Dwarf Pizza, of course ("look out, he has grissini!").

          1. Albert Coates

            Re: Who knew that putting all your eggs...

            " Italian Dwarf Pizza"

            There are those who prefer Glocks, and those who prefer Walthers or Sig Sauers, but I put my trust in my 9 mm Grissini.

            1. Jellied Eel Silver badge

              Re: Who knew that putting all your eggs...

              There are those who prefer Glocks, and those who prefer Walthers or Sig Sauers, but I put my trust in my 9 mm Grissini.

              New version of spray & pray: Get a Sig P320, throw it, and you're bound to hit someone. But a gentlemans choice might be a Rohrbaugh R9, but I don't know if they're still being made after Remington bought them. Taran seems to be cornering the gucci market, thanks to careful product placement, and I'm sure that in the right hands, a well-dried Grissini could be just as lethal as a pencil.

    3. Anonymous Coward
      Anonymous Coward

      Re: Who knew that putting all your eggs...

      The snake-oil Cloud sales people say that in the event of a failure of a particular region, you can move your application to another region, pretty seamlessly, especially if you pay for hot-standby and data duplication in another region.

      But they sell all the benefits, without detailing the problems and issues. If they don't mention them, they're not lying, are they. And I'll bet that any resilience promises about guaranteed uptime are either fenced to so little damages as to be laughable, or are actually meaningless weasel words that don't guarantee anything meaningful.

      1. David Hicklin Silver badge

        Re: Who knew that putting all your eggs...

        > you can move your application to another region,

        Which assumes that you can actually log onto the cloud services to move it in the first place...and if the application is only stored in that failed region.....

    4. PM.

      Re: Who knew that putting all your eggs...

      >>Does nobody do DR trials and simulations any more?

      No.

    5. phuzz Silver badge
      Unhappy

      Re: Who knew that putting all your eggs...

      Does nobody do DR trials and simulations any more?

      You have budget for DR? That must be nice.

      1. Anonymous Coward
        Anonymous Coward

        Re: Who knew that putting all your eggs...

        Budget is one thing, but we have to do a DR trial every year. Regulated industry and so. And we have to prove that we can hop for more than one day on one leg. Always done over a weekend.

        And often we find that some new-fangled system is not redundant and that the problem cannot be solved asap and we have to do a second trial that year.

        One year was so "interesting" that manglement announced the reserved date for a third run with the incentive, that the fourth would be over christmas with the group of the third run.

  10. Inspector71

    Time for a Coffee....

    We are locked out of our AutoDesk products, that's those Cloud productivity improvements for you....

    1. TimMaher Silver badge
      Windows

      Re: Time for a Coffee....

      No, it’s time for tea.

      See earlier comments.

    2. Huw L-D

      Re: Time for a Coffee....

      I've got 3 Architect clients. Effing nightmare.

    3. GNU Enjoyer
      Angel

      Re: Time for a Coffee....

      Those are clearly not yours, as those are refusing to operate for no reason.

      Should have used FreeCAD and other free replacement software - as that won't ever refuse to stop operating (if you need some certain functionality, you might need to pay a programmer to add it, but it seems that such costs will be less than what AutoDesk charges in the end).

  11. cookiecutter Silver badge
    Devil

    its just someone else's "shit" computer

    I still think, regardless of my feelings on the competition commission letting it happen, that on prem Vmware is still the best option.

    when you factor in shite like this, random outages, random cost rises, random engineers based on india that have been working 80 hour weeks unplugging stuff, VMware & Nutanix are still the way to go with on prem OpenStack too.

    Even with with 400%+ rises its cheaper than cloud

    1. Allonymous Coward
      Linux

      Re: its just someone else's "shit" computer

      As someone who spends a lot of time in AWS and mostly quite likes the power it brings - <gestures in the direction of Proxmox etc>

  12. Anonymous Coward
    Anonymous Coward

    Even coffee shops

    I've just opened the Gail's app and got a message about a connection error.

    Fortunately the app seems, unusually, to have competent developers because it then gave me a popup saying "anyway, here's your QR code to scan in a shop while this gets sorted out", which is a great deal more intelligent a response than I've had from apps from Boots or Tesco.

    1. Anonymous Coward
      Anonymous Coward

      Gail's app connection error

      The connection error I usually get says:

      Your social media profile shows insufficient links to the Home Counties, you are being redirected to your nearest Gregg's

      1. Anonymous Coward
        Anonymous Coward

        Re: Gail's app connection error

        Ah. I was wondering what "Gail's" was and why it had an app that customers seemed to require. My first thought (genuinely, until I saw the references to Boots and Tesco) was that it must be some sort of leftpondian thing…

        1. lordminty

          Re: Gail's app connection error

          Gail's is just a massively overpriced version of Greggs that sells sourdough loaves that look like they've been cooked by atomic fusion, as 'Artisan bread'.

          1. gryphon

            Re: Gail's app connection error

            Whenever I see a cafe, baker etc. marketing itself as artisan or supplying artisan products I always run a mile.

            Will inevitably be way more expensive and usually lesser quality than you could get for going 2 doors down the street to even a chain place.

            1. Anonymous Coward
              Anonymous Coward

              Re: Gail's app connection error

              I remember a craft brewery in France selling 'bière artisanale'. With the French love of puns they had noticed that it spells 'art is an ale' in English, which was their marketing tagline. I did think that was quite amusing.

          2. Anonymous Coward
            Anonymous Coward

            Re: Gail's app connection error

            But unlike Greggs they sell edible pastries, and passable coffee in real cups. Gail's is no more special than Costa or Nero, but they are a cut above Greasy Eggs.

          3. Dave559

            Re: Gail's app connection error

            "Gail's is just a massively overpriced version of Greggs"

            Even Greggs is an overpriced version of Greggs these days: showing my age here, but I remember when I first started buying Greggs cheese and onion pasties they cost around 37p or thereabouts, and when there was a price increase (maybe once a year or something) they went up a penny or two at a time… Yes, inflation and time passes and all that, but they're almost £2 now and seem to jump up in price almost every couple of months… :-(

      2. Anonymous Coward
        Anonymous Coward

        Re: Gail's app connection error

        Ah, we're in the posh part of the Midlands up here. Gail's or Costa...

        1. tiggity Silver badge

          Re: Gail's app connection error

          @AC

          Had a quick look - they do only seem to be in a few upmarket parts of the Midlands & North (which would explain why I have never seen one!)

  13. Soruk

    Just somebody else's computer.

    "The Cloud" is just somebody else's computer / network. If you are making your business rely on the cloud then you should have contingencies when the services go down. Unlike having your own kit you manage you are beholden to the cloud provider as to when they feel like bringing your services back - you're just one of thousands, your business is not a priority.

    1. Anonymous Coward
      Anonymous Coward

      Re: Just somebody else's computer.

      Unless it's literally 'your' network, it's always someone else's network. Even if it's in the same building as you. And especially if 'your' network relies on talking to 'other' networks to GSD*. Risk vs resilience isn't just a cloud thing. (*Get Sh*t Done).

  14. Anonymous Coward
    Anonymous Coward

    Building cross region resiliency is fairly easy

    So why dont AWS do it themselves?

    To be clear, this isnt a refutation of cloud as a solution, so all the snarky ' Its just someone else computer' comments merely show ignorance. Its a sign that well architected systems are necessary wherever they're hosted.

    1. David Hicklin Silver badge

      Re: Building cross region resiliency is fairly easy

      No its not ignorance, with on Prem you can physically see, feel and touch ALL the kit as well as being in control of it.

      The real challenge will be identifying anything that is dependent on a cloud based service and working out how to get around it when it is not there

      The JCB going through your network link is as always one other consideration, although in that case if you are clod based then EVERYTHING will be down.

      1. Korev Silver badge
        Coat

        Re: Building cross region resiliency is fairly easy

        > The JCB going through your network link is as always one other consideration, although in that case if you are clod based then EVERYTHING will be down.

        Isn't being Clod-based the problem here?

      2. xyz Silver badge

        Re: Building cross region resiliency is fairly easy

        nice typo...clod based

        1. David Hicklin Silver badge

          Re: Building cross region resiliency is fairly easy

          Thank you, I did notice it when typing and thought "that sounds better"

  15. Elongated Muskrat Silver badge

    allyoureggsinonebasket.com

    is down

    1. PM.

      Re: allyoureggsinonebasket.com

      To be fair, even if only *half* eggs were in this "basket" , you would possibly hosed anyway ....

      That's modern tech for you

      1. Elongated Muskrat Silver badge

        Re: allyoureggsinonebasket.com

        Especially since the other baskets are owned and operated by Microsoft and Google, neither of whom would I trust to walk my granny to the shops.

        1. John Brown (no body) Silver badge

          Re: allyoureggsinonebasket.com

          Yeah, it sucks[*]

          * Instructions on how to sucks eggs are not currently available due to technical difficulties.

  16. Diogenes8080

    The Downfall?

    Anyone have a link to the Downfall parody Hitler rant created for a major Amazon outage maybe 12 to 15 years ago? Those parodies are still as thick as fleas on Youtube and I find three more recent ones specifically for Amazon but not the original I'm thinking of. It would be a highly apt time to bump it back up the ratings.

    "Anyone who works for Cisco, HP or Veeam leave the room!" (it's that old)

    1. Paul Crawford Silver badge

      Re: The Downfall?

      Unrelated to today's outage, but one of the best Downfall parodies I remember was about ACS:Law who sued file sharers. A detailed article on the business is a good start to fully appreciate the parody:

      https://arstechnica.com/tech-policy/2010/09/amounts-to-blackmail-inside-a-p2p-settlement-letter-factory/

      Then enjoy the rant video at the end of this article:

      https://torrentfreak.com/acslaws-anti-piracy-downfall-sends-hitler-crazy-101004/

      1. Diogenes8080

        Re: The Downfall?

        I finally found it:

        https://www.youtube.com/watch?v=m3wrBFuGK2A

        The opening subtitles are uncanny...

  17. Anonymous Coward
    Anonymous Coward

    and in Ukraine...

    the active operations along the front line have stopped for the better part of the morning as both 'parties' use signal for comms.

    ...

    actually, they haven't, but the outage has caused some... delays.

  18. Anonymous Coward
    Anonymous Coward

    good news though

    because unless such shit happens, nothing happens to prevent it happening again. So there's a chance that something will happen to stop this happening again. Maybe.

  19. Merrill

    It's an old problem

    "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

    Leslie Lamport, 28 May 1987

  20. Pascal Monett Silver badge

    "breaks half the internet"

    Not my half.

    Then again, I'm only reading El Reg today . . .

  21. Anonymous Coward
    Anonymous Coward

    https://twitter.com/elonmusk/status/1980221072512635117

  22. J.G.Harston Silver badge

    Isn't the internet supposed to be designed that if part of it goes titups, it routes around it and continues working?

    1. Elongated Muskrat Silver badge

      It's not an "internet" problem, so much as a "proprietary systems that run big bits of the internet" problem.

    2. R Soul Silver badge

      "Isn't the internet supposed to be designed that if part of it goes titups, it routes around it and continues working?"

      It is and it does. However for too many people AWS (or Facebook or Tiktok or...) is the Internet.

  23. DCdave

    There'll be a lot of people going over their resiliency plans...

    ...to see if they tested resiliency properly. In most cases, I imagine they'll find they tested what they could and had at some point to rely on the cloud provider's assurances.

    There'll be even more people thinking maybe they should take a look at this resiliency plan thing, some time in the future, if they get round to it.

    1. Anonymous Coward
      Anonymous Coward

      Re: There'll be a lot of people going over their resiliency plans...

      Unfortunately, too many people think it's the inevitable sort of thing that happens from time to time when using computers.

      Just like when a whole generation in the 90's / 2000's were led to believe that having to reboot your computer daily was normal.

      Just wait until the critical infrastructure gets outdated, and the generation replacing them knows no better, and we have traffic lights breaking, aircraft feeling from the skies, and power stations in emergency shutdown, just because a minimum wage employee in another country typos a DNS update

  24. dlc.usa
    Holmes

    Meanwhile...

    Global credit card and airline reservation transactions just work. File this datum under things an IT architect should know (and have the guts to say to the bean counters when necessary).

  25. jonsg
    Facepalm

    What is HMRC doing in US-East-1??

    I would very much like to know why HMRC is doing anything in US-East-1, let alone having enough critical infra there to be vulnerable to a region failure.

    The gov.uk AWS sites ought to be in EU-West-2 (London), perhaps with fallbacks in EU-West-1 (Ireland) or other close-partner EU regions likeEU-Central-1 (Frankfurt) if needed.

    I would not expect the UK Government to be exfiltrating any more UK data than absolutely necessary (hint: it's not) to the US. And, no, I don't consider Five Eyes an excuse.

    1. Jan 0

      Re: What is HMRC doing in US-East-1??

      Probably because they've forbidden the use of any European Union sites for UK government uses yet haven't remembered that EU can be an abbreviation for "Europe" as well as "European Union".

    2. Dunstan Vavasour

      Re: What is HMRC doing in US-East-1??

      IAM, which manages access permissions around AWS, is a "global" service. It depends on DynamoDB in us-east-1 . No data goes there.

    3. MrWoo
      WTF?

      Re: What is HMRC doing in US-East-1??

      Why is it using a 3rd party foreign provider? It's the UK Government, employing millions of people (the NHS alone is 1.3 Million). Surely with that sort of scale it can run it's own datacentres - hell, it could even offer the services to local Councils (perish the thought that they'd think strategically about saving us all a few quid).

  26. Anonymous Coward
    Anonymous Coward

    An Age Thing

    It must be my age (and the fact I retired from paid employment several years ago) but ISTR the USP when the internet was being rolled out beyond the military and college networks was that it obviated a SPOF. But then, that was before it became a profit centre for some and “cloud” services were loaded onto a bandwagon. Now AI is being loaded up, I’m just waiting for the wheels to fall off.

    1. Anonymous Coward
      Anonymous Coward

      Re: An Age Thing

      If I hadn’t read it here, or seen it on the BBC news, I wouldn’t have known there was a problem. But then, my PC and phone aren’t biologically implanted (andI know I’m weird in that I actually talk to people).

    2. Solviva

      Re: An Age Thing

      And that's how the 'Internet' works today.

      What has changed is the definition of said Internet, from the original interconnected network(s), to meaning the applications which rely on the Internet to function. In this case the Internet didn't suffer any issues, however an application which uses the Internet to work did have an issue.

  27. Anonymous Coward
    Anonymous Coward

    Who unplugged the internet

    I blame Jenny

  28. xanadu42

    AWSome

    Amazon

    Web

    Services,

    one

    meagre

    excuse:

    It's DNS...

    You would think that after nearly two decades of operation the "engineers" at Amazon would know how to deal with DNS...

    Oops, forgot it is all "AI" agents now :(

  29. PM.

    At least El Reg works so we are good

  30. James O'Shea Silver badge

    Wasn't down this morning (08:00 Eastern)

    It's down now. (11:00 Eastern.)

    I get to be paid to visit El Reg because a certain project can't be accessed. Gee. Did I tell the 'C' bos to NOT set things up in the cloud? Didn't I say so _in writting_? Why, yes I did. I can't wait to see whose fault this is.

  31. myhandler

    Maybe this is the first test by AI to see how panicked the humans get.

  32. Anonymous Coward
    Anonymous Coward

    Microsoft sites were down as well

    People we having problems with Microsoft as well, even users of Azure.

    The main Microsoft admin login page was down, which is off - why was Azure affected by AWS?

    Spooky.....

    1. Elongated Muskrat Silver badge

      Re: Microsoft sites were down as well

      Possibly people who hadn't put all their eggs in one basket failing over to Azure and the sudden increased load causing issues?

  33. T. F. M. Reader
    Coat

    Follow the money

    The root cause is that people do not pay Amazon enough for redundancy and failover, so when a single region's DNS goes FUBAR...

    I am leaving, ok?

    1. Anonymous Coward
      Anonymous Coward

      Re: Follow the money

      So you have to pay extra for the things the cloud promised as its main features?

  34. xyz Silver badge

    17:16 CET

    Still up and down like a hooker's knickers.

  35. scrubber
    Trollface

    HMRC Down

    But your Digital ID still works, right? Right? Oh.

    1. Solviva

      Re: HMRC Down

      Might or might not do, but everything you can do now without Digital ID would still be do-able (or not thanks to AWS).

      1. Anonymous Coward
        Anonymous Coward

        Re: HMRC Down

        We can't do anything with Digital ID already. The latest AWS fuckup won't change that.

        1. Solviva

          Re: HMRC Down

          That's exactly my point, that which works now without digital ID won't suddenly stop working or degrade if/when Digital ID arrives. The nay sayers that jumped on Kier's "make it easier to access your money" seem to have turned this into it somehow would be harder to access your money without Digital ID (if/whenit exists) - well yes comparatively, but no harder than it would be today.

          1. Jellied Eel Silver badge

            Re: HMRC Down

            That's exactly my point, that which works now without digital ID won't suddenly stop working or degrade if/when Digital ID arrives. The nay sayers that jumped on Kier's "make it easier to access your money" seem to have turned this into it somehow would be harder to access your money without Digital ID (if/whenit exists) - well yes comparatively, but no harder than it would be today.

            I don't have any problems with accessing my money right now, so I fail to see how an extra layer of Digital IDiocy will make that easier. I don't have mobile banking apps on any of my phones, because phones get lost or stolen. Inserting a layer of Digital IDiocy that forces a dependency on a phone just increases the risks and hassle, if/when that phone is lost or stolen. So then a transaction might need

            1) A working/charged phone

            2) A working Digital IDiot system

            3) A working mobile network

            4) A working bank system

            Then ancillary stuff, like how will the IDiot card authenticate? Will it mean having to have Bluetooth enabled, draining battery and increasing security vulnerabilities? Then privacy aspects. Like my transactions with my bank are my business. Will the IDiot Card require location services enabled so HMG and approved 3rd parties know where I am and transaction details?

            But basically a whole lot of new risks and hassles for no actual benefit.

            1. Solviva

              Re: HMRC Down

              Well you've proved the point, Digital ID will make zero difference to how you access your money today (which sounds like you go outside and visit physical branches or cash machines).

              Digital ID is unlikely to be used for getting cash out (although it could be used to verify you are the card holder rather than entering a PIN). It's just another way of identifying yourself, supposedly more secure than a passport or driving license.

              1. Roland6 Silver badge

                Re: HMRC Down

                A friend has some official digital id app - it holds his passport, driving licence, and some other UK government id’s, It has proved so far to be of little real use outside of the few UK government bodies that use it, everywhere else that wants id, wants to see original hard copy id.

                So the latest was Halfords, where not only did they want to see his (paper) V5C but also his actual plastic card driving licence. When asked, the reply was that the app wasn’t on the list of approved formats.

              2. Jellied Eel Silver badge

                Re: HMRC Down

                Well you've proved the point, Digital ID will make zero difference to how you access your money today (which sounds like you go outside and visit physical branches or cash machines).

                It varies. So all the usual banking and payment things. Only now another layer of Digital IDiocy that might insert itself to make life harder and less convenient. Much as it promised last time Labour pulled this stunt. So it would be 'convenient' for me to get the states permission to buy alchohol, tobacco. Or not, if Nanny says 'No'.

                Digital ID is unlikely to be used for getting cash out (although it could be used to verify you are the card holder rather than entering a PIN). It's just another way of identifying yourself, supposedly more secure than a passport or driving license.

                One never knows with creeping compulsion. But there a lot of unknowns, costs and security vulnerabilities to discover. Along with perhaps needing a suspension of disbelief, or asking pointed questions, like why aren't passports or driving licences secure enough? But it's going to waste billions in direct costs, or indirect, if banks, retailers etc have to pay to support this system.

                1. Solviva

                  Re: HMRC Down

                  "like why aren't passports or driving licences secure enough"

                  Ask country X why their passports are easy to forge. Not everybody is eligible for a driving license.

                  The thing with (a properly implemented) digital ID is that it is simply an authentication mechanism, in this case run by UK.gov. As much as it would be possible to log when and where said ID has been used (and it would make sense to do that, whereby you yourself can view your history), that's the most that should be done by UK.gov. It's up to whatever organisation wants to participate in digital ID to perform authorisation on whether they allow digital Bob to use their services or not, the government simply provides verification that digital Bob really does represent real Bob.

                  1. Jellied Eel Silver badge

                    Re: HMRC Down

                    It's up to whatever organisation wants to participate in digital ID to perform authorisation on whether they allow digital Bob to use their services or not

                    So at the moment, that's just going to be every employer, and every employee. Well, except for employers who currently have no problem hiring illegal immigrants, and the IDiot card will do nothing to stop that.

                    the government simply provides verification that digital Bob really does represent real Bob.

                    Nope. The real Bob might still be a virtual Bob because ID theft is a thing, and IDiot cards have the potential to make this worse. And if its anything like the old ID Card proposals, there'll be fines for failing to provide information, providing incorrect information, but no compensation if the IDiot Card gets hacked or compromised. Pay say, £1m in compensation if that happens, then perhaps I'd get one. If it really is secure, government would have no problem with making that offer. Otherwise it might end up like India's card, their minister claimed it was secure and hackers promptly gave the minister a thorough fisking based purely on the data he flashed on his card.

                    1. Solviva

                      Re: HMRC Down

                      As for ID theft, that will be rather harder with a digital ID. The process of acquiring a digital ID should be secure enough that should a re-application come in for the same person, then flags will be raised requiring further verification.

                      As opposed to today where you can whip up a fake ID, some fake bank statements, fake utility bills and hey presto you now have credit in the name of Bob.

                      1. Jellied Eel Silver badge

                        Re: HMRC Down

                        As opposed to today where you can whip up a fake ID, some fake bank statements, fake utility bills and hey presto you now have credit in the name of Bob.

                        And tomorrow, you might be able to use that fake ID to enroll as Bob. Or you have Bob's phone, and therefore must be Bob. But still a lot of unknowns, like exactly how the enrollment process will work, or the authentication. Maybe if Alice has access to Bob's phone, Alice can tap it and becomes Bob. Or maybe the app will flash a pic of Bob, and someone will have to decide if the person with the phone is Bob, or not. Or if the ID requires a photo, then everyone will have to provide one, which will make facial recognition via surveillance more effective. Or not, because anyone who isn't in the database won't be easily identifiable..

                        1. Solviva

                          Re: HMRC Down

                          Well if Bob enrols Alice's biometrics on his device, or shares his passcode with Alice then sure Alice can pretend to be Bob if Alice has Bob's device. Is Alice Bob's mum, you didn't specify?

                          Meanwhile when Balaclava Boy swipes Bob's unlocked phone, the digital ID app still requires the same biometrics to be opened so Balaclava Boy is unable to identify as Bob.

                          1. Jellied Eel Silver badge

                            Re: HMRC Down

                            Meanwhile when Balaclava Boy swipes Bob's unlocked phone, the digital ID app still requires the same biometrics to be opened so Balaclava Boy is unable to identify as Bob.

                            Or Alice swipes Bob's phone, and unlocks it. Which is presumably easy enough to do given the number of phones that get stolen. They wouldn't be worth stealing, if they couldn't be unlocked and used. Then if the IDiot app depends on the phone's security, if that can be bypassed then Alice is still Bob, and the IDiot app might be none the wiser.. Which could also be an (in)convenience thiing. So use biometrics to unlock phone. Smile for the camera! Then would I need to do that again to authenticate to the app?

                            If that relies on the phone security, and that can be broken, then so is the assumption that this will provide secure ID authentication.

                            Or watch this video-

                            https://www.youtube.com/watch?v=fufnzxyv2Ps

                            Where my favorite Black Belt Barrister points out that "the GOV.UK One Login system — the UK Government’s new digital identity platform that is supposed to unify access to all online government services. Yet, behind the scenes, the system’s security operations have been outsourced to a company based in Romania, raising major questions about data protection, privacy, cybersecurity, and national security.

                            Especially after a government Red Team seemed able to hack the developers fairly easily, and pointed out that the 'secure' identity platform was being developed by unvetted staff on insecure systems..

                            1. Solviva

                              Re: HMRC Down

                              Funny that, your favourite BBB used to be my favourite BBB when he was creating videos about, well, barrister related stuff. Then he realised there's money to be made pandering to a certain section of the population who seem, how to say, anti-establishment (not sure I'm using that correctly, but hey). Then he started pumping out daily videos about nothing much, filled with sponsored segments.

                              Similar to how the venerable Dr (PhD) John Campbell was giving sound advice at the beginning of the pandemic, then again realised where the real money is to be made with his wink winks, and threw all the published science to the side and misinterpreted plenty of non-sensical fringe-science.

                              Anyway enough of reviewing YouTubers. So BBB is scared that it's outsourced to Romania, a country which adheres to GDPR legislation. When does BBB know the first thing about outsourcing and IT? Sure he knows law, he's a black belt, but no mention of being anything related to IT.

                              The alternative is to not outsource it and keep it all in house - back to my last statement, the government is hardly competent when it comes to IT (well perhaps a lot more than IT but I digress). So how would keeping it in house be better. Is there any information who this Romanian outfit are? Do they have a bespoke authentication product that's used by other companies/countries? Are they just some guys bedroom PC? More information would be good before one starts speculating about insecurity, but BBB is best when there's no actual information and plenty of speculation.

  36. Dizzy Dwarf

    I didn't realise 5 nines ...

    ... meant 9.9999% uptime.

  37. bazza Silver badge

    Who Needs Hackers...

    ...when Amazon pull the biggest D.o.S. there's ever been?

    Governments

    If things become too cloudy, you've then got whole economies at the mercy of the cloud owner or their mistakes. You'd think that this event would wake up governments as to the grave perils that lie in proprietary clouds that lock customers in. I'm not holding my breath on that one...

    1. Anonymous Coward
      Anonymous Coward

      Re: Who Needs Hackers...

      Grave perils one side... Large backhanders the other.

  38. Boris the Cockroach Silver badge
    Mushroom

    Meanwhile

    A message from a rogue engineer at AWS gets transmited to the resistance

    "I've built the system with one flaw, its well hidden but one blow and the whole system goes down"

    "We must get those plans......"

  39. rmullen0

    Oh no

    How is Pete Hegseth going to illegally communicate illegal military operations with Signal being down?

    1. Jamie Jones Silver badge

      Re: Oh no

      Easy. He'll ROT1 them and broadcast them on fox.

  40. Anonymous Coward
    Anonymous Coward

    It's just someone else's computer

    Whose fault is this? It's your own stupid fault! Shouldn't have relied on Amazon, Azure or Google, but on your own VPS server. Like I do, and everything's humming along nicely thank you.

    People are stupid and incompetent choosing one of the big Cloud providers. And it ain't cheap either. People are just lazy choosing AWS because it's a big name and they believe they can't go wrong with it. Their clueless boss will sign off on it regardless.

    1. ecofeco Silver badge

      Re: It's just someone else's computer

      The boss and the entire board of directors own shares of Amazon. Very likely preferred shares.

      Corporations don't buy from other corporations based on price and quality. They buy based on stock performance. And conflict of interest on a level that USED to be illegal.

      And of course, backhanders and collusion.

  41. TeeCee Gold badge

    .. working on multiple parallel paths to speed recovery.

    Aka: Running around like headless chickens.

  42. IceC0ld

    From the BBC :o)

    AWS is a US giant with a large global footprint, having positioned itself as the backbone of the internet.

    It provides tools and computers which enable around a third of the internet to work, it offers storage space and database management, it saves firms from having to maintain their own costly set-ups, and it also connects traffic to those platforms.

    That's how it sells its services: let us look after your business's computing needs for you.

    But today something very mundane went very wrong: a common kind of outage known as a Domain Name System (DNS) error.

    People who work in the tech industry will be rolling their eyes right now.

    This common error can cause a lot of havoc.

    "It's always DNS!" is something I hear a lot.

    https://www.bbc.co.uk/news/articles/cev1en9077ro

  43. ecofeco Silver badge

    I'm not surpsied

    And it's only going to get worse.

    I was browsing jobs in my region recently, and without exception, pay rates ARE falling.

    The occasional calls for out of region work are roughly the same. Stagnate or falling. And the contract schedules are getting shorter and shorter.

    The only people taking short term, low pay positions are NOT the people who are going to try their best. Ever.

    The tech bros ALL think their shiny is flawless and there is no need for maintenance and monitoring and if so, then one person can handle the 1000+ variables, no problem. Their hubris is breathtaking. In an auto-asphyxiation way.

  44. DS999 Silver badge

    Amazon ITSELF was broken for me

    I just happened to sit down in front of my TV at midnight PDT or just after to watch a movie. But the Amazon Prime app on my Apple TV didn't show any of my saved shows. Thought maybe it was some weird app problem so I rebooted my Apple TV, no luck. I thought I'd logout and back in on the app but I wasn't able to login - said I had the wrong password. So at this point I'm worried my account has been hacked and they've reset my password. I'm imagining someone ordering a bunch of stuff, me being unable to talk to a human at Amazon to stop it, and even if my credit card losses are covered they'd eat up the $300 credit I had from a recent return and Amazon might not give that back.

    So I go to amazon.com on my desktop and try to login, hoping I can tell it to reset my password if they haven't changed the email already but I can't even get past entering my username - it says I need to enable cookies. I temporarily turn off my cookie autodelete extension, no luck. I try Chrome which I hardly ever use and don't have configured with any extensions (it is sort of a sanity test or backup in case a web site refuses to cooperate with Firefox but I really need to access it) With Chrome it still says I need to enable cookies. OK, so this isn't someone hacking my account at least!

    Next I google "is amazon down" and a couple of the downdetector type sites show a huge spike in that in just the last 15 minutes or so! If I had sat down in front of my TV like 2 minutes earlier I probably could have started the movie and not had all that hassle.

  45. Random as if !
    Black Helicopters

    Localhost

    Is reqired

  46. yas1

    Coincidence?

    Just received spam emails with unbelievable giveaways from Currys and the links go to...

    amazonaws.com

    Coincidence, or am I just paranoid?

  47. Uh, Mike

    Backup Plan

    I've got cash in a drawer for when nobody can take my credit card for three days.

    I just hope I'm at home when it happens.

  48. GNU Enjoyer
    Angel

    I didn't even notice personally

    Enough written.

  49. Rogerborg 2.0

    On the bright side, US-EAST-1 has never had such a good vacuuming.

  50. Diamond

    Why is this a problem?

    Shouldn't this not be a problem? With a well-run cluster, shouldn't all these workloads automatically start back up in another region?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like