back to article AWS Tokyo outage takes down banks, share traders, and telcos

The AP-NORTHEAST-1 region of Amazon Web Services, located in Tokyo, has endured six hours of sub-optimal performance. The cloud colossus's status report states that AWS Direct Connect hybrid cloud networking service had trouble connecting to resources in the region due to "failures in core networking devices". As of 1530 …

  1. Anonymous Coward
    Anonymous Coward

    It is surely inevitable now that regulators intervene to mitigate the concentration risk of AWS and Azure. It's insane that essentially our entire financial services infrastructure runs from half a dozen cloud regions around the globe.

    1. Dan 55 Silver badge

      Don't forget governments moving to cloud either.

    2. Velv

      Perhaps we've become too reliant on the 24/7 nature of everything these days and need our expectations reset.

      Outages should be expected. It sucks when it happens to you, but you should have planned for it. Time for people to take more responsibility for their own lives and look to have contingency in place.

      I'm not really proposing this is either right or good, but there are some entitled people who think the world owes them everything for no effort on their part, and sadly the world is flawed, it does break, and we need to at least understand that fact.

      1. Anonymous Coward
        Anonymous Coward

        AWS' CTO is famous for the quip "everything fails. all the time.", and AWS (i doubt any cloud provider) don't suggest best practice guidance to implement a single point of failure?

        If a firm is sophisticated enough to be using Direct Connect, surely they are also sophisticated enough to have a fall back? It's not like AWS don't provide mechanisms for backup solutions?

      2. bombastic bob Silver badge

        the internet was designed to have a routing failover built in.

        Why not "the cloud" ?

        Electricity and phones and water in 1st world countries are expected to have 100% reliability. Why NOT "the cloud" ? I believe it CAN be done, and SHOULD.

        (see my earlier post, below)

        1. Ken Moorhouse Silver badge

          Re: Why not "the cloud" ?

          Interesting question which I presume because the internet is about routing data, it doesn't need to be aware of which data went where historically, which I assume the cloud must do in order to retrieve it later, a bit like RAID but involving network packets dynamically accessing storage as well as RAID techniques such as bit striping.

          So if a section of the cloud goes down, I surmise there has to be a mechanism for piecing together replica data from servers that are unaffected by the outage, and promoting these to be the primary version. Not an easy choice to make automatically where latency can cause a positive feedback situation to cause instability.

    3. Pascal Monett Silver badge

      What is insane is that a bank relies on someone else's server.

      Banks have a duty of security and confidentiality with their customer's data. How is that duty respected on a cloud server ?

      1. This post has been deleted by its author

      2. Anonymous Coward
        Anonymous Coward

        Probably the same ways they have when storing your data on servers run by IBM/TCS/HCL/Wipro/Accenture/Infosys/etc etc since time immemorial.

        "Someone else's server" is not a new concept.

  2. Potemkine! Silver badge
  3. Ken Moorhouse Silver badge


    ...will be a silent event.

    No Bank cloud.

    No Azure.

    No AWS.

    No Twitter...

    No Twitter??

    No Twitter OMG!!

    On-premises systems will be wheeled out, just like the shopping trolley in Cormac McCarthy's The Road.

  4. Imhotep

    And This Room Contains Our Data Center And Its Backup

    They lost "several" core networking devices? It would be interesting to hear how that happened and why there was no failover to backup devices.

  5. bombastic bob Silver badge

    private cloud failover?

    Here's a business opportunity for AWS: sell/rent end-users an inexpensive "private cloud failover system" in the form of a private cloud server that is capable of automatically handling the load (albeit slower) while the rest of the network is titsup...

    (It could also handle low bandwidth loads and include synchronization when normal services are restored)

    Just a thought, anyway. No more ALL eggs in ONE basket. The web publisher/whatever would basically rent or buy the server (maybe co-located on a different part of the internet, or physically local to the customer), and use regular AWS things to manage it (along with everything else). It would be for uber-high-reliability services, like banks, hospitals, governments, military, etc.. And it would handle a portion of the normal load. It would also have to specifically be set up to work when completely disconnected from the rest of the cloud services, and reliably sync up when services are restored.

    Something to think about.

    (assuming they do not already do this)

    1. Kevin McMurtrie Silver badge

      Re: private cloud failover?

      Managing data in multiple places requires completely different application design. Each location has different performance and availability, and it's all dynamic.

      AWS does recommend that everything be multi-region. Of course they would, because it's 3x money for them: 2x hosting + failover overhead + moving data

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like