back to article Fujitsu's Australian cloud suffers storage crash, outage

Fujitsu Australia has confirmed that its data centre in the Australian suburb of Homebush has experienced an outage. “A storage array in Fujitsu’s Homebush data centre became unavailable at 9.24PM on Saturday night and has affected services to customers,” a spokesperson told The Register. “We are treating this matter as a …

  1. Pascal Monett Silver badge

    "a major Australian financial institution", "the VMs cannot be recovered"

    Well as far as I'm concerned, that test was quite successful and the conclusions are obvious : do not trust the cloud. Not if you're a financial institution with an obligation of 100% accuracy in managing your customers' accounts.

    The cloud is still about as solid as its name. It must be great fun spinning up VMs and chucking all that load balancing in an environment that one does not have to manage, but it's a lot less fun being advised that all your data is lost and cannot be recovered.

    Companies will go down because of this.

    1. Anonymous Coward
      Anonymous Coward

      Re: "a major Australian financial institution", "the VMs cannot be recovered"

      This happened to me twice with Fudgeitso. Both times multiple disks in Netapp boxes were supposed to have experienced simultaneous failures.

      Far more likely imo that they didn't monitor them properly and just didn't fix them as they failed...

    2. TheVogon

      Re: "a major Australian financial institution", "the VMs cannot be recovered"

      "Companies will go down because of this."

      Quite. Have you seen what IT service providers pay for technicians to run their stacks? They must be filtering off the dregs from the bottom of the pile...

    3. GortonSM

      Re: "a major Australian financial institution", "the VMs cannot be recovered"

      - cummon ... this isn't "Cloud" its hosted virtualisation - not even close ......

      1. TheVogon

        Re: "a major Australian financial institution", "the VMs cannot be recovered"

        "this isn't "Cloud" its hosted virtualisation - not even close ......"

        So private cloud then. Pretty close...

        1. Anonymous Coward
          Anonymous Coward

          Re: "a major Australian financial institution", "the VMs cannot be recovered"

          No such thing as "Private Cloud", that's the old world of hosting and on-prem.

          Fuj calling it a cloud doesn't make it one, Public cloud from AWS and MS is secure and reliable, much more so than a traditional DC.

          Of course if you're scarred Public Cloud will take your job away I can understand you desperately trying to spread FUD about it.

  2. Your alien overlord - fear me

    Who knew, some cloud storage data is not backed up or duplicated to other off site clouds. Might as well have kept it in house!!

    1. Anonymous Coward
      Anonymous Coward

      Proper Public Cloud is, this is neither.

  3. Mad Mike

    The reality of cloud

    This shows the reality of cloud in all its glory.

    In complete contrast to the salesmens blurb, the cloud environment needs at least as much management and design as in-house. The salesmen banging on all the time about their company doing all the work and you can just sit back and relax is misleading in the extreme. All this was shown some time back by the Amazon S3 cloud outage in the USA, where some carried on OK, but many were floored by it. Who knew cloud storage could go down?

    This takes it one stage further by not only the storage going down (and obviously in such a way as to loose the data), but backups weren't even being taken. Cost cutting? Not understanding what was being provided? Who knows. Pretty poor show for a bank though.

    1. Sergiu Panaite

      Re: The reality of cloud

      The S3 outage that I think you're talking about shouldn't have been a problem if the people using it were aware they would need to set themselves up for potential relocation if (when..) it went down. As one may say, the cloud part was "working as designed".

      This Fujitsu one though, it goes against what "cloud" is supposed to be, Shirley? I thought the whole point was that geographically different locations would ensure that even if your primary location fails, it all "evaporates" away and "condenses" in another location. So this wasn't really a "cloud" deployment, but more of a "hosted environment, single DC" deployment. Just because you use the same tech that would allow you to deploy a proper cloud doesn't mean you deployed a cloud, but merely a bunch of steam like you'd get from a kettle in an unvented kitchen when the kettle doesn't stop itself from boiling when it was suposed to (don't ask...)

      1. Mad Mike

        Re: The reality of cloud

        @Sergiu Panaite.

        I think you're basically saying what I was. Just because it's a cloud, doesn't mean you don't need to architect it appropriately. Your second paragraph shows the misconception that people have with cloud and the idea that it sorts out problems for you. In some cases, it might, but not always. You still need to think of D/R. You still need to take backups (or risk loosing it all). With compute issues, it will normally self-heal within a location by bringing the VMs back up on other servers. If all the servers fail (unlikely), that's D/R, not cloud relocating you to another location. The issues with storage is that depending on how the storage is designed, it can make the data unavailable, hence bringing it up elsewhere is simply not possible. Unless, of course, part of the architecture is to replicate the data into another location.

        Clouds run by other companies such as Fujitsu, Amazon, Microsoft etc. are no better than one you could setup in your own sites. It all needs to be architected properly, with allowance made for each component that can fail and understanding of how you resolve that, or (depending on cost v risk) accept it and go down.

        1. Sergiu Panaite

          Re: The reality of cloud

          Looks like I was - hadn't had my first portion of crac... err, coffee for the day. Apologies.

    2. 2Nick3

      Re: The reality of cloud

      "Pretty poor show for a bank though."

      If it was just test/dev (as the article indicates) that is normally not treated with the same expectations as prod, so running it on Cloud isn't so egregious. If it is running in-house it would be on a secondary array, possibly old Tier1 storage, and they aren't always backed up.

      So the customer may have known exactly what they were getting here, and there may be no panic at all.

      Just some IT staff having a busier Monday repopulating the test/dev environment from prod.

      1. Mad Mike

        Re: The reality of cloud

        @2Nick3.

        You're right that there is too little information for a fully informed comment. From the reading, it sounds a bit like the VMs may be gone as well, but it isn't clear. I totally agree that test/dev is completely different to production, but backups (weekly, or even monthly) are still pretty normal. Not backing it up at all is somewhat cavalier. Be interesting to see how long it takes to bring those environments up again and that will show how ready and prepared they were for the event.

  4. DuchessofDukeStreet

    And of course customer beancounters are always quite happy to fund the costs of having proper back-up architecture for their dev and test environments. Because they're not really important, and if they go down, the outage can be accommodated and the platform restored at leisure.

    Until it can't, of course.

    1. Mad Mike

      @DuchessofDukeStreet.

      I think the issue here isn't so much the fact they went down. I'm sure a lot of organisations would accept 24hrs of downtime on test and dev to reduce costs. The issue is that they don't seem to be recoverable. Like they never heard of backups......or paid for them.

  5. Doogie Howser MD

    Cloud / Not Cloud

    Careful with the terminology here - public cloud and what Fujitsu offer may be somewhat different. In my experience, it's just managed services - i.e. your VMs on their kit and they manage it for you. They don't offer PaaS or SaaS or anything that resembles public cloud, including characteristics such as consumption monitored billing or automation.

    If the Aussie barn offers this then great, but don't lump Azure/AWS/GCP et al into the same bracket. They're not the same. And to re-iterate the point, don't keep a local (i.e. single DC) only copy of data that matters, I'd have thought in this day and age it would be obvious.

    It's not sales bullshit, it's simple public cloud / managed services design.

    1. Mad Mike

      Re: Cloud / Not Cloud

      @Doogie Howser MD

      It's all poor architecture. However, it can be very difficult when the salesmen are coming into the big wigs and claiming all sorts of things. Then, you put in a request for funding including things like resilience and DR and the big wigs say it's not necessary, as the salesmen said cloud solved everything and all their problems would go away.

      It all comes down to doing a proper architecture/design job, but the salesmen being somewhat simplistic to say the least, doesn't help senior management actually understand what they are and are not buying.

  6. Anonymous Coward
    Thumb Up

    "Strewth, Sheila!

    Someone's nicked me bits!"

    I expect they didn't say.

    1. kokoro

      Re: "Strewth, Sheila!

      You'll be lucky to find much ocker 'Strain spoken round Homebush and nearby Lidcombe where Fujitsu has its repair fac these days, lol

  7. TheBBG

    A salesman stated that his storage facility was 100% safe and no data could ever be lost. I asked him if his company would sign a contract giving 'me' total ownership if we signed up and subsequently lost any data. Never heard from him again :D

  8. GoldCoaster

    Unsurprising several luddites here are using this as a reason to spread FUD on cloud.

    Real, public cloud (AWS, Azure and GCP) has an exemplary uptime record, better than your private DC or co-lo, of course there not immune to failure and you need to architect for this, but its still better run than a private company can.

    Some other vendors "cloud washing" their old hosting offerings does not make them the same thing.

    1. Anonymous Coward
      Anonymous Coward

      GoldCoaster: cloud architect?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like