back to article Day 7 of the great Atlassian outage: IT giant still struggling to restore access

The great Atlassian outage is stumbling into a new week, with the biz reporting it has "rebuilt functionality for over 35 percent of the users who are impacted by the service outage," meaning the majority of those afflicted remain unable to access their sites. At this point it is fair to say the problem is severe. It kicked …

  1. Def Silver badge
    Facepalm

    But but but....

    Software as a Service and the Cloud are good things, right? Right? Riiiight?

    Yeah, right. Right up until they're not.

    1. CommonBloke
      Childcatcher

      Re: But but but....

      But they are good!

      Just not for the consumer.

      Who woulda thunk outsourcing your data would ever become a problem, eh? One in a million chance!

      1. Yet Another Anonymous coward Silver badge

        Re: But but but....

        We use Atlassian on-premises, so not affected

        Atlassian are stopping on-prem so we are currently trailing switching to Github cloud (?)

        1. djnapkin

          Re: But but but....

          They are stopping on-prem? I wonder if that will still go ahead after this disaster?

      2. TheWeetabix

        Re: But but but....

        Well, if we consider each second a chance of something happening, one in a million is twice a flipping month. Not exactly Five Nines.

      3. rototype

        Re: But but but....

        ... and we all know one in a million odds come up 9 times out of 10

    2. Plest Silver badge

      Re: But but but....

      They are good....if planned, implemented and managed properly. If you're taking money off customers, short changing them and cutting costs like crazy so your shareholders don't lose out, well you reap what you sow.

      1. werdsmith Silver badge

        Re: But but but....

        They are good at the moment, as I actively avoid using the fussy over busy Atlassian and JIRA I am only to happy to see this outage.

    3. oiseau Silver badge
      Facepalm

      Re: But but but....

      Software as a Service and the Cloud are ...

      One of the beancounters' ultimate wet dreams.

      But it is nothing but that.

      What are Atlassian's clients asking their beancounters now?

      "Say, how much is this royal fuck-up going to cost us and just how are we going to pay for it?"

      When everything goes haywire and there's no solution in sight after a week (cue Atlassian) ...

      What options do you think you really have?

      Just three, last one optional:

      1. grin

      2. bear it

      3. lay back and think of England and/or Boris Johnson

      When you have in house services you are certainly not problem exent.

      But if your IT line, from the manager to the last PFY is populated by technically competent people being paid a decent wage, be sure they will take care of it and things will be up and running as before with a minimum downtime.

      More expensive?

      Sure.

      More reliable?

      Yes.

      By a great many orders of magnitude.

      O.

      1. teebie

        Re: But but but....

        "More expensive?

        Sure.

        More reliable?

        Yes.

        By a great many orders of magnitude."

        ...and thus less expensive, when you look at the big picture.

  2. Anonymous Coward
    Anonymous Coward

    Business continuity

    Now is a good time for all of these project managers to reflect on the importance of resilience and business continuity.

    Scream and shout at Atlassian as much as you want, it won't restore it faster.

    1. FrankAlphaXII

      Re: Business continuity

      Amen. I'm in IT (now, again, whatever) but I come from an Emergency Management background. I really hope all of these affected customers had a decent continuity plan that had been exercised realistically and not merely as a means of checking off some VC's checklist to get funding. My standards are probably a bit high but think I know the answer to that by even a more reasonable measure if some of the Twitter threads I've read are any indication.

      BCP is like how security used to be back in the day, nobody took it seriously until it started to cost more to not give a shit.

  3. Anonymous Coward
    Anonymous Coward

    Ah....remember....."cloud" is cheaper......

    .....until you factor in the cost of NOT HAVING THE SERVICE AT ALL!!!

    1. Anonymous Coward
      Anonymous Coward

      Re: Ah....remember....."cloud" is cheaper......

      On-premise solutions are also capable of providing no service at all.

      1. Cav Bronze badge

        Re: Ah....remember....."cloud" is cheaper......

        Only if you are incompetent.

      2. Nate Amsden Silver badge

        Re: Ah....remember....."cloud" is cheaper......

        really it comes down to too many eggs in one basket. Certainly service failures can occur on premises. But pretty much universally those failures affect only a single organization. Granted there can be times when multiple companies are experiencing problems but it's still tiny compared to the blast radius of a SaaS provider having a problem.

        My biggest issue with SaaS at least from a website perspective is the seemingly constant need that the provider feels to change the user interface around and convinced everyone will love the changes. Atlassian has done that tons of times and it has driven me crazy. Others are similar, so convinced all customers will appreciate the changes.

        Go change the back end all you want as long as the front end stays consistent please.

        At least with on prem you usually get to choose when you take the upgrade, and in some cases you can opt to delay indefinitely (even if it means you lose support).

        Just now I checked again to confirm. Every few months I go through and bulk close resolved tickets(in Jira) that have had no activity for 60 days. I used to be able to add a comment to those tickets I would say "no activity in 60 days, bulk closing". Then one day this option vanished. I asked Atlassian support what happened and they said that functionality was not yet implemented on their new cloud product (despite us having being hosted in their cloud product for years prior). I can only assume it is a different code base to some extent. Anyway that was probably 3-5 years ago, and still don't have that functionality today. (there is an option to send an email to those people when the ticket closes I don't want that, I just want to add a comment to the ticket).

        Don't get me started on the editor changes in confluence in recent years just a disaster. Fortunately they have backed off of their plans to eliminate the old editor(for how long I don't know but it seems like it's about 2 years past when I expected them to try to kill it).

        Then there was the time they decided to change the page width on everything in confluence(I assume to try to make it printable), at least in that case they left an option(per user option) to disable that functionality(it messed up tons of pages that weren't written for that option).

        The keyboard shortcut functionality drove me insane in confluence as well, for years assuming it was there before(I don't know, I never used keyboard shortcuts in confluence going back to my earliest days of using it in 2006) it was not a problem but past couple of years I would inadvertently trigger a series of events on documents that I did not want just by typing. I was able to undo it every time, and finally disabled the keyboard shortcuts a few months ago.

        1. djnapkin

          Re: Ah....remember....."cloud" is cheaper......

          We also used to use the commenting when doing a bulk close on tickets. That was in previous job with on-prem. I shake my head that they haven't fixed that in their cloudy option.

        2. teebie

          Re: Ah....remember....."cloud" is cheaper......

          You can disable the keyboard shortcuts?

          Wow, with a single unticking life just got a little bit simpler.

          I checked if there was a 'be a javascript abomination' setting I could untick on the same page, but sadly there isn't.

          1. deep_enigma
            Trollface

            Re: Ah....remember....."cloud" is cheaper......

            No, that one's buried in about:config in your browser....

      3. Zippy´s Sausage Factory
        Devil

        From the ZSF book of quotations...

        "But if it's in the cloud, that's always cheaper and safer because we don't need local people managing it? And if it goes wrong we can just sue them, right? Right?" -- way too many middle managers

        1. A Non e-mouse Silver badge
          Unhappy

          Re: From the ZSF book of quotations...

          When writing contract specs, our legal dept. insist on putting in KPIs to keep the vendor honest. The vendor duly agrees to these KPIs when they sign the contract.

          I asked our legal people: What would happen if the vendor breached the KPIs? Would we sue them? Terminate the contract? All I got in return was a shrug. The legals are keen to add all this boiler-plate into the contract, but not keen on actually doing anything when asked to.

          The vendor knows our legal team aren't keen on taking any action so don't on the KPIs anyway.

        2. Anonymous South African Coward Silver badge

          Re: From the ZSF book of quotations...

          And if it goes wrong we can just sue them, right? Right?

          A hollow voice says "Fool".

      4. An_Old_Dog Bronze badge

        Re: Ah....remember....."cloud" is cheaper......

        True, but on-premise problems, you at least have a fair-to-excellent chance of fixing yourself. For off-premise problems, there's not a damned thing you can do.

        For those of you with cloud service SLAs, how are those working out for you?

    2. thondwe

      Re: Ah....remember....."cloud" is cheaper......

      Cloud isn't cheaper - you're paying for all the bodies to do the hard graft of rebuilding a broken system for you and taking the political flack

      The on premises alternative means you having availability of knowledgeable staff (not off sick with Covid/Holiday) plus spares for any server/network tin/data centre pieces/rooms/... and taking the political flack...

      Pick your risk profile...

      1. Cav Bronze badge

        Re: Ah....remember....."cloud" is cheaper......

        Which you will have in place, if you are not incompetent.

        1. A Non e-mouse Silver badge
          Mushroom

          Re: Ah....remember....."cloud" is cheaper......

          And big enough to have:

          * Enough knowledgeable staff to cover for sickness, holiday, COVID, etc

          * Enough kit/capacity to cope with systems(s) going down

          You pick the right tool for the job. I work for a large company and we have a mixture of on-prem and cloud. On-prem when the problem is big enough to tick the boxes above; Cloud when the product/service is too small/niche for us to keep skilled up to manage.

  4. NoneSuch Silver badge
    FAIL

    IDEA: Let's put our core contact with clients into another companies hands and hope for the best.

    "Welcome to Itchy and Scratchy Land, where nothing can possibli go wrong ... Er, possibly go wrong...that's the first thing that's ever gone wrong."

  5. John Miles

    Could be 2 more weeks according to a Reddit comment

    Seen a Reddit comment - Link

    got email from the community manager, that some instances can be down for further two weeks.

    This is not how a billion dollar company build the system or handles recovery, I am going to look for an alternative and dump Atlassian as soon as possible.

    ==== snip of the email I got ====

    What this means for your company

    We were unable to confirm a more firm ETA until now due to the complexity of the rebuild process for your site. While we are beginning to bring some customers back online, we estimate the rebuilding effort to last for up to 2 more weeks.

    I know that this is not the news you were hoping for. We apologize for the length and severity of this incident and have taken steps to avoid a recurrence in the future.

    1. JoeCool

      Re: Could be 2 more weeks according to a Reddit comment

      Just the vagueness of that "update" should raise alarms.

    2. John 104

      Re: Could be 2 more weeks according to a Reddit comment

      @John Miles

      The people responsible for sacking the people responsible for taking steps have been sacked.

    3. nematoad Silver badge
      Stop

      Re: Could be 2 more weeks according to a Reddit comment

      "...and have taken steps to avoid a recurrence in the future."

      And so should you by looking to move to somewhere else.

      On-prem there is usually someone to carry the can.

      SaaS, "Who ya gonna call?"

    4. arachnoid2

      Re: Could be 2 more weeks according to a Reddit comment

      Is this how long it takes to pay a ransom these days?

  6. Anonymous Coward
    Anonymous Coward

    Compensation ?

    Be curious to know how much compo Atlassian think is appropriate, and then compare it to how much they get sued for.

    When RBS went down, some people lost jobs and houses.

    1. wolfetone Silver badge
      Trollface

      Re: Compensation ?

      One month's free trial should be enough, surely?

      1. Anonymous Coward
        Anonymous Coward

        Re: Compensation ?

        One month's free trial of no service?

        Sounds like a deal to me*

        *Not saying it's a good deal...

      2. Anonymous Coward
        Anonymous Coward

        Re: One month's free trial should be enough, surely?

        > One month's free trial should be enough, surely?

        Don't forget a one year free trial of Equifax credit monitoring.

        I know this is downtime/loss and not a breach, but it seems to be the traditional catch-all compensation these days.

  7. fidodogbreath Silver badge

    We suspect that the "dedicated team" Atlassian assigned to sorting out the problem has yet to take down the bunting from World Backup Day before the incident occurred. [...] The irony of [Jira Service Management] collapsing into a heap due to an issue with a maintenance script will not have been lost on the affected users."

    ^^^ This is why I read El Reg. ^^^

  8. Plest Silver badge

    On a positive note...

    I bet there's a lot of customer re-negotiating their contracts to start running local copies of Confluence and Jira again, and I bet they're going to get a nice deal on those licenses under threat of taking their PM software needs elsewhere.

    Before Atlassian ask, "Where you going to go?". RIght now, you got nothing, no service and no app so anything's gotta be better than absolutely nothing that Atlassian are offering for their £20,000 a year JIRA cloud license that worth less than the paper it's printed on.

    1. Korev Silver badge
      Childcatcher

      Re: On a positive note...

      Yeah, right now emailing copies of project_plan_and_tickets_v3_final_old.xlsx about would be working better

  9. iron Silver badge
    Facepalm

    There's a hole in your Bitbucket

    There's a hole in your Bitbucket,

    Dear Atlassian, dear Atlassian,

    There's a hole in your Bitbucket,

    Dear Atlassian, my dear.

  10. -tim
    Facepalm

    Options?

    It is amazing how much of Atlassian's stuff can be replaced in a single weekend by two coders with a private usenet server, git, some perl template toolkit web pages, markup to html scripts, and a html friendly newsreader.

  11. Anonymous South African Coward Silver badge

    The Cloud

    The Cloud is somebody else's computer. And most likely, the contract that was signed, absolved said cloud operator from any blame should services go TITSUP.

    A BOFH's wet dream. Punt a service, have people use it, make lots of money easy, and when things goes TITSUP, you just point to the relevant contract clauses.

    Lather, rinse, repeat.

  12. Anonymous Coward
    Anonymous Coward

    I've interviewed with them a few times, i think 2020 was the last time I talked to them.

    I was never all that impressed with them.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022