back to article Atlassian Jira, Confluence outage persists two days on

Atlassian's Jira and Confluence issue tracking and collaboration services have been down for some customers for two days now and the outage remains ongoing. Starting around ​​Tuesday at 0902 UTC, Atlassian's status page reported problems affecting customers using Jira Work Management, Confluence, Jira Service Management, Jira …

  1. Anonymous Coward
    Anonymous Coward

    We continue to work on the resolution of the incident for a number of our Jira Work Management, Jira Service Management, Confluence, Jira Software, Atlassian Access, Jira Product Discovery, and Opsgenie Cloud customers. You can follow the progress in jira ticket...... oh wait.

    1. teknopaul

      Rest assured

      They are working on... something, they just first need to login, find out whats wrong, triage, and check the wiki to find resolutions from last time, execute the required ops procedures,

      and then they can report on it.

      Anyone know if they use their own cloud?

      1. Kabukiwookie

        Re: Rest assured

        Our DR plan is in the wiki...

  2. VoiceOfTruth Silver badge

    The vocabulary of our age

    -> a small number of sites

    It's always always a 'small number' or a synonym. Perhaps I can enlarge (geddit) the vocabulary of the people who write these status messages. Let me suggest bijou, dinky, teeny-weeny, stubby, piddling, measly...

    1. Gene Cash Silver badge

      Re: The vocabulary of our age

      I'm thinking "a small number" is a short phrase for "95% of our customer base"

      1. Doctor Syntax Silver badge

        Re: The vocabulary of our age

        Or "not more than 100%".

    2. Anonymous Coward
      Anonymous Coward

      Re: The vocabulary of our age

      To be fair, it doesn't seem to have affected our Jira systems.

      At least, it doesn't seem to be any more sluggish or clunky than usual...

      Quite why any modern system needs you to manually refresh the entire page to see new comments - or to pick up new values in the search boxes - is completely beyond me!

      1. Ace2 Silver badge

        Re: The vocabulary of our age

        I just want a way to comment that doesn’t include their garbage editor auto-formatting everything badly. Plain text isn’t an option anymore!

  3. jeff_w87

    Cloud vs On_Premise

    This is why moving your "stuff" to the cloud is probably not the best option for a lot of people. Centralized target for hackers of all stripes and a central point of failure when someone on staff screws up an update or maintenance activity. This happens way too often with all these big players and the fact that they promise resiliency with their cloud offerings, it just doesn't seem to pan out and customers are down for hours or days. Hopefully their customers in this case, since it looks like they've violated their SLA will be getting some major refunds, but I'm sure the CEO will still get his/her bonus for the year.

    1. Ben Tasker

      Re: Cloud vs On_Premise

      > This is why moving your "stuff" to the cloud is probably not the best option for a lot of people.

      Statistically, there are likely some affected users who were all but forced into the cloud.

      Atlassian axed their on-prem versions (unless you want to pay for a datacentre license) about a year back.

      1. Anonymous Coward
        Anonymous Coward

        Re: Cloud vs On_Premise

        They axed it for new orders, luckily (for me in any case) renewals are still an option until 2024. Though unfortunately we’ll have to start moving next year since maintenance for us end in February 2024.

        The price hike in our case : over 30% (at current prices), which in absolute numbers is a LOT more than maintaining the 2 measly VM’s our setup requires. That 30% does not include the cost we’ll have to change internal applications that integrate with their crap, and does not include the price hike we’ll have to pay for some addins that are sold either in a Jira cloud or a Jira on-prem version.

      2. Nate Amsden

        Re: Cloud vs On_Premise

        While I knew the situation I had a good laugh anyway. I recently renewed a 10 user server license for confluence that I purchased(?) about 10 years ago(for extremely limited personal use) but had lapsed. The cost to renew was $110, to "true up" the license to current time. That's fine, not a big deal.

        Then saw the suggestion hey you can move to data center edition. Again I knew the situation but was curious anyway. To see the $10 price on the left for my existing license(to renew again for another year), vs the lowest cost data center offering of a mere $27,000 I think it was on the right(for 500 users).

        But at least the license is still perpetual(for the given version of the product anyway).

        I've been using confluence since early 2006, and with the cloud version(inherited from the orgs I worked at) the experience has significantly gone downhill in many aspects. My favorite version of confluence I think was probably version 3(guessing here it was a long time ago, the last version to support editing wiki markup). I have been somewhat relieved that their cloud folks have seemed to have postponed indefinitely their forced migration to the new editor. I had so many issues with it and tickets and phone calls. They kept saying that the new editor will be forced soon and I'd have no choice but to use it. But that was about 2 years ago now and that hasn't happened. Surprising they have not been able to address whatever edge cases the old editor allowed that the new one does not yet. At least they fixed one of my most annoying issues which was keys getting stuck and just printing the same character over and over and over again. Took them weeks to figure it out after trying to blame my computer/browser for the issue.

        I use JIRA regularly as well but much less often. I don't use any other Atlassian products.

        My regular wiki at home is Xwiki which seems to work quite well, confluence is just for some other stuff that I want to be able to access that I haven't moved(yet).

        1. teknopaul

          Re: Cloud vs On_Premise

          Confluence has some cool concurrent editing features. Useful even for small teams.

          It become a bit of a beast of late and I gave up trying to keep up with new features.

          I do wish the wiki syntax actually worked and you could post code and log output without hassle.

          New features seem to cause breakage. I haven't found the _insert this text verbatim_ feature.

    2. Anonymous Coward
      Anonymous Coward

      Re: Cloud vs On_Premise

      We moved from on-prem Jira to their cloud version (ours is still working), and the main reason for us was that nobody in the office wanted to be responsible for maintaining it. It worked out easier for us to pay Atlassian to deal with that, so we could concentrate on maintaining things for our customers who pay us money.

  4. Anonymous Coward
    Joke

    "Our team identified this immediately..."

    > Our team identified this immediately...

    Perhaps not quite immediately - it probably took an "oh no second" :-)

  5. Woza
    FAIL

    Waiting for

    The "who, me?" article

  6. aki009

    RIP Jira/Confluence

    When Atlassian started forcing customers to go to their Cloud, these formerly independent customers became vulnerable to this sort of thing. Less stuff for more bucks. It's the Oracle model or something.

    1. Anonymous Coward
      Anonymous Coward

      Re: RIP Jira/Confluence

      It's not that long ago that Oracle replaced their in-house applications with Confluence and Jira. People complained about how crappy the Atlassian stuff was, this really feels like karma.

  7. hoofie

    Gets Expensive

    You move off the on-premise JIRA because that's costing you time and money to maintain the VM.

    Once you are on the cloud version you find your yearly expenditure is matching or more than what you paid to manage and maintain the VM.

    QED

    1. aki009

      Re: Gets Expensive

      I'm sure there is more than one company looking for a replacement that's not cloud based.

  8. Anonymous Coward
    Anonymous Coward

    change freeze

    "If there's a lesson here, it may be that what happens in Vegas doesn't stay in Vegas if your cloud service is down. "

    And another one is "freeze any darn maintenance/change while we have a big show running on, FFS !". How could this go past CAB ???

    1. Robert Grant

      Re: change freeze

      I remember CABs.

  9. Anonymous Coward
    Anonymous Coward

    The Cloud...

    Other peoples computers you have no control over

    1. Doctor Syntax Silver badge

      Re: The Cloud...

      Maybe we should update that along the lines of "Other peoples' computers who have control over you."

  10. Anonymous Coward
    Anonymous Coward

    Atlassian have been bugging us constantly to move on-prem to their cloud service, hmmmm...glad we held off

    1. aki009

      We ain't going to the cloud...

      For us there's no possibility of us moving to the Atlassian cloud. We'll rather spend the effort going to another solution than be entirely dependent on Atlassian's infrastructure working right. Especially in light of their "maintenance scripts".

  11. pip25
    WTF?

    Maintenance script?

    It's hard to imagine how such a script can cause several days of outage, unless it involved "maintaining" the customer database and their backups by deleting them all.

  12. Abominator

    That and they have been price gouging their long time customers on server/data centre.

    Fucking grade ad arse holes.

  13. Tzhx

    How rude

    Just my luck.... stuff I use and would like to keep using, whenever it's down for a "small number of users", I'm somehow in that small number.

    And whenever I'd rather have the excuse that "oh that pile of shit is broken so I'll just work on what I think is most important"... Jira continues to be there with it's horrific interface and tedious ceremonies.

  14. DubyaG
    Mushroom

    Routine Maintenance, Right

    If this was routine, that implies something that is done on a regular basis. In the world of Change Management, I was under the impression that changes should always have a back out plan ready to go or some other Plan B. So a routine task shouldn't take down the whole enchilada. In my case, we use on-prem Jira and when notified of the "You vill all go to the cloud" announcement a search was started for another solution. Bye Bye Atlassian.

  15. Rob Davis

    Host your own opensource with Drupal Burndown agile/kanban, though self-host will not suit all

    https://www.drupal.org/project/burndown - this provides a Kanban or Agile Board ticketing system.

    1. Get a digitalocean vps for about 10-20 pounds per month. Or host on a raspberry pi. Rationale for these smallish servers is that if the ticketing system is only used by the business, then a large scale server for a public site is not necessary.

    2. Map to a web domain, if hosted on a raspberry pi over a non-static IP, use a registrar with a dynamic dns server, like joker

    3. Setup a LAMP stack on the server

    4. on the server get and install drupal open source CMS, and add the burndown module, also open source, free

    5. setup email to send notifications, perhaps use zoho mail - low cost

    6. manage backups yourself and security updates

    No affiliation, financial incentive or referall benefits of the above to me. There are likely other similar open source solutions out there. Bugzilla comes to mind but I'm not sure how well it works for most recent processes these days, having not used it for many years.

    The above will not suit all. There is always going to be a cost and risk whatever approach and that includes consider self-hosting. With the greater control of self-hosting, comes a requirement to have the knowledge in-house and manage the server which means billable time out of one or more staff, along with a process for absence and when those staff move on, offboarding, onboarding.

    Cloud has the benefits to those whose business core competence/reason for existing is not to manage a ticketing system. Which makes the cloud a valid consideration. I've used cloud services includingJIRA in some roles as a developer, as well as trello and pivotaltracker.

    Still, having options is healthy and the self-hosting may suit some who already have teams in the business of doing that kind of work already with the related skillset.

  16. Anonymous Coward
    Anonymous Coward

    TITSUP

    Ticket Information Totally Suspended, Unknown Prognosis

    Why?

    Typing In The Script Upset People

  17. Kevin McMurtrie Silver badge

    Little Bobby Tables' older brother

    Nolan Romy Riff-Star, short name \nrm -rf *

    Such a trouble maker.

  18. Enigman

    Routine until it's not

    At my last job all our changes were peer reviewed and any scripts that were to be run also had to have a peer review by someone with equivalent or greater scripting skill/experience. Production level changes went before a CAB except where they were declared 'routine' - routine being the change was successfully executed 10 separate occasions with no backout required, no incidents attributed to the change. Changes to the script would result in the counter being reset and the change could end up needing to be presented to CAB.

    If the script hadn't changed since it was last run the peer review of the script wasn't mandatory but we did have to provide evidence that the script hadn't been changed for Production changes (timestamps, checksum or similar.)

    Scripts had to be like for like across environments (Dev, Test, Prod (DR was classed as Production) unless there was a specific reason (different setup in Dev to Prod - client tried to save money with lower specs for some environments.)

    At the time I got shoved out the door they were working on 'streamlining' processes and pushing the Agile line to speed things up. I had heard on the grapevine later on that my former offshore colleagues had tried to blame one of my old scripts for a major incident. What eventually transpired they had taken one of my scripts as a base, changed some of it's functionality for what they wanted and in the process removed the parts that backed up existing configurations/settings etc. They had presented it as being 'unchanged' so that they didn't have to go through CAB.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like