Atlassian Jira, Confluence outage persists two days on

Atlassian's Jira and Confluence issue tracking and collaboration services have been down for some customers for two days now and the outage remains ongoing. Starting around ​​Tuesday at 0902 UTC, Atlassian's status page reported problems affecting customers using Jira Work Management, Confluence, Jira Service Management, Jira …

[Anonymous Coward]

We continue to work on the resolution of the incident for a number of our Jira Work Management, Jira Service Management, Confluence, Jira Software, Atlassian Access, Jira Product Discovery, and Opsgenie Cloud customers. You can follow the progress in jira ticket...... oh wait.

[teknopaul]

Rest assured

They are working on... something, they just first need to login, find out whats wrong, triage, and check the wiki to find resolutions from last time, execute the required ops procedures,

and then they can report on it.

Anyone know if they use their own cloud?

[Kabukiwookie]

Re: Rest assured

Our DR plan is in the wiki...

[VoiceOfTruth]

The vocabulary of our age

-> a small number of sites

It's always always a 'small number' or a synonym. Perhaps I can enlarge (geddit) the vocabulary of the people who write these status messages. Let me suggest bijou, dinky, teeny-weeny, stubby, piddling, measly...

[Gene Cash]

Re: The vocabulary of our age

I'm thinking "a small number" is a short phrase for "95% of our customer base"

[Doctor Syntax]

Re: The vocabulary of our age

Or "not more than 100%".

[Anonymous Coward]

Re: The vocabulary of our age

To be fair, it doesn't seem to have affected our Jira systems.

At least, it doesn't seem to be any more sluggish or clunky than usual...

Quite why any modern system needs you to manually refresh the entire page to see new comments - or to pick up new values in the search boxes - is completely beyond me!

[Ace2]

Re: The vocabulary of our age

I just want a way to comment that doesn’t include their garbage editor auto-formatting everything badly. Plain text isn’t an option anymore!

[jeff_w87]

Cloud vs On_Premise

This is why moving your "stuff" to the cloud is probably not the best option for a lot of people. Centralized target for hackers of all stripes and a central point of failure when someone on staff screws up an update or maintenance activity. This happens way too often with all these big players and the fact that they promise resiliency with their cloud offerings, it just doesn't seem to pan out and customers are down for hours or days. Hopefully their customers in this case, since it looks like they've violated their SLA will be getting some major refunds, but I'm sure the CEO will still get his/her bonus for the year.

[Ben Tasker]

Re: Cloud vs On_Premise

> This is why moving your "stuff" to the cloud is probably not the best option for a lot of people.

Statistically, there are likely some affected users who were all but forced into the cloud.

Atlassian axed their on-prem versions (unless you want to pay for a datacentre license) about a year back.

[Anonymous Coward]

Re: Cloud vs On_Premise

They axed it for new orders, luckily (for me in any case) renewals are still an option until 2024. Though unfortunately we’ll have to start moving next year since maintenance for us end in February 2024.

The price hike in our case : over 30% (at current prices), which in absolute numbers is a LOT more than maintaining the 2 measly VM’s our setup requires. That 30% does not include the cost we’ll have to change internal applications that integrate with their crap, and does not include the price hike we’ll have to pay for some addins that are sold either in a Jira cloud or a Jira on-prem version.

[Nate Amsden]

Re: Cloud vs On_Premise

While I knew the situation I had a good laugh anyway. I recently renewed a 10 user server license for confluence that I purchased(?) about 10 years ago(for extremely limited personal use) but had lapsed. The cost to renew was $110, to "true up" the license to current time. That's fine, not a big deal.

Then saw the suggestion hey you can move to data center edition. Again I knew the situation but was curious anyway. To see the $10 price on the left for my existing license(to renew again for another year), vs the lowest cost data center offering of a mere $27,000 I think it was on the right(for 500 users).

But at least the license is still perpetual(for the given version of the product anyway).

I've been using confluence since early 2006, and with the cloud version(inherited from the orgs I worked at) the experience has significantly gone downhill in many aspects. My favorite version of confluence I think was probably version 3(guessing here it was a long time ago, the last version to support editing wiki markup). I have been somewhat relieved that their cloud folks have seemed to have postponed indefinitely their forced migration to the new editor. I had so many issues with it and tickets and phone calls. They kept saying that the new editor will be forced soon and I'd have no choice but to use it. But that was about 2 years ago now and that hasn't happened. Surprising they have not been able to address whatever edge cases the old editor allowed that the new one does not yet. At least they fixed one of my most annoying issues which was keys getting stuck and just printing the same character over and over and over again. Took them weeks to figure it out after trying to blame my computer/browser for the issue.

I use JIRA regularly as well but much less often. I don't use any other Atlassian products.

My regular wiki at home is Xwiki which seems to work quite well, confluence is just for some other stuff that I want to be able to access that I haven't moved(yet).

[teknopaul]

Re: Cloud vs On_Premise

Confluence has some cool concurrent editing features. Useful even for small teams.

It become a bit of a beast of late and I gave up trying to keep up with new features.

I do wish the wiki syntax actually worked and you could post code and log output without hassle.

New features seem to cause breakage. I haven't found the _insert this text verbatim_ feature.

[Anonymous Coward]

Re: Cloud vs On_Premise

We moved from on-prem Jira to their cloud version (ours is still working), and the main reason for us was that nobody in the office wanted to be responsible for maintaining it. It worked out easier for us to pay Atlassian to deal with that, so we could concentrate on maintaining things for our customers who pay us money.

[Anonymous Coward]

"Our team identified this immediately..."

> Our team identified this immediately...

Perhaps not quite immediately - it probably took an "oh no second" :-)

[Woza]

Waiting for

The "who, me?" article

[aki009]

RIP Jira/Confluence

When Atlassian started forcing customers to go to their Cloud, these formerly independent customers became vulnerable to this sort of thing. Less stuff for more bucks. It's the Oracle model or something.

[Anonymous Coward]

Re: RIP Jira/Confluence

It's not that long ago that Oracle replaced their in-house applications with Confluence and Jira. People complained about how crappy the Atlassian stuff was, this really feels like karma.

[hoofie]

Gets Expensive

You move off the on-premise JIRA because that's costing you time and money to maintain the VM.

Once you are on the cloud version you find your yearly expenditure is matching or more than what you paid to manage and maintain the VM.

QED

[aki009]

Re: Gets Expensive

I'm sure there is more than one company looking for a replacement that's not cloud based.

[Anonymous Coward]

change freeze

"If there's a lesson here, it may be that what happens in Vegas doesn't stay in Vegas if your cloud service is down. "

And another one is "freeze any darn maintenance/change while we have a big show running on, FFS !". How could this go past CAB ???

[Robert Grant]

Re: change freeze

I remember CABs.

[Anonymous Coward]

The Cloud...

Other peoples computers you have no control over

[Doctor Syntax]

Re: The Cloud...

Maybe we should update that along the lines of "Other peoples' computers who have control over you."

[Anonymous Coward]

Atlassian have been bugging us constantly to move on-prem to their cloud service, hmmmm...glad we held off

[aki009]

We ain't going to the cloud...

For us there's no possibility of us moving to the Atlassian cloud. We'll rather spend the effort going to another solution than be entirely dependent on Atlassian's infrastructure working right. Especially in light of their "maintenance scripts".

[pip25]

Maintenance script?

It's hard to imagine how such a script can cause several days of outage, unless it involved "maintaining" the customer database and their backups by deleting them all.

[Abominator]

That and they have been price gouging their long time customers on server/data centre.

Fucking grade ad arse holes.

[Tzhx]

How rude

Just my luck.... stuff I use and would like to keep using, whenever it's down for a "small number of users", I'm somehow in that small number.

And whenever I'd rather have the excuse that "oh that pile of shit is broken so I'll just work on what I think is most important"... Jira continues to be there with it's horrific interface and tedious ceremonies.

[DubyaG]

Routine Maintenance, Right

If this was routine, that implies something that is done on a regular basis. In the world of Change Management, I was under the impression that changes should always have a back out plan ready to go or some other Plan B. So a routine task shouldn't take down the whole enchilada. In my case, we use on-prem Jira and when notified of the "You vill all go to the cloud" announcement a search was started for another solution. Bye Bye Atlassian.

[Rob Davis]

Host your own opensource with Drupal Burndown agile/kanban, though self-host will not suit all

https://www.drupal.org/project/burndown - this provides a Kanban or Agile Board ticketing system.

1. Get a digitalocean vps for about 10-20 pounds per month. Or host on a raspberry pi. Rationale for these smallish servers is that if the ticketing system is only used by the business, then a large scale server for a public site is not necessary.

2. Map to a web domain, if hosted on a raspberry pi over a non-static IP, use a registrar with a dynamic dns server, like joker

3. Setup a LAMP stack on the server

4. on the server get and install drupal open source CMS, and add the burndown module, also open source, free

5. setup email to send notifications, perhaps use zoho mail - low cost

6. manage backups yourself and security updates

No affiliation, financial incentive or referall benefits of the above to me. There are likely other similar open source solutions out there. Bugzilla comes to mind but I'm not sure how well it works for most recent processes these days, having not used it for many years.

The above will not suit all. There is always going to be a cost and risk whatever approach and that includes consider self-hosting. With the greater control of self-hosting, comes a requirement to have the knowledge in-house and manage the server which means billable time out of one or more staff, along with a process for absence and when those staff move on, offboarding, onboarding.

Cloud has the benefits to those whose business core competence/reason for existing is not to manage a ticketing system. Which makes the cloud a valid consideration. I've used cloud services includingJIRA in some roles as a developer, as well as trello and pivotaltracker.

Still, having options is healthy and the self-hosting may suit some who already have teams in the business of doing that kind of work already with the related skillset.

[Anonymous Coward]

TITSUP

Ticket Information Totally Suspended, Unknown Prognosis

Why?

Typing In The Script Upset People

[Kevin McMurtrie]

Little Bobby Tables' older brother

Nolan Romy Riff-Star, short name \nrm -rf *

Such a trouble maker.

[Enigman]

Routine until it's not

At my last job all our changes were peer reviewed and any scripts that were to be run also had to have a peer review by someone with equivalent or greater scripting skill/experience. Production level changes went before a CAB except where they were declared 'routine' - routine being the change was successfully executed 10 separate occasions with no backout required, no incidents attributed to the change. Changes to the script would result in the counter being reset and the change could end up needing to be presented to CAB.

If the script hadn't changed since it was last run the peer review of the script wasn't mandatory but we did have to provide evidence that the script hadn't been changed for Production changes (timestamps, checksum or similar.)

Scripts had to be like for like across environments (Dev, Test, Prod (DR was classed as Production) unless there was a specific reason (different setup in Dev to Prod - client tried to save money with lower specs for some environments.)

At the time I got shoved out the door they were working on 'streamlining' processes and pushing the Agile line to speed things up. I had heard on the grapevine later on that my former offshore colleagues had tried to blame one of my old scripts for a major incident. What eventually transpired they had taken one of my scripts as a base, changed some of it's functionality for what they wanted and in the process removed the parts that backed up existing configurations/settings etc. They had presented it as being 'unchanged' so that they didn't have to go through CAB.