Atlassian boasts strong Q3 revenue growth in wake of two-week outage Atlassian has reported robust figures for its Q3 2022 ended March 31 in which revenues were up, net income was in the red and questions over the recent outage continue to linger. As a reminder, Atlassian's very own Who, Me? moment occurred April 4 when a script used to deactivate instances of a standalone application instead …
However successful the figures - this is a major corporate/cloud "thingie" - quarterly losses on that level are not wonderful. How much of that money is going into support burden / vulnerability remediation / building sustainable code for the future (and unifying product codebases and features as necessary) and how much is just loss?
A common theme for many outages is how complex systems have unpredicted (and often unpredictable) failure modes, often leading to cascading failures. A risk for anyone operating complex systems. It could happen to us too.
Then there's this one. Which is a display of spectacular incompetence.
Incompetence you say? All they did was create an internal API to unquestioningly and permanently delete things by number, and passed the wrong number to it. 800+ times. Because no-one noticed the wrong numbers were supplied.
Could happen to anyone!
Paris is looking at the list of IDs to be submitted to the API...
Updated Atlassian has warned users of its Confluence collaboration tool that they should either restrict internet access to the software, or disable it, in light of a critical-rated unauthenticated remote-code-execution flaw in the product that is actively under attack.
An advisory dated June 2, 1300 PT (2000 UTC), does not describe the nature of the flaw, and reveals "current active exploitation" has been detected. No patch is available.
The flaw is present in version 7.18 of Confluence Server, which is under attack, as well as potentially versions 7.4 and higher of Confluence Server and Confluence Data Center. Version 7.4 is a long-term support edition.
The Atlassian outage that began on April 5 is likely to last a bit longer for the several hundred customers affected.
In a statement emailed to The Register, a company spokesperson said the reconstruction effort could last another two weeks.
The company's spokesperson explained its engineers ran a script to delete legacy data as part of a scheduled maintenance for unidentified cloud products. But the script went above and beyond its official remit by trashing everything.
Atlassian's Jira and Confluence issue tracking and collaboration services have been down for some customers for two days now and the outage remains ongoing.
Starting around Tuesday at 0902 UTC, Atlassian's status page reported problems affecting customers using Jira Work Management, Confluence, Jira Service Management, Jira Software, and Atlassian Access Cloud.
The company's last update, on today at 1204 UTC, says the issues are still being worked on.
The great Atlassian outage is stumbling into a new week, with the biz reporting it has "rebuilt functionality for over 35 percent of the users who are impacted by the service outage," meaning the majority of those afflicted remain unable to access their sites.
At this point it is fair to say the problem is severe. It kicked off on 5 April and the IT giant said that while it was "running a maintenance script, a small number of sites were disabled unintentionally".
Atlassian reported in a status update this morning that: "The rebuild stage is particularly complex due to several steps that are required to validate sites and verify data." This will be of little comfort to the 65 percent of affected customers wondering if signing up for Atlassian's cloud was such a good idea after all.
Who, Us? Atlassian has published an account of what went wrong at the company to make the data of 400 customers vanish in a puff of cloudy vapor. And goodness, it makes for knuckle-chewing reading.
The restoration of customer data is still ongoing.
Atlassian CTO Sri Viswanath wrote that approximately 45 percent of those afflicted had had service restored but repeated the fortnight estimate it gave earlier this week for undoing the damage to the rest of the affected customers. As of the time of writing, the figure of customers with restored data had risen to 49 percent.
Atlassian opened its Las Vegas Team '22 event this week with products aimed at giving management an insight into projects – and developers an insight into what is actually going on.
The timing of the launch is just a little awkward: Atlassian is suffering a significant ongoing outage, with Jira, Confluence and Opsgenie unavailable for some customers. Trello and Bitbucket remain operational.
What's also unfortunate is the stuff unveiled on Wednesday is all about communication and keeping an eye on projects. The three new products, in various stages of early access, are Atlas, Compass, and Analytics.
Atlassian is still scrambling to recover from a recent software script fiasco and is hoping no customer data gets lost, which may be more than Microsoft can manage if OneDrive, as some have reported, has been intermittently corrupting large uploads for at least two months.
Four days after some Atlassian customers began encountering problems with the cloud giant's collaboration software, recovery efforts continue and a few folks are worried they may not get their data back.
One wrote to The Register wondering about that possibility after the company, via Twitter, responded to a request to confirm that customer data is backed up and failed to actually do so.
Atlassian has demonstrated the interconnectedness of all things with a warning that some versions of Bitbucket Data Center and Confluence Data Center require patching courtesy of the Hazelcast Java deserialization vulnerability.
Hazelcast is an in-memory data grid and spreads data over the nodes of a cluster and is used for efficiency and performance via its in-memory tech. It is also relatively environment agnostic, running happily on-premises or in Microsoft, Amazon, and Google's clouds.
The vulnerability affects products running as a cluster; the Server and Cloud versions of Bitbucket and Confluence are not affected. Exploitation is via a specially crafted JoinRequest with the potential result of arbitrary code execution.
How does one rebuild an aeroplane, at 40,000 feet, while it is still full of passengers? That was the question posed by Daniel Tao, head of engineering at Bitbucket, while discussing the source shack's move to AWS data centres.
The outfit found itself having to migrate 50 million repositories out of parent company Atlassian's bit barns and shunt them across the billion or so daily interactions into Amazon's cloud without falling into a heap. Oh, and it had to sort it out during a global health pandemic.
Bitbucket Cloud has been around for over 10 years and, along with an on-premises version, is Atlassian's take on source wrangling. It was augmented back in April with Open DevOps, built around Jira, Confluence and Opsgenie as well as Bitbucket, but, unlike the rest of the Atlassian line, had remained firmly in its own data centres.
Atlassian has warned users of its Confluence Server that they need to patch the product to remedy a Critical-rated flaw.
The company's not saying a lot about CVE-2021-26084, besides describing it as a "Confluence Server Webwork OGNL injection vulnerability … that would allow an authenticated user, and in some instances unauthenticated user, to execute arbitrary code on a Confluence Server or Data Center instance."
The bug scores 9.8 on the ten-point Common Vulnerability Scoring System.
The Register - Independent news and views for the tech community. Part of Situation Publishing
Biting the hand that feeds IT © 1998–2022
