Atlassian ran a tabletop DR simulation that revealed it lived in dependency hell

Australian collaborationware company Atlassian has revealed it’s spent four years trying to reduce dangerous internal dependencies, and while it has rebuilt its PaaS, it still has issues – but thinks they’re now manageable. As explained in a Tuesday post by Senior Engineering Manager Andrew Ross, “Atlassian runs a large …

[Michael Hoffmann]

Artifactory?

I guess they can call their registry whatever they want, internally, but does that imply they don't also run jFrog's software for all their other packages and artifacts?

[Caver_Dave]

I'm doing a different level of disaster recovery at the moment for a charity I have been associated with for 40 years.

The one part time member of staff has resigned and now I'm compartmentalising the files and email to allow different volunteers to take over control of their areas of responsibility. (We estimate that it will take us 3 months before we can get a new person in.) Sorting the files and emails into silos e.g. moving from general minutes to minutes for committee A, committee B, etc. has taken 60+ hours.

But the most frustrating part, and still not resolved, is Microsoft charity account licenses - we had one license before for the single member of staff, now it seems that we need 6 or more to cover all the volunteers doing all their individual parts, and we will go back to one license when we have a new staff member. I keep being told that someone from a different Microsoft department will contact me in 2-3 days, and then get the same message from the next person. Ad infinitum it seems!

I also have to bring the email into the Microsoft environment (currently multiple independent emails set up during Covid) so that we can have centralised, but compartmentalised access. I can certainly see the business model for Microsoft certification now, as every online resource seem to have at least one crucial piece of information missing to stop you doing it yourself. e.g. I know that I need to change the MX to move the email routing to the Microsoft infrastructure (I wrote an email system decades ago), but it's not clear where to set it to point to and how to set Exchange up to receive it with our domain name. And don't get me started on the number of disparate places you seem to have to configure users in Exchange!

I can see why no-one wants to give free Exchange advice to a charity.

Needless to say that Microsoft is not my area of expertise, but there is little chance of getting a new part-time member of staff without everything being in their environment.

Anyway, our simple disaster recovery plan (checked each Christmas break) has worked in so much as we can still communicate via email and see our files (although it all has to be farmed out manually by me to volunteers to deal with at the moment.)

A disaster recovery plan is only as good as the last test of it, so I applaud Artifactory for going public with their findings.

[Richard 12]

Free advice?

Do not use Microsoft for email.

You're very small. You really don't need it, and it will cause you nothing but pain and suffering.

MS365 Exchange is confusing and continually changes, they seem to assume every organisation has one or two FTE employees looking after their 365 estate. If you're big enough for that then it's probably fine and may even be worth it. But if you're not, when whoever looks after it (likely actually you) makes a mistake or burns out, the charity will be screwed, and it could even close.

Find a few small to medium-sized cloud providers who own their own servers in a few datacentres. They will be cheaper overall and unlike Microsoft, they'll actually help you.

You've probably already got a website host, if so then it is very likely that they also provide email over IMAP and webmail. It may even already be included as part of your domain and web hosting contract.

If not, a couple of GB of email across hundreds of mailboxes can be had for under £2 a month.

[Doctor Syntax]

Re: Free advice?

Agree completely. Small organisation, huge vendor - you're just a miniscule source of income, your problems are far too small to be noticed. The charity I look after uses Mythic Beasts for email and more recently for NextCloud.

[Quando]

Re: Free advice?

> Do not use Microsoft

FTFY

[A Non e-mouse]

I agree with Richard 12. Office 365 is a beast and unless you have resource to run a tenant yourself (Hint: It sounds like you don't) Then you need to either jump ship off of Office 365 or pay a 3rd party for an Office 365 service.

BTW - Microsoft rarely deal with licensing themselves (We spend a lot more than you on Office 365 and MS still refuse to talk to us about it) Virtually every customer has to deal with a reseller for licensing.

I'm sure there will be companies out there who specalise in providing IT for the charity sector.

[Roland6]

> is Microsoft charity account licenses

These have become more difficult to get hold of.

Previously (UK) you could purchase via Charity Digital, now it is best to use a local IT company who will purchase on your behalf and would set up your 365 tenancy etc., although you will need to have registered as a charity/non-profit on MS's charity site.

However, like others here I would caution around MS Exchange 365. I had a charity with many independent emails and it took time to sort out, the sorting out was made easier by using an ISP's non-365 mail service, which made it easier to have multiple role based email accounts (eg. Accounts, Enquiries, People, IT) each with their own GiffGaff phone - needed for SFA et al (although setting GiffGaff up for a small organisation is a pain as they don't seem to support more than one phone per account).

If you really need MS office software, just buy Office 2024 LTSC from one of the long established web resellers (this also means you can use the free Grant 365 accounts). Obviously, you will need to link the software to Dropbox/Google Drive to facilitate file sharing etc. if you want to avoid OneDrive/Sharepoint.

[I could be a dog really]

As the others, just don't. Don't inflict pain on yourself and the charity. Don't subscribe to a broken by design* email system.

Either go to someone small enough to care and actually talk to you if/when you have problems, or if you have some admin skills - run your own mailserver (though that isn't without it's own risks.) I run my own, and TBH I'd need a vary good reason to go back to useing someone else's.

* AFAIK, every major provider runs a broken by design system - in what universe is it acceptable to state "Yes, I've accepted that message for delivery" and then decide (based on vague "because spam" rules) to toss it in the bit bucket instead, leaving neither the sender nor recipient any wiser about it's non-delivery. Either accept and deliver it, or don't accept it - it's really, really not hard to do.

[Anonymous Coward]

every major provider runs a broken by design system

Fair disclosure, I'm an ISP mail admin wrangling our custom layers on top of open software.

There's one gaping huge problem with only ever rejecting something flagged as spam.

How does the *sender* go about figuring out what was offensive about their message....

... when the recipient never even knows that they tried, and their provider very very extremely rarely offers any insight even on the receiving end as to why a message they finally found out about, was blocked? (And all too often can't be bothered to tune the filter or even tell their recipient how to say "Yes, really, accept mail from this sender".)

I've had to go around in circles on this point from the sender's side more than once, and when all I can tell my sender is "You'll have to contact the recipient by some other method to get their provider to tell them/us why the message was blocked", few senders appreciate it. Especially when it's personal or business-conversation mail with well-established contacts.

All too many providers that reject spam, do it quite badly. At least if you accept and file in a Spam folder, the recipient can retrieve it, and let you know so you can fine-tune your filtering.

[doublelayer]

Re: every major provider runs a broken by design system

There is rarely any good way to do it, but telling the sender at least tells them they need to do something. They can try sending an innocuous message like "I just tried to send you something but it got returned. Please call me", and that is more likely to get through. It's annoying for everyone involved, but at least it gets resolved. Dropping mail just makes it worse because the sender thinks the receiver received and is ignoring their message, the receiver thinks the sender never sent one, and neither knows what to do so it stays not working until one of them asks for an update, and that's in the best-case scenario.

[Roland6]

Re: every major provider runs a broken by design system

The CIX mail service seems to have a good spam filtering service and drops spam into the Spam folder, separate to the Outlook Junk folder.

Their Email 25 @ £24 Pa looks suitable for Caver_Dave’s needs.

https://www.cix.uk/web-hosting/email-only

Yes finding out why emails get blocked can be interesting… Some years back I discovered the company that had set up a client’s Panda email security had decided to block all inbound emails from the major public ISPs: BTInternet, VirginMedia, Sky etc…

An issue with Outlook, is that it seems the third-party spam filters only support Outlook 2-19 and previous.

[Charlie Clark]

Why would you want to stick with Microsoft? Presumably for the apps and not "Exchange Online". There are good, free alternatives: OpenOffice/LibreOffice but OnlyOffice if you want something that looks like Microsoft. And pretty much every mail client is better than Outlook, but Thunderbird now has a fairly good track record and support system. If the next person can't work with slightly different software, then I suspect your patient work with files will also be quickly undone.

[Anonymous Coward]

You might be surprised at the resistance to using anything but MS Office tools.

It's really one of MS's greatest achievements that death and bankruptcy are see as preferable options to using Libraoffice for many.

[Anonymous Coward]

Size matters

How many users are there?

Depending on privacy aspects it may be better just to buy a bunch of individual accounts or use a small provider if you want your own domain etc. Even if the charity needs strong privacy I personally wouldn't trust any of the big guys more. If privacy is key Proton might be a solution, they offer strong privacy. I believe it is end to end encrypted within their system so they can't read your emails. If sending outside you can set a password so it can't be read in transit or on the recipients system only by password holders.

You might even be able to move to a free office suite although that is not so clean. LibreOffice is decent but people get used to Microsoft and don't like change. It also can get formatting a bit messy when exchanging with Microsoft but can read & write their file formats mostly. You know bullet points & fonts changed. If you're using lots of Macros it could be hell but just writing letters or creating presentations it can do it quite well.

Also, you may find that careful selection of office tools will leave more money for the charity's recipients.

[Anonymous Coward]

My Mrs was chairing a very small charity until recently. She was happy to have acquired free charitable 365pro licences and was starting to build out a modernisation of the charity's setup using 365 tools to create workflows for them etc.

Then MS pulled the rug on them and withdrew the 365pro license and only allows charities the basic tier now. So they've lost access to all the power tools they going to use.

MS now offers a 'charitable rate' for the pro license which is unaffordable to them (their finances are only ever on a knife edge).

I'd suggested that FOSS equivalents were available but there is little interest in doing anything non-MS. I'm sure they would be fine if they'd just be open to it.

They have had some help through the Scottish Tech Army which provides volunteer IT help to charities.

If you are south of Hadrians Wall I think the "Tech For Good Alliance" might be able to help you with your Exchange woes

[Roland6]

> So they've lost access to all the power tools they going to use.

That’s probably a blessing.

I also took a look at the “power tools” with the idea of using them to create workflows and very quickly discovered just how limited they were. Something simple like on-boarding a new member of staff:

1. Inform HR so they can send out contract etc.

2. Inform Accounts so they can set up payroll on the accounts system and bank.

3. Inform IT so they can have a user account set up and prepare a computer/phone/tablet.

4. Collate resulting information (from above) and create on-boarding pack.

Whilst possible, could only be done for a single specific instance, to on-board the next person, you had to copy and then edit the workflow…

Plus only those with the Pro license could access the created workflow etc. The tools were significantly worse than the forms flow engine MS shipped with Exchange and Outlook in the mid 1990’s.

You can see why companies like Monday.com have sprung up.

[Anonymous Coward]

You are very probably right that they may be better off without it. Even assuming that they were able to get everything working exactly as they wanted, without an IT resource on the books to keep it ticking over it would only take one update from MS to screw the whole thing up or a few years with a gradual drift to uselesses through MS usual dev practices.

They'd be better with an off the shelf MIS but again just no money to spare.

[Caver_Dave]

Thank you all for your helpful replies

[Dr Paul Taylor]

can only have hard dependencies on lower layers

So programmers have finally discovered well founded relations!

Maybe they should read Dimitry Mirimanoff (1917), who sorted out the "paradoxes" of early set theory. Search for his name together with mine to read an English translation.

[shodanbo]

Re: can only have hard dependencies on lower layers

Its not like this is not well understood in the realm of computer science. All across the field this problem of dependency management occurs, and the layered approach is the well understood solution.

The problem is enforcing the rules, and even knowing for complicated distributed systems when the rules are being broken.

Its easy to break the rules. It costs time and performance to always stick within the rules. Understanding why the rules are important can require an big picture understanding of a complex system that not all working within that system even possess,

And the costs associated with breaking the rules often do not manifest immediately. Instead those costs slowly accrue over time. Eventually the failures in dependency management cause enough problems that they become tech debt that needs to be paid down just to continue treading water.

[Anonymous Coward]

Re: can only have hard dependencies on lower layers

It requires will power and understanding at a senior level all the way down from shareholders & fund holders. Hence difficult in a world culture based on short term. You get into the realms of corporate culture, people motivation and measurement. Unfortunately, long term integrity and resilience have lost value in people's minds. I theorize we have a big disaster (WW2) that illustrates the value and then it declines until the next big disaster. Unless people suffer badly it gets forgotten. If you want resilience hire a CEO that lost his own business through lack of resilience.

[Bluck Mutter]

wow!!!!

"works on over 40,000 DynamoDB tables and 80,000-plus Amazon Relational Database Service (RDS) tables"

I don't think that is anything to be crowing about.

I have worked on OTLP databases with millions of tables (and therefore multi-millions of indexes) and on an OLTP system with >4 Trillion rows (much of which was historical in nature but still needed to exist).

And these had local failover and remote DR... instantiating the replica's of these was an interesting exercise.

Kids today and all that!!!

Bluck

[Anonymous Coward]

Re: wow!!!!

That number of tables says to me "bad design, we don't know what we're doing". My experience of their products on prem confirms that opinion.

[Anonymous Coward]

Re: wow!!!!

It's probably setting up tables per client. Maybe that is providing isolation & security? Even if not technically more secure, it can provide admin security from mistakes. They may be balancing cost vs isolation because a separate DB instance would be better.

[Fruit and Nutcase]

Confusion

Confluence Confusion or Confliction may be a more apt name. The amount of duplicate and conflicting information that appears to be normality when information is placed in Confluence makes searching for, and getting the right information a very tedious task.

They should add some AI functionality to warn/dedupe whenever users create such entries. Not necessarily needed in real time

[Bebu sa Ware]

The before and after images

could be a particularly aggressive malignancy metastasizing. ;)

Not entirely surprised given some of the problems that Atlassian has had over the last vew years.

Perhaps their experience will prod other organisations into review their systems—if they have anyone left that can still to make of the mess.

Or they could pass the whole can of worms to their AI which exhibits a certain elegant (sic) symmetry between the humungous and problematical dependency graph, and the even larger ineffable LLM that is expected to solve the problem.

[Pascal Monett]

"Atlassian recently announced a plan to ditch its on-prem products"

And thousands of their customers are soon going to announce that they don't give a fuck.

Atlassian should check with SAP to see how well forcing clients to go the The Cloud^TM is going.

[Anonymous Coward]

Re: "Atlassian recently announced a plan to ditch its on-prem products"

Cloud forcing: Aaagh!

There's reasons for and against cloud, if you have to force it there's a problem. Maybe listening to customers is a good idea.

[David Hicklin]

Re: "Atlassian recently announced a plan to ditch its on-prem products"

Sadly the bosses at last $work place just loved that as it ticked two boxes at once for going both Agile and Cloud....when it was down we were neither

[Duncan Sellars]

Jira hell

If that's their architecture, it kind of explains why Jira is so bad...

[steelpillow]

Layers of learning

Sudden discovery that a layered model for a system architecture is a Good Thing. Wow, who'd 'a thunk it! What do they teach these cloud developers?

[Anonymous Coward]

Re: Layers of learning

As a company or operation grows, it becomes hard to keep things under control unless you slow growth. I once knew a CEO of a major, household name corporation, CEO of the whole European arm, not in IT directly but various high tech engineering. He explained the nightmare they have trying to keep everything in strategy, policy and direction. You attend to one area and the area you fixed 2 years ago goes out of line. He described it as a roundabout. This guy was not the normal just grab what you can type. He had been a class athelete and was still training & competing as an amateur, was pleasant to talk to with an appropriately sized ego, never bigged himself up. So I would say performance oriented not ego oriented. So, I listened to what he had to say. I also had a relative who was a CEO of quite a big food company. His advice was you need a board that challenges you and exceeds you in their area of specialisation. But alas I think people like that running corporations are either old or dead now.

Without these people at the top things either go badly or immorally. It gets out of control. At least Atlassian took the step of recognising and admitting a problem. Unless you do that it isn't going to get better.

[David Hicklin]

Re: Layers of learning

Not helped by the Agile model of just throw it together and get it out of the door,,,, we can sort the bugs and dependencies out one day.....

[xyz]

PaaS

from the diagram it looks like Pasta as a Service, specifically spaghetti.

[Fred Daggy]

Re: PaaS

Pasta as a Service? That might be first ever cloud service I sign up my own money for.

Easy on the Rigatoni, its a very lazy pasta.

[Anonymous Coward]

AI ?

Atlassian Incompetence.

[cd]

Spock, what is that...thing?

I think we've discovered a new cluster, Captain.

[Anonymous Coward]

It makes web software. Of course it makes careless use of dependencies. This is even encouraged by the communities broken ideas!

[Sudosu]

KISS

My guess is that it is not easy to detect intrusions with that pile of spaghetti

[Taliesinawen]

Atlassian the sophisticated high-wire act

Atlassian: a highly unstable high-wire act, because of its complex microservices architecture. The system balances numerous independently deployable services coordinated via APIs, demanding sophisticated orchestration and constant oversight. ref

[bazza]

This is all pointless. Whilst they still have one circular dependency - as the article's tail end reports - then they cannot bring it all up from cold.

Not impressed.

There are ways in which all of this can be avoided. For example, if one adopts Communicating Sequential Processes as the basis for one's system architecture, and then one uses the algebra Tony Hoare created for it, you can algebraically prove system correctness (lack of livelock, deadlock, etc) before ever cutting code.

This isn't especially hard, though of course such design formalism and analysis is anathema to many. In this case, Atlassian are spending a lot of money despaghettifying their own mess and haven't actually achieved anything substantive in doing so. Whereas some comparatively cheap design analysis up front would have saved them all of this.

Their current mess translates into business risk, increased costs, and ultimately surpressed share price.

[steelpillow]

The Truth Will Out

The truth is, they are all Pastafarians. They believe that if they can sphaghettify enough of it, then they themselves will be touched by one of those noodly appendages. Sadly, they forgot about the meatballs.

[Anonymous Coward]

Quelle Suprise

I'm sure they're not alone.

In the simplest of examples I wonder what portion of websites rely on users accessing 3rd party libraries / modules via CDNs because it's easier than keeping a copy under their control? I've often come across failures of all different sorts because a link was down or the source changed something. Whilst it generally works for major items like Google's fonts some people end up relying on less common items that weren't even meant to be accessed through code or automation. Then we have governments baying for war and generally behaving like idiots waving egos around, what do we think would get attacked first? International comms! Probably already happening low key, we see the hacking, the accidental? anchor drags. Are they tests?