Matrix.org homeserver grinds to a halt after RAID meltdown

A RAID failure has taken the Matrix.org homeserver offline, leaving users of the decentralized messaging service unable to send or receive messages while engineers attempt a 55 TB database restore. To be clear, those with their own homeservers, such as government organizations, are unaffected, but anyone using Matrix.org as …

[ParlezVousFranglais]

"A whole series of things happened at exactly the wrong time in unison, which then led to the situation that we see"

Yep - been there, done that, still have the t-shirt somewhere. I don't envy the inevitable stress of that level of failure, followed by a 55Tb restore and replaying that many logs (just hope they are all intact) but while they may not see it at the moment, that will become a learning exercise, and the team involved will hopefully eventually be better off for the experience. Hope they have a well deserved beer at the end of it all...

[ChoHag]

I don't know what's worse. That the central node's failure took the decentralised service down or that it has 55TB of data in it.

[MatthewSt]

Probably the latter, as the former didn't actually happen (as per the words written in the article).

It's like email: anyone using a matrix.org address is offline, everyone else is still online and can communicate with anyone else not on matrix.org

[ChoHag]

A domain can have several MX records to handle the situation when one fails. That's what not having a central point of failure looks like.

Your comment is as helpful as pointing out that when DNS resolution fails the internet still works. It doesn't, in all the ways that matter. This article is describing the failure, in all the ways that matter, of a system which prides itself on being decentralised because of excessive centralisation (not to mention a truly ridiculous amount of data in a single, one might even say central, place).

[John Deeb]

You're right. But. In a way matrix.org is an out-sized demo for the Matrix protocol and concept. Registration there is a convenience and its purpose to gather interest. It seems to have become out-sized, likely because the maintainers, who develop also the most popular client, are prioritizing growth over strict decentralization. However they offer convenient server products to deploy for communities and businesses. Obviously this is not free although people can go and do the work themselves. So this "system" does not keep a central place. People who register a free account at matrix.org do.

[tulir]

matrix.org's failure did *not* take the decentralized service down, as the article explicitly says if you read it to the end. All other nodes are still fully functional.

[ara4n]

The point is that it's *not* a central node (there's no such thing in Matrix) and it *didn't* take down the decentralised service...

[druck]

Correct, but of little consolation.

If you allow a node to become so big it serves the majority of your users, they aren't going to give a frig that it is a decentralised system and a few other people aren't affected when it goes down.

Decentralised should mean lots of small nodes.

[m4r35n357]

SW or HW raid?

It is sort of relevant here! Otherwise comes across as vague bitching about PostgreSQL.

[alain williams]

Re: SW or HW raid?

It does not matter what sort of RAID. PostgreSQL sits on top of whatever it is. This fu-bar is nothing to do with the database.

It does show that even if you have RAID you still need backups; they protect you in different ways.

[VoiceOfTruth]

Re: SW or HW raid?

You can read more about it on the Postgres mailing list here:

https://www.postgresql.org/message-id/flat/CAPo1J60Vcu%2B5G0EvvAZtYgTn6U6ADij3aVJ8WFVz77jP%2BBd_Tw%40mail.gmail.com

It would appear to be hardware that caused the corruption, which then manifested itself in Postgres.

[m4r35n357]

Re: SW or HW raid?

Then I hope they have a reliable source of replacement hardware!

[YetAnotherLocksmith]

Re: SW or HW raid?

I'm sure the state data snaffler entities of several major powers will be eager to assist.

[David Hicklin]

Re: SW or HW raid?

> It would appear to be hardware that caused the corruption, which then manifested itself in Postgres.

Had that myself when a disk space increase operation on an IBM Unix was screwed up on a production system, not fun sitting there through the night for the Oracle DB to restore and then run the transaction logs overnight with the start of production time getting closer and closer....could have sworn there was a time dilation effect as the restore seemed to slow down and down...

Of course the stress reaches its peak with that final "alter database open" and you hope it has all worked.....thankfully it did!

[VoiceOfTruth]

Re: SW or HW raid?

There is nothing like a really good restore. I've been there and done that a few times. A couple of times through hardware faults, more usually Bob deleted something by mistake.

[simonlb]

Re: SW or HW raid?

"even if you have RAID you still need backups"

Yes, it's no fun when the RAID card in your server blows up and kills ALL the disks in the array at the same time.

[Anonymous Coward]

Re: SW or HW raid?

"Yes, it's no fun when the RAID card in your server blows up and kills ALL the disks in the array at the same time."

You were only supposed to blow the bloody doors off!

[el_oscuro]

Re: SW or HW raid?

^^^ 100% ^^^. With a 35 year career as a DBA, my take is if you haven't practiced a restore recently, you don't have backups. I have seen every type of "redundant" RAID system fail, and like to keep multiple copies of critical files on different types of hardware, In one example, a SAN started failing and silently corrupting redo logs. Corruption written to backup tapes too, all without any visible errors, rendering those backups completely useless.

Fortunately when I set up that database, I multiplexed the redo logs on 2 different RAID devices on the SAN, plus a separate copy on the local hard disk. The local hard disk copy was what saved our asses.

[ara4n]

Re: SW or HW raid?

yesterday's outage started with a HW raid failure (two disks were added to the array, which the HW raid controller unexpectedly promoted into the array, corrupting the whole array).

the postgres corruption problem from July (https://matrix.org/blog/2025/07/postgres-corruption-postmortem/) was also the same HW raid controller - our theory there is that we lost some writes during a NVMe drive failing which then caused the slow-burning corruption.

[Colin Bull 1]

What RAID

Is it RAID 10 or 5 or an even crappier version. WE SHOULD BE TOLD.

[alain williams]

Re: What RAID

The link given by VoiceOfTruth above says that it is RAID10 using the Linix MD subsystem.

[FirstTangoInParis]

Re: What RAID

After some experience with even Raid 1 on Ubuntu using Seagate iron wolf disks, I wouldn’t recommend it for data that changes a lot. On my own server I have long term storage plus laptop backups. MD would lose sync for a pastime on the backup partition. I took the backups to its own disk not with raid and it’s all been fine since.

[Anonymous Coward]

Re: What RAID

https://matrix.org/blog/2025/07/postgres-corruption-postmortem/

"Our filesystem is nothing fancy, just ext4, and we're using stock Debian kernels.

"For the curious: our current generation of database servers run Linux kernel 6.1, and each server uses eight 15TB Intel NVME SSDs in a RAID10 configuration to give us 64TB of storage

A seriously long time ago (~2006) I used to run SCSI disks in an mdadm raid and tuned ext4 file system which once or twice had serious problems when one of the underlying drives developed intermittent faults. Of course SSDs bring a whole new range of faults to the table.

I haven't had much to do with rdbms that run atop file systems, naively I might consider ZFS over ext4+md as it covers a nasty or two that afflicts ext4 etc.

A quick goggle of the interwebs indicate a few brave souls are running large postgres instances on ZFS so not not an entirely demented idea.

[John Deeb]

No you're redefining decentralized to mean peer-to-peer. Like decentralized government is not about every human having to decide everything by first talking with everyone else around him. It's decentralized in the way that the protocol does not require a centralized service from one company or institute. But a localized one. A localized service is not a centralized one unless one does not look at the wider world. If the local is everything you know, it becomes a centralized service again. But only in that circumstance.

[IGotOut]

"To be clear, those with their own homeservers, such as government organizations, are unaffected, but anyone using Matrix.org as their homeserver"

It's like saying wordpress.org is down, but those with their own Wordpress sites are unaffected.

[ara4n]

P2P Matrix to the rescue

arewep2pyet.com shows the work which has been going on on fully decentralising Matrix :)

[brother-wolf]

Re: P2P Matrix to the rescue

Only issue is it's been 2 years since their last entry on what appeared to be tracking projects moving at pace:

From the webpage:

```

Last updated: 2023-05-01

```

Although, the last commit under the "History" section was actually `2023-07-06`, confirmed in the GitHub record.

It would be interesting to know what happened.

[Anonymous Coward]

An open network for secure, decentralised communication

An open network for secure, decentralised communication

How can it be described as decentralized if it falls over when a single server crashes? Even so why didn't the back-up matrix.org server seamlessly kick-in? What is peoples decentralized msgs even doing on a raid array ?

[Anonymous Coward]

Re: An open network for secure, decentralised communication

Why the need to store end-to-end user encrypted msgs on homeservers at all. Opening up the service to abuse - such as:

a. A malicious homeserver can add users under its control to encrypted rooms without legitimate consent.

b. A malicious homeserver can add devices under its control to user accounts within rooms. That can still receive session keys to decrypt messages, breaking confidentiality.

[John Deeb]

Re: An open network for secure, decentralised communication

Maybe because a room might need management, history or rules not depending on who is online or if they have the room even in their system?

Same thing with all useful chat applications by the way.

[tulir]

Re: An open network for secure, decentralised communication

The network did not fall over, all other nodes have been fully functional the entire time. Only users whose accounts were on matrix.org were affected.

Both the primary database server and the backup failed on the same day, which is why the entire server went down.

[SnailFerrous]

The admins knew there was a problem when a black cat walked by the server, followed by another, just like it.

[elsergiovolador]

There

Been there few weeks ago. Took almost a week to restore dozens of TB.

Still running Ubuntu 12.04 and home made orchestration.

When asked customer why they don't want to upgrade, they said: "People capable of hacking it are already dead."

[K555]

Re: There

I take that to mean that the distro is so old that it's off the radar.

But it also sounds a little like the customer had taken a hit out on the people that might go after their system!

[Anonymous Coward]

Re: There

I love the idea of a server assassin. No sleek metal endoskeletons, just the scrape of shuffling rack on concrete following them down an alleyway. Power LEDs glow and flicker from under the brim of a hat.

"Hey kid", the voice issues forth from the chassis speaker. "Do you want to play a game?"

[DS999]

They've still got a leg up on some

It appears they actually had valid backups, based on their indication that they've completed the restore and are moving onto the transaction log replay. This is the point where more than a few similar small organizations have said "oops it turns out our backups haven't been working for months now, but the way we were validating them was incorrectly indicating success" or words equivalent to that, and everyone relying on them is SOL.

[Anonymous Coward]

Re: They've still got a leg up on some

It's not just small companies that tick the box rather than ensuring backups can be restored. Many times I and others have pointed out to companies they have no idea if they can restore to watch 'commercially' oriented folk willfully ignore. That willfull ignorance goes across government entities and large businesses. Because it costs quite a bit and it "might" not go wrong on their watch.

Related: My all time favourite was asking to buy a new storage system as the current was maxed out to be told there was no money, my response was; but I created the costings for the environment and ensured there was provision for this happening. The reply was, yes but corporate took that as extra profit. Doh!

[David Hicklin]

Re: They've still got a leg up on some

Agree, at one place I worked at the "sandbox" play system would routinely be refreshed from the prod version and the process required a backup restore to another location of the prod system which proved that not only did the backups work but that we could also restore them somewhere else if needed.

[sedregj]

Workls well

Notice how all that happens to messages is that they will be delayed - not lost! This is a catastrophic fail for a node in a matrix but it seems to be failing safe very nicely.

I remember a time when a bluey used to take a week to cross the world. My grandparents remembered a time when comms were impossible in many cases. I could go on ...

[Anonymous Coward]

Re: Workls well

Our expectations have become quite high. Not necessarily for the better.

The whole short messaging obsession is a human disaster. Things go better when you have conversations with people; first choice face to face, second voice, third email and worst of all a single sentence text message. Not to say we shouldn't have instant messaging capability but the way we all make it first choice is the cause of a lot of misunderstandings.

[Fred Daggy]

Re: Workls well

That all depends. Your priority is not my priority. And my priorities change depending upon what I am doing.

If you walk in to my office and speak to me face to face when I am deep in concentration, I am liable to march you out the door. An email or text message might get a response ... at my time of choosing.

Face to face has an implied urgency, but for who? Body language and other contextual clues can be highlighted over a conversation, a big advantage.

Telephone removes the need for proximity.

Instant messaging and email allow prioritization.

TL:DR - horses for courses.

[TeeCee]

There is no charge for using Matrix.org...

Well, you got what you paid for.

[greenwood-IT]

Deja vu...

"Decentralised" aka "on prem server" - who knew?

[GNU Enjoyer]

Wow, matrix is more terrible than I thought

What kind of chat software results in the server needing 55TB of storage for a relatively small amount of messages and images?

No ircd uses more than a few gigs of storage and even if a temporary file upload service (up to a month of storage) is provided as well (to offer the file upload feature of matrix), I don't see the required storage space being more than a few hundred gigabytes.

As for encryption, TLSv3 for all clients with a server host that can be reasonably trusted mitigates the need for end-to-end encryption (and you can use OTR if you want it).